Hardware load balancing and JDE EnterpriseOne 9.0 HTML Servers

[unitas99007]

We've recently upgraded our Environment from 8.10 to 9.0 running the 8.98 tools release. We've setup multiple HTML servers, and are planning on running two instances of HTML servers on each physical box. Each instance will run on a separate port. (Probably 82 and 83). Ideally, 4 physical servers with 8 instances total.

That being said, we want to hardware loadbalance these instances. We setup a "virtual cluster"on our hardware loadbalancer (a Coyote Point E350GX) with a virtual IP and port number (say, 80).

Here's the crux of the issue. If the virtual cluster is set to listen on the same port as the server is listening on (port 82 and 82 respectively) then the load balancer performs as it should and the page loads appropriately. If the virtual cluster is set to listen on a different port (like 80) and the servers listen on 82 or 83, then the page does not load and recieves a HTTP 404 error.

It seems to me the HTML servers hardcode the return port into the HTTP headers.

Anybody have an idea what might be going on here, or how I can influence this behavior??

Thanks in advance.

Unitas99007

 

[JEMILLER]

In your situation you generally need to change the httpd.conf to include:

UseCanonicalName Off

It defaults to on.

You may also need to add this to the JAS.INI

[PORTALCONFIGURATION]
localhost=node_ip_address:jas_instance_listening_port (if not 80)

The localhost setting is mentioned in the 8.97 JAS install guide but not the 8.98 guide which I assume you used for your 9.0 install. I do not have any sites running load balancing under 8.98 tools so I can't confirm it is still required.

 

[unitas99007]

Justin,

I've tried switching "UseCanoncalName" to both on and off, and both with and without the [PORTALCONFIGURATION] section in my jas.ini. This still doesn't work.

I'm assuming 8.97 tools and 8.98 are somewhat similar as far as how a loadbalance would go. How would you set this up for 8.97 and the 8.10 environment? Any other tweaks that might be affecting this?

Thanks!

Unitas99007

 

[cdawes]

The really easy way to fix this is to have everything listen on port 80. I've recently configured 2 clients with hardware load balancing on tools 8.98. Both cases had 2 physical servers with 3 JVM's on each.

All you really need to do is create Virtual Host entries, modify the HTTPD.CONF file and update the DNS. I have everything looking for "http://PD812.COMPANY.COM". This address is published in the DNS and I use a Cisco ACE 4710 to do the load balancing. The Cisco ACE could do the redirection but it's just just as easy and clean to do this in the JAS/HTTP config.

What are you running WAS? OAS?

Colin

 

[unitas99007]

Colin,

I'm running WAS 6.1 on Windows Server 2003 boxes. Could you give an example of the VirtualHost syntax? I'm assumming you have more than one IP assigned to the physical box and each VirtualHost Listens on a different IP, right?

Once again, thanks for the help.

Unitas99007

 

[unitas99007]

Oh yeah, just curious, how beefy were your boxes? We've got dual-core Xeon 5420's with 4 GB of RAM in each box, and are planning on running two instances of the HTML server configured with a 512 MB Java heap size.

 

[brother_of_karamazov]

Why use anything other than port 80:


http://jeffstevenson.karamazovgroup.com/2009/01/simple-e1-url.html


[ QUOTE ]
We've recently upgraded our Environment from 8.10 to 9.0 running the 8.98 tools release. We've setup multiple HTML servers, and are planning on running two instances of HTML servers on each physical box. Each instance will run on a separate port. (Probably 82 and 83). Ideally, 4 physical servers with 8 instances total.

That being said, we want to hardware loadbalance these instances. We setup a "virtual cluster"on our hardware loadbalancer (a Coyote Point E350GX) with a virtual IP and port number (say, 80).

Here's the crux of the issue. If the virtual cluster is set to listen on the same port as the server is listening on (port 82 and 82 respectively) then the load balancer performs as it should and the page loads appropriately. If the virtual cluster is set to listen on a different port (like 80) and the servers listen on 82 or 83, then the page does not load and recieves a HTTP 404 error.

It seems to me the HTML servers hardcode the return port into the HTTP headers.

Anybody have an idea what might be going on here, or how I can influence this behavior??

Thanks in advance.

Unitas99007

[/ QUOTE ]

 

[cdawes]

In both cases I have 2 machines and each are 2 way Quad Core E5450 with 8 GB RAM. In one case I'm running WAS ND 6.1.0.21 on Windows 2003. On each machine I have one instance of WAS ND with 3 vertical cluster members.

Are you running separate WAS instances or are you using vertical clustering?

I am NOT using horizontal clustering at the WAS level --> using the Hardware load balancers for this. The dmgr isn't fault tolerant and theres no need to have 2 levels for horizontal load balancing.

Just a couple of notes:

1. Make sure you're at 6.1.0.21 - issues before this
2. Heap Sizes may be off. Check the IBM Docs (how many concurrent users are you planning for?)See URL below. WAS 6.0 and WAS 6.1 are TOTALLY different in terms of JVM tuning so make sure you follow up on this.
http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100946

My config supports a typical load of 200 - 300 users (100 users per JVM) per server. However, the IBM bench mark is 300 users per JVM so in theory I could have almost 1000 users on each server (don't recommend this).


Colin

 

[cdawes]

Check Brother of Karamazov/Jeff Stevenson's Technology Blog. He's got a good article on this.

You don't need more than one IP address per server. You can have one server with 100's of web sites all on the same port (80) and IP address.

Colin

 

[unitas99007]

All,

We've got one WAS instance. Two HTML servers on the instance. I'm not sure if that's a vertical cluster or not, still fairly new to this stuff.

Planing for 100 users per physical box, 50 per instance. It's a bit oversized, but I wanted the extra power and redundancy.

As far as port numbers, we used to set things up on 80 everywhere, from what I'm told, but then the bossman read you're supposed to set stuff up for different environments on different ports- 81 for DV, 82 for PY, and 83 for PD, etc. So that's the reasoning behind that.

Also, I tried setting up two instances on the same box with same port. I added an IP to the NIC in Windows, and in the scf_servername.conf file (appended by Enterpriseone at end of HTTP.conf during setup) i changed the "Listen 82" statement to "Listen IP#1:82" on the first instance, and "Listen IP#2:82" on the second instance. I can login via both IP's, but in server manager, it just appears as two users on the first instance. Something else I'm missing?

If I can get that working, then the hardware loadbalance problem is a non-issue, as i can configure it a different way.

Unitas99007

 

[brother_of_karamazov]

Well, you're not going about it the way I would but you're almost there. Have you boss read about Apache Name-based Virtual Hosting.

In the Integrated Solutions Console (fka WAS Console), under Generic JVM Arguments enter


<font class="small">Code:</font><hr /><pre> -DcloneId=AS_PD1 -Dwas.home=true </pre><hr />

where AS_PD1 is your J2EE instance.

Spelling and capitalization are critical.



[ QUOTE ]
All,

We've got one WAS instance. Two HTML servers on the instance. I'm not sure if that's a vertical cluster or not, still fairly new to this stuff.

Planing for 100 users per physical box, 50 per instance. It's a bit oversized, but I wanted the extra power and redundancy.

As far as port numbers, we used to set things up on 80 everywhere, from what I'm told, but then the bossman read you're supposed to set stuff up for different environments on different ports- 81 for DV, 82 for PY, and 83 for PD, etc. So that's the reasoning behind that.

Also, I tried setting up two instances on the same box with same port. I added an IP to the NIC in Windows, and in the scf_servername.conf file (appended by Enterpriseone at end of HTTP.conf during setup) i changed the "Listen 82" statement to "Listen IP#1:82" on the first instance, and "Listen IP#2:82" on the second instance. I can login via both IP's, but in server manager, it just appears as two users on the first instance. Something else I'm missing?

If I can get that working, then the hardware loadbalance problem is a non-issue, as i can configure it a different way.

Unitas99007

[/ QUOTE ]

 

[cdawes]

Yes you MUST install each instance on it's own port number otherwise Server Manager will not let you install. Then you change the configuration to add a Virtual host for each instance.

So for example the JDE default would be something like *:8091 for PD but you can add PD812.COMPANY.COM:80.

The * means that the you're looking for anything on port 8091. The other line above means that you're only looking for "PD812.COMPANY.COM" on port 80.

Then on the PY instance the JDE default could be something like *:8092 but you also listen for "PY812.COMPANY.COM:80".

Notice that JDE default to listen on all IP addresses on the same port? What we're doing isn't rocket science we're just looking for a specific host name on port 80.

I would suggest that you have a CNC Expert come in and help you with your configuration.

You've got a good start but there is a lot more. For instance what do you monitor to check to see if an individual application server is up or down? With WAS it's easy to monitor the WC_defaulthost port for each JVM.

You can do an entire hardware clustering config for 2 servers start to finish in much less than 1 work week and seeing how you have a lot done it would not be very involved at all and you'd learn from someone who's done it many times before.


Colin

 

[unitas99007]

All;

Thanks for the overwhelming support and response. I feel like I've almost got it wrapped up.

brother - I added the section to my Generic JVM arguments. Thanks for the tip!

Colin- I would hire a CNC guy, but we already hired one before the loadbalancer was in the mix to set up the enterprise servers, deployment server and the first HTML server. So for now it's all me. Plus, I like to tackle a new challenge.

Here's one of my virtual hosts configuration:

<font class="small">Code:</font><hr /><pre>


# The following configuration was added by the EnterpriseOne Server Configuration Framework

Listen 172.20.61.54:82
NameVirtualHost 172.20.61.54:82
<VirtualHost 172.20.61.54:82>
Alias /jde "C:\WebSphere61\AppServer\profiles\AppSrv01/installedApps/vjdeweb01Node01Cell/vweb01_was61_82.ear\webclient.war"
DocumentRoot "C:\WebSphere61\AppServer\profiles\AppSrv01\installedApps\vjdeweb01Node01Cell\vweb01_was61_82.ear\webclient.war\owhtml"
ServerName 172.20.61.54
ServerAlias vjdeweb01_A
ServerAlias vjdeweb01.root_A.local
</VirtualHost>

<Directory "C:\WebSphere61\AppServer\profiles\AppSrv01/installedApps/vjdeweb01Node01Cell/vweb01_was61_82.ear\webclient.war\WEB_INF">
Order Deny,Allow
Deny from All
</Directory>

<Directory "C:\WebSphere61\AppServer\profiles\AppSrv01/installedApps/vjdeweb01Node01Cell/vweb01_was61_82.ear\webclient.war">
Order Deny,Allow
Allow from All
</Directory>

</pre><hr />

And here's the second VirtualHost entry:

<font class="small">Code:</font><hr /><pre>

# The following configuration was added by the EnterpriseOne Server Configuration Framework

Listen 172.20.61.55:82
NameVirtualHost 172.20.61.55:82
<VirtualHost 172.20.61.55:82>
Alias /jde "C:\WebSphere61\AppServer\profiles\AppSrv01/installedApps/vjdeweb01Node01Cell/vweb01_was61_82_B.ear\webclient.war"
DocumentRoot "C:\WebSphere61\AppServer\profiles\AppSrv01\installedApps\vjdeweb01Node01Cell\vweb01_was61_82_B.ear\webclient.war\owhtml"
ServerName 172.20.61.55
ServerAlias vjdeweb01_B
ServerAlias vjdeweb01_B.root.local
</VirtualHost>

<Directory "C:\WebSphere61\AppServer\profiles\AppSrv01/installedApps/vjdeweb01Node01Cell/vweb01_was61_82_B.ear\webclient.war\WEB_INF">
Order Deny,Allow
Deny from All
</Directory>

<Directory "C:\WebSphere61\AppServer\profiles\AppSrv01/installedApps/vjdeweb01Node01Cell/vweb01_was61_82_B.ear\webclient.war">
Order Deny,Allow
Allow from All
</Directory>

</pre><hr />

So my next and final question is this- now that both HTML servers are up and seen by servermanager, I can hit both webpages via shorturl (thanks again brother), here's the next bit. I log into the environment, but the User Sessions in Server manager show two active sessions on the first instance, not one session on each instance. Any ideas why?

Thanks again.

Unitas99007

 

[cdawes]

No problem.......good to see people up to a challenge.

Can you post a diagram with your config to make it easier to follow? Are these 2 different boxes with different IP's or one physical box with 2 IP's? I'll assume 2 different boxes unless told otherwise.

Listen 172.20.61.54:82

--> 0.0.0.:82 or *:82 unless you have multiple NIC's simply makes it easier to understand and you won't get burned by an IP address change
--> technically coding in the IP address is faster but no one will ever notice

NameVirtualHost 172.20.61.54:82
--> same as above - loose the IP addresses
--> if listening on port 80 thn it would be something like PD812.COMPANY.COM:80

ServerName 172.20.61.54
--> should be the actual host name of the machine if using port 82 (some may do different configs)
--> if using port 80 then this is the hostname (unqualified) of the web site (eg PD812)

ServerAlias vjdeweb01.root_A.local
--> think you have a typo here
--> also should be HOSTNAME , FQDN eg JDEJAS1, JDEJAS1.COMPANY.COM
--> you only need one line but you can use multiple lines like you have done

If you can hit both web servers by the URL and the web servers are on different physical machines then you have a hardware load balancer issue.

If the web servers are on the same machine (different partitions) but on different IP's then you also have a load balancer issue.

Check the logs on the load balancer to see what is occurring.

Spend some time on http://httpd.apache.org/ and you'll learn a few great tips.

Also you need to tune the HTTPD significantly - you've got the out of the boxconfig and there is some really simple changes you can make to improve performance.

Check the IBM 8.12 Tuning Guide for JDEdwards (just Google it).


Colin

 

[unitas99007]

Colin,
Thanks for the quick response.

Single box, one NIC, two IP's. One install of WAS 61, running two JDE 9.0 HTML servers, both on port 82. The goal is to bind each HTML server to a single IP listening on port 82.

The testing I did yesterday was hitting the direct URL, not the loadbalanced one.

I'll post a sever and network diagram later today.

Unitas99007

 

[Jaise James]

Hi There,


Two years ago , I had implemented this type of solution using F5 switches. I should have some post here somewhere too.

In my extensive testing , I found that if you let H/W load balancer do the job of load balancing between different JVM's, it really does not work. It always returns requested back to port 80. After much of frustration, the way I configured was to use ND.

So I use H/W load balance to load-balance between the servers ( You can add as many server you want) and on each server we have ND installed that load balances the JVM's on the box.

So H/W load balnce always goes to port 80 or 81 ( what ever you configure) on each server and ND load balance between 9081 9082 etc)

I have been live with solution for two years without any issues.


By the way, my solution goes further and it load balances the App server as well.

 

[unitas99007]

I found a post by riko44 that seems to be *exactly* what i want to do.

[ QUOTE ]

Thanks everyone for responses, we got many very useful ideas!

Just wanted to give some update as to what we have accomplished so far ..

We are trying to implement following method/workaround.
Biggest issue for us is that F5 doesn't forward properly traffic from on port to another as expected.

So after reading Jeff Stevenson's whitepaper
“Eliminating the Use of Port Numbers in Websphere/HTTP”
http://www.thestevensons.org/node/view/647
I thought it may be a good idea to use similar approach so F5 will only deal with port 80 (or any particular single port for that matter), while Websphere server(s) will take care of ‘rerouting’ to the specific Webserver instance within the box.
The first obstacle we came up with is that F5 doesn’t allow having different hosts names under the same IP address.
So, we decided to experiment with somewhat radical idea and assign multiple IP addresses to the same Windows 2003 box (WAS server).
So let’s say we have WAS box webserver (10.1.1.100).

We attached two more IP addresses to this box (Windows 2003 machine) - (10.1.1.13 and 10.1.1.14)
We Created 2 more host names in DNS (e1test85 10.1.1.13 and e1test86 10.1.1.14)
We Created two virtual hosts in Websphere - e1test85 and e1test86 to listen on port 80
Modified httpd.conf on the Webserver to redirect e1test85:80 and e1test86:80 traffic to the actual pdjas1:85 and pdjas1:86
(this is different from what mentioned above whitepaper recommends)
<VirtualHost e1test85:80>
Redirect / http://webserver:85
</VirtualHost>

Created a pool in F5 to load balance between 10.1.1.13:80 and 10.1.1.14:80
Created a F5 virtual host 10.1.1.30 (DNS name e1test) to point to the newly created pool ….
And voila, it seems to be load balancing. No iRules so far , we can add / remove hosts on the fly (I tested and it seems to be working).
Now the unexpected thing.. it looks like as result of these manipulations (I suspect redirect command ) Big IP actually is not carrying much of the traffic through itself. So lets say when I am typing http://e1test (virtual host name) it immediately directs traffic to http://webserver:86 or :85 and I can see the webserver:86 in the address bar.
I believe that is because we haven’t done exactly what whitepaper is suggesting (I will still try to make that part work)
I am not sure that is exactly what we want, but nevertheless load balancing seems to be working.
As you can see main point is to assign multiple IP addresses to the same WAS server.
We will keep researching and I let you know if we find anything else

Thanks



[/ QUOTE ]

I'll test this out tonight to see if it works.

 

[unitas99007]

Worked flawlessy.