Web Logic Server sizing

Dear All,
I'm sizing a new JDE upgrade on red stack.

From some Oracle doc I found that a single Web Logic Managed Instance can serve 100 + users (running standard programs).

I never seen more then 30-50 users per JVM; never on OAS or WAS or WLS too but I'm curios to know your experience.

Thanks in advance

Bruno Condemi

The issue with OAS and earlier stack releases was with the limitation of memory. With WLS - you have the ability to use a 64bit JVM - hence the 2Gb limit is easily overcome. You still need to size the architecture properly - the standard sizing estimate is around 32 concurrent users per 2Gb. Don't forget that the application servers need to scale up based on the number of concurrent user sessions - and that the concurrency changes based on the type of use as well (sales order users uses up a LOT more memory and CPU compared to a journal-entry user)

Don't forget that the AdminServer also requires memory (between 512Mb to 1Gb) - and the OS needs memory too.

Thanks Jon.
I don't have any serious test on this but I assume (from what I know about JVM architecture) that a cluster (vertical) of two 2 GB jvm instances is more convenient that a single 4GB jvm instance in terms of performance.

Increasing the heap size of the jvm can lead to long garbage collection time; the CPU need to allocate and deallocate more object too in the heap and fragmentation occurs.

If I run two clustered instance, each JVM process take a single CPU, the heap is small enoug h to run a fast GC, there are less chance to have CPU peek and heap fragmentation.

Is it only "paranoia"?

There is no easy way to create multiple JVM's under WLS. Not like OAS. You have to go through the entire install each time to create multiple instances.

As for the long garbage collection time - I haven't actually experienced the issue ! There is a document that talks about setting up verbose garbage collection on metalink - 1294574.1.

I think the big issue is whether you are using jRockit or SunJDK. I only use jRockit on all the weblogic servers - and I've never seen issues with garbage collection. I have servers on both Linux (Redhat 5.7) and on Windows 2K8R2.

Yes it's true; It was so easy to add JVM to OAS.

About jRockit... it's the only jdk supported in production environment.

Back to the original question, so a good starting point is 60 user per JVM (4GB) per CPU with a cluster for failover and scalability.

[ QUOTE ]

Back to the original question, so a good starting point is 60 user per JVM (4GB) per CPU with a cluster for failover and scalability.

[/ QUOTE ]
No need for a cluster - unless you're prepared to spend something in the order of $ 500K for Enterprise Edition WLS - use a load balancer instead for failover and scalability. For smaller implementations - my recommendation is looking at Zen Load Balancer (free and open source) - for larger instances, look at using hardware based load balancers like the F5.

60 users per 4Gb sounds right - the CPU really depends on what each user is actually doing - but 1CPU per 60 users does seem low. I personally size 30 users per 2Gb per 1 core CPU.

Ok, now I understand, the WLS standard license doesn't include the cluster option.

Thanks for your input.

Right. None of the standard edition licensing (Websphere NOR WLS) includes clustering. The price difference between redstack (standard) and Enterprise Edition is substantial. $45k per CPU core. So it can hit $250,000 per server pretty quickly.

But nobody needs clustering at that level. The only time you want clustering at that level is if you're running Oracle SOA Suite - which takes full advantage of intra-transaction failover. But if the WLS goes down and those users have to reconnect to the same URL that passes them to another server - nobody is really going to complain about that too much. Zen is a phenomenal product (and has the ability to provide full clustering at the loadbalancer level) - so theres really no single point of failure.

I've always been a very big proponent of open source. It was my presentation at Collaborate 2004 (or was that Fusion 2004) that pushed JDE hard to adopt Linux as a platform. Oracle finally taking over Peoplesoft was the natural progression to Linux, but I'd already documented the performance of Websphere on Linux 3-4 years earlier !

Yes, it's true, JDE web application doesn't take advantage from JEE features like ejb, jms, and so on. Perhaps that new mobile applications, built on top of ADF framework, can benefits of wls clustering but I never used until now.

Now I'm pretty sure I'll go to Zen, I like the idea of open source software built as appliance.
Just a question: How to prevent Zen failure itself. I think in production environment I have two instance of Zen with some virtual ip in order to switch in case of application failover.

[ QUOTE ]

Just a question: How to prevent Zen failure itself. I think in production environment I have two instance of Zen with some virtual ip in order to switch in case of application failover.

[/ QUOTE ]

You install two copies of Zen on two servers, and cluster them together. Works like a charm ! Simples !

Theres a video that explains the setup - here :

http://www.youtube.com/watch?v=bleU7B8hAJQ

Fantastic, it's so simple.

Okay wow.....seems like a one sided conversation here so I think I'll have to chime in.

32 users per 2 GB.......no I don't think so. This isn't Citrix.it's web. Double your numbers and you might be close. Triple them and you're in the sweet zone.

If you need confirmation go do a sizing with any hardware vendor. Yes they will give the 300 users per JVM, other tuning guides will give you 150 users per JVM but the reality is that 50 - 75 users per JVM is standard and achieved in 99% of all somewhat current E1 installs.

On a 32 bit JVM your max JVM size is 1280 MB. Yep that's it. Increasing it is done at your own peril and you will likely run into issues. I've got lots of scientific research to back this up as well (the explaination requires a PhD to understand).

On a 64 bit system.........well you have no limit but the normal useful size of the JVM is 2 GB but I've tested up to 6 GB with no ill efect (and I mean really tested).

32 users per 2 GB is based on Citrix......not web. Lot's of tuning guides out there to support a higher number.

Also check your JVM's by 1) looking in Server Manager at the Java memory usage or enable verbose GC and check it overtime. It's likely the system is very rarely approacing your maximum allowable size.

Colin

Colin - you're suggested numbers are a very small factor in difference, especially to cost, and it seems you're arguing for the sake of argument - but I'll step up to the plate.

I have not seen a single EnterpriseOne benchmark that specifically points to any more than the original benchmarks I ran myself. Yes - I admit that the 32 per 2Gb recommendation was my benchmark and it was done originally on Citrix - it was based on 40 users per 2Gb but with a 20% redundancy factor built in. So suggesting that a web client can run 60 users per 2Gb is only actually a 50% increase for the java-based client.

However, you have to also understand that the 40 users per 2Gb value was BEFORE ANY MEASURED IMPACT to performance over the minimum hard ceiling. The actual tests ramped up to over 100 users per 2Gb - but the memory performance was dramatically different above the 40 user limit. The performance restriction was that all 18 scripts could not take more than 1s of average performance hit (as measured by the end-user). All the benchmarks in the past show that the performance hit is under 1s for specific platforms.

However, for the web benchmarks I HAVE seen, JDE immediately doubled the acceptable performance hit to 2s - this was because Java graphic routines were taking longer to "Draw" than the Windows C routines. If 2s was an acceptable performance under windows - then you'd also expect a much larger number of concurrent users under citrix too !

96 "redundant" users per 2Gb of memory under the original 18 scripted benchmark is a hell of a claim. That would anticipate a sizing estimate of 120 concurrent users per 2Gb of memory - an average of 17Mb of usage per concurrent session.

Again, I haven't seen a true apples to apples comparison of users per JVM. I therefore prefer to be cautious, and would absolutely oversize memory for the web servers rather than be in the terrible situation of undersizing them. Remember, the 18 scripts do NOT take into consideration any of the memory-utilizing functions such as import/export or even multiple concurrent applications per user.

Now, I am willing to test this myself - I have performed my own personal independent benchmarks in the past to back up my claims - and I'm always willing to accept that I might be wrong.

But given the fact that the difference between 2Gb and 4Gb is approximately $50-$100 of Enterprise Memory cost - I'd rather err on the side of caution.

One thing we can agree on, however, is the use of larger 64bit JVM's for production. I also use 6Gb JVM's in production.

Hi Colin,
thanks for your response.

So in your experience a single JVM with 2GB (and of course using a single core) serve 50 - 75 users?

I have some others questions but I want to be sure to understand your point.

[ QUOTE ]
On a 32 bit JVM your max JVM size is 1280 MB. Yep that's it. Increasing it is done at your own peril and you will likely run into issues. I've got lots of scientific research to back this up as well (the explaination requires a PhD to understand).

[/ QUOTE ]

Actually, you don't need a PhD to understand the above. Let me make it simple for you.

The maximum JVM Heap size for a 32Bit JDK is technically 1280Mb under Windows. This is because Windows uses memory at above the 1280Mb addressible area. The 32bit Sun JDK can only technically use contiguous free memory - hence the reason the limitation at 1280Mb - even using /PAE in Windows (giving a total of 3Gb memory) doesn't help.

However, under Linux or any other 32Bit OS that doesn't place memory at the 1.2Gb addressable space, the limit for the JDK is actually a lot higher - around 2.2Gb, since there is a lot more contiguous memory available.

The Oracle 32Bit JRockit JDK however CAN use non-contiguous memory, and therefore could theoretically also use about 2.2Gb on a 32bit windows system with /PAE.

There really isn't much limitation on a 64Bit JDK on a 64Bit OS - but remember, there is a limitation of thread use with each JVM. If you have substantially more cores, you want to have at least one JVM per core to take advantage of the CPU power - and that alone might force you to take a smaller memory per JVM.

An easy way to test this is to use the following Java code :

<font class="small">Code:</font><hr /><pre>
public class OutputMaxMemory {
public static void main(String[] args) {
Runtime rt = Runtime.getRuntime();
long totalMem = rt.totalMemory();
long maxMem = rt.maxMemory();
long freeMem = rt.freeMemory();
double megs = 1048576.0;

System.out.println ("Total Memory: " + totalMem + " (" + (totalMem/megs) + " MiB)");
System.out.println ("Max Memory: " + maxMem + " (" + (maxMem/megs) + " MiB)");
System.out.println ("Free Memory: " + freeMem + " (" + (freeMem/megs) + " MiB)");
}
}
</pre><hr />
compile this with javac OutputMaxMemory.java - then try
<font class="small">Code:</font><hr /><pre>
java -Xmx9000M OutputMaxMemory
Total Memory: 85000192 (81.0625 MiB)
Max Memory: 9432858624 (8995.875 MiB)
Free Memory: 83588920 (79.71660614013672 MiB)
</pre><hr />

play around with the -Xmx number to see if you can get above 1280Mb in your system. Under Windows XP Professional 32bit with hotspot JDK, I managed to get only up to 1610Mb - with Windows 2008R2 using jRockit I couldn't get to a maximum without insane numbers !

Yep...the difference is small but significant.

In your case you're telling everyone who has more than 30 concurrent users they have to cluster.

In my case I'm saying "don't worry, no issues until you hit more than 60+ concurrent users".

If we look at the JDE client footprint and see where the average client lies this makes a huge difference.

Yep if I'm at a billion dollar plus company who has oodles of cash then sure let's just cluster the heck out of the system and away we go.

However if I'm at a small shop with a few IT people and a limited budget the difference between 30 and 60 concurrent users may infact mean a doubling in their business.

I have infact ran 100+ concurrnet users for an extended period of time on a 32 bit JVM with each user having multiple apps open. While I do prefer 60 users per JVM you can't always anticipate user behaviour.

If we look at the various vendor and JDE benchmarks they all point to very significant benchmarks with no significant impact on response time as the number of concurrent users increases on the later tools releases.

Of course pre 8.97 was a totally different story.

I'm not saying don't user extra memory. All Web Servers on 9.1 support 64 bit so that should be used almost exclusively. However no reason to push a client with 50 users into doing a more elaborate configuration then warranted.

Colin

Well 2 GB isn't the total server memory. You have to account for the O/S and whatever else is running.

Also I always have atleast a few cores available. For 50 - 75 concurrent users I would have a minimum of 2 cores.

Are we taking about WAS or WLS here?

If you have the money I would always prefer to split my load between 2 O/S instances - 25 - 38 on one machine and 25 - 38 on the other machine (hardware load balanced).

So yes in short 50 - 75 users with 2 GB of memory is more than possible and is the norm at my clients (2 on 9.1, 15+ on 9.0, 15+ on 8.12).

Colin

It is my understanding that the WAS edition included in the Blue Stack license is Network Deployment which does allow clustering.

[ QUOTE ]
Right. None of the standard edition licensing (Websphere NOR WLS) includes clustering.

[/ QUOTE ]

[ QUOTE ]
Yep...the difference is small but significant.

In your case you're telling everyone who has more than 30 concurrent users they have to cluster.

In my case I'm saying "don't worry, no issues until you hit more than 60+ concurrent users".


[/ QUOTE ]

Nope. I never suggested that. I just suggest that you utilize more memory on your 64Bit OS for each concurrent user.

I have never, ever recommended clustering for EnterpriseOne Web users. I do recommend using Webtier Utilities together with a realistic load balancer. My suggestion is that ALL customers consider using a loadbalancer - whether its an open-source load balancer like Zen - or a hardware load balancer. It just makes sense, and can provide a LOT more cost effective solution since it provides high availability between users.

ok - heres a nice little chart that explains all. I only work these days with 64bit JVM's on 64bit OS - I'm not expecting any of my customers to be using 32bit anything these days. I'm also never going to suggest implementing E1 HTML on anything less than 2 cores. Therefore, heres an interesting sizing chart :

32 users : 1 64bit JVM : 2Gb MaxHeap : 2 CPU Core
64 users : 1 64bit JVM : 4Gb MaxHeap : 2 CPU Core
128 users : 1 64bit JVM : 8Gb MaxHeap : 4 CPU Core
256 users : 1 64bit JVM : 16Gb MaxHeap : 8 CPU Core
512 users : 1 64bit JVM : 32Gb MaxHeap : 16 CPU Core
1024 users : 1 64bit JVM : 64Gb MaxHeap : 32 CPU Core

So, with my chart above, you can see that in THEORY you can have 1000 users on a single 64bit JVM with 64Gb memory and 32 available cores - but in reality, you'd probably hit issues with very long Garbage Collection timing issues above 8Gb.

Lastly, and this is more important to production use, you probably want more JVM's to isolate any issue that might develop with a single JVM and to ensure that less users are impacted by a single system failure.

A better way to look at the above is the following :
32 users : 1 64bit JVM : 2Gb MaxHeap : 2 CPU Core
64 users : 1 64bit JVM : 4Gb MaxHeap : 2 CPU Core
128 users : 1 64bit JVM : 8Gb MaxHeap : 4 CPU Core
256 users : 2 64bit JVM : 8Gb MaxHeap ea: 8 CPU Core
512 users : 4 64bit JVM : 8Gb MaxHeap ea: 16 CPU Core
1024 users : 8 64bit JVM : 8Gb MaxHeap ea: 32 CPU Core

If you are using 16core CPU machines, then you can have 1024 users spread across 2 machines, each running 4 JVM's - and will use 32Gb on each after the OS, together with a loadbalancer ensuring that all user sessions are round-robin balanced between the 8 JVM's.

So, after all that, my chart is still a good yardstick - without fully understanding what type of transactions the users are performing.

[ QUOTE ]
It is my understanding that the WAS edition included in the Blue Stack license is Network Deployment which does allow clustering.


[/ QUOTE ]

You may be right. I always assumed that Blue Stack only came with Standard Edition Websphere, but Oracles edelivery site has the network deployment download for IBM Technology Foundation - which would certainly include the advanced clustering technology. That is certainly a more valuable product than Oracles redstack offering. Is IBM still offering the Blue Stack license for new JDE implementations ?