Citrix - why the limit on Number of users

amwalshjde

Well Known Member
We have been using JDE Xe, with Citirx XPe for years. The rule I have always followed was to only have about 35 users per citrix box. When I look at the utilization of our 3 citrix boxes, I see that they dont appear to be overwhelmed. For instance, we have dual proc machines with 6 GB ram, with 35 users we are still only using about 2.8 GB RAM, and CPU usage sits below 20%. Network bandwidth isnt an issue either. If I remove a server from the farm (we have 3) and run on two, and the number of users on each box goes above 50, within a few hours we get wierd issues. Such as the JDE applciation not opening after connecting to citrix, and "unable to submit job" to enterprise server, etc.

Where does this rule of no more then 35 (or 50) come from? If the citrix box dosent appear to be over utilized, what is the contraint?



Thanks
Aaron
 
Patient: "Doctor, it hurts when I do this"
Doctor: "Don't do that"


[ QUOTE ]
We have been using JDE Xe, with Citirx XPe for years. The rule I have always followed was to only have about 35 users per citrix box. When I look at the utilization of our 3 citrix boxes, I see that they dont appear to be overwhelmed. For instance, we have dual proc machines with 6 GB ram, with 35 users we are still only using about 2.8 GB RAM, and CPU usage sits below 20%. Network bandwidth isnt an issue either. If I remove a server from the farm (we have 3) and run on two, and the number of users on each box goes above 50, within a few hours we get wierd issues. Such as the JDE applciation not opening after connecting to citrix, and "unable to submit job" to enterprise server, etc.

Where does this rule of no more then 35 (or 50) come from? If the citrix box dosent appear to be over utilized, what is the contraint?



Thanks
Aaron

[/ QUOTE ]
 
...I figured that....
wink.gif


Any clue as to the why? Is it CPU, Memory, disk IO? Or just the way JDE works? I dont like having to tell people we need more hardware, when it appears that our current hardware is under utilized...at least not without being able to tell them why.

Aaron
 
I had a feeling I was going to end up giving a follow-up
cool.gif


My experience has been that it is usually memory leaks, E1 does not do a very good job of memory cleanup and older versions such as your are worse. Others on here may have different/better answers.

Have you considered a Tool Release (Service Pack) upgrade?


[ QUOTE ]
...I figured that....
wink.gif


Any clue as to the why? Is it CPU, Memory, disk IO? Or just the way JDE works? I dont like having to tell people we need more hardware, when it appears that our current hardware is under utilized...at least not without being able to tell them why.

Aaron

[/ QUOTE ]
 
I am monitoring the memory usage, ans every night I see the memory usage go down from about 2.5-3GB used down to 10% used. Then the next day as people sign in the memory usage goes back up. If there was a memory leak, I would expeect the mem usage to slowly rise over time.

In our case. We can reboot all three server's and only put two in the farm, and within an hour start getting those odd errors.


Thanks,
Aaron
 
Hi,

What Windows release are you using?
Does your version support more than 4 Gb RAM?
If so, have you configured AWE?
2.8 Gb is a pretty suspicious number, it's usually the
roof for Windows releases that support up to 4 Gb RAM,
there's 1 Gb reserved for OS internal activities).
Have you thought on running on Win64 bits?
 
Which monitors are you using for memory?

If you are not already, try looking at Private Bytes and of a slightly lesser importance, Working Set.

You can also keep an eye on Non-Paged Pool Bytes but that is probably not an issue.

Create a perfmon Counter Log from the attached file, edit it to change the output location, let it run for a couple of days, analyze the result with graphs.



[ QUOTE ]
I am monitoring the memory usage, ans every night I see the memory usage go down from about 2.5-3GB used down to 10% used. Then the next day as people sign in the memory usage goes back up. If there was a memory leak, I would expeect the mem usage to slowly rise over time.

In our case. We can reboot all three server's and only put two in the farm, and within an hour start getting those odd errors.


Thanks,
Aaron

[/ QUOTE ]
 
Have you tried using Citrix measures in Performance Monitor?



I have seen the behavior you describe when there are excessive Context
Switches/Sec. Strange things happen when there are too many running threads
for the processor(s) to service.



Benchmark this value and see if the strange behavior starts at a particular
threshold.

_____

From: [email protected] [mailto:[email protected]] On
Behalf Of amwalshjde
Sent: Monday, February 02, 2009 1:33 PM
To: [email protected]
Subject: Re: Citrix - why the limit on Number of users



I am monitoring the memory usage, ans every night I see the memory usage go
down from about 2.5-3GB used down to 10% used. Then the next day as people
sign in the memory usage goes back up. If there was a memory leak, I would
expeect the mem usage to slowly rise over time.

In our case. We can reboot all three server's and only put two in the farm,
and within an hour start getting those odd errors.


Thanks,
Aaron

OneWorld Xe SP22 AS/400 V5R2

_____


The entire
<http://www.jdelist.com/ubb/showflat.php?Cat=3D&Board=3DOW&Number=3D14248 JDELIST thread is available for viewing.


Looking for a job? Check out the Job
Opportunites
forum


This is the JDELIST EnterpriseOne Mailing List.
JDELIST is not affiliated with JDEdwards=AE.

To unsubscribe from this list via email, Click
<mailto: [email protected]?Subject=3DUnsubscribe&Body=3DSirs,% 0A
Pl
ease remove this address from the JDELIST EnterpriseOne% 20M
 
Sebastian-

we are running on windows 2000 Advanced Server, using 5.2 GB of ram with PAE enabled (see screen shot).

We havent put much thought into 64bit, as we should be able to use the 5-6 GB of RAM with the PAE switch.
 
[ QUOTE ]
We have been using JDE Xe, with Citirx XPe for years. The rule I have always followed was to only have about 35 users per citrix box. When I look at the utilization of our 3 citrix boxes, I see that they dont appear to be overwhelmed.....

[/ QUOTE ]

I just saw this thread, and thought it might be apt to comment on it, since I'm the chap who initially came up with the 32 users per box sizing estimate. I've commented on this before, and it requires a little bit of a history lesson of how OneWorld and Citrix/Terminal Server came to be.

First of all, you have to be aware that OneWorld has been supporting Terminal Services since 1997 - more than 10 years. The sizing "estimate" was scientifically created in 1998 - TEN YEARS AGO ! So of course, it isn't necessarily accurate with todays technology. But that is also why the W environments aren't accurate either.

Secondly, remember if you're using OneWorld Xe - it is also a product that appeared in 2000 - a product that is more than 8 years old. So you're trying to use old software on new technology using old sizing methodologies that have never been updated.

In 1997, I was given the task at ATG (The Advanced Technology Group in Denver) to test OneWorld under Terminal Services and Citrix (then, the products were in beta and were named "Microsoft Hydra" and "Citrix Picasso") to show that they DID NOT scale - that the technology was NOT as good as general CNC practices !!

Our test bed was a DEC Intel server with quad 180MHz Pentium Pro processors and 2Gb of memory. Yup, this was in 1997 - so the box back then was (almost) state of the art !

Our initial test was against a SQL 6.5 database server and we had a script set up in Macro Scheduler to plug away orders in P4210 - 5 line sales orders every minute or so per user.

When we loaded up Oneworld, we noticed that at around 5-6 users, there was a huge spike in CPU processing - ramping up to 100% - and although we could load a couple more, soon each of the sessions were becoming unusable due to this CPU utilization and each session "hanging". Our initial test, therefore, proved that Terminal Servers (and therefore Citrix) was NOT a solution that JDE should be recommending.

However. I sat back and looked at what exactly was hammering the CPU's - and thought to myself that if I mapped the business functions away from the Terminal Server (say, onto the Enterprise Server) - using our latest CNC practices - that we might get over the initial CPU utilization issue. Lo and behold, almost all CPU utilization disappeared, and suddenly we were able to start scaling the terminal server. We were able to show, therefore, that with the business functions mapped off the terminal server - that Citrix/Terminal Services COMBINED with CNC practices produced a scalable, recommended solution.

The next limit we found occurred at around 40-42 users or so. In effect, the memory was being utilized and when the physical memory started to "top out" that the amount of paging from each of the sessions started to impact the CPU utilization again - and we started to see some performance degradation that would impact end users.

Looking at the average memory utilization per session - each user seemed to be utilizing around 40Mb of memory. Therefore with a standard windows NT 4.0 Terminal Server Edition setup, we saw around 40 concurrent users with 2Gb of memory.

However, because of the stability (back in 1998) of the Terminal Server and the amount of "memory leaks" we saw both in the OS and in the product, we felt that a company with, say, a few hundred concurrent users - we wanted to scale the Terminal Server to 80% of the number of expected users - so that for every 5 servers, we would have the ability for one of those terminal servers to fail and still be able to support the customers users. 80% of 40 users is 32 users.

Fast forward to 2000. By now, almost every customer of OneWorld utilizes Windows Terminal Servers and Citrix. (for the comparison history between Terminal Servers RDP and Citrixs' ICA protocol - thats altogether another story). Lots of customers have a requirement to deploy across a WAN - and many customers like the terminal server since it dramatically reduces the management but also provides predictable network utilization.

But in 2000, the Pentium III processor was available - two full generations beyond what we had tested in 1997 - and the processor speed was able to peak to 1.4Ghz by 2001. That was a total of more than SEVEN times the amount of processor speed compared to the pentium pros'.

Therefore, by the time that Pentium III 1.4Ghz processors were available - together with the improved Windows 2000 server OS - the requirement for mapping business functions to scale to 40 concurrent users had disappeared. Since JDE hadn't fully got their business functions to work on application servers by then, there was a convoluted list of BSFN mappings - around 450 or so - that the install team introduced with Xe and called the "W" environment. Of course, logically, the "W" environments were a terrible idea since developers sitting on fat clients weren't viewing the same type of environment as the end users on terminal servers with the W environments !

Over the next few years, I went through a huge struggle to reverse the "W" environment mappings - instead recommending either running fully 2 tier (only mapping business functions for business reasons) - or going fully 3 tier using something that is now referred to as the "J" environments, and running ALL users on that environment type - no matter what client type they are running. That way, the users are on a consistent environment and support issues between fat clients, Java clients and Citrix clients cease to become and issue.

[ QUOTE ]
For instance, we have dual proc machines with 6 GB ram, with 35 users we are still only using about 2.8 GB RAM

[/ QUOTE ]

So, why did the 32 user sizing limit stick ? After all, in your example you have 6Gb of memory - so could conceivably utilize 96 concurrent users on the box !

Well, until recently, the limit for Windows 2000 server has been limited to 2Gb of Memory. This was increased to 4Gb for Windows 2003 server - though the "enterprise" edition could be utilized - but at a significant increase in cost. Secondly, the COST for slamming lots of memory into a single box was FAR in excess of the cost for two boxes with the same memory. Therefore, since you can provide more REDUNDANCY with more smaller boxes compared to fewer large boxes, it was recommended to stick to the 2Gb limit. Today its probably wise to size to 64 users for every 4Gb of memory, providing the latest CPU's are being used - and, as the price point falls, the sizing should be re-evaluated.

If you ARE seeing issues, however, and you're not seeing paging issues or CPU utilization - then maybe the number of context switches is becoming an issue ? Windows has some limits there - and a simple run of Perfmon and a google of performance monitoring will identify issues like this.

So - that is the reason. There are a couple of interesting points that come out of this.

First of all, the 40Mb utilization is specifically around the oexplorer.exe process (the OneWorld Explorer). If you are using Solution Explorer (activera.exe) - then expect DOUBLE the memory utilization. I've noticed that customers using solution explorer literally can scale to half the users compared to OneWorld explorer users. So, 8.9 and 8.10 - since they require solution explorer - requires more memory per concurrent user.

Secondly, a web client uses about the same amount of memory per session as a OneWorld explorer users - ie, about 40Mb. Therefore the JVM recommended size of 1.2Gb means that you can run about 30 concurrent sessions per JVM - coincidence ? I doubt it ! Of course, you can load up dozens of JVM's on a single server - especially using 64bit OS's - providing you have the physical memory to handle them...!

I hope that provides some explanation. I can forward links to the original Terminal Server testing documents (which are on my website) for anyone interested in the history behind these numbers !!!
 
Older versions of Windows Server Std had a limitation of 2 GB memory available for terminal servcies. With a registry entry you could bump it up to 3GB. I don't know if this applied to Windows 2000 Advanced Server.
 
What measure have you used to see the memory used per session for web users? Our experience would tend to agree in practice with the 40Mb and 30 users being a good rule of thumb number to avoid exceeding, although individual JVMs will run with higher numbers than that.
 
First off, thanks to everyone for all the information.

Regarding Context Switching:
I have started to moniter the context switches on our citrix box, it's still early here (East Coast, USA) so we have 25 users signed in and the average context switch / sec is 8,000-10,000. I did some googling and had a hard time finding a definitive number of context switches where things started to go bad. The closest I could find was that 14,000 / per CPU. If that's the case with our dual proc we should be good up to 28,000. Dose that sound about right?

Regarding PAE/AWE:
I read that that even with PAE enabled that in windows 2000 apps need to be designed to work with the AWE interface: (source)
http://books.google.com/books?id=4za7eLykh5cC&pg=PA35&lpg=PA35&dq=Windows+2000+Server+citrix+2gb+limit&source=web&ots=SxR7FQSDBn&sig=hl1b8obc-1N-EoxH_ORBCpgK6b0&hl=en&sa=X&oi=book_result&resnum=2&ct=result#PPA36,M1

If that is the case, dose that mean the wer are truly limited to having the applications only use 2gb of the 5.25 installed?

Again, thanks for everyone's insight.
Aaron
 
Re: Citrix - why the limit on Number of users - W2K Memory Limits

[ QUOTE ]
Well, until recently, the limit for Windows 2000 server has been limited to 2Gb of Memory.

[/ QUOTE ]

Actually, Windows 2000 Std supported 4GB of memory like Windows 2003 Std.

Windows Memory Limits by Version
 
Re: Citrix - why the limit on Number of users - W2K Memory Limits

Thankyou for the correction. HOWEVER, 2Gb of memory is ONLY available to the application (whereas the other 2Gb is available to the system) - so the limit is still only 2Gb in effect UNLESS the PAE switch is used, which increases the application memory to 3Gb. THEREFORE, 6Gb under any 32bit windows is overkill and is not being fully used.

The Microsoft link is here :

http://msdn.microsoft.com/en-us/library/aa366778.aspx

And the discussion of 4Gb Tuning is here :

http://msdn.microsoft.com/en-us/library/bb613473(VS.85).aspx
 
Re: Citrix - why the limit on Number of users - W2K Memory Limits

Which is the same rule for Windows 2003 Std.


Agreed?
 
Re: Citrix - why the limit on Number of users - W2K Memory Limits

I thought it was the /3GB switch that changed user-system from 2-2 to 3-1?


[ QUOTE ]
Thankyou for the correction. HOWEVER, 2Gb of memory is ONLY available to the application (whereas the other 2Gb is available to the system) - so the limit is still only 2Gb in effect UNLESS the PAE switch is used, which increases the application memory to 3Gb. THEREFORE, 6Gb under any 32bit windows is overkill and is not being fully used.

The Microsoft link is here :

http://msdn.microsoft.com/en-us/library/aa366778.aspx

And the discussion of 4Gb Tuning is here :

http://msdn.microsoft.com/en-us/library/bb613473(VS.85).aspx



[/ QUOTE ]
 
[ QUOTE ]
Regarding Context Switching:
I have started to moniter the context switches on our citrix box....The closest I could find was that 14,000 / per CPU. If that's the case with our dual proc we should be good up to 28,000. Dose that sound about right?

[/ QUOTE ]

Correct. 14,000 per core is the general "rule of thumb". As you load up, this number can spike pretty quickly. With only 25 users, you're already seeing 10,000 context switches per second - and when you load up even more users this can increase exponentially causing the users sessions to "lock".
 
Re: Citrix - why the limit on Number of users - W2K Memory Limits

It is.

This is what PAE does:

PAE Explanation

PAE is kind of risky because a substitute Windows kernel is loaded called ntkrnlpa.exe instead of ntoskrnl.exe.

Here is a good explanation of the /3GB switch:

3GB Switch
 
Re: Citrix - why the limit on Number of users - W2K Memory Limits

Are you saying that there is no reason to put more the 4GB of RAM into a 32bit OS running as a citrix server?

In our case with just the OS, Citrix and JDE being used on the server, it is possible that the following is occuring regarding our 6GB RAM (with PAE enabled on windows 2000 Advanced Server):

First 2gb being used by system/kernel
Next 2gb being used by Citirx and JDE
Last 2gb not being used

Then when either the System 2gb if fully used or the Application 2gb is used we start to see performance issues?
If that's the case is there a way to measure the usage of the 2gb system space and the usage of the 2gb application space?

Here is a link with long discussion on this topic (not JDE specific)
http://www.brianmadden.com/blogs/brianmadden/archive/2004/02/19/the-4gb-windows-memory-limit-what-does-it-really-mean.aspx
 
Back
Top