E920 WEB Server stability

[Alex_Pastuhov]

I'm doing some load testing of JDE WEB Servers and finding that E910 on WL with Java6 is _a_lot_ more stable compared to E920 with WL on Java8. <- that's with all the default settings, mostly.

I.e.: it runs fine in general, but even 50-70 concurrent users can easily kill it. E910 can run with 100 users for days without any issues.

Are the early E920 adopters seeing any similar issues with E920?

Or did you have to do any special setup to make it work?

 

[altquark]

Do you have -Xms -Xmx set the same in your weblogic startup parameters ? If not, thats probably why you're keeling over.

Also - Java 6 is 32bit. Java 7 and Java 8 are 64bit and have the jRockit extensions. I certainly wouldn't try pushing more than 1.5Gb RAM with Java 6....

 

[Alex_Pastuhov]

Thanks Jon!

The defaults for -Xms & -Xmx are set to 32 & 200 respectively. It does seem a bit on the low side, but it works with E910. I am going to set them to 1024 both in E920 and re-test.

The E910 Java was also x64, I think:

java version "1.6.0_24"
Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
Oracle JRockit(R) (build R28.1.3-11-141760-1.6.0_24-20110301-1430-windows-x86_64, compiled mode)

 

[Alex_Pastuhov]

Besides, I see no out of memory errors in the logs...

 

[RussellCodlin]

Those numbers are very low for memory allocation. 1024 would be a minimum even if you're only running a few JDE users. If you're using Java 7 you also should increase the PermGen Space to 192M (-XX:MaxPermSize=192m).

You probably won't see out of memory errors with Xmx being reached. Instead, the JVM will just continuously run a Full GC to claw back memory which will significantly impact performance and may make WebLogic unresponsive.

If I had a system that needed to support 50 concurrent users I'd be running at least 2 JAS instances within the same WebLogic container with 2048M of RAM allocated per instance.

 

[Alex_Pastuhov]

Thanks Russell! Of course, these are just test systems, I'm just trying to get a good feel of what these values should be. I'll be bumping those up now...

 

[RussellCodlin]

Understand. More for reference if someone else is reading the thread. For our 9.2 lab environment we haven't seen any JAS server instability on WL12c. We have seen instability and bugs in the tools with regards to kernels but we've been working through those with Oracle. Our instance now seems pretty good but it is 9.2.0.3 with POCs from Oracle so not technically a GA release.

 

[altquark]

If you're running a 64Bit system, and you're trying to get 100 users running - I would strongly recommend setting -Xms 4096 -Xmx 4096 (ie, 4Gb to the weblogic instance).

My calculation per concurrent user is based on 32Mb RAM per concurrent session, and round up to the nearest Gb.

 

[Alex_Pastuhov]

Thanks, Jon, I'll try this.

 

[Alex_Pastuhov]

An update: running with 4GB of RAM (-Xms 4096 -Xmx 4096), on 8 CPU's, testing it with only 5 heavy concurrent users for 30 minutes, after ~18min the response time starts climbing up and the pages per second goes down until it becomes irresponsive ~20-22 min from the start.

This can't be right, people would be screaming.

This is the Java version I'm using:

java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)

It's WLS 12.1.3 and JDE TR is 9.2.0.3 with all the ESU's as of a few weeks ago.

I do not suspect the test harness we use, because it works fine against an older E910 server, i.e.: 50 users with less RAM for many hours and no issues.

I can't see any immediate issues and it does work fine in general until it hits this issue.

Any advice would be most welcome.

 

[RussellCodlin]

Which OS are you using?

Our 9.2 system is tools 9.2.0.3 / WL 12.1.3 on Windows 2012 R2 with JDK1.8.0_73. We don't do load testing against our webserver so can't comment under load but under normal usage (max of 3 concurrent users doing either config, testing or training) we don't see any stability issues or memory problems and our web instances would be lucky to get a restart once a month. We run PS, DV, PY and PD all on the same web container as well as their AIS counterparts. We then run ADF and OVR on a second server.

Is your WL instance running in production or development mode? If you set the memory settings to -Xms512m -Xmx4096m do you see continuous memory growth over the test period or a sudden spike when performance dips? When the system becomes unresponsive what happens if you manually trigger garbage collection (you can do this through server manager)? Is there high CPU load at all during the test? Are you running AIS on the same container or a separate container or not all? What is your load test actually doing? When the load tester starts to fail can you start a new session from a separate client machine and see the same issue just manually testing?

 

[craig_welton]

What do the other servers (DB and Enterprise) look like when this happens? Anything pinned there?

 

[Alex_Pastuhov]

Thanks for your responses!

I'm sorry for the delay: I do not have all the details at hand now and it may take me some time to collect all the answers, but what I do know is: it's in production mode, I have both ms & mx set to 4096, so no spikes, I'll look at the garbage collection later and get back to you, I will also look at the CPU in more detail later, but I was under an impression it was not high, I do have AIS there in a separate container, but it wasn't started at that time, the test is to sign in, run an application, do a find, open a line in a detail form, close it and sign off, I'll need to experiment more to see what a manual sign-on looks like at that time, but in the end, when it all stops working after the test, the manual sign-on does not work anymore either.

I have the testing done at another site and wasn't doing it myself, so I'll need to coordinate it a bit more to get more details. Hopefully in a week or so.

Similarly, I have no details at the moment about what's happening on the enterprise server at the time, but I will post anything I find out when I know more.

I think I may update the Java installation there as well and see if this makes any difference. Your version .73 is newer than my .60, so maybe that's the reason, who knows?

 

[ice_cube210]

Hi Alex,

Running a 9.1 Apps system on TR 9.2.0.3 on Windows 2012 R2 , WLS 12c , no issues . Have not load tested 50 users , but have not seen any issues you mention when 5-10 users are on.

My java version - 1.8.0_92 is more recent than yours too , so may be a JDK update might be worth looking into

Good Luck