Jrockit faulting on weblogic 10.3.5

[RaptorOfWest]

Hi

We are currently based on weblogic platform JAS version 10.3.5 with jrockit_160_24_D1.1.2-4. The OS is windows 2008 R2 staffed with 4 cores and 16gb ram. There is single JAS instance created and peak user load is around 90 though the average load is 40-50 users mostly using PO and AP modules.
We have our production running from past 2 months and have never seen any issues.

A day before the system suddenly stopped and restarted the jvm. The error we got was below

Faulting application name: java.exe, version: 1.6.0.24, time stamp: 0x4d6cf5a1
Faulting module name: jvm.dll, version: 28.1.3.11, time stamp: 0x4d6cf51c
Exception code: 0xc0000005
Fault offset: 0x0013b8bc
Faulting process id: 0xa2c
Faulting application start time: 0x01cd4217c927419c
Faulting application path: E:\Oracle\MIDDLE~1\JROCKI~1.2-4\jre\bin\java.exe
Faulting module path: E:\Oracle\MIDDLE~1\JROCKI~1.2-4\jre\bin\jrockit\jvm.dll
Report Id: 6695d384-af60-11e1-94c8-005056a86016

The JVM is created with following arguments as per Oracle case when the 9.1 was installed. Also the load when the crash happened was 26 users only.

SET JAVA_OPTIONS_WIN=-jrockit -XnoOpt -XXnoJITInline -Xms1024m -Xmx1024m -XXtlasize:128k -XXlargeobjectlimit:128k -Dtoplink.xml.platform=oracle.toplink.platform.xml.jaxp.JAXPPlatform

I am wondering if any of folks here have seen any such issues with jrockit. Any insight would be welcome.

 

[altquark]

your JRockit is a pretty old version - 1.6.0_24_D1 is old. Weblogic 10.3.5 is a recent version - but my recommendation is to install a newer version of the jrockit. I just installed jrockit-jdk.1.6.0_31-R28.2.3-4.1.0 - which seems to work just fine. I've used this latest jRockit version with WLS 10.3.5 and also 10.3.6 too.

Secondly, your JVM size seems relatively small - you're only using 1Gb of memory. For a 6Gb Windows 2008 R2 64Bit server dedicated as a JDE HTML server, I use the following java startup options :

-XX:MaxPermSize=1024m -Djavax.xml.rpc.ServiceFactory=oracle.j2ee.ws.client.ServiceFactoryImpl –Xms4096m –Xmx4096m –verbose:memory

I'm sure others have tons of different WLS size settings.

But with 16Gb of available memory - you really ought to make the min/max JVM Size a lot larger than 1Gb !

 

[fbrammer]

I also read somewhere that either OWL or JRocket has trouble with a path longer than 10 characters. This issue may have been fixed.

 

[RaptorOfWest]

Thanks very much altquark

I will apply the settings tonight and going to test it through. Another interesting observation we have found is when the crash happens the amount of users is ~20 however there is process called Dell KACE running which coincides with the timing of crash.

We will disable the agent and then change the JVM size, see what happens

Thanks again.

 

[ice_cube210]

I apologize for taking this thread a bit off topic , but are more people now using just one JVM on Weblogic servers because they are now 64 bit(WebLogic) and can use a lot more memory than before ?

With 2 GB JVMs and one CPU per JVM you could typically handle anywhere between 40 - 80 users depending on how interactive they were.

Jon , in your example you mentioned a 6 GB server with 4GB memory as the Min , Max for Weblogic. Which leaves 2 GB for the OS , so I presume you are only running one JVM on this server. How many concurrent users are you able to handle on this one server ?

If the new JDKs are now multi-threaded then I guess I could see why we would not need more than one JVM to take advantage of multiple CPU's on a server.

 

[RaptorOfWest]

Hi

Thanks for replies. We resolved the issue. The problem was the java was being used by another monitoring tool by Client to determine the health. The weblogic jrockit memory was in-accessible for garbage collection during the specific time as the monitoring tool was trying to interfere with it.
The solution was to disable the monitoring tool, and remove the aggressive collection option in the weblogic. The currect arguments we added are below and the server is now functioning for more than 4 days without any errors.

-XX:MaxPermSize=1024m -Xms4096m –Xmx4096m -XXaggressivept

We will upgrade our jrockit to 28.2.3 just to keep it updated.

As far the use of single JVM, most of our clients who are small-medium shops prefer to use them as they are not keen to train their personnel on clustered setups. With 64 bit we have been running 100-150 users in few shops and so far no issues. The usual scenario is load balanced JVM on two machines with one being used as failsafe....some of them we used Zen balancer..an open source product

Thanks

 

[altquark]

first of all, I tend to avoid clustering for a couple of reasons

1. It is more difficult to set up, and more difficult to train people to administer clustering
2. It requires a different license that Oracle Technology Foundation doesn't support (Enterprise rather than Standard edition)

But all my customers DO have load-balanced WLS servers - something I do for all customers, no matter the size. Currently, I am load-balancing 6 WLS servers in production - with the ability to go higher if necessary. I am SIZING my WLS servers at 32 users per 1Gb RAM - so a 4Gb JVM should be able to support a total of at least 128 concurrent users. Currently I haven't seen that many users on our systems, since we're actually running 6 servers and are eavenly loadbalancing our user pool (we are not deployed yet).

In my customers environment, they're using a clustered F5 for the load balancing.

I introduced the idea of using the Zen Loadbalancer this year at Collaborate - I hope that my presentation is what prompted you to consider using it. I also hope you're clustering the Zen Loadbalancer as well - you certainly don't want the loadbalancer being a point-of-failure ! It runs fantastically - I've pushed Zen Loadbalancers for smaller customers and for sand-box implementations to great success.

Glad you've increased the heap size. By the way - 1Gb might be too much - I was re-reading the MaxPermSize setting, and on my servers, I'm looking to drop that down to 512Mb instead of 1Gb. But it certainly should be much higher that the default size in my opinion. I also like removing the aggressive option for garbage collection.

glad I could help !

 

[RaptorOfWest]

Thanks Jon ..it certainly was your presentation that led us to use Zen, a wonderful product indeed.

 

[ice_cube210]

Re: ZEN Load Balancer

A question on the zen load balancer for those that are using it.

Does the free software based solution only work on a Linux based OS ? Are there any ways to do install this on a Windows OS or a commercial version available for Windows ?

Thanks in Advance

 

[altquark]

Re: ZEN Load Balancer

Zen only works on its version of Debian that it comes bundled on. There really isn't any point installing it on windows - you should view it as an "appliance". I use it on vmware - so if you want to, you can always install vmware or virtualbox on windows and then run Zen as a guest machine !

 

[ice_cube210]

Re: ZEN Load Balancer

Thanks for the response Jon.

Will give it a try

 

[78aspide78]

Re: ZEN Load Balancer

Jon,
I saw your slide on ZEN product. I'm really happy to see some open source software in the JDE ecosystem.

Just a question, from a customer point of view, running on red stack, why I need to use ZEN?
Does WLS provide load balancing support (via WebLogic proxy plugin)?

Thanks in advance.

 

[78aspide78]

Re: ZEN Load Balancer

Ok, you just give me the answer. WLS provides load balancing only for clustered instances and the cluster option is not available on the standard edition.

 

[fschygiel]

Re: ZEN Load Balancer

Jon,
Do you have an idea of how "scalable" zenloadbalancer is?
Is it reliable for a 100/200/500, etc users JDE installation?

thanks
Fabio