Weblogic procees uses at least 60% of CPU

Piotr Tatarek

Member
Hello

On our server we have an issue that since last week the CPU is getting up to 95%. We have find out that process WEBLOGIC is using more than 60% and should be max 20%.

What could cause such an issue as we are looking for a root cause for this ?

Please adivse

Rgs
 
Hi piotr

i recently upgraded our JDE 8.10 to 9.1 with weblogic for the HTML server we had the same issue and changing the heap size fixed our issue.
the url below will help you to start your weblogic server with specifying size of the memory allocation pool.

https://docs.oracle.com/cd/E13222_01/wls/docs81/perform/JVMTuning.html#1131866

hope this helps
 
Hi,

You haven't said what release you're on but we've had a similar issue on Websphere with the CPU hitting 100% (on 9.1).
There is an issue reported, which is resolved in 9.2 TR - E1: JAS: *** INFINITE LOOP DETECTION WITHIN APPLICATION ER:**** (Doc ID 1905029.1). Might not be what you are experiencing but we had the intermittent 'Infinite Loop' errors on some of our JAS Servers.

We haven't implemented 9.2 TR yet - we have played with the timeout cache and min / max heap memory. This has appeared to make a difference but we're still monitoring.

Regards,

Sanj
 
Other than high CPU during full garbage collection (already mentioned) there is a known bug/issue referred to as the "Stuck Threads" issue.

When stuck threads are the issue you will have high CPU on the Java process but the users will not notice or not notice very much. This is because the stuck thread(s) are running at a lesser priority than the active threads that are actually processing user requests. So while the CPU is high the threads that are serving user requests take priority. Usually the CPU will hit 99%/100% versus 95% in this situation but it may depend on how you are monitoring the metric.

The issue seems to be nearly gone since tools 9.1.5 but I still see it occasionally on newer tools releases. It happens intermittently. Some of my sites can go months before running into it. Some sites never see it.

The only solution is to stop and start the Web container.

See:
E1: WLS: JAS Server Running Weblogic Reaches 100% CPU Load Prior to the 9.1.5.0 Tools Release (Doc ID 1616726.1)
 
Guys,
Thank you very much for your time and help.

In logs from WEB PY we found below information:
[COLOR=olive !important]28 Jan 2016 09:06:39,818[WARN][RUNTIME]Error retrieving element active from UI state document. [/COLOR]
[COLOR=olive !important]28 Jan 2016 09:06:39,821[WARN][RUNTIME]Error retrieving element active from UI state document. [/COLOR]
[COLOR=olive !important]28 Jan 2016 09:06:39,824[WARN][RUNTIME]Error retrieving element active from UI state document. [/COLOR]
[COLOR=olive !important]28 Jan 2016 09:18:13,268[SEVERE][RUNTIME]*** INFINITE LOOP DETECTION WITHIN APPLICATION ER:****[/COLOR]

Form Name :p3112_W3112E_VJDE0001

On Event: Button clicked, at ER Line = 12

While Statement: VA[266] != LI[String] = 6

On Control-ID: 13

It looks like we have a problem with P3112 application.

This is article from Oracle about infinite LOOP and propably reason.
https://support.oracle.com/epmos/faces/DocContentDisplay?id=1905029.1


Do you have experience in that subject?
Our JDE platform is E900 and TR 9.1.5.3

Much appreciated.
Thanks


 
I think you are right based on those log messages that the infinite loop is the source of your high CPU. The web client sessions are maintained on the web server and event rules code is run within the JVM. An infinite loop in ER code can be in standard JDE code or custom code. I wasn't aware of the document you link to. I think it describes a particular source of ER looping where the user session is left running and then cleaned up which then orphans the loop. I am not certain that simply changing back to the default 20 minute session timeout will solve the issue (assuming you have customized it). My experience with infinite loops is that usually the only temporary solution is to restart the JVM and the only permanent fix is to address the issue in the ER code.

Looking at the P3112 bug (Bug 20925693 : P3112 OCCURS INFINITE LOOP IN E1 ROOT LOG) is seems it isn't fixed until 9.1 apps. I suggest you may need to implement the 9.1 fix in your 9.0 code. You can certainly try your luck for a POC fix back to 9.0 but 9.0 Update 2 being now being out of premier support you probably won't get far. The fix looks to be delivered in 9.1 ESU JM18820.
 
We seem to also have the "stuck threads issues", as well as, just recently java.lang.OutOfMemoryError: GC overhead limit exceeded. Now our JVMs seem to be stable for 2 weeks or so, then, all of a sudden, issues start happening. We have 4 WLS servers with 2 production JVMs and 2 test JVMs. The max users is approx 50 per JVM and we are using Microsoft Network Load Balancing.

Our current HEAP settings are -XX:MaxPermSize=256m -Xms1024m -Xmx1280m - which we were going to change the MAX to them to Xmx2048m. Thoughts? Suggestions?

Our JDE platform is... E811 SP1 with Tools Release 8.98.4.7, with our WLS 10.3.5, with JDK 1.6

Thank you in advance....
 
kkelly10, I second your idea to increase Xmx to 2048m.

Be sure you are running 64-bit Java, otherwise Xmx=2048m will prevent your server from starting up at all.
 
Thank you Brandon. Another quick question -- what are your thoughts about disabling the garbage collection feature, by adding the option -XX:-UseGCOverheadLimit to the startup scripts. I saw pros/cons about doing this. Thoughts?
 
You should make your -Xms and Xmx the same values. If you're running 64bit Java - then you should have the values at least

-Xms2048m -Xmx2048m

However, your platform (JDK 1.6) I believe is 32bit - so you must not increase your max to above -Xmx1280m

You should reinstall your weblogic server and use a 64bit weblogic so you can go higher than 1.2Gb of memory.
 
Thank you Jon...we are running 64 bit, so I will make the change. As for the -XX:MaxPermSize=256m -- that is fine or should it be increased?
 
-XX:permSize -XX:MaxPermSize are used to set size for Permanent Generation. This is where class files are kept. These are the result of compiled classes and jsp pages. If this space is full, it triggers a Full Garbage Collection. If the Full Garbage Collection cannot clean out old unreferenced classes and there is no room left to expand the Permanent Space, an Out‐of‐ Memory error (OOME) is thrown and the JVM will crash.

In Java 8 - this has been removed and is now called "Metaspace" to address some of PermGen's drawbacks (as you can see here, one of the drawbacks is that it had a fixed size).

256Mb is probably fine. I don't usually increase the MaxPermSize parameter - and, of course, under the latest Java version (with the latest Weblogic version - 12c) - the parameter isn't used anymore.
 
Back
Top