HTML instances consuming 100% CPU beasvc

Soul Glo

Soul Glo

VIP Member
Hi,

We are E1 9.1 TR 9.1.3.2 WLS 10.3.6. We have 4 HTML servers running 4 JVM instances. We have been experiencing issues where the CPU utilization will jump to 100% after a day or so. We have the E1 instances starting as a windows service using the beasvc.exe as recommended.

I worked with an Oracle associate to create the create service scripts, they install fine and they run (Below is the script). In troubleshooting the issues with Oracle we tried several ways to get dumps with the service running as windows service with little success. Yesterday we started an instance of E1 using the startManagedWebLogic.cmd on the command line, this instead of launching beasvc launches a native java process. Since doing so I have not see the java process consume any more that 4% and there are currently 25 users active on that instance.

my question is has anyone seen this with 9.1 and is there another way to create windows service so that it launches java and not beasvc. or is there something in the WebLogic layer that is not reading the process correctly or cleaning up it's processes?

echo off
SETLOCAL
set JAVA_VENDOR=Sun
set DOMAIN_NAME=ERPJDEPVWB02_D1
set USERDOMAIN_HOME=D:\Oracle\Middleware\user_projects\domains\ERPJDEPVWB02_D1
set SERVER_NAME=wls_e1_pd96
set PRODUCTION_MODE=true
set ADMIN_URL=http://erpjdepvwb02:7001
set JAVA_VM=server
set JAVA_OPTIONS=-Dweblogic.Stdout="D:\Oracle\Middleware\user_projects\domains\ERPJDEPVWB02_D1\stdout.txt" -Dweblogic.Stderr="D:\Oracle\Middleware\user_projects\domains\ERPJDEPVWB02_D1\stderr.txt"
set MEM_ARGS=-Xms4096m -Xmx4096m
call "d:\Oracle\Middleware\wlserver_10.3\server\bin\installHTMLSvc.cmd"
ENDLOCAL

Thanks
Cleola
 
you should be using Nodemanager services to launch your adminserver (and thence automatically launch your instances). Lots of information on the web about how to set up nodemanager...
 
Jon is right, you should use the node manager but I don't think the problem is here.I have a client where sometimes (after 3-4 days) some instances go to high cpu usage (using the node manager to start the managed server).
 
Can anyone point me to where any of these documents are. Someone mentioned this to me from the Oracle community.

For example, leave nodemanager and AdminServer as services (adminserver should not be problematic as a windows service), and then create a small text file with the following lines:

connect('weblogic','welcome1','localhost:7001')
start('MYINSTANCE')


Then save it for example as C:\MyInstance.wlst
Finally create a batch file like C:\StartMyInstance.bat like:

@echo off
C:
cd C:\Oracle\Middleware\wlserver_10.3\common\bin
wlst.cmd C:\MyInstance.wlst


Notice how the batch file just calls the WLST scripting environment to execute the commands in C:\MyInstance.wlst.
On this example, WLS is installed on C:\Oracle\Middleware\wlserver_10.3, the administrator credentials are weblogic/welcome1, the AdminServer instance is on the default port 7001 and the managed instance is named MYINSTANCE (case sensitive).

Thanks
 
Jon,

There's really no right way of doing it but, sure, using the WebLogic Node Manager to manage startup can be easier than setting up Windows services for the Admin Server and Managed Servers.

Starting these things via services is not "wrong".

One advantage to using Node Manager is whatever you set for the Managed Server properties in the Admin Console will be present when the instance is started, versus having to set many of the same settings in the Windows registry for those server started as a service.

A disadvantage to using the Node Manager is if the instances are down when the server reboots, they'll remain down.

That's all I have time to say about that now.
 
I tested the process I got from Oracle Support Community and it worked fine. I will test some more and then I will have to create an alternate script to perform the stop instance.

What I liked about windows service is that it is easy for any system administrator to verify if the instance is running. With WLST or NodeManager they will have to look in task manager to see if the java processes are running. It will also be a bit more complex for Orion to monitor the services as they may be other java process running that are not for E1 that can trigger alerts if they fail (not likely) but possible.
 
We do also see this 100% cpu utilization on one jas instance issue. We are E1 9.1 apps, 9.1.2.0 Tools on Linux WebLogic servers. Our users are not experiencing performance issues during the times the cpu is 100%.
We restart the jas instances from Server Manager.
It can run for days or weeks before having the issue.
 
Are you proactively doing anything to resolve the issue? It does not seem to create issues for our users as they have not reported any. Ours will go for a few days to a week. It does not matter if we use windows script or server manager the behavior is the same. We are on Intel WLS servers and iSeries DB, 9.1 tools 9.1.3.2.

Are you using both Server Manager and Windows services?
 
We have similar issues with Tools 9.1.3.1 (E1 9.1)
Linux WLS
It takes anywhere from couple hours to couple days for a specific JVM to start getting 100% CPU Load.
It starts one by one but eventually all JVMs in cluster go nuts.

In our case we noticed that 'problematic' JVMs start generating huge number of log files related to DD error (but no mentioning of a specific DD item)
That's when CPU goes 100% .

We have been working with Oracle on this for quite some time, but no resolution yet.

=== log example ==

19 Dec 2013 15:29:41,538[WARN][xxxxx_HTML_xxx][RUNTIME]A JASServiceException is caught when fetching data dictionary information from JAS. Please exit and regenerate the form. szDict=DCTOjava.lang.NullPointerException: User name is null
 
We have been working with Oracle for over a month and they keep asking for logs and more logs and jstack dumps, thread dumps and on and on.....This past week we reduced the number of JVM's form 4 to 3 on each server and that seem to perform better. I also noticed that the JVM's that were started via Server Manager had more issues, that one I had to re-start.

I believe that this might have something to do with Java being used and not JRockit with the new release and Oracle E1 Support and WLS support are out of their element on this one.
 
I would absolutely try and use jRockit if possible. Under jRockit and the same tools release, I haven't seen the utilization issue on a regular basis.

However, I have indeed seen it on one Linux machine once - running 9.1.3.3 tools on jRockit, CPU utilization maxed out a thread on one instance after about 4 days of uptime, then maxed out the other threads a little later on. I restarted the instances on that machine, and haven't seen the issue since - that was at least 3 weeks ago, and the instances are running fine. I put it down to a network issue (disconnect or something).
 
We were told that JRockit is no longer supported with the 9.1 release and WLS 10.3.6.

If we have to downgrade to 10.3.5 and JRockit that means having to re-create all our servers. I applied the latest WLS patch to a server today and will see if it has any improvements. I also reduced the JVM's from 4 to 3 and that helped some instead of every few days of 100% we can go almost 4-5 days. This does not seem to be affecting the users.
 
[ QUOTE ]
We were told that JRockit is no longer supported with the 9.1 release and WLS 10.3.6.

[/ QUOTE ]

I don't see this.

I just checked Oracle's Certifications on the support website - this is what I found.

For 9.1.3.3 tools release, Oracle Weblogic 10.3.6 is required. There is no reference under the JDEdwards HTML to what Java version is supported - but under the 10.3.6 Weblogic, I see Oracle jRockit 64Bit 6u37 R28.2.5+ being the supported java version.

I would like to know what contradicts this ? Was it Oracle Support ? Or some piece of documentation ?

Many customers are using jRockit - so a change like you stated would worry many implementors.
 
Hi Jon,

They may be referring to the Java 7 support which as far as I can tell eliminates JRockit as a separate product. My reading of it is that they either have or are in the process of integrating the JRockit features such as aspects of its JIT compiler, Garbage collector and Mission Control into the Java 7 platform.

See this announcement: http://www.oracle.com/us/corporate/press/1603497

From 9.1.3 tools and Weblogic 10.3.6 you will see Oracle JDK 7 listed in the certification matrix for the HTML Server. It also appears that onky JDK 7 is support for 10.3.6 which is why for my recent installs over the last 6 months I have not been using JRockit as I have been installing WLS 10.3.6.

See: JD Edwards EnterpriseOne 9.1.X HTML Server Certifications (Doc ID 1506201.1)

Take note that with Java 7 on Windows there is a MaxPermSize of 350M that should be used to avoid issues with Weblogic startup (see http://docs.oracle.com/cd/E23943_01/doc.1111/e14142/jdk7.htm) if you startup using certain methods. I am using NSSM to wrap the Weblogic startup into a windows service instead of the Node Agent and therefore ran into the issue with the default MaxPermSize.

I have not seen this 100% CPU issue under the configuration above.

JRockit may still be the better for JDE at this moment. I have seen various comparisons between JRockit and Java 7 with the "JRockit" options (whatever they may be) in it as well as the new G1 garbage collector. It is not clear to me that either one is best and it may depend on workload patterns. JRockit seems to have the edge in some situations. I am also not clear on which features of JRockit are in Java 7 at this moment and whether any of them require the commercial license that might not come with the JDE technology foundation. If the JRockit deterministic GC has been integrated into Java 7 then that might need a commercial license to be legal with E1.

It would be interesting to see a WLS 10.3.5 JRockit against 10.3.6 Java 7 comparison. But assuming that Java 7 is the way forward and that you are cut off from JRockit after 10.3.6 it is probably only interesting from an academic standpoint. There was a Weblogic tuning presentation for November's E1 Technical task force. It seems that the work was done under JRockit so there may be additional recommendation coming out this year. The general recommendation in the November presentation was to avoid esoteric JVM options and go with a 2 to 3 GB heap size. I have been using 3 GB under Java 7 64-bit. There was nothing else offered in the way of JVM tuning with the rest being WLS Threading, TCP and JDBC tuning. It seems that at least from the E1 standpoint that the recommendation for the JVM configuration is pretty much default and the stategy is to throw enough JVMs and memory at the concurrent user pool to keep heap usage below 70%.
 
I am using a windows service to start the service using beasvc and this is what the memory arguments are set to for each E1 instance. But I have tested this using java and startWebLogic.cmd and get the same CPU spikes.

-server -XX:MaxPermSize=1024m -Djavax.xml.rpc.ServiceFactory=oracle.j2ee.ws.client.ServiceFactoryImpl -Xms4096m -Xmx4096m

and this is for admin server - -server -Xms1024m -Xmx1024m -XX:MaxPermSize=512m

Do I still need to adjust -XX:MaxPermSize=512m to 350?

I just applied the latest WLS patch but not sure yet if it will make a difference with the CPU issue.
 
Back
Top