java.lang.OutOfMemoryError: Java heap space

Larry_Jones

Larry_Jones

Legendary Poster
Any tips / experiences that can point us into resolution of this issue will be greatly appreciated!
-------------------------------------------------------

Last week we did a Load Test on our new 9.0 configuration.
We're in the process of upgrading from 8.9 to 9.0 and nearly ready to go live on 9.0.

Our Load Test did its job magnificently - it uncovered the fact that our configuration fails (JVMs out of memory) well before the anticipated load / number of users.

We don't believe we have an issue with the hardware / OS setup.
Our config is:
-------------------------------------------------------------
Database: Oracle 11G on separate Win 2003 64bit server - 16gb memory.
Enterprise: 8 core Xeon - 8 GB Ram - Win 2008 64bit
Batch Server: ditto
Web Server1: ditto
Web Server2: 16 core Xeon - 12 GB ram - Win 2008 64bit

For testing we used a single PY900 instance on each web server with a setup that basically balances the users between the servers as they logged in (I'm not the CNC - not sure of the mechanism).

JDE.ini, JAS.ini and JDBJ.ini files are in attached zip
--------------------------------------------------------------

So anyway, < 10 minutes into the load test user sessions start failing with a variety of errors, HTTP 500 errors, applications hanging, general slowdowns or freezes, etc. Max user count reached was 50. Max anticipated concurrent user load after go-live is 150.
---------------------------------------------------------------
Checking logs we saw the following:
Web Server log:
10/01/22 09:06:12.433 Internal server error
java.lang.OutOfMemoryError: Java heap space
at java.math.BigDecimal.<init>(BigDecimal.java:444)
at java.math.BigDecimal.<init>(BigDecimal.java:647)
at com.jdedwards.base.datatypes.tagMathNumeric.parseNumericString(Unknown Source)
at com.jdedwards.base.datatypes.tagMathNumeric.<init>(Unknown Source)
at com.jdedwards.runtime.er.erobjects.ERObject.createInitialValue(Unknown Source)
at com.jdedwards.runtime.model.grid.FormGridModel.insertGVRow(Unknown Source)
at com.jdedwards.runtime.engine.FormGridEngine.insertModelRow(Unknown Source)
at com.jdedwards.runtime.vtcomponent.grid.VTFormGrid.addDataFromGC(Unknown Source)
at com.jdedwards.runtime.engine.FormGridEngine.addGridData(Unknown Source)
at com.jdedwards.runtime.vtcomponent.grid.VTFormGrid.addGridData(Unknown Source)
at com.jdedwards.runtime.engine.form.FormEngine.populateGridFromResultSet(Unknown Source)
at com.jdedwards.runtime.vtcomponent.form.VTForm.populateGridFromResultSet(Unknown Source)
at com.jdedwards.runtime.model.BusinessView.processData(Unknown Source)
at com.jdedwards.runtime.model.BusinessView.fetchDetailData(Unknown Source)
at com.jdedwards.runtime.vtcomponent.form.VTFetchManager.runtimeFetchStartPage(Unknown Source)
at com.jdedwards.runtime.vtcomponent.form.VTFetchManager.runtimeFetch(Unknown Source)
at com.jdedwards.runtime.engine.form.FindBrowseEngine.fetchDetailData(Unknown Source)
at com.jdedwards.runtime.engine.form.FindBrowseEngine.fetchDetailData(Unknown Source)
at com.jdedwards.runtime.engine.form.FindBrowseEngine.processFind(Unknown Source)
at com.jdedwards.runtime.vtcomponent.form.VTForm.processFind(Unknown Source)
at com.jdedwards.runtime.engine.ButtonEngine.onClick(Unknown Source)
at com.jdedwards.runtime.vtcomponent.VTHyperMenuExit.sysPressButton(Unknown Source)
at com.jdedwards.runtime.vtcomponent.VTHyperMenuExit.onClick(Unknown Source)
at com.jdedwards.runtime.vtcomponent.VTHyperMenuExit.processVirtualEvent(Unknown Source)
at com.jdedwards.runtime.virtual.OWVirtual.processVTEvent(Unknown Source)
at com.jdedwards.runtime.virtual.OWVirtual.processEventLoop(Unknown Source)
at com.jdedwards.runtime.virtual.OWVirtual.run(Unknown Source)
at com.jdedwards.base.util.ThreadPool$WorkerThread.run(Unknown Source)
10/01/22 09:06:14.914 Internal server error
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedReader.<init>(BufferedReader.java:80)
at java.io.BufferedReader.<init>(BufferedReader.java:91)
at com.peoplesoft.pt.e1.metadata.dataAccess.SpecDatabaseService.createIMetaDataObject(Unknown Source)
-------------------------------------------------------------
PY900 Instance Log (e1root_***):
...
22 Jan 2010 06:35:06,443 [WARN ] MPUGSLEY - [RUNTIME] Could not generate NER: N7003390_PlugAndPlay_F43121
22 Jan 2010 08:13:23,702 [WARN ] - [RUNTIME] The application, P3102, does not have an ASVRHDR spec.
22 Jan 2010 08:37:39,628 [WARN ] LJONES - [RUNTIME] Error in VTGrid.processVirtualEvent - SET_FOCUS_FOR_INSERT_GB_ROW: try to set focus on an invisible row. The model row index is 1
22 Jan 2010 08:37:39,628 [WARN ] LJONES - [RUNTIME] Error in VTGrid.processVirtualEvent - SET_FOCUS_FOR_INSERT_GB_ROW: try to set focus on an invisible row. The model row index is 2
22 Jan 2010 08:37:59,580 [SEVERE] LJONES - [LIB] Error: Invalid number inside MathNumeric [ .]
22 Jan 2010 08:37:59,580 [SEVERE] LJONES - [LIB] trying new java.math.BigDecimal(buf.toString()) java.lang.NumberFormatException
java.lang.NumberFormatException
at java.math.BigDecimal.<init>(BigDecimal.java:368)
at java.math.BigDecimal.<init>(BigDecimal.java:647)
at com.jdedwards.system.lib.MathNumericImpl.readMathNumeric(Unknown Source)
...
22 Jan 2010 08:38:07,583 [SEVERE] LJONES - [LIB] Error: Invalid number inside MathNumeric [ .]
22 Jan 2010 08:38:07,583 [SEVERE] LJONES - [LIB] trying new java.math.BigDecimal(buf.toString()) java.lang.NumberFormatException
java.lang.NumberFormatException
at java.math.BigDecimal.<init>(BigDecimal.java:368)
at java.math.BigDecimal.<init>(BigDecimal.java:647)
at com.jdedwards.system.lib.MathNumericImpl.readMathNumeric(Unknown Source)
at com.jdedwards.system.lib.MathNumericImpl.<init>(Unknown Source)
at com.jdedwards.system.net.JdeNetInputStream.readMathNumeric(Unknown Source)
...
22 Jan 2010 08:58:10,632 [WARN ] SCHMAUTZ - [RUNTIME] Error in VTGrid.processVirtualEvent - SET_FOCUS_FOR_INSERT_GB_ROW: try to set focus on an invisible row. The model row index is 1
22 Jan 2010 08:59:46,198 [MANDATORY] - [JAS] User is not Authenticated.
22 Jan 2010 09:00:07,258 [WARN ] SCHMAUTZ - [RUNTIME] Unhandled exception in populateWebVersionTable
22 Jan 2010 09:00:07,258 [WARN ] SCHMAUTZ - [RUNTIME] Failed to select versions from F983051 table
22 Jan 2010 09:00:07,258 [WARN ] SCHMAUTZ - [RUNTIME]
22 Jan 2010 09:00:07,258 [WARN ] SCHMAUTZ - [RUNTIME] java.lang.NullPointerException
java.lang.NullPointerException
at com.jdedwards.runtime.generator.engine.ReportGenHelper.populateWebVersionTable(Unknown Source)
at com.jdedwards.runtime.generator.engine.ReportGenEngine.buildWebVrsnList(Unknown Source)
 

Attachments

  • 155092-temp.zip
    8.8 KB · Views: 166
Have you set the heap size of your JVMs? This is not in the INI files. You must set this through the OAS EM console.
 
Hi Ken.

Max Heapsize initially was at default of 768M.
We upped to 1024M but no real diff/impact in doing so.

Thx for the response.
 
My heap size was taken to 1.5Gb, which is pushing it. I still only get 15-20 users per JVM before it starts to go wrong. This is on 8.91.1.2 but we do have a session timeout of 90 mins, which is very bad, but one that I cannot reduce.

I've been very disappointed with OAS in terms of scalability, the only way I could get the servers to load balance properly over the JVMs on a single machine was to set the number of JVMs = No Processors - 1

So for each quad core server I only get 3 JVMs supporting 45-60 users in total, which IMHO isn't very good. Its for this reason I will be looking at WebLogic when its available for Windows\SQL Server.
 
Paul,

thanks for the reply.

Sounds like your experience is basically the same as ours. We too were pushed to use a longer session timeout.

However even changing session timeouts and cache settings to Oracle defaults (minimum values) we see little change.

Our consultant tells us that a single JVM should be able to handle 50 users with no problem, so we still have issues here I think.
 
What tools did you use for the load testing?
Appreciate if you can share the information as we cannot get 50 users to bang away on the system simultaneously.
 
Larry,
What is your Min heap size ? If the difference in the Min and Max is a lot there can be memory leak issues. I would set both Min and Max to 1024 for this test and later reduce it to 512 and 1024 since it is PY.
 
Martin,

we used People Tools . . .
smile.gif


specifically we had close to 50 people banging away continuously at their tasks. We're in something of a business lull here so we were fotunate.
 
try to set JVM starting parameter with maxPermSize to 256 M.
If I remember well default settings is 128M.
gg
 
In my previously experience java.lang.OutOfMemory Error due to a bad development methodology. I try to explain better.
If you customize a find/browse to put a new data dictionary column on the grid and to populate this column you call, in "Grid Record is fetched" event, a fetch on a table, this statement has great impact on the web interactive performance in terms of RAM and CPU usage.
It's very much better to create for exemple a NER where you write your fecth and call this NER from event "Grid Record is fetched".

sorry for my english
gg
 
Joel,

thanks for the reply. I'll use your post to update all the issues on this thread.

After seeing the out of memory error we modified the JVM Heap size default values from 32M -> 768M to 1024 ->1024. Later we changed these to 512M -> 1024M.

We actually had several configuration related issues on our system.

1. Too many Call Object Kernels configured. I believe we had the default value - 50 - which for our test was basically allocating 1 COK per user. Not a cause for the JVM out of memory errors - but once we had all users on the system we probably would have seen unneeded paging ocur. Reduced to 30 right now.

2. In general Cache settings and Timeouts set too high. In some cases we reduced this to the defaults or lower. In others we compromised between the default settings and what we'd really like to have. This was "probably" the primary cause of the Java Heap space out-of-memory error.

3. As mentioned above we changed the JVM Heap size Min and Max from defaults to 512M>1024M. This change just provided a bit more breathing rooom though in my opinion.

4. After making the above changes we no longer were seeing the "OutOfMemoryError" but we still experienced bottlenecks, freezes and miscellaneous weird errors once more than 30 users were on. We finally tracked this down to a setting on the HTTP server - the ThreadsPerChild setting (note - this setting has different behavior for Windows servers). The default value - 50 - was far too low. There are many different opinions on what to set this to. Based on what it does however I would argue that it should be at least 1.5 times the max expected concurrent users. After upping this number substantially our bottlencks and freezes finally went away.

Just to cover the other issues shown in the logs I posted:

A. The BigDecimal / MathNumeric error was traced to the Initialize Event of Form C - Time Entry and Revisions - in Application P311221 - Work Order Hours and Qty entry. It actually occurs somewhere inside a called BSFN . . . we have a open call into Oracle support on this but are not too hopeful of a resolution since we cannot tie this error in the log to a specific problem.

B. Errors such as "Could not generate NER: N7003390_PlugAndPlay_F43121" apparently are junk error messages that can be safely ignored.

C. Errors such as "SET_FOCUS_FOR_INSERT_GB_ROW: try to set focus on an invisible row" apparently are related to the coding trick to set focus on a specific grid row (which no longer works in 8.98). The error message may be ignored.

D. Error messages related to F983051 and populateWebVersionTable are more bogus messages that may be ignored.

FYI - we also changed JDK versions on all servers from 1.4.xx to 1.5.0_22. This was to correct a bug related to time - without the upgrade the servers would always be on Greenwich time.
 
Back
Top