Larry_Jones
Legendary Poster
Any tips / experiences that can point us into resolution of this issue will be greatly appreciated!
-------------------------------------------------------
Last week we did a Load Test on our new 9.0 configuration.
We're in the process of upgrading from 8.9 to 9.0 and nearly ready to go live on 9.0.
Our Load Test did its job magnificently - it uncovered the fact that our configuration fails (JVMs out of memory) well before the anticipated load / number of users.
We don't believe we have an issue with the hardware / OS setup.
Our config is:
-------------------------------------------------------------
Database: Oracle 11G on separate Win 2003 64bit server - 16gb memory.
Enterprise: 8 core Xeon - 8 GB Ram - Win 2008 64bit
Batch Server: ditto
Web Server1: ditto
Web Server2: 16 core Xeon - 12 GB ram - Win 2008 64bit
For testing we used a single PY900 instance on each web server with a setup that basically balances the users between the servers as they logged in (I'm not the CNC - not sure of the mechanism).
JDE.ini, JAS.ini and JDBJ.ini files are in attached zip
--------------------------------------------------------------
So anyway, < 10 minutes into the load test user sessions start failing with a variety of errors, HTTP 500 errors, applications hanging, general slowdowns or freezes, etc. Max user count reached was 50. Max anticipated concurrent user load after go-live is 150.
---------------------------------------------------------------
Checking logs we saw the following:
Web Server log:
10/01/22 09:06:12.433 Internal server error
java.lang.OutOfMemoryError: Java heap space
at java.math.BigDecimal.<init>(BigDecimal.java:444)
at java.math.BigDecimal.<init>(BigDecimal.java:647)
at com.jdedwards.base.datatypes.tagMathNumeric.parseNumericString(Unknown Source)
at com.jdedwards.base.datatypes.tagMathNumeric.<init>(Unknown Source)
at com.jdedwards.runtime.er.erobjects.ERObject.createInitialValue(Unknown Source)
at com.jdedwards.runtime.model.grid.FormGridModel.insertGVRow(Unknown Source)
at com.jdedwards.runtime.engine.FormGridEngine.insertModelRow(Unknown Source)
at com.jdedwards.runtime.vtcomponent.grid.VTFormGrid.addDataFromGC(Unknown Source)
at com.jdedwards.runtime.engine.FormGridEngine.addGridData(Unknown Source)
at com.jdedwards.runtime.vtcomponent.grid.VTFormGrid.addGridData(Unknown Source)
at com.jdedwards.runtime.engine.form.FormEngine.populateGridFromResultSet(Unknown Source)
at com.jdedwards.runtime.vtcomponent.form.VTForm.populateGridFromResultSet(Unknown Source)
at com.jdedwards.runtime.model.BusinessView.processData(Unknown Source)
at com.jdedwards.runtime.model.BusinessView.fetchDetailData(Unknown Source)
at com.jdedwards.runtime.vtcomponent.form.VTFetchManager.runtimeFetchStartPage(Unknown Source)
at com.jdedwards.runtime.vtcomponent.form.VTFetchManager.runtimeFetch(Unknown Source)
at com.jdedwards.runtime.engine.form.FindBrowseEngine.fetchDetailData(Unknown Source)
at com.jdedwards.runtime.engine.form.FindBrowseEngine.fetchDetailData(Unknown Source)
at com.jdedwards.runtime.engine.form.FindBrowseEngine.processFind(Unknown Source)
at com.jdedwards.runtime.vtcomponent.form.VTForm.processFind(Unknown Source)
at com.jdedwards.runtime.engine.ButtonEngine.onClick(Unknown Source)
at com.jdedwards.runtime.vtcomponent.VTHyperMenuExit.sysPressButton(Unknown Source)
at com.jdedwards.runtime.vtcomponent.VTHyperMenuExit.onClick(Unknown Source)
at com.jdedwards.runtime.vtcomponent.VTHyperMenuExit.processVirtualEvent(Unknown Source)
at com.jdedwards.runtime.virtual.OWVirtual.processVTEvent(Unknown Source)
at com.jdedwards.runtime.virtual.OWVirtual.processEventLoop(Unknown Source)
at com.jdedwards.runtime.virtual.OWVirtual.run(Unknown Source)
at com.jdedwards.base.util.ThreadPool$WorkerThread.run(Unknown Source)
10/01/22 09:06:14.914 Internal server error
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedReader.<init>(BufferedReader.java:80)
at java.io.BufferedReader.<init>(BufferedReader.java:91)
at com.peoplesoft.pt.e1.metadata.dataAccess.SpecDatabaseService.createIMetaDataObject(Unknown Source)
-------------------------------------------------------------
PY900 Instance Log (e1root_***):
...
22 Jan 2010 06:35:06,443 [WARN ] MPUGSLEY - [RUNTIME] Could not generate NER: N7003390_PlugAndPlay_F43121
22 Jan 2010 08:13:23,702 [WARN ] - [RUNTIME] The application, P3102, does not have an ASVRHDR spec.
22 Jan 2010 08:37:39,628 [WARN ] LJONES - [RUNTIME] Error in VTGrid.processVirtualEvent - SET_FOCUS_FOR_INSERT_GB_ROW: try to set focus on an invisible row. The model row index is 1
22 Jan 2010 08:37:39,628 [WARN ] LJONES - [RUNTIME] Error in VTGrid.processVirtualEvent - SET_FOCUS_FOR_INSERT_GB_ROW: try to set focus on an invisible row. The model row index is 2
22 Jan 2010 08:37:59,580 [SEVERE] LJONES - [LIB] Error: Invalid number inside MathNumeric [ .]
22 Jan 2010 08:37:59,580 [SEVERE] LJONES - [LIB] trying new java.math.BigDecimal(buf.toString()) java.lang.NumberFormatException
java.lang.NumberFormatException
at java.math.BigDecimal.<init>(BigDecimal.java:368)
at java.math.BigDecimal.<init>(BigDecimal.java:647)
at com.jdedwards.system.lib.MathNumericImpl.readMathNumeric(Unknown Source)
...
22 Jan 2010 08:38:07,583 [SEVERE] LJONES - [LIB] Error: Invalid number inside MathNumeric [ .]
22 Jan 2010 08:38:07,583 [SEVERE] LJONES - [LIB] trying new java.math.BigDecimal(buf.toString()) java.lang.NumberFormatException
java.lang.NumberFormatException
at java.math.BigDecimal.<init>(BigDecimal.java:368)
at java.math.BigDecimal.<init>(BigDecimal.java:647)
at com.jdedwards.system.lib.MathNumericImpl.readMathNumeric(Unknown Source)
at com.jdedwards.system.lib.MathNumericImpl.<init>(Unknown Source)
at com.jdedwards.system.net.JdeNetInputStream.readMathNumeric(Unknown Source)
...
22 Jan 2010 08:58:10,632 [WARN ] SCHMAUTZ - [RUNTIME] Error in VTGrid.processVirtualEvent - SET_FOCUS_FOR_INSERT_GB_ROW: try to set focus on an invisible row. The model row index is 1
22 Jan 2010 08:59:46,198 [MANDATORY] - [JAS] User is not Authenticated.
22 Jan 2010 09:00:07,258 [WARN ] SCHMAUTZ - [RUNTIME] Unhandled exception in populateWebVersionTable
22 Jan 2010 09:00:07,258 [WARN ] SCHMAUTZ - [RUNTIME] Failed to select versions from F983051 table
22 Jan 2010 09:00:07,258 [WARN ] SCHMAUTZ - [RUNTIME]
22 Jan 2010 09:00:07,258 [WARN ] SCHMAUTZ - [RUNTIME] java.lang.NullPointerException
java.lang.NullPointerException
at com.jdedwards.runtime.generator.engine.ReportGenHelper.populateWebVersionTable(Unknown Source)
at com.jdedwards.runtime.generator.engine.ReportGenEngine.buildWebVrsnList(Unknown Source)
-------------------------------------------------------
Last week we did a Load Test on our new 9.0 configuration.
We're in the process of upgrading from 8.9 to 9.0 and nearly ready to go live on 9.0.
Our Load Test did its job magnificently - it uncovered the fact that our configuration fails (JVMs out of memory) well before the anticipated load / number of users.
We don't believe we have an issue with the hardware / OS setup.
Our config is:
-------------------------------------------------------------
Database: Oracle 11G on separate Win 2003 64bit server - 16gb memory.
Enterprise: 8 core Xeon - 8 GB Ram - Win 2008 64bit
Batch Server: ditto
Web Server1: ditto
Web Server2: 16 core Xeon - 12 GB ram - Win 2008 64bit
For testing we used a single PY900 instance on each web server with a setup that basically balances the users between the servers as they logged in (I'm not the CNC - not sure of the mechanism).
JDE.ini, JAS.ini and JDBJ.ini files are in attached zip
--------------------------------------------------------------
So anyway, < 10 minutes into the load test user sessions start failing with a variety of errors, HTTP 500 errors, applications hanging, general slowdowns or freezes, etc. Max user count reached was 50. Max anticipated concurrent user load after go-live is 150.
---------------------------------------------------------------
Checking logs we saw the following:
Web Server log:
10/01/22 09:06:12.433 Internal server error
java.lang.OutOfMemoryError: Java heap space
at java.math.BigDecimal.<init>(BigDecimal.java:444)
at java.math.BigDecimal.<init>(BigDecimal.java:647)
at com.jdedwards.base.datatypes.tagMathNumeric.parseNumericString(Unknown Source)
at com.jdedwards.base.datatypes.tagMathNumeric.<init>(Unknown Source)
at com.jdedwards.runtime.er.erobjects.ERObject.createInitialValue(Unknown Source)
at com.jdedwards.runtime.model.grid.FormGridModel.insertGVRow(Unknown Source)
at com.jdedwards.runtime.engine.FormGridEngine.insertModelRow(Unknown Source)
at com.jdedwards.runtime.vtcomponent.grid.VTFormGrid.addDataFromGC(Unknown Source)
at com.jdedwards.runtime.engine.FormGridEngine.addGridData(Unknown Source)
at com.jdedwards.runtime.vtcomponent.grid.VTFormGrid.addGridData(Unknown Source)
at com.jdedwards.runtime.engine.form.FormEngine.populateGridFromResultSet(Unknown Source)
at com.jdedwards.runtime.vtcomponent.form.VTForm.populateGridFromResultSet(Unknown Source)
at com.jdedwards.runtime.model.BusinessView.processData(Unknown Source)
at com.jdedwards.runtime.model.BusinessView.fetchDetailData(Unknown Source)
at com.jdedwards.runtime.vtcomponent.form.VTFetchManager.runtimeFetchStartPage(Unknown Source)
at com.jdedwards.runtime.vtcomponent.form.VTFetchManager.runtimeFetch(Unknown Source)
at com.jdedwards.runtime.engine.form.FindBrowseEngine.fetchDetailData(Unknown Source)
at com.jdedwards.runtime.engine.form.FindBrowseEngine.fetchDetailData(Unknown Source)
at com.jdedwards.runtime.engine.form.FindBrowseEngine.processFind(Unknown Source)
at com.jdedwards.runtime.vtcomponent.form.VTForm.processFind(Unknown Source)
at com.jdedwards.runtime.engine.ButtonEngine.onClick(Unknown Source)
at com.jdedwards.runtime.vtcomponent.VTHyperMenuExit.sysPressButton(Unknown Source)
at com.jdedwards.runtime.vtcomponent.VTHyperMenuExit.onClick(Unknown Source)
at com.jdedwards.runtime.vtcomponent.VTHyperMenuExit.processVirtualEvent(Unknown Source)
at com.jdedwards.runtime.virtual.OWVirtual.processVTEvent(Unknown Source)
at com.jdedwards.runtime.virtual.OWVirtual.processEventLoop(Unknown Source)
at com.jdedwards.runtime.virtual.OWVirtual.run(Unknown Source)
at com.jdedwards.base.util.ThreadPool$WorkerThread.run(Unknown Source)
10/01/22 09:06:14.914 Internal server error
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedReader.<init>(BufferedReader.java:80)
at java.io.BufferedReader.<init>(BufferedReader.java:91)
at com.peoplesoft.pt.e1.metadata.dataAccess.SpecDatabaseService.createIMetaDataObject(Unknown Source)
-------------------------------------------------------------
PY900 Instance Log (e1root_***):
...
22 Jan 2010 06:35:06,443 [WARN ] MPUGSLEY - [RUNTIME] Could not generate NER: N7003390_PlugAndPlay_F43121
22 Jan 2010 08:13:23,702 [WARN ] - [RUNTIME] The application, P3102, does not have an ASVRHDR spec.
22 Jan 2010 08:37:39,628 [WARN ] LJONES - [RUNTIME] Error in VTGrid.processVirtualEvent - SET_FOCUS_FOR_INSERT_GB_ROW: try to set focus on an invisible row. The model row index is 1
22 Jan 2010 08:37:39,628 [WARN ] LJONES - [RUNTIME] Error in VTGrid.processVirtualEvent - SET_FOCUS_FOR_INSERT_GB_ROW: try to set focus on an invisible row. The model row index is 2
22 Jan 2010 08:37:59,580 [SEVERE] LJONES - [LIB] Error: Invalid number inside MathNumeric [ .]
22 Jan 2010 08:37:59,580 [SEVERE] LJONES - [LIB] trying new java.math.BigDecimal(buf.toString()) java.lang.NumberFormatException
java.lang.NumberFormatException
at java.math.BigDecimal.<init>(BigDecimal.java:368)
at java.math.BigDecimal.<init>(BigDecimal.java:647)
at com.jdedwards.system.lib.MathNumericImpl.readMathNumeric(Unknown Source)
...
22 Jan 2010 08:38:07,583 [SEVERE] LJONES - [LIB] Error: Invalid number inside MathNumeric [ .]
22 Jan 2010 08:38:07,583 [SEVERE] LJONES - [LIB] trying new java.math.BigDecimal(buf.toString()) java.lang.NumberFormatException
java.lang.NumberFormatException
at java.math.BigDecimal.<init>(BigDecimal.java:368)
at java.math.BigDecimal.<init>(BigDecimal.java:647)
at com.jdedwards.system.lib.MathNumericImpl.readMathNumeric(Unknown Source)
at com.jdedwards.system.lib.MathNumericImpl.<init>(Unknown Source)
at com.jdedwards.system.net.JdeNetInputStream.readMathNumeric(Unknown Source)
...
22 Jan 2010 08:58:10,632 [WARN ] SCHMAUTZ - [RUNTIME] Error in VTGrid.processVirtualEvent - SET_FOCUS_FOR_INSERT_GB_ROW: try to set focus on an invisible row. The model row index is 1
22 Jan 2010 08:59:46,198 [MANDATORY] - [JAS] User is not Authenticated.
22 Jan 2010 09:00:07,258 [WARN ] SCHMAUTZ - [RUNTIME] Unhandled exception in populateWebVersionTable
22 Jan 2010 09:00:07,258 [WARN ] SCHMAUTZ - [RUNTIME] Failed to select versions from F983051 table
22 Jan 2010 09:00:07,258 [WARN ] SCHMAUTZ - [RUNTIME]
22 Jan 2010 09:00:07,258 [WARN ] SCHMAUTZ - [RUNTIME] java.lang.NullPointerException
java.lang.NullPointerException
at com.jdedwards.runtime.generator.engine.ReportGenHelper.populateWebVersionTable(Unknown Source)
at com.jdedwards.runtime.generator.engine.ReportGenEngine.buildWebVrsnList(Unknown Source)