Excessive delay in Login and first time opening of any application - 9.1 tools 9.1.5

[RonWeasley]

Hi,
We have a JDE lab instance running application 9.1 with tools 9.1.5.7. Our WebLogic 12c based HTML server is having unexplained performance issues. Each time we login, it takes more than a 1.5 mins to come to home screen. Each time a new application is opened, it takes almost an eternity to open it up. Subsequent openings are fast. Unfortunately, the debug log hardly reveals anything with unexplained time gaps ranging from 15 secs to 40 secs at various stages. Here are examples.

This is a lab and we don't have the permission to create SR and pursue it with Oracle. Any idea on what might be wrong? Please note that our server is pretty good by lab standards (32 GB RAM, 2.8 GHz Intel Xeon CPU 4 cores). Each time its slow, I checked the performance through windows task manager and hardly see any significant load in the system. There are no error messages in the system as well.

Any idea on how to proceed troubleshooting would be really appreciated...

Logs from Login Activity (Debug - Technical format with threads)
-------------------------------------------------------------------------

Excerpt #1 (unexplained 37 second gap in debug log)
------------------------------------------------------------
24 Feb 2016 13:11:00,225 [Line -1] [Timer-3] [DEBUG ] - [JDBJ] JDBj service [UDC table] found key [98-JS- D-null] in the cache, returned 1 records.
24 Feb 2016 13:11:00,225 [Line -1] [Timer-3] [DEBUG ] - [JDBJ] JDBj service [Environment Lookup Service] getting key [JDV910].
24 Feb 2016 13:11:00,225 [Line -1] [Timer-3] [DEBUG ] - [JDBJ] JDBj service [Environment Lookup Service] found key [JDV910] in the cache, returned 1 records.
24 Feb 2016 13:11:00,225 [Line -1] [Timer-3] [DEBUG ] - [JDBJ] JDBj service [Table group lookup] getting key [98-JS-UDC].
24 Feb 2016 13:11:00,225 [Line -1] [Timer-3] [DEBUG ] - [JDBJ] JDBj service [Table group lookup] found key [98-JS-UDC] in the cache, returned 0 records.
24 Feb 2016 13:11:37,131 [Line -1] [[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'] [DEBUG ] - [KERNEL] complete
24 Feb 2016 13:11:37,146 [Line -1] [[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'] [DEBUG ] - [JDBJ] Execute Business Function:LeftJustifyUDCValue, Host: E1NJINTES01:6016
24 Feb 2016 13:11:37,146 [Line -1] [[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'] [DEBUG ] - [KERNEL] execute
24 Feb 2016 13:11:37,146 [Line -1] [[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'] [DEBUG ] - [KERNEL] complete


Excerpt #2 (unexplained 15 sec gap - I see multiple gaps like this)
---------------------------------------------------------------------------
24 Feb 2016 13:12:27,588 [Line -1] [[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'] [DEBUG ] - [JDBJ] Checking exclusive application security, User = TESTUSER, Environment: JDV910, Role:*ALL
24 Feb 2016 13:12:27,588 [Line -1] [[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'] [DEBUG ] - [JDBJ] No application name is specified. Exclusive application security is not in effect., User = TESTUSER, Environment: JDV910, Role:*ALL
24 Feb 2016 13:12:27,588 [Line -1] [[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'] [DEBUG ] - [JDBJ] Setting connection runtime property No Where Paren to false.
24 Feb 2016 13:12:42,603 [Line -1] [[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'] [DEBUG ] - [MAFLETCONTAINER] A URL of /WatchlistManagerService.mafService?e1.namespace=&e1.service=WatchlistManagerService&RENDER_MAFLET=E1Menu&cmd=getNamesMenu&e1.state=maximized&e1.mode=view was Requested
24 Feb 2016 13:12:42,635 [Line -1] [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] [APP ] - [RUNTIME] WL:WatchlistManager:getManager:Obtaining the WatchlistManager singleton
24 Feb 2016 13:12:42,635 [Line -1] [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] [APP ] - [RUNTIME] WL:WLMetaDataManager:loadWatchlistForUserByDropdown: :TESTUSER
24 Feb 2016 13:12:42,635 [Line -1] [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] [DEBUG ] - [JAS] Calling server component com.jdedwards.jas.JDEQueries#processServerRequest


Hope this gives you some idea as I am running out of it! Thanks in advance for your help...

Regards...

 

[Tom_Davidson]

Ron, While your times are a bit longer than ours, it may be a caching issue. I know on Monday mornings we used to get complaints about JDE being slow. Our 'fix' was on Sunday evening we have the help desk log into JDE and hit a couple of the big programs, P01012, P42101, etc. and do a 'find' this reloads many of the 'basics'. Once we did that, no more complaints on Monday morning.

Just an idea...

 

[RonWeasley]

Tom,
Thanks. But the issue is not the delay in the opening of the application but also during log in. It is lab environment for testing purpose - there are hardly any users. I still get a more than a minute delay to login to my HTML environment. And any new application takes an eternity to open up. This delay is not the usual Monday morning delay that many of us experience.
I also think it is something to do with my HTML server or WLS cache; but with just gaps and no errors mentioned in the logs; I am unable to pin-point it.

Regards....

 

[Chan Rana]

Not sure how your servers are setup on LAN/WAN but when we moved recently on WAN there was same behavior we found specially bcus the Systems DB was across the network.

Chan

 

[ice_cube210]

Have you checked logs on the Enterprise Server , specifically security kernel to see if any delays or gaps there in the authentication process.

What platform is the Database on ?

Does the user signing on have a valid address book number defined ?

 

[Larry_Jones]

A little more information would be helpful.
Server platforms? Database Platforms?
Is just the Weblogic web server in a test/lab environment? What about Enterprise Server?
Why do I have the feeling you're running everything on 1 VM Host?

 

[AlWasio Consulting]

Need more information on what servers and connectivity being used.

 

[RonWeasley]

Hi All,
It is a lab (practice) environment. I have two windows 2012 R2 server (Intel Xeon octacore 2.8 GHz) with Deployment Server having 16 GB and Ent Server having 32 GB RAM. The Enterprise Server also hosts the Oracle 12c database and WebLogic 12c for HTML, BSSV, AIS servers. Hardly anyone uses it - (1 or 2 users at a time). This setup was done for our practice and PoC stuffs.

There is hardly any CPU / memory consumption (as evident from Windows task manager) when I try to login and the system generates the above unexplained gaps. However, I do see hogging thread in WLS console. I have attached the thread stack dump for everyone's reference. I am trying to figure out if that indicates something - any directions on it would be really helpful!

Thanks.

 

[RonWeasley]

no entries in the security kernel. WLS shows some hogging thread ... updated the thread with details.