JDBC interface: JAS 5.0.2.3 and MS SQL 2000 - All wintels w2k3

[Mitch]

I wonder if other E1 8.10 shops with similar configs are having performance, and stability issues specifically in Websphere 5.0.2.3.

By mid-week, JAS server becomes unstable - you know this just by looking at the Summary info and numbers under SAW. This is happening even after extensive performance adjustments, and on top of the known unicode conversion overhead.

Something is strange about memory and cache mgt in JAS server. ie. When no users logged on in weekends, the JAS server could take up 750MB out of 786 MB heap size, and it may or may not recover after refresh.

For instance, while the Fat client runs an app in seconds, the same application may take 10 to 15 min to run on the web, and other times the web is as fast as the FAT client. It's also intermitten. The same business views are called by both clients.

App side: Applied related SARs, reindexed tables.
System side: E1 8.93 R1, all w2k3 servers, JDBC V2.2 of MS. Serialize Web objects using IBM JDK 1.3.1 w patch.

We're down to JDBC interface right now based-on some lines in jas, and rt debug, ie. "removing old transactions" lines in jasdebug match with timing on the select statement in RTdebug. It's hard to understand this. Eventhough it's generally slow in web, if it were just a JDBC driver issue, shouldn't be happening to other apps as well - since we're having this performance issue in the worst form in just one application: the voucher receipt match.
Regards,
Mitch.

 

[brother_of_karamazov]

You don't mention what service pack you are on but when we applied SP23_F1 we found the following issue in HTMLClient.ini:

#byPassSerializableObjCache:
# TRUE: turn OFF serializableobject caching
# FALSE: turn ON serializableobject caching
# DEFAULT value is FALSE
byPassSerializableObjCache=TRUE

We sarred this with PeopleSoft and the following was generated:

ID# 7441538

[WEB CLIENT PERFORMANCE]
in HTMLClient.ini byPassSerializableObjCache=FALSE is the
new setting which allows for the WebServer to cache the
objects from database

Check your HTMLClient.ini to ensure that byPassSerializableObjCache=FALSE

 

[Jay Paff]

Mitch,

We are running WS 5.0.2.3 (Wintel) with our Xe implementation and we had some "stability" issues also. Our WS would run for about 25 hours and then hang. While running the Tivoli Performance Monitor I saw the thread counts on one of my instances start to grow very rapidly. This would cause all the instances to hang because the ND server would still try to direct traffic to this instance with the runaway thread counts. After working with PS, they sent me 3 class files, JDEThread.class, URIComponentManager.class and URIComponentManager$ComponentThread.class. PS indicated that we had implemented things in a way they had never seen before. They couldn't tell me what it was we had done, all they would say is that technically it wasn't incorrect and it revolved around the recovery of destroyed threads. To make a long story short, as soon as I implemented the new class files, my stability problems went away.

Hope this helps.

Jay

 

[Mitch]

We're on SP:R1 (S1 already out). Oh ..what a nighmare that SP:F1 was with the web tracing enabled in compiled code, and shipped to customers like us! The warnings is still there even on SP: R1.

Yes. byPassSerializableObjcache is already FALSE.

This may sound strange -as we don't run pgms on the web server. To me those instances are still programs eventhought we converted to services. But I think something isn't right in cache/mem mgt in websphere, on the OS side, I switched w2k3 server's performance options for memory usage to 'programs' as opposed to 'cache'. the default is cache but it was worst before. Now, I've decided to restart web services everyday.

Also websphere 5.0.2.3 runs on top of IBM http 1.3.26.2. (When I first checked IBM site months ago, the readme file had no references to Windows 2003 server at all - only 2000. But recently noticed that they listed w2k3 server on the web site under supported environments for 5.0.2.3. Not sure if they ever tested 5.0.2.3 on w2k3 extensively).
By the way, as I understand no IBM fix pack for 5.0.2.3.

Mitch
E810, 8.93 R1, all w2k3 no-sp1, MS SQL 2000 sp3a, Websphere 5.0.2.3 on IBM http 1.3.26.2

 

[Mitch]

Jay,
Our PD thread counts starts at 70s and stays around 170,180s. We also tried different class files but didn't help. During my next full serializaton, I will enable both error logs - it's going to be long and big logs.

I get mixed signals from GSC as to allocate 768MB or 1024MB heap size. Few of them says 1024 may corrupt data etc. Any issues with this one?

What I hear that 8.94 tools release has a lot of improvements in web performance but we're not there yet. We're new to E1. Are your other app and db servers are intels too?
Thx.
Mitch
E810, 8.93 R1, all Wintels w2k3, MS SQL 2000 sp3a, Websphere 5.0.2.3 on IBM http 1.3.26.2

 

[Jay Paff]

Hey Mitch,

Our heap is configured at 768 min, 1024 max and it never grows beyond min. I used to have min at 512, and it grew just a little bit, to about 550. I was always told to increment in 256 chunks, thus 768.

Jay

 

[Jay Paff]

Mitch,

Just thinking a bit more about your threads, you may be experiencing what I experienced. My thread min and max are default of 10 and 50 and I don't see my thread grow beyond 10. If yours grows to 170 or 180 you may be hitting the ceiling on your settings and thus your problem. What are your min and max for your threads? (I am talking about the Web container threads and not the ORB threads).

Jay

 

[Mitch]

Jay,
Ours maxthread in web container (of Admin console) was set to 50 too (default). But GSC recommended we increase it to 100 (as our maxusercount=100), also set it to growable (check box) . We were told OK as long as it's under < 200.

Oor SAW WEB summary example:
Heap memory allocated by VM: 805,698,048 bytes
Heap used memory in VM: 358,327,688 bytes
Available (free) memory in VM: 447,370,360 bytes
Total threads in VM: 177

Also, we're told to keep max and min same as 768(mistakenly put 786 earlier). Maybe i set it to 786(min) and 1024 (max).

Thanks,

 

[brother_of_karamazov]

I have heard both. Oddly IBM in their official documentation says to *never* have them set the same. I have always set them to 768-768 or 512-512 with no issues.



[ QUOTE ]


Also, we're told to keep max and min same as 768(mistakenly put 786 earlier). Maybe i set it to 786(min) and 1024 (max).

Thanks,

[/ QUOTE ]

 

[Jay Paff]

Mitch,

I don't know if I would rely solely on the thread count from SAW. This is all threads, including ORB and those from Web Container. If you can, get Tivoli Performance Monitor going, this will break out what your various thread counts are, plus it will monitor your garbage collection. Additionally, there is an advisor piece to the TPM that may help you.

Jay

 

[Mitch]

By the way, I never touched the OBR pool parameters in Admin Console. The defaults are sames as the web container threads. Mn: 10, max: 50, Growable button not checked.

 

[Mitch]

Jay,
Did you make any changes to your ORB thread counts? You're right. We don't look at at thread counts alone. the GSC is focusing on the JDBC driver. The supported SP is v2.2 from MS. The GSC did not recommend SP3 at this point. It's either something about that 'housekeeping process' that takes forever to clear, or the select statement for SQL.
Mitch,