Who is live on 8.11 using HP-Unix on a WAN

mtrottier

Well Known Member
I have a customer who live in on 8.11 in with no "Local" users and we are having fits with stability of Websphere. After about a half a day errors start increasing to the point that we have to bounce Websphere.

Is anyone out there with a similar configuration? If so what are your experiences.
 
How much memory is allocated to your WebSphere JVM? Perhaps the JAS server is starved for memory and as user activity increases you get increasing failures.

When you say errors are increasing what errors are you seeing in the JAS logs?
 
I've tried bumping up the Heap size as high as min=1024 max=2048.

Error message vary but here are some:
[TRANSACTION_STATE_NOT_VALID] Transaction state is not valid

Error encountered getting menu data: null

ERROR: com.jdedwards.system.none - <?xml version="1.0"

ERROR: com.jdedwards.jas - trying jmiMethod.getMethod().invoke (e instanceof InvocationTargetException) java.lang.OutOfMemoryError

ERROR: {com.jdedwards.taskexplorer} - Unable to load children nodes

An SQL exception occurred: Bigger type length than Maximum.
ERROR: {com.jdedwards.jas} - Security Server return error status: 16/ Invalid Token

Then we get strange things like when "USER A" tries to navigate the menus they get a "critical system" erorr and yet "USER B" can see the same menu just fine.

The real killer is that eventualy every user who tries to login gets the "A unknown Jas error has occured" message when trying to login.

While all this is happening on say the websphere App running on Port 83 users can use the App running on port 82 just fine. Then if we stop and start the websphere app on port 83 the errors go away.
 
You can check and try several things...
1) you are not using more than one environment per instance in PathCodes=... in [OWWEB].
2) Instead of rebooting the JAS instance next time, try "Clear EnterpriseOne Menu cache" from SAW first; you can also clear other caches (security, OCM, connection pool, JDENET) as well and see if those help you identify the problem.
3) check security server log, compare jde.ini settings with TokenGen.ini, also clocks, timezones.
4) Are you single or multiple foundation and what Tools Release?
5) Did your tasks/roles get "upgraded" incorectly from a prior release? Anything in the solution explorer merge logs?
 
Answers to above questions:

1) you are not using more than one environment per instance in PathCodes=... in [OWWEB].
No. Users can only use JPD811 on the Web instance

2) Instead of rebooting the JAS instance next time, try "Clear EnterpriseOne Menu cache" from SAW first; you can also clear other caches (security, OCM, connection pool, JDENET) as well and see if those help you identify the problem.
Yes we have tried clearing all caches but this did not help.

3) check security server log, compare jde.ini settings with TokenGen.ini, also clocks, timezones.
I haven't looked at this.

4) Are you single or multiple foundation and what Tools Release?
We are single foundation 8.94I1

5) Did your tasks/roles get "upgraded" incorectly from a prior release? Anything in the solution explorer merge logs.
This was a fresh install not an upgrade.
 
OK, a couple of more:

1) HC and other user overrides in F98950 -- ahhh if we all had a penny for each time we had to delete those... They used to crash fat clients, they crash JAS servers too
smile.gif

2) What version/fixpack of Websphere or OC4J and most importantly, what are your JDK levels? And, if using WAS, there are Java fixes needed for certain WebSphere fixpack combinatons, discussed in the later posts of this thread:
http://www.jdelist.com/ubb/showflat.php?Number=101731
 
If it were a user override problem I don't think bouncing the Web server would resolve them.

We are running Websphere 5.0.2.3 on HP Unix and I'm fairly confident that we have the proper fixpacks and JDK levels.

I did find some interesting entries in the JDE.log files for the security kernal that appear to be created at the time the users were getting the "Unknown Jas error" message at login:

18195 Mon Apr 10 15:47:38.061526 jdeksec.c2409
Unable to find user record from table F98OWSEC

18195 Mon Apr 10 16:21:20.589019 userctx.cpp109
PSEOneUserCtxFactoryCreateAuthToken: Invalid token length or frame.

18195 Mon Apr 10 16:21:20.589222 jdeksec.c3986
Failed to validate auth token: token library unable to validate token


I checked the timeout settings in the JAS.INI and websphere and I think they look OK:
USERSESSION=7200000
and
Session Timeout: 60 min
 
I know I may be asking the obvious, but do you have password policies, long srver names, SSO, LDAP, portal ...
 
We are just using the basic E1 sign on security.
Again even if this were the problem I don't think bouncing the Web instance would resolve it.

Again the symptoms are that everything works great for a while and then as time goes on things get worse and worse until no one can log in. This usually happens in less than a day.

Another interesting fact is that during development, where most users were "local", we could go weeks without seeing any problems and no bouncing was required. It seemed to get worse only when the majority of the users were "remote".

This is why I’m asking if anyone out there is setup in such a way that the majority of their users are remote?
 
So here is the Oracle "Help Desk" response:

"The recommended heap size in WebSphere 5.0.2 for our JAS instances is 768/768. This is large enough to prevent the internal server error message, but small enough to avoid the CPU spikes that the higher level heap size setting seems to create.

Isn't that helpfull?????
 
Mike,

in almost all cases where I've seen this issue it's mostly related to memory leaks from JAS. How many ESU's have you applied? Are you fix current? There are several ESU's available for performance and memory leaks for all versions of E1. You can use Change Assistant to identify these, download them and install them. If you don't have these ESU's on the system the JVM will crash under load.

Also I'd recommend getting to a more recent version of WAS. I'd try atleast 5.0.2.6.......all my clients are at 5.0.2.12.


Colin
 
According to Change Assistant there were no "Memory" related ESUs for the HP 9000.

Of course it does suggest installing the latest Service Pack.
 
The error that now appears to be related to the "Unknown JAS Signon" error is the following:

11 Apr 2006 17:38:31,917 [Servlet.Engine.Transports : 63] ERROR: {com.jdedwards.jas} - JDESignOn Fail [SQL_EXCEPTION_OCCURRED] An SQL exception occurred: Bigger type length than Maximum.
java.sql.SQLException: Bigger type length than Maximum

From what I can find with Google is that this might be related to my classes12.jar file on the server. Does anyone know how I make sure I have the "Latest" "Compatable" version of this file?
 
Are you running 9i or 10g?

If you're running 9i then the default classes12.jar file is bad. You would need to download the updated one from Metalink.

Colin
 
Mike,

the ESU's are platform independent. Remove the platfrom when doing your search. Right now there are 27 Memory ESU's and 36 Performance ESU's for 811. Without them you'll have an unstable JVM.

The updated classes12.jar should help.

Updating WAS to 5.0.2.12 with the updated JDK should help WAS stability.

The newest Tools Releases may help but aren't really required since your should be pretty good.


Colin
 
I got someone new at Oralce Help desk and she seems to think it has something to do with the fact that all our users are coming in over the WAN. Here is her response:

<font color="blue"> </font> Here is some information that I have for using web clients over a WAN environment...

TaskExplorer in Low Bandwidth /High Latency Network issues
1) Excessive Round Trips
Task Explorer when rendering the menuing tree makes a lot of Round-Trips to Web Server.
e.g. : in Staging package Internet Explorer makes 87 round trips to WebServer to render the initial menu

2) Voluminous Data being transferred
Task Explorer Rendering sends a lot of HTML data for the browser to render
e.g.: in initial rendering 124,000 bytes are sent , in subsequent rendering of child menus ~50,000 bytes are sent over the wire

1) Configuration Fix:
a) HTTP Caching : We have configured IBM HTTP Apache WebServer to set Expires header on resources it serves - in particular images
This would ensure that a HTTP Compliant Browser would not make superfluous roundtrips to retrieve the same image 50 times
With this setting Mozilla and FireFox make only 2 round trips down from 70+ to render the initial page

b) HTTP2 Data Compression: We have configured IBM HTTP2 Apache WebServer to compress HTML
with this setting the amount of Data compresses from average 50,000 bytes down to average 4,500 bytes

if we lived in a perfect world these two settings would result in low roundtrips and less data being transferred
however Internet Explorer has a bug which ignores the Cache Expires HTTP Header in inner HTML and still makes 70+ round trips

2) Code Fix
http://support.microsoft.com/default.aspx?scid=kb;en-us;319546
This is the bug in Internet Explorer which exhibits itself when we run innerHTML

This fix reduces the round trips to 24 down from 87 ,
this is the best we can do with InternetExplorer as it makes a new Connection for each unique resource
FireFox keeps the connection open and makes multiple HTTP Gets in the same connection

How Customer X got around the problem
Customer X turned on HTTP 2.0 compression which helped a great deal with general response time and screen rendering. However, they had an additional problem with their overseas sites when rendering the task views in the HTML client. This was caused by a 0 byte .GIF file which is part of the rendered task view (menu). There are dozens of these on a page and even though they are effectively empty, the IE browser had to go back to the web server every time one was called (because the local copy was 0 bytes) which resulted in a long delay due to network latency. To fix this, as well as to reduce the overall impact of other static HTML content, Customer X put in a piece of network equipment at their overseas sites that cached this type of content locally so the round-trips were on the local LAN and not over the WAN. This resolved the screen rendering issue.


Workaround for IE Bug

"To workaround the Internet Explorer bug as Microsoft says:
Create a brief time-delay so that Internet Explorer has enough time to verify whether the image is in the cache, and then call the innerHTML property."

to achievethis workaround: set this JVM property in the WebSphere's Generic JVM properties
-Dcom.peoplesoft.e1menu.ie.timeout="10" (or any other number)
this is delay in millisecs before E1 Menus will load the inner HTML

<font color="black"> </font>

Anyone have any thoughts on this?
 
Mike,

Did you resolved the stability issue of Websphere 5.0 on hp-ux? We are getting ready to go live in about a month on 8.11 SP1 Tools 8.95_F1 with WebSphere 5.0.2.12. Our enterprise server is hp-ux and we have a separate hp-ux box for our web server. Any tips/warnings/advice would be greatly appreciated.

Maria
 
Back
Top