WAS EGENs runnig slow and Web application crashing on a regular basis

KENNEY

Well Known Member
We are experiencing a slowness on running EGENs on our WAS 4.0.5 environment.

A full GEN on an environment was taking about an hour on average now it is running 6 to 14 hours and seems to be getting worse, too top it off our WEB server crashes about three times a day which I am sure has to be related.

I have checked the JAS logs and EGEN logs as well as the Oracle database and nothing seems abnormal.

Can anyone suggest anything that I might look at, Oracle support has not been able to offer much help as yet.
 
OK obvious question first - what's changed - don't forget to consider personnel, hardware, O/S patches, DB patches, Anti Virus, java in that thought.

Are you egen'ing in direct mode?
What's performance like just through say SQL Plus from the gen machine?
Have you tried truncating F989999 and F989998 before a gen?
Is it equally bad for all pathcodes?

What sort of web server crash are you getting?
 
Thanks for your input ,
What has changed, We installed a new application server, both old and new
were AIX on basically the same AIX release I will verify that all the
patches have been applied, One issue originally overlooked was the C
compiler was not installed, does the EGEN process use the C compiler on
the application server.

The WEB server itself is an AIX box and has not changed.
Another issue that has occurred since our last full EGEN is some patches
were applied to the Oracle Db.

As for your other answers
We are generating in direct mode.
I always truncate the tables F989998,F989999 before the direct
generation .
The generation problems are occurring in all paths. Initially after
noticing the problem in PD the DV generated normally, now it too is
crawling.

As for the crash on the Web it basically runs fine, If someone is signed
on at the time of the crash and DB operation will result in a messgae to
the effect "failure to find security server" , any users to are not on but
trying to connect to the Web server will simply fail to connect to the
server.

A few thing I will check on
1. Check to see what patches were applied to the Oracle DB.
2. Check to see any differences on the operating system between the old an
new systems.
3 Check on the compiler settings between the old an new system.
4 Roll back to an older full package for DV on both EGEN client and
server and try EGEN against this package.


Michael Kenney
IT Services
University of Windsor
(519)253-3000 ext 2754
 
Hi Kenney,

Do you have any logs from your crashing JAS instances and
WAS server?
 
Yes I have copies of the jas.log STDOUT, STDERR and RT.log, They don't show anything to abnormal , it seems that last thing logged in all of them was something related to passwords (invalid or about to expire), but these errors are contained throughout the log .

i.e Security Server return error status: 2/Invalid Password

About the pnly common with the server stopping is every business day it occurs between 11a.m and 12 noon and then usually about 3 hours later, however doenot occur on weekends. For what its worth I will attach a copy of a JAS.log.
 
Hi,

I have seen something similar (but not around e-gen) with an oracle installation. Check your alert log on the Oracle database and see if your system is suffering from connection problems.
 
Had DBA check the alerts and informs me that they are clean. Can anyone suggest any other logging that can be enabled i.e Websphere, Apache etc.
 
Hi,

you said in one of your earlier replies that you attached your log file, but I do not see it....
Your explanation about the security server error led me to get you to review your Oracle logs as logicallly if the user profile you are using for the generation has a problem you would get the security server error at login and not half way through the generation.
These errors tend to come when you lose connection to the database. Have you checked your security server logs on the server marked as your security server?
Do your web app / fat clients lose connectivity at these times of the day?
As a work around / test you could look at generating locally (create a new generation profile that points to the genertion tables in OCM to OneWorld Local - xxxxx for the environment that you are trying to generate for and if that works OK - (no problem with access / msde size limit) then you can do a copy table to the database location that you would normally generate to.

Without knowing more I would monitor your database at the times you mention whilst you are generating (to the database) and see if you can notice anything unusual.
 
I'd have a look at that new application server, particularly if its also acting as the security server. Security gets checked more often and in more places than I'd expected but it might also be symptomatic of a network level issue. We had an app server which would periodically take itself off the lan because of some IP packets it really took exception to. Took a while to prove but that was genuinely the case.
 
Terry I will try to attach a set of logs,

And to Richard you may be on to something, I notced the other day at the point it crashed, I also experienced a failure to connect to security error on my full client for a brief moment, I will talk to my DBA
 

Attachments

  • 131891-set4logs.zip
    6.6 KB · Views: 100
Any more info?
You doing direct table generation? How many threads you allowing to the data server, they recommend like 25 to 50. Does the evnironment have JITI on/off?

Does your Data server logs show any problems with connections? Backup strategy locking database? Just random guesses...
 
Well to give anyone interested with an update of the issue.
We isolated the web crashing issue to the WAS persistent processes on our Apache server, for some reason when a process starts it is not terminating, we had the max set to 500 which is about 3 hours worth. To bandaid the problem we upped the maximum allowed processes to 2000 and have scheduled nightly restarts on both the APACHE and WAS server.

As for the slow Egen it may have been related to hardware problems on our EGen work station, I am currently in the process of rebuilding it.

Thankyou to all who responded with suggestions.
 
Back
Top