UBE Permonace Issue 9.2 - Sporadically

Soumen

Reputable Poster
We are facing an issue with a specific custom ube where the run times would widely fluctuate between couple of minutes to hours. There is no difference between the processing options and data selections of these 2 runs and they are mostly a resubmit of the same UBE by the user. But what would usually take 20 mins would countinue to run till 6-7 hours sometimes. Another version of the same UBE which usually takes 3-4 mins would run down to 20-25 mins sometimes. So almost a 7 fold increase in processing times happening sporadically.

We are currently in the middle of an upgrade and this is a 9.2 (TR 9.2.2.4) system which users are testing. The server is an AS400 server with V7R3 and is currently on the same host as our current production, just different partition. So resource consumption should not be an issue as far as I think and moreover there are not much users currently on the system. I have been monitoring the CPU during the time job runs and have not noticed any spikes.

More importantly the same UBE doesnot behave the same on the current 9.0 production system. So I highly doubt if the code could be an issue here.

Attached are some of the runtimes between 9.0 and 9.2 for compartitive analysis we did.

I am looking for some ideas on how to narrow down this issue and get to the root cause. Any help would be deeply appreciated.

And yes I have checked the jde.ini setting and Callobj Monitor Working Directory is set to 0.

Thanks,
Soumen
9.0 & 9.2
IBMi V7R1 & V7R3
WebLogic 10g, 12c

P.S:- Sorry on the typo in the subject line... !
 

Attachments

  • UBE_Performance.jpg
    UBE_Performance.jpg
    151.6 KB · Views: 22
Last edited:

Tom_Davidson

VIP Member
There are lots of things that could be the cause, what else is going on in the machine at the time? Does batch run in it's own memory pool? If so do you have autotune turned on? If this is an update UBE are users (or another batch job) using some of the records this UBE needs (record locks)?

These are just a few things.

Tom
 

ice_cube210

VIP Member
Have you tried turning on UBE introspection or logging when the job is running for 6 to 7 hours when you expect it not to? Also, have you looked at the records processed in execution detail to compare the ones that ran long versus ones that did not.
 

Soumen

Reputable Poster
The UBE is actually an extraction UBE where it extacts records to a flat file on IFS. We are in UAt testing so there is not much activity in the server which can create records locks. I even compared records count between 2 runs (ran long versus ones that did not) with different times and they match.

My next step is to run this UBE on FAT client and try to duplicate the issue and if I can ... then turn the debug and see if I can find anything. But considering that the issue does not happen everytime makes it harder to replicate.
 

eydeak

Reputable Poster
Are you extracting directly from transactional tables? Or from something like EDI tables? I ask, because it could be some sort of specific data related issue. If it was a work area like EDI, you could copy the EDI data down to a test environment to see if you get the same results. And if you get the same results, you can get some debug logs to dive into. If you copy it down and don't get the same results, then it sounds more like a system issue.
 

Aarto

Reputable Poster
Hi There! The screenshot does not show the actual runtime of the UBE. Date/time when job was submitted is not necessarily when it started processesing as the job could have been waiting for another UBE in the same queue to complete first. Have you checked the "Job audit" info to see the actual effective runtime of the UBE and how many records it processed?
(Highlight the job in "submitted jobs" and take row/exit "Execution detail" and then select tab "Audit data" This will show the actual runtime of the job and how many records it processed
It could be that you have user-submitted jobs running in the same queue as some long running scheduled jobs for example

Hope this helps
Aarto
 

Richard Lee

Active Member
Hi Soumen,

have you try the old tools the performance monitor?
file name: Performance Workbench.exe
zip packed file: PW.exe
it parsed the ube log file and help you to identify in more detail.

I used to happen analyze a batch update report that best run time was 1 hour but some days it jumped to over 10 hrs.
I was using the oracle database 11g-12c. Using the oracle enterprise control, i quickly identify the SQL plan jumped to a worse plan and i asked dba to do something.
in some case in my env, there are many report queries in our database, some query huge ledger table (f4111, f42119....) , they caused CPU high load and increase the runtime per query.
in the worse case, the database is stuck until it's session time expired or we kill the session.

R/L
 

Soumen

Reputable Poster
Hi Soumen,

have you try the old tools the performance monitor?
file name: Performance Workbench.exe
zip packed file: PW.exe
it parsed the ube log file and help you to identify in more detail.

I used to happen analyze a batch update report that best run time was 1 hour but some days it jumped to over 10 hrs.
I was using the oracle database 11g-12c. Using the oracle enterprise control, i quickly identify the SQL plan jumped to a worse plan and i asked dba to do something.
in some case in my env, there are many report queries in our database, some query huge ledger table (f4111, f42119....) , they caused CPU high load and increase the runtime per query.
in the worse case, the database is stuck until it's session time expired or we kill the session.

R/L

Thanks Lee, I will try that out.
 
Top