Job Queues Backing Up

kmcevoy

Member
We process 2000-3000 orders per day. We ship confirm each order, generate
and print the invoice at the warehouse packing stations, and put the invoice
in the box with the shipment. We use WTS in 6 of 7 locations. For the
WTS locations we have to generate the invoice by submitting a batch job.
The process works fine until the queues start to backup. The busiest time
of the day is in the afternoon. The users will report that the invoice
jobs submitted to the queues are backing up. When we check the queues,
sure enough there are numerous jobs in a Wait status. When everything is
running normally, generating a single invoice will take a few seconds. We
setup a multi-threaded queue that allows 6 jobs to process at one time. When
things are running normally we rarely see more than 3 jobs running at a time
because they get processed so quickly. When the queues backup, it appears
that only 1 job will process at a time (acts like it is single threaded) and
its execution speed is a little slower. The queue service delays before
processing the waiting jobs and then treats the queue as single threaded.
No database locks are being held when this problem occurs. Enterprise
server performance is not slow at this time. Interactive users don't
notice any performance problems.

We have done the following to try to alleviate the problem:
- We drop and rebuild the F986110 table on a weekly basis. We analyze the
table and rebuild the indexes several times a day.
- While a backup is occurring, we do an analyze on the F986110 table and
sometimes the slowness goes away immediately, sometimes it doesn't.
- Every morning, we delete the completed jobs from the previous day. We
usually create about 2500-3500 jobs per day.
- The queues run on the enterprise server and it gets rebooted every night.
- Sometimes we can cycle the queue services and the problem goes away.
Sometimes the problem comes right back. On the really bad days we have had
to reboot the server.
- We can move a waiting job to a different queue and it may process
immediately; sometimes it waits before processing.

Has anyone encountered this kind of problem before? Any suggestions on
what else to check? Any feedback would be greatly appreciated.

Thanks,

Kevin McEvoy
J.A. Webster, Inc.
[email protected]

B733.2, SP11.1, NT, Oracle 8.05
 
Kevin,

what's the disk I-O like when this occurs?

The reason I ask is that we had a periodic problem with table F986110 under Oracle 8.0.5 that caused the queue jobs to perform full table scans of the F986110 table instead of using the indices. This behavior caused a tremendous amount of disk I-O to occur. Analyzing F986110 sometimes had a positive effect, sometimes not. The problem went away for us when we upgraded to Oracle 8i. Our jobs volume is nothing like yours I should note - one or two hundred a day.

Hope this helps.

Regards,

Larry Jones
[email protected]
OneWorld B733.1, SP 11.3
HPUX 11, Oracle SE 8.1.6
 
we only do about 1800-2300 jobs a day but I did post our solution to this a
couple days ago if you want to search the archives (search for queues and
you'll see my post in the last 3 weeks) which might help deal with some of
your problems.

- We drop and rebuild the F986110 table on a weekly basis. We analyze the
table and rebuild the indexes several times a day.

No way you should have to do this. we currently have 6 months worth of jobs
in the F986110 table (I know this is ridiculous but we just rolled out
distribution starting 6 months ago and we still have some paranoia about
what reports we might need to view in a moments notice.)


- Every morning, we delete the completed jobs from the previous day. We
usually create about 2500-3500 jobs per day.

Shouldn't have to do this either the only thing that would take time would
be when a user goes to look for their job in submitted jobs. That is all
this table is for basically. It might take a little longer for the ube
process to update the record but shouldn't affect ube performance.

- The queues run on the enterprise server and it gets rebooted every night.

Well that should make sure your memory is nice and clean but again shouldn't
have to do that. :)

- Sometimes we can cycle the queue services and the problem goes away.
Sometimes the problem comes right back. On the really bad days we have had
to reboot the server.

- We can move a waiting job to a different queue and it may process
immediately; sometimes it waits before processing.


These last two (in addition to the fact that you have this prob at your
heaviest load time)lead me to believe that you are experiencing a resource
issue. I'm no NT guy but we have found different points in our
implementation where we had to up our semaphores (aka communication
channels/memory resources) on the server and in the JDE.INI on the server.
In your case I would turn on performance monitor and see what is being
hammered the most when you start seeing problems. It might just be that your
box is spending a lot of time swapping and in this case limiting the jobs
that can run because it is at max swap capacity. Also get your DBA to verify
that the DB doesn't have some kind of unwieldy SQL going on (we all know
that OW doesn't produce the most elegant select statements all the time). If
most of your jobs are the same at the same time maybe you need a little
Oracle tuning. I'm lucky my DBA and my UNIX admin are awesome. Like I said
we do about 2000 a day at the moment but on our 12 way HP9000 they don't
even really cause a blip now that we have our queue director (see my post)
and some quality tuning. Sorry nothing definite but maybe it will get you
going in the right direction.

Mark Siebenschuh
HP9000/Oracle 8.0.5/B733 Base (XE on the way)
 
You can solve the performance problem by issuing

alter table f986110 cache;

Which will cause the blocks on the table to be moved to the end of the least
recently used list and will keep them in memory longer. The default
behavior in Oracle for a full table scan is to put the blocks at the front
of the LRU list in memory which means they get pushed out rather quickly
causing the excessive disk I/O. Another option, but less palatable is to
create a custom index on the columns related to the SQL causing the full
table scans and then rebuild it every half hour so.

HTH,
[email protected]
 
Mark,
Are you running the jobs that have to be run in a single threaded
queue in a single threaded queue? There is documentation on the KG
about which jobs must be run in a single threaded queue and which don't.
Most financial jobs should be run in a single threaded queue.

Lisa G. Stinebuck
Senior Service Delivery Technician
Logical EBOC Cincinnati
513-412-7950 x1012
[email protected]
 
Are you doing any package deployments to the
enterprise servers during the time the queues get
backed up? If you are this is the source of your
problem. When you deploy a package the ube will put a
lock on the specs. It will wait for all the jobs that
are currently processing to end and it will put the
new jobs on hold. Not sure if this is a case.
--- kmcevoy <[email protected]> wrote:
>
> We process 2000-3000 orders per day. We ship
> confirm each order, generate
> and print the invoice at the warehouse packing
> stations, and put the invoice
> in the box with the shipment. We use WTS in 6 of 7
> locations. For the
> WTS locations we have to generate the invoice by
> submitting a batch job.
> The process works fine until the queues start to
> backup. The busiest time
> of the day is in the afternoon. The users will
> report that the invoice
> jobs submitted to the queues are backing up. When
> we check the queues,
> sure enough there are numerous jobs in a Wait
> status. When everything is
> running normally, generating a single invoice will
> take a few seconds. We
> setup a multi-threaded queue that allows 6 jobs to
> process at one time. When
> things are running normally we rarely see more than
> 3 jobs running at a time
> because they get processed so quickly. When the
> queues backup, it appears
> that only 1 job will process at a time (acts like it
> is single threaded) and
> its execution speed is a little slower. The queue
> service delays before
> processing the waiting jobs and then treats the
> queue as single threaded.
> No database locks are being held when this problem
> occurs. Enterprise
> server performance is not slow at this time.
> Interactive users don't
> notice any performance problems.
>
> We have done the following to try to alleviate the
> problem:
> - We drop and rebuild the F986110 table on a weekly
> basis. We analyze the
> table and rebuild the indexes several times a day.
> - While a backup is occurring, we do an analyze on
> the F986110 table and
> sometimes the slowness goes away immediately,
> sometimes it doesn't.
> - Every morning, we delete the completed jobs from
> the previous day. We
> usually create about 2500-3500 jobs per day.
> - The queues run on the enterprise server and it
> gets rebooted every night.
> - Sometimes we can cycle the queue services and the
> problem goes away.
> Sometimes the problem comes right back. On the
> really bad days we have had
> to reboot the server.
> - We can move a waiting job to a different queue and
> it may process
> immediately; sometimes it waits before processing.
>
> Has anyone encountered this kind of problem before?
> Any suggestions on
> what else to check? Any feedback would be greatly
> appreciated.
>
> Thanks,
>
> Kevin McEvoy
> J.A. Webster, Inc.
> [email protected]
>
> B733.2, SP11.1, NT, Oracle 8.05
>
>
>
>
> --------------------------
> To view this thread, visit the JDEList forum at:
>
http://198.144.193.139/cgi-bin/wwwthreads/showflat.pl?Cat=0&Board=OW&Number=3677
>
>
*************************************************************
> This is the JDEList One World / XE Mailing List.
> Archives and information on how to SUBSCRIBE, and
> UNSUBSCRIBE can be found at http://www.JDELIST.com
>
*************************************************************
>


=====
Michael Moorman
7009 Wallace Ave.
Cincinnati, OH 45243
513-604-6786

__________________________________________________
Do You Yahoo!?
Yahoo! Photos - Share your holiday photos online!
http://photos.yahoo.com/
 
Back
Top