Best way to segregate batch jobs?

  • Thread starter brother_of_karamazov
  • Start date

brother_of_karamazov

Legendary Poster
I am looking for ideas on what you think is the best way to balance jobs between

a. Queues
b. Batch Servers

Assume that you have two or three (or four if you insist) batch servers available and that you are not using batch server load-balancing.

Assume that you have a number of single and multi-threaded queues on each server.

Assume that you are only concerned with the Production environment.

Assume that you seek the most highly optimized solution for processing the most amount of batches (~1500/day) in the shortest amount of time.

Assume that you wish users to sense no delay in a job processing.

If you have any other assumptions in your solution please state them.


What's you best configuration?
 
I half expected a holy war to break out over this question. Since there are no other comments, I'd like to say something; why bother with segregation when you can consolidate onto a "real" batch server which can scale (like iSeries/pSeries or even big Sun or HP boxes?!?)

....I know, for the sake of this argument we must assume separate Windows batch servers. Bleh. I'd move all of the quickly processing multithreaded jobs to their own server and max it out until CPU or memory hits the wall. Then I'd add another multithreaded server for those jobs if there is any spill over. A separate server would then be used for the long running, single-threaded queue jobs. Perhaps another server could be used as the spill over for those jobs, or as a hot spare or ad-hoc troubleshooting server. With all of those servers, you're probably increasing the likelihood of a problem. It is always good to have a method for troubleshooting in production should your report or posting output not match the desired results.
 
I will go with Charles suggestion "why bother with segregation when you can consolidate onto a "real" batch server which can scale (like iSeries/pSeries or even big Sun or HP boxes?!?)"

We currently process approximately 1500 to 2000 jobs on most of the days - with very limited job wait times.

We have implemented a segregation by function (single threaded queue), default queue is multi-threaded and for a few users (specific queues - for all their normal jobs).

We used to get a lot of queue wait times as most of the queues were single threaded and the default queue was limited to 3 jobs at a time.

Changed the configuration and created two multi-threaded queues to allow 10 jobs to process at the same time.

For some users - who required quick processing of jobs - changed their ini file to point them to specific job queues or changed their default priorities to get their jobs processed quicker.

We can process 12 to 14 jobs at the same time without any significant system performance issues.

Sun E450 / 4CPU / 4Gb Memory / XE SP22 / Oracle 8.1.7
 
BOK,

I agree with Charles. No need for multiple batch servers if your job queues are correctly configured and your hardware correctly sized. We currently run on average 6000 jobs per day on a single Windows 2000 Advanced Server box running on 8 CPUs and 8 GB of RAM configured for 15 single threaded jobs queues and 5 multi threaded jobs queues. No major issues.
 
Thanks for the feedback. I suppose a unique situation that I am running into is the presence of multiple Real Estate UBE's that run for longer times than normal jobs. I would say roughly a third of my jobs run for more than ten minutes each, and at certain periods (weekly, monthly, etc.) I am deluged with twenty jobs that run for thirty minutes each. An odd mix to say the least.

When the system was architected, the sizing partner was told to expect around 150 jobs/day so the company ended up with a single 2 CPU batch server. The option was to scale horizontally so I took on two more 2 CPU boxes. I have what I have so I am trying to figure out the best (not just a good) way to portion out the load. I am well aware of how many jobs can run on certain size servers and am also aware of the pros and cons of vertical vs. horizontal scaling. What I really want to know is how others distribute their UBE loads either across queues or horizontally scaled batch servers.

Thanks.







[ QUOTE ]
BOK,

I agree with Charles. No need for multiple batch servers if your job queues are correctly configured and your hardware correctly sized. We currently run on average 6000 jobs per day on a single Windows 2000 Advanced Server box running on 8 CPUs and 8 GB of RAM configured for 15 single threaded jobs queues and 5 multi threaded jobs queues. No major issues.

[/ QUOTE ]
 
BOK,

I have a customer who has three 2 cpu application servers, a setup similar to yours. This customer started with 1 application and then scaled horizontally as their business grew and their budget stayed at a minimal. This customer runs all legacy system interfaces on server 1, all long running jobs on server 2 and runs everything else (usually jobs that run under 10 minutes) on server 3. They run about 500 jobs daily.

In a configuration like yours, having three application servers available, I would suggest having one server for all jobs that run under say 5 minutes like maybe Pick Slips/Bill of Lading/Posting/..., one server for your Real Estate jobs and one server for all other long running jobs like maybe Sales Update/Invoicing/scheduler jobs. Your mileage may vary. There is some analysis and monitoring and ajustments to be made having three application servers, but the time it takes is well worth it in the long run. As a hint for managing submitted jobs, you could have all three application servers redirect the PrintQueue to one common file server by modifying their JDE.INI. Then you would have all three Server Maps pointing to one common database/library. All that you have left to do would be to modify the Work With Servers application P986116 to see all jobs from all application servers : With With Servers Form, Menu/Toolkbar Exit..., Server Jobs, Event Rules, modify the Call(App: P986110B, Form: W986110BA) event rule to specify '*' in the data structure parameter 'Executionhostname'. I have done this and it works great.

Just a thought,
 
Back
Top