Single threaded jobs after load balancing

I currently have 1 batch processing server for EOne. It consists of:
1 - Single Threaded Job Queue
1 - Multi-Threaded Job Queue (8 wide)

I plan on installing a second batch server and load balancing between the two servers. How do others handle the single threaded queue issue? Do I force all single threaded jobs to run on only one server?

Thanks

How are you going to do the load balancing? Manually using OCM entries or true load balancing using a device of some sort?



[ QUOTE ]
I currently have 1 batch processing server for EOne. It consists of:
1 - Single Threaded Job Queue
1 - Multi-Threaded Job Queue (8 wide)

I plan on installing a second batch server and load balancing between the two servers. How do others handle the single threaded queue issue? Do I force all single threaded jobs to run on only one server?

Thanks

[/ QUOTE ]

With either a Juniper or Foundry load balancer

Can you give me a bit more detail as doing batch server load balancing is not as simple as putting a sprayer in front of your batch servers.

I do think that you have raised a very serious issue on load balancing and single-threaded jobs. I think I have the answer for you but am working on it some more.



[ QUOTE ]
With either a Juniper or Foundry load balancer

[/ QUOTE ]

Are you going to be following Darryl Shakespeare's white paper on the subject?

[ QUOTE ]
With either a Juniper or Foundry load balancer

[/ QUOTE ]

Okay, I did a lot of thinking on this subject today and attached is what I have so far. I call it a "thinking paper", the kind I usually write as I think through a subject so it may be a little rough and it is not quite done yet.

After remembering what I know about kernels and how UBE's are processed through the system, putting that together with what I know about load balanced batch servers, I think I have a fair idea that it is entirely possible for two single-threaded jobs to be processing at the same time in a load-balanced setup- not a good thing.

The basic idea is that the "single-threadedness" is determined at the kernel level, on a particular physical server. The load balancing occurs at a virtual level above and before the kernel level. So, the control of single-threaded jobs is at the physical server level, too late to control whether another single-threaded job can run on the other server.

Any contrary opinions guys?

Here is my current setup and the available options that I think I have:

All servers are HP Proliant with 3.0G x 2 processors (4 with HT) and 4 GB RAM.

1 - UBE processing Server (JDE-BAT)
1 - BSFN processing Server (also processes 4 UBEs for Create!Form, did not want EVERY job to have metadata in the pdf files) (JDE-APP)
1 - SQL Server
8 - Citrix machines


I would like to replace JDE-BAT with 2 3.2G quad processor machines. This should allow me to have up to 32 concurrent jobs.

Option 1:
Citrix machines are balanced for users, adding Host table entries
CTX1 - JDE-BAT1
CTX2 - JDE-BAT2
CTX3 - JDE-BAT1
CTX4 - JDE-BAT2
CTX5 - JDE-BAT1
CTX6 - JDE-BAT2
CTX7 - JDE-BAT1
CTX8 - JDE-BAT2

PROs:
- Cheap, nothing but the server to buy
- Simple to implement

CONs:
- Not true load balancing
- Unable to tell if 1 of the batch servers is fully loaded and the other is waiting for jobs


Option 2:
Purchase network level load balancer (Cisco or Foundry).

PROs:
- Round Robin to alternate jobs to the batch servers
- Cost for load balancer should be around $10-20K, depending on setup
- Use load balancer to Round Robin Citrix servers

CONs:
- Round Robin still does not tell if the server is loaded with Big jobs.
- Additional hardware involved (point of failure)


Option 3:
Purchase Juniper (Redline) layer 7 load balancer.

PROs:
- Be able to truly load balance based on processor usage
- HTTP compression for 8.11-8.12 upgrade
- Load balance citrix servers

CONs:
- Additional hardware involved
- Cost should be around $20-40K, depending on setup and options


No matter which option I choose, I think I will need to have multithreaded jobs process on the batch servers, single threaded jobs either process on the JDE-APP server or 1 of the batch servers. I would think the nature of "single threaded" is the jobs should be queuing up behind each other to prevent data corruption. You may be able to have multiple single threaded job queues, but they should process different single threaded jobs that are not dependant on the jobs in another single threaded queue.

I have not reviewed Darryl Shakespeare's white paper, I guess I will pull it down see what it has to offer as well.