[ QUOTE ]
Antivirus = McAfee
DEP = On during problem periods
System Restore = Off during problem periods
I'll have to get back to you on the server specs. They are fairly new EMC servers with tons of new SAN disk.
[/ QUOTE ]
Apart from understanding your hardware infrastructure - heres a couple of troubleshooting tips :
Try turning off DEP on some of the workstations to see if that solves the issue - often DEP can cause major lag on a system
McAfee always seems to cause issues - especially with any java operations. Turn off McAfee during the problem period to see if it is the antivirus causing issues.
I am assuming you're not running Application Code 9.1 - since you're running MS SSE locally ? I'm presuming you're on 8.12 or an early 9.0 ?
Now, to the server specs - the SAN should be EMC - but the servers will be something else usually. I'd be interested in seeing what your disk I/O performance is - theoretically it should be super-fast with a properly configured SAN - but I've seen frames with appalling performance due to a mis-configuration such as badly set up fiber switches etc. It might be worth running some I/O tests against the disk using some common benchmark utilities - just to make sure the disk I/O is really operating at the right speed.
Note, that because you've stated you haven't seen CPU utilization - everything I'm recommending is disk I/O - there are only going to be a number of issues you could be experiencing :
1. CPU (you eliminated that when you cannot see CPU utilization)
2. Memory (I'm taking the 4Gb as adequate memory and am guessing that MS Task Manager isn't obviously complaining over lack of memory)
3. Network performance (ie, to the database and/or application servers) - to test this, try some pings, and if they're <10ms, then try some FTP File Transfers to make sure that you're adequately connecting. Your firewall and any other software device might be causing issues if you're having bandwidth issues.
4. Disk I/O Performance. This could be either latency or bandwidth - just like a network - but can be more difficult to monitor. Try watching PerfMon and see if the disk queue length isn't growing while you run the process. Again, copying large amounts of data on the disk also should provide information on whether the disk is really adequately performing.
Hope those help. If turning off the antivirus helps, then you need to identify what files McAfee is hammering, and ideally exclude them from the antivirus scans.