First off, do your logs show any errors?
The SvrBuild.log, and BuildError.txt (in the root directory of your package build) and the BuildReport.txt and buildlog.txt (in the work subdirectory of the package build) can give some indication if there are errors ocurring.
Next, are your server jobs dropping into a single or a multi threaded queue? We had issues when our server jobs were falling into an 8 job multithread. The contention was unbelievable! We finally settled on a 3 thread drop (though others have suggested that a two thread drop is optimal - see other threads on this subject).
Last, do you have a nightly backup process that is contending with the full package build? This was also another problem that we had, as our save was programmed to 1) shut down OneWorld services, and 2) terminate job queues. Until we figured out what was happening, the package builds were in essence falling asleep during the save process, and waking up much later...during our daily processing time.
If these items don't hit it, I'm betting the heavy hitters on the list can nail the issue with the info from the logs...