Slow performing UBE's (why oh why?)

Slow performing UBE\'s (why oh why?)

Hi

We have thrown some serious hardware at JDE. We have our terabyte SQL database now running on EMC DMX with 15k disks (database is hosted on a new w2k3 server - 32 gig RAM). The sad thing is that we have experienced minimal performance improvement on our UBE run times.
We have low CPU usage on out Enterprise server (windows 2000, 8 processors, 32 gig RAM), low network usage, a gigabyte backbone and a disk queue of 4 on the DMX.
Firstly the JDE UBE only runs on one processor (too be expected) but does not seem to try to claim more CPU, memory, cache, nor is waiting on SQL replies from the DB, nor making requests as quickly as we expect. The UBE's seem to idle.......

Any ideas as to where and issue might reside? I must be honest and say I am becoming very disheartened about the JDE app as it almost seems to be hardcoded to only be able to run at certain speeds.

(Background - its not that the UBE's have slowed down but rather that they do not want to run faster)

Regards
Larry

Re: Slow performing UBE\'s (why oh why?)

Are all of your UBE's taking a long time to run, or just certain reports? If it's just certain reports, you can try using Performance Workbench to pinpoint which parts of the UBE are taking a long time to run or just aren't optimized (for instance, fetching the same record thousands of times in a report). Here's the link to it, you'll need to sign in to Peoplesoft's website to access it...

http://www.peoplesoft.com/corp/en/iou/performance/perf_workbench.jsp

Re: Slow performing UBE\'s (why oh why?)

It not specific to any UBE's. Those with performance problems we are looking at UBE by UBE. What I am trying to get at is that if you throw more processing power, memory, faster disks - why do the UBE's not run any faster? (Why do they not take advantage of the processing power, the memory, disk speed? They still just seem to idle along not making requests to the database any quicker. The database is responding almost instantly to all requests that the UBE is making, but JDE does not seem to utilise any more of the resources available)

- To clarify, on friday we were using a SQL database on a FastT700 SAN. Today we are using the same database, same fragmentation issues etc on a new Symmetrix DMX1000 15k disk array. Surely there would be some speed improvement. Our stats are showing that UBE's are running the same times as normal. The SQL server hosting the database is also a W2K3 - fresh install (it was W2K before). We have been assured by microsoft to get at least a 30% performance gain just from using W2K3 in our environment. After some investigation we discovered that it does not seem as if JDE/UBE's are making the requests quick enough - the database is definately responding quicker and the disk queues are much lower but there are no significant changes in run times of UBE's.

Regards
Larry

Re: Slow performing UBE\'s (why oh why?)

Hi,

Have you tried tuning the following...

1. JDE (take a look at JDE.INI parameters)
2. SQL (indexes, fragmentation, tempdb, filegroups, among
many others...)
3. Windows 2003

I may tell you that default JDE installation is far from
being optimal, you'll have to tweak JDE, SQL and W2003
parameters to get decent results.

Regards

Re: Slow performing UBE\'s (why oh why?)

I understand what you are telling me but the bottome line is if you are throwing more grunt at JDE - even if the parameters are not optimal nor is the database completely defragmented. There should still be a performance gain comparing like with like, before and after.

I have seen this on a Unix/Oracle system before and thought it was unique but I am seeing the same thing on Windows/SQL. I am very familiar and comfortable with JDE settings, tweaking and tuning within JDE - database and application.

I have run a UBE that ran three hours with no other processing happening on the system. No users, no SQL activity. The JDE processor that it ran on the the Enterprise server peaked at time at about 80% but averaged at about 20%. The database idled at 4% with peaks up to about 10%. Network had minimal activity. We had 7 processors not being used on the Enterprise server and 1 processor running at 20%. Cache ustilisation on the server also at about 20%.... Nothing in our entire system was being maxed out - ie. no bottlenecks that we know of.

Regards
Larry

Re: Slow performing UBE\'s (why oh why?)

Have you tried adjusting your network queue settings? Changed the UBE priority etc?

Re: Slow performing UBE\'s (why oh why?)

As far as I know changing the priority within JDE does not make the UBE get more processing time but rather progress through the queue faster - ie. above lower priority jobs. In this scenario I only had one UBE running.
We did however prioritise it from the windows server perspective and put it on a high priority. This did not provide any improvement as the resources are not being "fought" for by any other process.

Regards
Larry

Re: Slow performing UBE\'s (why oh why?)

It just means, that you have not eliminated the bottleneck(s), whatever they are.

Bottlenecks have a tendency of moving from one resource to another with any significant change. I.e.: you may have addressed the old bottleneck, but introduced a comparable new one.

As usually, 99% of the solution is to locate it, fixing it is typically very easy (albeit expensive)...

As Master Yoda used to say: "Do not underestimate the power of the dark side" ;-) Sorry, couldn't resist...

Re: Slow performing UBE\'s (why oh why?)

Have you run one of the UBEs in question in full debug mode? If not do so and take a good look at the jdedebug.log. Search for the "select" statements and see how long between submittal and the next line in the debug log, that's your SQL processing time.

I had a FASTR that was taking 1.5 hours. After looking in the debug log I found a SQL select statement that was taking about 3 minutes to process. My hardware is almost as fast as yours, and obviously a 3 minute SQL select really bothered me. I looked further, there wasn’t another SQL statement anywhere in the log. So for 90s minutes of UBE time, 3 was spent waiting on SQL and 87 was spent processing the results.

So I ran the SQL statement in query analyzer. 3 minutes to process, however it brought back 2.1 million records. The UBE was then doing it’s calculations per line on 2.1 million records, a long process to say the least. The real issue was the programmer not using the updated/correct business view. Once the report was rewritten with the correct business view processing time fell from 1.5 hours to about 10 minutes.

Adding more processor will not speed up your system at all unless you have enough jobs to support the CPUS. Faster CPUS with more internal cache will help some, Xeon 2 meg cache chips really move along, but they also cost much more than the lower cache CPUS. Faster drives won’t help at all if the processing time is slow in the UBE and not in the actual select/update statements. You’ll notice a UBE that takes 10 minute to process the actual select and update statements might take 20 seconds. So if you doubled the speed of your SQL server, your processing time drops from 10 minutes to 9 minutes and 50 seconds. Again, all of this can be confirmed via the debug log.

While JDE will never use more than one processor per job, you can speed the system up with a mutli-threaded queue if you don’t already have one setup. You’ll need to find the document listing all the jobs that can’t be run multithreaded, move them to a single file queue, then move the rest to a multi-threaded queue. We have our MT7334 queue set at 14, so we can process 14 jobs at once. Doesn’t seem to matter if were processing 1 or 14 jobs, the jobs always take the same amount of time. We’ve moved a lot of single threaded jobs into different queues like AR7334, FS7334 etc. This takes in depth knowledge to know which jobs can be moved where, some single thread jobs can run while other single thread jobs in different subsystems run. We kept all of our posting jobs in a single ‘single threaded’ queue. Add in our single threaded queues and our 14 MT queues and it’s possible we might have 18 jobs running at once. This keeps our 4 XEON processors busy .

We have all of our UBEs running on a separate 4 way Xeon server. Another 4 way xeon runs the SQL server. Even if we have 10+ jobs running, interactive doesn’t take a hit this way, our SQL server is rarely if ever maxed out.

Overall our processing time is down from our old World AS400 system. However FASTRs have taken a hit in performance, even with a good deal of tweaking. Nature of the beast..

Re: Slow performing UBE\'s (why oh why?)

Try turning debug logging off. Our reports were painfully slow in 8.10 until I modified the log settings in the JDE.INI file on the Enterprise server to not create debug logs at all. Our Pre-payroll was taking over two hours to process. With debug logging turned off it only takes 15 minutes.

Re: Slow performing UBE\'s (why oh why?)

Thank you for a couple of ideas. I will try them during out maintenance slot on sunday.
I agree with Alex that the bottleneck is somewhere else. It is currently eluding us.
I will report back after the weekend.

Regards
Larry

Re: Slow performing UBE\'s (why oh why?)

I agree with Alex - you'll need to find the cause of the problem and fix it.

I believe the assumption that you're making isn't valid - ie: If a UBE is consuming all of the available resources on a given hardware platform, it'll only use a fraction of the resources (and run way faster) on a platform that has 10 times more capacity. In my experience, inefficient UBE's expand to consume anything in their path...