Slow Fat Client Tools on VMWare with TR 9.1 FC

We finished an upgrade from 8.x tr 8.97 to E1 9.0. Throughout the upgrade the FCs were comparatively slow. Most recently we went to TR 9.1.0.2 on our FCs. Now the FCs are painfully slow in terms of development tools. Worse yet is that the problem seems to be sporadic.

Nothing in Windows task manager / resource monitor is showing high utilization. The problem seems to be worse using the design tools more than anything else.

These are the specs:

E1 9.0 with TR 9.1.0.2
VMWare FCs
Windows 7 32-bit
2 VPs + 4GB RAM
iSeries DB services / all DB2/400 DBs

E1 is otherwise normal performance.

Anyone seen any specific problems with FCs on VMWare at this TR?

That is strange.

Have you turned off DEP and system restore on the FC's ? Secondly are you running antivirus - what type ? Try turning off as much as possible to identify the issue.

Can you provide details of your VMWare server - ie, specifications, including your drive configuration. It SOUNDS like you've got poor I/O performance for some reason.

One thing I have noticed on dev clients is the local Oracle database seems to be much slower than local SQL server used in earlier releases. Saving object specs is markedly slower for me anyway.

To John's point, it's usually I/O on the VMs.

Craig

Antivirus = McAfee
DEP = On during problem periods
System Restore = Off during problem periods

I'll have to get back to you on the server specs. They are fairly new EMC servers with tons of new SAN disk.

I am SSE but thanks for the tip on Oracle Express.

[ QUOTE ]
Antivirus = McAfee
DEP = On during problem periods
System Restore = Off during problem periods

I'll have to get back to you on the server specs. They are fairly new EMC servers with tons of new SAN disk.

[/ QUOTE ]

Apart from understanding your hardware infrastructure - heres a couple of troubleshooting tips :

Try turning off DEP on some of the workstations to see if that solves the issue - often DEP can cause major lag on a system

McAfee always seems to cause issues - especially with any java operations. Turn off McAfee during the problem period to see if it is the antivirus causing issues.

I am assuming you're not running Application Code 9.1 - since you're running MS SSE locally ? I'm presuming you're on 8.12 or an early 9.0 ?

Now, to the server specs - the SAN should be EMC - but the servers will be something else usually. I'd be interested in seeing what your disk I/O performance is - theoretically it should be super-fast with a properly configured SAN - but I've seen frames with appalling performance due to a mis-configuration such as badly set up fiber switches etc. It might be worth running some I/O tests against the disk using some common benchmark utilities - just to make sure the disk I/O is really operating at the right speed.

Note, that because you've stated you haven't seen CPU utilization - everything I'm recommending is disk I/O - there are only going to be a number of issues you could be experiencing :

1. CPU (you eliminated that when you cannot see CPU utilization)
2. Memory (I'm taking the 4Gb as adequate memory and am guessing that MS Task Manager isn't obviously complaining over lack of memory)
3. Network performance (ie, to the database and/or application servers) - to test this, try some pings, and if they're <10ms, then try some FTP File Transfers to make sure that you're adequately connecting. Your firewall and any other software device might be causing issues if you're having bandwidth issues.
4. Disk I/O Performance. This could be either latency or bandwidth - just like a network - but can be more difficult to monitor. Try watching PerfMon and see if the disk queue length isn't growing while you run the process. Again, copying large amounts of data on the disk also should provide information on whether the disk is really adequately performing.

Hope those help. If turning off the antivirus helps, then you need to identify what files McAfee is hammering, and ideally exclude them from the antivirus scans.

If you are using SSE what might have happened is that one of the databases might have gotten its autoclose flag set. It seems to happen on both the deployment server and fat clients. I have never pinned down what causes it. What you will see is very slow performance for all spec related activity such as loading FDA. Take a look at your Windows event log and if you see many messages related to a database being opened then you have the issue.

You can query the setting using the OSQL utility that will be installed with SSE.

osql -E -S (local)\JDESSELocal

Check for Auto Close On:

select name from master.sys.databases where is_auto_close_on = 1
go

If you get any results from the above query then run:

alter database xxxxx
set auto_close off
go

The server and SAN are EMC. EMC has gotten into the server business. The fancy word for their servers is "appliance".

How do you turn DEP completely off? I can see a radio button to omit it for specific processes and I see a radio button to just run it for Windows processes. I don't see a way to turn it off completely.

Thanks for the tip.

I ran the SQL query and the query results were null.

Here you go :

http://geekswithblogs.net/PsudoKnowledgeBase/archive/2009/06/03/turning-off-data-execution-prevention-dep.aspx

or for Windows 7

http://www.fanhow.com/knowhow:Enable_or_Disable_DEP_in_Windows_7_27019737

Both should point you in the right direction. DEP can easily be turned off completely or back on again using the above instructions.

I like how EMC use their server term as "appliance" - where VMWare has a different use for the same word ! Everyone it seems has gotten into the blade system environment - we're using Cisco UCS - which is actually certified for JDE !

But even so, I had issues with our Cisco UCS connecting to our EMC SAN - there was a configuration issue with the frame, and it was easy to show the symptoms to the frame guys (poor Disk I/O). Use IOMeter (http://www.iometer.org/) to see if you can identify an issue.

Thanks for the info. I'll let you know how I make out.

FWIW, I found a link on disabling it with Windows 7.

http://www.mydigitallife.info/disable-and-turn-off-or-enable-or-turn-on-data-execution-prevention-dep-in-windows-vista/

We shut off DEP and all of McAfee and things improved by about 70%. The strange thing is that it is just the development tools. And, the now slowness is isolated to ER statements involving table I/O & business view I/O. Regular if / then, do, bsfn calls, etc. respond quickly. Table and BSVW call statements take more that 4-5 seconds.

I turned up logging where the ER Table I/O problem occurs and noticed a couple of sql calls to OL. Other than that I have no idea why it is so slow.

I have heard that virus scanning is particularly problematic with VMWare. I was told that there is a special virus scanning software made just for VMWare.

So, my efforts will now focus on find the problem with I/O.

If you run the SQL Statements you captured to OL from a native SQL Client - do you also get a bad response time ? You might want to check that the OL tables just aren't indexed right !

I now always check the I/O from the OS level to make sure that whatever VM's I'm provided by infrastructure folks really perform. Often an entire implementation can be blamed for bad performance because one person didn't configure a network card correctly.

It's never the hardware that gets the blame - always the software !