8.19 / 9.0 Performance issues, tuning tricks

[gregglarkin]

List,

I have a very broad question here - does anyone hae any 8.12 / 9.0 performance gripes? Any war stories like "gee thingw were SO much faster on XE opening {insert favorite modual here}, but now that modual is so much slower in 8.12." or the like?

How about tuning tips? has anyone come across any great tuning tips vs how 9.0 was loaded out of the box?

How about client performance stories - web client delivered over citrix verses delivered straight to the desktop? Anything that you would like to share?

We have one small but growing instance of 8.12 in Europe, and are starting to plan out our 9.0 migration for North America, so we'd like to hear your performance stories.

Gregg Larkin, MBA
North American JDE Systems Engineer
Praxair Incorporated

 

[altquark]

Hi Gregg

Under 8.12, using Tools Release 8.97 - over a high-latency, low bandwidth connection, I just performed a web test vs Citrix test. Heres an overview (your results might be DRASTICALLY different)

Over Web (tried using Riverbed AND used SSL and Compression technologies)
Time to log in - 2 minutes
Time to click "find" and retrieve results in Item Master - 1 minute

Over Citrix (published IE through citrix)
Time to log in - 20 seconds
Time to click "find" and retrieve results in Item Master - 2 seconds

Its obvious that the web client, deployed to machines over a very high latency connection with little bandwidth will cause major performance issues. Obviously I cannot share much more than this - but certainly 8.97 isn't any improvement for this customer, and 8.98 (on the test environment) shows similar results.

There are some modules that are faster (I'll let others comment on the functionality improvements) - and there are lots of performance tuning tips - many are already on JDEList, and many are in the minds of those who get paid to help customers with performance issues (!) - but I've got documentation that shows it is not really possible to get the same performance delivered straight to the desktop over a WAN compared to a citrix published session.

Feel free to call me if you want help with any specific performance-related issues, I've been performance testing 8.96, 8.97 and 8.98 through to the latest version for the past couple of years...!

 

[Sebastian Sajaroff]

Hi Greg,

I definitely recommend to publish your JDE browser
link via Citrix or Windows Terminal Services to your
end users.

Performance difference is usually impressive!

You'll also gain stability (you know exactly which patches,
Windows Service Packs and IE releases you're running) and
you get rid of all that crapware like chat messengers,
toolbars, funny gadgets, etc... that will harm your
Web client performance.

Do not hesitate : go Citrix or Windows Terminal Services.

 

[gregglarkin]

Thanks Jon and Sebastion!

So, in your two opinions, the biggest out of the box performance issue is the Web client delivered over a WAN? Not surprising based on previous posts. Does anyone else have any Citrix vs Web stories or other glaring performance tweaking stories to share?

Here's a follow up question, are you guys seeing a substantial performance difference between a farm of VM-Ware based OAS servers vs. similarly sized hardware OAS servers.

To be specific, if I had a farm of five hardware OAS servers, sitting next to a dedicated VM-Ware host serving up five OAS servers (same ram and stuff), is there a big hit using VM?

Gregg Larkin, MBA
North American JDE Systems Engineer
Praxair Incorporated

 

[Sebastian Sajaroff]

Hi Greg,

There's a performance penalty, which is not noticeable
for a low number of users (<100) but becomes quite
noticeable when you exceed 100-150 users per VM host.

(At least that was my experience on 3 accounts).

Let's start with the fact that JAS servers are CPU and
RAM intensive. Then we have the fact that VMWare virtual
machines share a single RAM bus (the pipe that connects
CPUs to RAM on the motherboard), so that will definitely
slows down CPU access to RAM contents.

The more VMs you add to your physical host, the slower
your VMs will run, no matter how much RAM you dump there.

On the other hand, I find VMs are an interesting option
for development and testing scenarios.

 

[altquark]

Under large loads, the VM's will utilize more memory, and will start paging. You need to ensure that your OAS servers - whether physical OR virtual - do not page, since paging causes extreme amounts of performance degradation.

However, a virtual OAS server seems to perform at about 90% of a physical OAS server - so in my opinion, its worth being virtualized due to the ease of management. But, as sebastian said, an OAS machine that is paging is WAY worse than a physical machine paging.

Its always down to how well the architecture is designed (!)

 

[gregglarkin]

not to ignite a flame war, but when I passed on your replies to our Senior Enterprise Architect, this was his response, which I was asked to post and pass on your responses.

"My testing has not shown any need for Citrix. Internal testing by JDE shows that the web client takes less bandwidth than Citrix. The SSL issue is not a problem with modern CPUs and the issue about serialization and packet acknowledgements is just plain wrong."

Gentlemen, your responses?

 

[ebearman]

Gregg,
My two cents is somewhere between Jon and your EA. I have done quite a bit of testing for clients as well and have found that a 16-bit color published Citrix apps can perform the same as a client HTTP (or HTTPS) session. For example, I just connected my AT&T 3G modem to my laptop (~256kbps measured), fired up the Cisco VPN client, logged in to their 8.97.3.0/8.12 system and measured:
HTTP (with compression on) - between 4 and 5 seconds to login, 5 seconds to open P0411 app
Citrix 16-bit - 4 seconds to login, between 3 and 4 seconds to open P0411
Citrix 8-bit - 3 seconds to login, 4 seconds to open P0411

Hope that helps a little...

-Ethan

 

[altquark]

I actually have written out an answer for you - detailing you exactly why this occurs. But, if I publish this information here - I'll get nothing, and lots of people will get lots of knowledge, and I'll be out of work in a week.

But, to disprove Larry's company is a ballsy thing to put out there. And, I'm not really comfortable doing that without some sort of compensation. So, in the same vein as Larry making the bet with the Fortune 1000 over the Exadata2 machine - here's my suggestion.

I'm guessing your "Senior Enterprise Architect" gets $150,000 a year salary. If you want, I'll set up a test to prove what I'm talking about is correct, and will publish both a whitepaper AND will demonstrate the difference between a web client over citrix and a web client directly over a WAN.

If I'm right, then you pay me half of his salary in a single lump sum. What you do with the results is up to you. If I'm wrong, then your architect is correct, and you save hundreds of thousands on citrix licenses.

I'll demonstrate this in your offices in Buffalo, and will even pay for the airfare myself. It'll take me less than an hour to demonstrate that LATENCY is a big issue with JDE - and much more so than BANDWIDTH.

Up to Praxair now. You know my number, call me back. I can be in Buffalo on Friday if necessary.

 

[unitas99007]

I definitely don't know as much about JDE as you, gregg, Jon or Sebastian, but here's our setup for what it's worth. Our company was running Citrix in JDE 8.12/Tools 8.97. We then did a migration to JDE 9.0 / Tools 8.98.05. We moved off the Citrix environment to just HTML servers. Our users, in both scenarios, were accessing the servers in a datacenter from multiple locations over a MPLS network. The HTML servers seem to perform really well. We're planning on 50 users per HTML server.

Unitas99007

 

[altquark]

I'm stating, as in the past, that the mileage will vary.

If your latency is high, then your performance will suck. If latency isn't an issue, then performance will be fine.

LATENCY has nothing to do with BANDWIDTH

When I say "too high" - we're talking here more than 100ms of latency becoming abusively noticeable.

Here is a simple test. Set up standalone in one location, and access the web portion of standalone from other locations. Simple, simple test to see how well it works over the network. If the latency is really high, then you'll have lots of issues.

Some companies don't have ATM and high-speed connectivity to all their branches. Some companies have ships in the middle of the arabian gulf that have to connect through satellite. The latency of those remote points could be hundreds of milliseconds.

I'm not impressed that this is being even discussed. If you have an issue, then Citrix will solve it. If you don't have an issue - then why talk about it ?

 

[ebearman]

Jon,

Your statement I'm not impressed that this is being even discussed is a bit too dismissive. If it is being discussed, there is value for someone in the discussion.

One other note that I have found with Citrix, depending on release/client/etc., is that too high of latency will cause the connection to drop as well. Jon you describe satellite connectivity, that is the exact scenario where latency can be too high and cause problems. Furthermore variable latency, such as from an MMS style microwave connection, can wreak havoc on Citrix, RDP, VoIP, etc.

You are absolutely correct that latency and bandwidth are separate issues. Both the modern JDE HTTP (8.97 tools and newer) and Citrix connections require little bandwidth per user session. That is easy to measure with simple packet capture tools like Ethereal.

-Ethan

 

[altquark]

[ QUOTE ]

One other note that I have found with Citrix, depending on release/client/etc., is that too high of latency will cause the connection to drop as well. Jon you describe satellite connectivity, that is the exact scenario where latency can be too high and cause problems. Furthermore variable latency, such as from an MMS style microwave connection, can wreak havoc on Citrix, RDP, VoIP, etc.


[/ QUOTE ]
Right - but using a web client directly over such a connection is completely unuseable. Citrix might be plagued with disconnects - but at least it can be tuned to be relatively "useable". Direct web connect is completely unusable over such a connection.

My whole point of this (and why I said it shouldn't even be discussed) is that I never EVER mentioned ANYWHERE that bandwidth is the issue. Therefore - why discuss sausages when potatoes were on the menu ?

Latency is the issue. Always has been, always will be. Citrix is better over high-latency connections - but over really REALLY insane latency connections (seconds worth) - it also has issues.

The recommendation is that if you're < 100ms, then direct connect using the web is fine. >100ms but <700ms, then you're going to have to use Citrix (or RDP or whatever streaming technology you prefer). >700ms, and you're going to have to send carrier pigeons OR you're going to have to re-architect using some sort of replication/synchronization methodology. I think thats a rule that everyone can agree on.

Lastly, the additional benefit of citrix is the fact that you cease to have to manage the web browser on the client. Instead, you have one, centrally controlled, web browser that is being "shared" through a secure ICA connection. End of support issues.

 

[cdawes]

Okay...........I've stayed out of this long enough and I love watching this tennis match of Citrix vs. Web.


Here's my view:
1. Citrix is expensive. If you don't have it already why invest in it? If you already ave it then you can consider using it.

2. The JDE Web Client is good. Yes it is more "chatty" than Citrix. Greg --> sorry but I have to respectfully disagree with your network guy on this. It's simply not as thin as ICA

3. JDE has 3 different levels of interactivity. For high latency try setting a separate JAS instance to LOW interactivity

4. Web Clients disconnect and you have to log back in. Here is the real benefit of Citrix - I can reconnect to my session.

5. Latency of >100ms but <700ms does not require Citrix. I have a client running from the Northern Canadian Tundra to a data center in Sweden and Latenct is 200 ms - 400 ms. Web Client runs well.

6. Of my many client 20+ are live on 8.12. Only 4 are using Citrix or Terminal Services. 2 are in the process of ditching Citrix.

BTW how did this toic turn into a Citrix vs. Web Client flame war?

 

[brother_of_karamazov]

[ QUOTE ]
Instead, you have one, centrally controlled, web browser that is being "shared" through a secure ICA connection. End of support issues.

[/ QUOTE ]

You mean I don't have to support the Citrix boxes and associated software?

Cool...

 

[cdawes]

To answer your original question if you are on tools release 8.97 or higher with 8.12 or 9.0 you should not have ANY significant issues.

While I haven't done deployments of the !0,000 users you have I have done many 1000+ user deployments on the recent releases and customers have been ESTATIC with the performance. If you need references or want to talk to them let me know off line.

The only module that we've ever had performance issues with was Job Status Inquiry on 8.12 and this is being improved on 9.0. The customer that had this issue had 10's of thousands of jobs (construction).

For tuning there is so much I don't know where to begin. Really depends on the architecture configuration.

One final note on the Citrix vs. Web debate. With Citrix the PDF output is all local to the servers. With web when you view a PDF it gets pulled over the WAN.

JDE has reduced this issue via compression and linerization of the PDF's. However if using Optio, CreateForm or FormScape you need to turn this off.

One alternative is BI Pubisher.


Colin

 

[gregglarkin]

Colin,

Wow, I'm getting a lot of mileage out of this thread. Lots of different opinions in here, but good information. I'm going to analyze all of the input and create a white paper out of this. A few follow ups for you:

"The JDE Web Client is good. Yes it is more "chatty" than Citrix. Greg --> sorry but I have to respectfully disagree with your network guy on this. It's simply not as thin as ICA"

Do you have some stats on that handy? I'm going to set up some tests, but I'd like some outside data to compare it with.

"JDE has 3 different levels of interactivity. For high latency try setting a separate JAS instance to LOW interactivity"

Excellent point - back in XE SP22 (I believe SP22J) they added in the higher interactivity levels so that the application could start doing field level updates and data verification rather than waiting until the user to fill out the whole screen and hitting enter before it goes and validates data. (that tidbit was for others reading this, I know you know that). Setting a seperate JAS instance to low interactivity to combat latency issues is a very clever solution.

Gregg Larkin, MBA
North American JDE Systems Engineer
Praxair Incorporated

 

[tonycoval]

Okay gentlemen,
I am new to the conversation and want to throw in some more fodder.
I have watched a great debate, however I want to get to the "why's" of this.
Jon is always on target with the Citrix contention for latency, and Colin some great points, but this is for the web guys...

We all know that Citriz rules in latency because it is streaming, not doing page requests.

Each and every JDE application averages 47 httprequests to finish. Oh yes, it is java and Ajax and and the miracle stuff, but Salesforce averages 11 requests.
By the way the average page size (based on calling from parameterized URL as the MAF conceals the app size from the menu frame) is 1.2 MB!

So with low interactivity, optimizing the proper webcaching and browser settings, you can make it work! No not on as Jon put it the insane latency, but with the 98% norm.

They really need to optimize all the .js in this app, please!
Lets go one step further! The portlets each load a full javascript set, with 1.3 MB each and 47 requests, so if you want to build a portal, and want to throw in 4 portlets on a page, and aren't fully optimizing or on a dedicated pipe, just plan on brewing some coffee whilsty awaiting your page!

We have all compared notes on results and opinions, but how about we suggest ideas for cutting this "chattiness" in half? They will get there on their own I guarantee it - they have strong people in Denver still... but boy Gregg, wouldn't THAT make you a nice whitepaper!

 

[altquark]

I've been really busy with a customer emergency since last week - but have a couple of mins to add a comment here. I agree, Colin has a very good point - I don't think anyone has ever actually tested the difference with the interactivity levels, and I think it definitely warrants a look.

Certainly theres a lot of javascript - but a lot of that can be cached locally with the browser. All those funky JDE controls can be cached, and never have to be re-downloaded. With the interactivity level set to low, I'm certain the number of turns will be greatly reduced - and that could conceivably be a LOT more efficient on a high-latency connection.

I'm going to see if I can test this next week on an 8.98 install and grab some stats...

 

[altquark]

ok - I tested this on an EnterpriseOne 8.97 installation. And the results were interesting.

First of all, the setting is as follows :

On the JAS.INI for the instance :
[ QUOTE ]

[ERPINTERACTIVITY]
InteractivityLevel=HIGH, MEDIUM or LOW



[/ QUOTE ]
Secondly, there is information in Oracle Metalink article ID 946773.1 that states
[ QUOTE ]

The Go To End button is not available when the Interactivity Level is set to Low. Only the Previous and Next buttons are visible under Low Interactivity. The Interactivity Level is set in the jas.ini file under the [ERPINTERACTIVITY] section on the InteractivityLevel line.

NOTE: When a user's profile has Accessibility Mode set to Yes they will automatically be set to Low Interactivity Mode. This only affects that user and does not have a global impact.


[/ QUOTE ]

So, evidently, this CAN be set by USER ID from 8.98 onwards !!!!! This is very important to note.

[ QUOTE ]

Set user profile accessibility mode to No by doing these steps:

Go to P0092 (Work With User / Role Profiles)
Do an open Find and Select the correct Userid
Set radio button for "Set Accessibility Mode" to NO
Click OK and Close


[/ QUOTE ]

The value that is triggered is ULFUTTIME1 in F00921

However, since we are running 8.97 in this test, we had to manually change the instance using the JAS.INI.

Here are some tests we ran between the two instances.

local-office (<1ms latency, high bandwith)
high-interactivity interactive process : 9 seconds
high-interactivity find screen : 3 seconds
low-interactivity interactive process : 3 seconds
low-interactivity find screen : 2 seconds

remote-office (145ms latency, high bandwidth)
high-interactivity interactive process : 21 seconds
high-interactivity find screen : 5 seconds
low-interactivity interactive process : 12 seconds
low-interactivity find screen : 6 seconds

remote-rig (730ms latency, low bandwidth)
high-interactivity interactive process : 23 seconds
high-interactivity find screen : 7 seconds
low-interactivity interactive process : 14 seconds
low-interactivity find screen : 9 seconds

As can be seen, the interactive process is at least HALF the time with the setting. The "Find" (ie, going into a specific application and timing how long it takes to "find" some data) takes exactly the same amount of time - therefore is unaffected.

If you find this type of benchmarking interesting - please call me to help with your benchmarking resolution.

Thanks has to go to Colin for the initial idea behind this...!