Users Experiencing Slowness on Web

peterbruce

peterbruce

Legendary Poster
JDEList,

Our users are virtually always reporting that the JDE (8.11 sp1 TR8.96C WAS 6) response is slow. Minutes instead of seconds. They obviously have experienced reasonable repsonse times. This siutaion has sort of sneaked up. There has been no dramatic event that identifies when it started. I suspect it has something to do the configuration of our web server.

I have asked our system guys and they virtually always report the servers are idling (Enterprise, Database, Web).

About two weeks ago we got 15 to 20 javacore dumps with OUTOFMEMORY errors and then today we got 1, but nothing before or in bewteen.

I was surprised by the number of possible problems reported in the variety of logs.

The jderoot_??????.log on the web server contains things like:
- An exception has been caught by the Web client.
- java.lang.NullPointerException
- exception in processEvent java.lang.NullPointerException
Followed by a lot of:
- at com.jdedwards.....(Unknown Source)

The jde logs for CALL OBJECT KERNEL on the enterprise server contain lots of real time event errors:
- Failed to get event defintion for event RT?????. Please check is event definition correct?
A number of:
- KNT0000099 - JDEK_DispatchCallObjectMessage: Pooled Connection: ???????? timed out!
A few:
- Fail to fetch Error Text ????? from Data Dictionary.

I don't know a whole lot about the web. Fat clients I can handle well. Sometimes I wish we stayed on Xe with the fat clients.

Can someone please let me know if the above items found in the various logs etc. can possibly have anything to do with the "slowness" of the user's experience. What do these things mean? How can they be fixed? I have done some research, but in the main I have found it difficult to find any meaningful (to me anyway) information.
 
Hi,

May you, please, post your Enterprise Server JDE.INI
and your JAS server JAS.INI, JDBJ.INI, JDELOG.PROPERTIES?
Please, remove passwords from those files before posting!
Thanks,
 
Sebastian,

Thanks for replying.

While going through the files you requested (see attached) I noted a couple of odd things, which I have noted in the attachment. They included a couple of passwords that have not been changed since installation and are therefore incorrect and a server that does not exist. Where they occur may or may not be important.

All the files you requested are in the one text file.
 

Attachments

  • 130258-JDEProdINIs.txt
    41.5 KB · Views: 1,791
Peter,

We've had similar situations in our shop and we've looked at things like this in the past:

1) If you're using multiple JAS servers, ensure that they're properly load-balanced using, for example, a round-robin protocol.

2) Although somewhat obvious, the latency in response time from the Users' end may be due in part to heavy network traffic. At any given point in time, Users are running reports, invoking queries, etc. This all eats away at the network bandwidth and this can result in performance issues if there's not enough of bandwidth to go around.

Hope this helps,

Michael
 
Michael,

Thankyou for your insights.

We only have the one JAS/Web server.

The network traffic issue is something that we have looked at on previous occaisions. We have had issues with heavey network traffic slowing everything down - email, mapped share drives, other applications, the internet. However, on most occaisions when JDE Web slowness is reported, the users report that it is only JDE - no other application of function is having problems.

The only other network issue, and I believe this is your point, is that JDE itself may be overloading our network capacity and the other applications and functions are not in use at the time. Our network team report that this not the case.

Combine the above with (apparent) errors in log files and javacore dumps with out of memory errors and, to my mind at least, it points to our web setup.

The other thing I would like to mention is that our web server is a machine with a fair amount of grunt and it always seems to have considerable unused RAM - even with the javacore dumps with out of memory errors.
 
Michael,
Look at thread #120145 Thread #120145

We made several of these changes about 3 weeks ago and have not had a heap dump since. The primary change is adding the -Xloratio0.3 in the Generic JVM Arguments.
 
For reference, the hardware specs of our JAS/Web server are as follows:

CPUs: 4 x dual core 3.6ghz Intel Xeon
RAM: 8 gb
 
Peter,

We had a similar issue. With us, there were certain users that were reporting the slowness, while others around them were fine. All we had to do was apply some Windows updates to their workstations and they sped up. It was probably some IE updates that helped, but we just applied a bulk load, so i couldn't point you to a certain one.

Not sure if this is exactly the same issue, but it worked for us.

Dave
 
Peter,

Out of memory errors and heap dumps are a sure sign that you are exceeding the memory capacity of your JAS JVM. An individual JVM will only use the memory configured for it regardless of how much is installed on the box. To make maximum use of the RAM on your box you will need to look at vertical partitioning (cloning). Vertical cloning will let you run multiple JVM's behind a single JAS port. This will spread out your user load allow you to make use of all of your server's RAM.

On windows I generally us a 1GB maximum JVM size and create clones to use the remaining physcial memory less about 1 GB for the OS. These days I have seen recommendations from JDE as high as 1280m per JVM.

You may be able to get away without vertical cloning for a while by tuning your single JVM's memory cluster settings. Take a look at the following solution Id 201018061 on the Customer Connection. It discusses the -Xk setting to the JVM that controls cluster size. The value I use is -Xk9000 which is the general recommendation. The solution document discusses how to come up with the optimum size for your platform.

Have the usage patterns on your system changed recently? Have you suddenly added a block of new concurrent users? Is scroll-to-end or data browser enabled? I have found Scroll-to-end and Data Browser to be likely culprits when you start heap dumping. Until tools 8.97 grid fetches are entirely cached in memory. Even with the cluster setting in place it is still possible to push a JVM beyond its memory maximum. If you still heap dump with the cluster size parameter added you will probably need to look at vertical cloning.
 
Dave,

Thanks for the "heads up" on the windows front. All our desktops are managed centrally and are kept pretty well up to date with patches etc. But I'll check to make sure.
cool.gif
 
Justin,

That is really good information.

I have also seen references to horizontal cloning. How do you set up virtical and horizontal cloning and what is the difference?
 
Peter,

Clustering is also referred to as cloning or scaling. Horizontal scaling increases the number of physical boxes serving a particular web application while vertical scaling increases the number of Java Virtual Machines on a single box serving a web application. Interaction between the HTTP server and the JVMs is what allows these multiple JVM's and machines to service users from a single url:port combination. This interaction is managed through the HTTP server WebSphere plugin. Horizontal and Vertical scaling can also be combined so that a web application is hosted across multiple boxes and across multiple JVM on each box.

There are many ways to lay a multi-node, multi-JVM WebSphere installation. If high-availability of each component is a requirement there will be a number of components in addition to the JAS application instances that must also be clustered. Here are a few IBM Redbooks on the subject that I have found useful:

http://www.redbooks.ibm.com/abstracts/sg246688.html?Open
http://www.redbooks.ibm.com/abstracts/sg246198.html?Open

There is a lot to consider in those Redbooks. I find most of it overkill for E1 clustering needs. I personally prefer to implement vertical scaling only and use a hardware load balancing device to provide horizontal scaling.
 
Hi Peter

Just some 2c worth from me.

1. With 8Gb RAM - are you using Windows 2003 Enterprise Edition ? Obviously Standard Edition won't address more than 4Gb RAM, and its good to provide as much memory as possible

2. Is your JAS server connected to the DB server via gigabit ? Confirm your network connectivity and speed - run some simple ping tests and some FTP's to ensure the network isn't the bottleneck

3. Consider moving to a newer service pack - there are often performance improvements in service pack releases

Just a quick 3 suggestions - you've probably already addressed these of course (!)
 
Justin,

Thanks for the information.

I was hoping that setting up vertical clustering/scaling/cloning would be a simpler process than it apparently is and that there existed a set of step by step instructions that explained how to accomplish the task. I have not been able to locate any such set of instructions.

However, due to the research I have done and the information provided, I have enhanced my knowledge considerably.
 
Jon,

Thanks for your USD 0.02 worth, valuable at any price. In answer to the points you raise:

1) Yes, our Web Server does use Windows 2003 Enterprise Edition and all 8gb is available. We just need to configure WebSphere to use it - thus my interest in vertical clustering/scaling/cloning.

2) Yes again. Our Web/JAS, Enterprise and Database servers and filer are all connected to each other by a gigbit network as is our network backbone between buildings on campus. The only network related bottle neck could be the 100 megabit network within buildings.

3) We installed a WebSphere SP about 6 months ago. I checked early this year and there was nothing new. However there may be something over the past couple of months.
 
Peter,

Vertical cloning is discussed in the "IBM WAS Network Deployment 6.0 Guide for JD Edwards Tools Release 8.96". Get it here and see page 19:

http://www4.peoplesoft.com/contdoc.nsf/0/20B62DED5935AD1E8825715D006EF35A?OpenDocument

Having to install the Network Deployment instance and node agent is a hassle when all you want to do is vertical cloning. WebSphere is an over engineered powerhouse. I am sure its complexity makes sense for big eCommerce sites. As I said in my previous post I feel it is overkill for E1. OAS is so simple. You just specify multiple JVMs for the E1 web app and you are vertically cloned. Since you are on WAS you will have to following the instructions in the above document. It will be well worth it though. You will be able to make full use of your server's RAM and you should have happier users for your trouble.
 
Justin,

Many thanks - you've saved me a week trying to find that (well may be not quite a week).
 
Back
Top