XML request is killing the Enterprise Server

Gisbert

Member
Hi, we have a custom middleware that process interop with JDE 9.1 via XML Request, we have the logic to reuse the session generated for each user, after some time the server doesn't response anymore to xml requests, but still works for web client.
In the server manager the errors looks like: jdeSawImpGetKernelUserListV3:Error:JDENET_ReceiveMsg:le_net_error 11:<srvbd02>,<6016>,15925416,5

Here is a sample of the XML sent:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<jdeRequest sessionidle="900" environment="JPD910" session="17760508.1448400516.2" comment="" user="XXX" type="callmethod" role="*ALL" pwd="XXX">
<callMethod name="FormatLocation" app="cme">
<params>
<param name="szCostCenter"> 103</param>
<param name="szDisplayFormatLocation">00000000</param>
<param name="cValidate">2</param>
<param name="cMode">2</param>
</params>
</callMethod>
</jdeRequest>

As you can see the session is reused with a timeout of 15 minutes until jde return it's expired, then we create a new one; so there is no clue why the overhead.

If someone can share any tip would be great.
Thanks.

Som
 
I'd be checking a couple of things. Firstly, check to see whether your middleware is correctly reusing the session and not opening new sessions on each call and thus running out of resources. 15 minutes is a long timeout for kernel threads and not something we recommend when customers are deploying our solution. The overhead of creating a new session isn't that significant and if you have a high volume of transactions going through you would obviously be hitting the request a lot more frequently than 15 minutes.

The other thing to check is whether you have kernel recycling turned on. Again, because your sessions are open for so long you may be recycling the call object kernel with an open session. If the XML dispatch kernel session is still valid but the call object kernel that is referenced by the session id has been recycled then the dispatch kernel will just hang until it times out which is set to 10 minutes by default. If your middleware attempts to resubmit again with the same session id you'll get the same result.
 
Also - what tools release are you running ? Perhaps there is an update to your tools release that might address this ?
 
Hi Russell, thanks for your response; we have been checking the COU and COK with the usage of the application.
We have changed the settings to 50 COK, and the timeout is 3 minutes, we are reusing the sessions until jde returns it's expired or broken, but it looks like the 15 minutes timeout is causing overhead and the volume of users operating with the application is 50~70 at the same time.
The kernel recycling is turned on but runs at 1 AM, so nobody is working.

The last settings we have right now are:
COK availables: 50
XML Session timeout: 3 minutes

But still the enterprise is dying... Can you share please some settings that you have used before?

Thanks.
 
With your 50 - 70 users, are they all running their own sessions or do they all share the one session? When you say it dies, does it stop responding to all requests or just a single session is dying? Do you have to bounce the enterprise server services to get it going again or can you just wait until the session expires?

For that sort of load on our solution, assuming unique sessions for each of the 70 users and ignoring all other user types, we would be running probably 10 COK with multi-threaded turned on. The only time we've seen the issue you're experiencing has either been because the COK has become a zombie or because kernel recycling has kicked during active sessions. The zombie kernels were always caused either by a tools bug or a BSFN bug and were fairly easy to replicate. The kernel recycling issue is just a bug that we handle within our solution.

I will say that none of our 9.1 customers are still on 9.1.2.x tools though as they are fairly old. The oldest tools we have out there are 9.1.3.3 but most are at least on 9.1.4.x. In general I would recommend a tools upgrade to at least the latest 9.1.4.x tools release (9.1.5.x introduces the new UI so this is a much bigger exercise).
 
Hi Russell, each user is running his own session as recommended by XML Request performance guide in oracle support, when I say the server dies it's because all the XML request stops working, there is no response for the requests, but JDE Web still works without problem, the only solution we have tried to recover is bouncing the services in the enterprise, we didn't wait until the sessions expires. (Because the operation needs to continue ASAP)

Well that's a problem we have found; they have only 1 enteprise in PD and there are around 80-90 Call Object Users that are from JDE Web and aren't being recycled, we are considering this as part of the problem, there are no zombies when the server stops working, it's something like lack of resources to process the requests. We don't have COk recycling issues because it's done after work hours.

UPDATE: We had to change the solution to expire each user session after the transaction ends, after that change the server is still alive, but I don't feel happy with that fix, I don't know if I'm missing something in the middledware side or if this only can be fixed improving infrastructure, can you share some experience about this?

Thanks for your responses.
 
When you say transactions are you running a series of jde function calls (eg begindoc, editline, enddoc type series), are you using XML Transactions to interface with the Z tables, or are you just running simple function calls like FormatLocation?

How many XML dispatch kernels (kernel type 22) are you running?

When the system fails you should be able to see from the logs which kernels are processing XML requests and if those kernels are still up. If all of your sessions are failing at once it sounds like the XML Dispatch Kernel(s) are having an issue.
 
We have custom BSFNs that encapsulates the begin doc, edit, etc in a single call, every transaction is a single bsfn call; we finally had to close the connection after every xml request; we still have some issues but is due to all the other JDE (forms, ubes) processes running in the same server, we will create a new enterprise server to isolate all this functionality. Will tell you later after the implementation.
 
Back
Top