E9.2 Calling Orchestrations with Logic Extensions from Boomi with multiple threads - not threadsafe?

[jolly]

We have an orchestration with some LEX steps being called from Boomi. Boomi is configured to call the Orchestration multithreaded (up to 5 threads) so the Orchestration will have 5 simultaneous instances running when it has lots of records to process.

We are getting exceptions from the Logic extensions in the Orchestration - about 1 in 10 randomly. Re-running the orchestration manually by itself always succeeds.

If we limit Boomi to one thread, the orchestration calls always succeed.

If I setup a Postman flow on three workstations to call the Orchestration in a loop continuously it always succeeds. Each workstation has a different token.

If I replace the Logic Extension with something else such as a Form Request it always succeeds.

Boomi is calling the orchestration with an authentication token that is shared between the threads.

We are finding this in more than one Orchestration using more than one Logic Extension

What to make of this? The only thing I can deduce is that Logic Extensions running simultaneously within the same connection are not threadsafe.

Has anyone had any experience with multithreaded Boomi to Orchestration calls, especially where the Orchestration contains a LEX?

NB: Tools 9.2.7.4.

Cheers
JohnO

 

[lfurino]

Hi John,

First, I would make sure you are using the v2 version of the tokenrequest api. We have seen similar issues with using the original tokenrequest api.

If you are using the v2 tokenrequest and still having issues you could play around with the different jobQueues.
Here is a list from Oracle on the different options:

Orchestrator Guide

You can specify how you want to process an orchestration: in a single-threaded queue, in a fire-and-forget multithreaded queue, or in the default queue. Using this feature, you can have more control over the runtime processing of orchestrations.

docs.oracle.com

To use a different queue you can add an additional Name:Value pair in your JSON body like this:
"jde__jobQueue": "JDE_USER_SINGLE"

I hope this helps, but if not you might want to log an SR with Oracle.

Regards,
Larry

 

[jolly]

Hi John,

First, I would make sure you are using the v2 version of the tokenrequest api. We have seen similar issues with using the original tokenrequest api.

If you are using the v2 tokenrequest and still having issues you could play around with the different jobQueues.
Here is a list from Oracle on the different options:

Orchestrator Guide

You can specify how you want to process an orchestration: in a single-threaded queue, in a fire-and-forget multithreaded queue, or in the default queue. Using this feature, you can have more control over the runtime processing of orchestrations.

docs.oracle.com

To use a different queue you can add an additional Name:Value pair in your JSON body like this:
"jde__jobQueue": "JDE_USER_SINGLE"

I hope this helps, but if not you might want to log an SR with Oracle.

Regards,
Larry

Hi Larry,

Good idea.

I have tried the v2 tokenrequest but we still get exceptions from various LEX intermittently. It might be a little more reliable though - my Postman stress test of 5 threads soon fails on one thread but the remaining 4 seem to keep going for a while until a second thread failed. Subsequently 3 threads seems to remain alive.

The Orchestrations are all v3. I assume that the {{url}}/jderest/orchestrator and {{url}}/jderest/v3/orchestrator will work the same as there is only a v3 orchestration?

I could run a single thread queue for orchestrations but really prefer not to as the performance drops noticeably.

Our CNC has working through the SR process with Oracle. We'll see if they are interested and responsive on it.

Thanks
JohnO

 

[Gustavo Barizon]

Issue 5 different tokens, spread them on each thread from Boomi, and manage them for this process. Don't forget to "Log out" after you complete the process. LEX runs at the JAS level, and that may be causing inconsistencies. I had several multi-thread issues with Boomi at a customer, and we set the Orchestration for "Single Thread" to fix the issue, but the performance wasn't our focus, and that worked as we expected. With 5 threads, and 5 tokens, you'll be fine, and no LEX issues.

 

[jolly]

In this case we need to be multithreaded as there are many sites and if they are bottlenecked into single thread queues the response will be too slow. I can't convince or coerce the Boomi team to manage their tokens and am stuck with their use of shared tokens.

For the time being we have replaced all the LEX with either Form Requests or NER Custom Requests. It is now rock solid.

 

[EricGagne]

Hi,

We have been experiencing the same issue recently. This an issue with logic extensions regardless of what calls them, they simply cannot run in parallel. Like you we have to migrate our code to NERs and Form Requests.

Edit: We are also on 9.2.7.4

 

[DaveWagoner]

@jolly @EricGagne would you two work with me to write this up into an enhancement request? I assume Oracle will treat it as such on their end, but we would want to get it in front of the user community so that Oracle could prioritize it accordingly. I think it's important to frame it as not just a boomi > orchestration enhancement, but a LEX threading enhancement.

 

[jolly]

Would be happy to. Agree not just Boomi as I was able to demonstrate this very simply with a Postman Flow too.
However we have a case open and are currently working with Oracle on it.

 

[DaveWagoner]

Would be happy to. Agree not just Boomi as I was able to demonstrate this very simply with a Postman Flow too.
However we have a case open and are currently working with Oracle on it.

Sweet. Oracle has asked us to "duplicate efforts" and tie a Oracle Support SR with a "Tech SIG" enhancement request where we track user community comments. Your duplication via postman will be really valuable.

Also, Oracle wants interested customers to "add themselves" to the oracle support SR's, once published, to register user community support. I'd be happy to sign on to this in both spots to help make it better.

All of this is null and void if Oracle treats this as a bug rather than an enhancement-- so if they're leaving it a true bug (or if it's still up in the air) then we should hit pause.

I'll toss you my email in a DM

 

[EricGagne]

I'll be happy to help too if I can.

 

[jolly]

Update: Oracle have sent my client a couple of iterations of a JAR file for the AIS server and this has helped. No more errors from LEX calls.
We were also getting some instability and zombie kernels from a P4210 FormRequest based Orchestration. We set the callObject kernels to single threaded mode on the integration server. This has surprisingly not impacted throughput at all in our stress testing but has made all our orchestrations reliable, including the P4210 process.

 

[jolly]

UPDATE: Oracle have delivered a bug fix - will be released in Tools 9.2.9.1. Kudos to Oracle.

 

[DaveWagoner]

Woohoo!!! Good news