Random IPC errors usually when running jobs from Tidal

[msouterblight1]

Hello,

I am trying to track down the cause, but more importantly the resolution for random IPC errors we are experiencing in our environment. When the jobs fail we are seeing errors similar to:

INFO: Entering kernel signal handler, process exiting soon: iParam: 1310364011
INFO: In kernel signal handler, starting alarm handler: iParam: 1310364011
INFO: Done setting IPC Handle State structures to abandoned, process exiting immediately: iParam: 1310364011
Jul 10 10:56:27.781013 DEBUG INIT0 - 13083 **** jdeDebugInit -- output disabled in INI file.
Jul 10 10:56:27.783190 jdemem.c141 - 13083/-166132016

I have increased the IPC range in the past, but I was wondering if there was anything different I could do to alleviate this situation. We are on Linux (RHEL 5) with Tools Release 8.98.3.3.

Thanks!

 

[ebearman]

Are you completely up-to-date on your agents and other Tidal software? They have released quite a few patches in recent months...

 

[Tom_Davidson]

Matt,

FWIW, this doesn't look like an IPC issue to me, it looks more like a memory allocation issue.

The setting IPC to abandoned is normal when a job goes zombie.

Are you able to create a debug log for this?

Tom

 

[Ghost Sailor]

Hey Matthew, we're getting the same type errors on an IBM AIX (Unix) platform. The UBE suddenly aborts with these entries at the end of the log:

INFO: Entering kernel signal handler, process exiting soon: iParam: 1310045511
INFO: In kernel signal handler, starting alarm handler: iParam: 1310045514
INFO: Done setting IPC Handle State structures to abandoned, process exiting immediately: iParam: 1310045514

It initially only happened when Tidal submitted it, but now we've had it error out the same way manually submitting the UBE from the EnterpriseOne 9.0 menu option. So I'm suspecting it is not Tidal related...

Have you had any recent progress?

 

[Jim Miller]

Anyone have any further information on this? We receive this same type of error on EnterpriseOne 9.0 running MRP submitted from Tidal on AIX Unix.

 

[msouterblight1]

Re: Random IPC errors usually when running jobs from Tidal - Possible Fix

Oracle support got back to me on this one. This is apparently a known bug that affects all 8.98.3.X Tools Releases. It is supposedly fixed in 8.98.4.X. I have asked for a one-off for our current Tools Release, 8.98.3.3 on Linux. I have attached the article on the bug that Oracle sent to me....

 

[Jim Miller]

Re: Random IPC errors usually when running jobs from Tidal - Possible Fix

Matt thank you for exchanging emails with Hall. That helped us to convince Oracle support that this was not just an MRP issue, but a tools release issue in dealing with large batch jobs.

The fix was identified for us on Thursday 07/14/2011. We applied the tools fix on last Friday 07/15/2011 and have run large batch jobs, including MRP runs, over and over again, to see if any would fail like this again. It has not happenned, again. We are very glad Oracle was able to very quickly give us the fix!

Oracle was very confident that this fix addresses the issue, and had given it to other customers that reported the problem. It was NOT however on any "critical patch" announcements. Tidal and Oracle have also (already) worked on additional issues and enhancements for Tools Release 8.98.4 and above, so this is already incorporated.

Again, thanks Matthew for posting this as it allowed us to triangulate a Tools Release issue, vs an MRP or ini or other server settings (IPC, Semamphores, etc) type issue. Without your info here, we would have continued chasing MRP data selection, processing options, corrupt data and server settings issues at 10 hours MRP run time, those tests had already consumed several weeks of our time troubleshooting red herrings!!!