Why some scheduled jobs were not launched?

Harry Chen · Feb 3, 2012

Since 2009 we have had incidents that scheduled job failed to launch at night. Totally we have 447 jobs failed to launch among 365981 jobs a year. Even though the fail-to-launch percentage is as low as 0.2% it is still important to know why it happens.
We have many jobs in scheduler and most of them are Batch Posting which runs on single-threaded job queue every 5 minutes - which is not critical as the next run will post the rest batches. But some scheduled jobs are for sales invoice printing and sales updates and we have to manually run them if auto-launch fails.
We tried to analyze to find any pattern to see if the fail-to-launch has common things like server, job queue, object name but we were not able to find any. We also submitted a service request to Oracle Support but there was not suggestion provided.

Does any of you have the experience seeing the scheduled jobs not launched?
Have you had it solved finally?

Thanks, --Harry

peterbruce · Feb 5, 2012

Harry,

We have had a similar situation. A particular UBE would not launch every now and then. The UBE is R03B525 and we run 4 different versions of it at half hour intervals on the first of every month. The problem was that occaisionally went it was run the future schedule stopped being inserted and when all the previously inserted. At the time I did not have the time to investigate and fix the problem. The decision was taken to use the native Solaris scheduler and run the ube with the runube utility. This has been running effectively. Funny it was only the one UBE.

DSauve · Feb 6, 2012

Peter and Harry,

This could possibly have been caused if a scheduled job was added using a version of P91300 that has a processing option set with a blank value for Maximum number of job schedule records for recurring jobs with no end date. Other than that, have you checked the log files for the scheduling kernel for errors, or were the jobs scheduled with a username that had changed its password?

Just a few things to check out.

gerd_renz3 · Feb 6, 2012

I had a similar problem some time ago with a client who had a "very busy" scheduler, very many scheduled jobs.
We had two Unix batch servers, lets call them A and B. I noticed that the jobs only failed on server B. Server A was the only security server then. I started a Security Server on server B as well and ever since I had no more failing jobs.
I guessed back then that server A got so busy that the security server could not reply properly or in proper time which then caused the job to fail.

My two centavos.

Cheers, Gerd

msouterblight1 · Feb 7, 2012

That's a large amount of Batch Jobs to trust to the JDE Scheduler. I would recomend looking at a package that allows for dependencies and notifications. We use Tidal at my company, but I know there are a few other packages.

Just my .02.

gregglarkin · Feb 8, 2012

Harry

I'll add my $.02 to Matthew's. In my old job we ran several thousand jobs a night through Tidal scheduler. With Tidal we could set up complex dependencies, run streams of jobs for multiple systems (we had a stream with fusion, JDE, and createprint jobs, that fed data to a Cognos stream right after). Tidal can automatically rerun a job that fails. We had it set to rerun a failed job. If that failed, it sent a message to the operators. They tried to run it. If that failed, they called me. That ensured that the critical streams always got processed.

- Gregg

vasoov · Feb 14, 2012

We had exactly the same issue because maxKernelProcesses did not match the number of jdenet kernel defs in JDE.ini. We were stumped for a while, but on correcting the maxKernelProcesses the "unknown" issue of batch jobs not being launched was resolved.

Harry Chen · Feb 17, 2012

Thanks all for your input.
Hi Vassov,
Could you please point out more clearly where I should check?
Once I have the answer from you I will ask our CNC to check our servers.

Here is the related sections I found in my own JDE.INI on fat client (not server):
---------------
[JDENET]
serviceNameListen=6015
serviceNameConnect=6015
maxKernelRanges=27
netTrace=0

[JDENET_KERNEL_DEF27]
krnlName=CLIENT DISPATCH KERNEL
dispatchDLLName=jdeuser.dll
dispatchDLLFunction=_JDENET_ClientDispatch@28
maxNumberOfProcesses=0
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF13]
krnlName=WORK FLOW KERNEL
dispatchDLLName=workflow.dll
dispatchDLLFunction=_JDEK_DispatchWFServerProcess@28
maxNumberOfProcesses=5
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF2]
# for dispatching UBE jobs
krnlName=UBE KERNEL
dispatchDLLName=jdekrnl.dll
dispatchDLLFunction=_JDEK_DispatchUBEMessage@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=1
---------------

Thanks, --Harry

vasoov · Feb 20, 2012

Can you post your server INI file?

bwr2006 · Feb 22, 2012

If you are dealing with that many jobs you should really consider TIDAL it is an amazing job management software. You can tweak it however you want.

Harry Chen · Mar 1, 2012

Sorry I am still waiting for CNC to give me the server INI. Also we would check the scheduler kernel log. We had more jobs failed to launch in scheduler this week.
By the way we already bought Active Batch to run scheduled jobs outside JDE. Perhaps we can use it for JDE as well, with RUNUBE.exe capability.

Harry Chen · Mar 2, 2012

Hi Vassov
Here is our server JDE.ini section you asked:
--------------
[JDENET]
serviceNameListen=6015
serviceNameConnect=6015
maxNetProcesses=7
maxNetConnections=400
maxKernelProcesses=64
maxKernelRanges=32
netTrace=0
ServiceControlRefresh=5
enablePredefinedPorts=0
enterpriseServerTimeout=60000
netBroadcastAddress=INADDR_BROADCAST
useKeepAlive=1
netCoreDump=0
kernelDelay=0
krnlCoreDump=0
statsUpdateInterval=30
netChildCheck=15
maxNumSocketMsgQueue=200
maxIPCQueueMsgs=100
internalQueueTimeOut=30
FilePacketBufferSize=32768
netShutdownInterval=5
netTemporaryDir=D:\JDEdwards\JDE_HOME\targets\ERP9_ENT_ECOAPP5_DVPY\temp

[JDENET_KERNEL_DEF1]
krnlName=JDENET RESERVED KERNEL
dispatchDLLName=jdenet.dll
dispatchDLLFunction=_JDENET_DispatchMessage@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF2]
krnlName=UBE KERNEL
dispatchDLLName=jdekrnl.dll
dispatchDLLFunction=_JDEK_DispatchUBEMessage@28
maxNumberOfProcesses=10
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF3]
krnlName=REPLICATION KERNEL
dispatchDLLName=jdekrnl.dll
dispatchDLLFunction=_DispatchRepMessage@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF4]
krnlName=SECURITY KERNEL
dispatchDLLName=jdekrnl.dll
dispatchDLLFunction=_JDEK_DispatchSecurity@28
maxNumberOfProcesses=12
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF5]
krnlName=LOCK MANAGER KERNEL
dispatchDLLName=jdekrnl.dll
dispatchDLLFunction=_TM_DispatchTransactionManager@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF6]
krnlName=CALL OBJECT KERNEL
dispatchDLLName=XMLCallObj.dll
dispatchDLLFunction=_XMLCallObjectDispatch@28
maxNumberOfProcesses=175
numberOfAutoStartProcesses=0
singleThreadedMode=N
ThreadPoolSize=20
ThreadPoolSizeIncrement=5

[JDENET_KERNEL_DEF7]
krnlName=JDBNET KERNEL
dispatchDLLName=jdekrnl.dll
dispatchDLLFunction=_JDEK_DispatchJDBNETMessage@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0
ThreadPoolSize=50
ThreadPoolSizeIncrement=5

[JDENET_KERNEL_DEF9]
krnlName=SAW KERNEL
dispatchDLLName=jdesaw.dll
dispatchDLLFunction=_JDEK_DispatchSAWMessage@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF10]
krnlName=SCHEDULER KERNEL
dispatchDLLName=jdekrnl.dll
dispatchDLLFunction=_JDEK_DispatchScheduler@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF11]
krnlName=PACKAGE BUILD KERNEL
dispatchDLLName=jdekrnl.dll
dispatchDLLFunction=_JDEK_DispatchPkgBuildMessage@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF12]
krnlName=UBE SUBSYSTEM KERNEL
dispatchDLLName=jdekrnl.dll
dispatchDLLFunction=_JDEK_DispatchUBESBSMessage@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF13]
krnlName=WORK FLOW KERNEL
dispatchDLLName=workflow.dll
dispatchDLLFunction=_JDEK_DispatchWFServerProcess@28
maxNumberOfProcesses=5
numberOfAutoStartProcesses=1

[JDENET_KERNEL_DEF14]
krnlName=QUEUE KERNEL
dispatchDLLName=jdekrnl.dll
dispatchDLLFunction=_DispatchQueueMessage@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF15]
krnlName=XML TRANS KERNEL
dispatchDLLName=XMLTransactions.dll
dispatchDLLFunction=_XMLTransactionDispatch@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF16]
krnlName=XML List Kernel
dispatchDLLName=xmllist.dll
dispatchDLLFunction=_XMLListDispatch@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF19]
krnlName=EVN KERNEL
dispatchDLLName=jdeie.dll
dispatchDLLFunction=_JDEK_DispatchITMessage@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF20]
krnlName=IEO KERNEL
dispatchDLLName=jdeieo.dll
dispatchDLLFunction=_JDEK_DispatchIEOMessage@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF21]
krnlName=OPE KERNEL
dispatchDLLName=jdeope.dll
dispatchDLLFunction=_BBOPE_DispatchMessage@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF22]
krnlName=XML DISPATCH KERNEL
dispatchDLLName=xmldispatch.dll
dispatchDLLFunction=_XMLDispatch@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF23]
krnlName=XTS KERNEL
dispatchDLLName=xtskrnl.dll
dispatchDLLFunction=_JDEK_DispatchXTSMessage@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF24]
krnlName=XML SERVICE KERNEL
dispatchDLLName=xmlservice.dll
dispatchDLLFunction=_XMLServiceDispatch@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF30]
krnlName=METADATA KERNEL
dispatchDLLName=mdserializer.dll
dispatchDLLFunction=_MetadataDispatch@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=1
ThreadPoolSize=50
ThreadPoolSizeIncrement=5
MaxNumberOfConcurrentThreads=50

[NETWORK QUEUE SETTINGS]
QKActive=1
QKOnIdle=300
JDENETTimeout=60
UBEPriority=5
SaveOutput=TRUE
UBEQueue=QBATCH
PrintImmediate=TRUE
OutputDirectory=

Harry Chen · Mar 9, 2012

Is there anything wrong in the JDE.ini settings I posted?
Thanks,

vasoov · Mar 12, 2012

Harry,

I noticed that the maxKernelProcesses=64 is too small. It should equal the total number of maxNumberOfProcesses found under all the Kernel defs.

In your case: 1 + 10 + 1 + 12 + 1 + 175 ...

Regards,

Vasoo

Harry Chen · Mar 13, 2012

Thanks so much for sharing this. I will try to test the solution.

Harry Chen · May 24, 2012

Sorry I have to re-post again my server JDE.INI as the one with mismatched maxNumberOfProcesses was from testing environment.

Here is our production server JDE.INI, I counted the total and the number matches to 64:
-----------------
[JDENET]
serviceNameListen=6015
serviceNameConnect=6015
maxNetProcesses=7
maxNetConnections=400
maxKernelProcesses=64
maxKernelRanges=32
netTrace=0
ServiceControlRefresh=5
enablePredefinedPorts=0
enterpriseServerTimeout=60000
netBroadcastAddress=INADDR_BROADCAST
useKeepAlive=1
netCoreDump=0
kernelDelay=0
krnlCoreDump=0
statsUpdateInterval=30
netChildCheck=15
maxNumSocketMsgQueue=200
maxIPCQueueMsgs=100
internalQueueTimeOut=30
FilePacketBufferSize=32768
netShutdownInterval=5
netTemporaryDir=D:\JDEdwards\JDE_HOME\targets\JDE9_APP02_PD\temp
netHostName=

[JDENET_KERNEL_DEF1]
krnlName=JDENET RESERVED KERNEL
dispatchDLLName=jdenet.dll
dispatchDLLFunction=_JDENET_DispatchMessage@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF2]
krnlName=UBE KERNEL
dispatchDLLName=jdekrnl.dll
dispatchDLLFunction=_JDEK_DispatchUBEMessage@28
maxNumberOfProcesses=10
numberOfAutoStartProcesses=10

[JDENET_KERNEL_DEF3]
krnlName=REPLICATION KERNEL
dispatchDLLName=jdekrnl.dll
dispatchDLLFunction=_DispatchRepMessage@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF4]
krnlName=SECURITY KERNEL
dispatchDLLName=jdekrnl.dll
dispatchDLLFunction=_JDEK_DispatchSecurity@28
maxNumberOfProcesses=5
numberOfAutoStartProcesses=5

[JDENET_KERNEL_DEF5]
krnlName=LOCK MANAGER KERNEL
dispatchDLLName=jdekrnl.dll
dispatchDLLFunction=_TM_DispatchTransactionManager@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF6]
krnlName=CALL OBJECT KERNEL
dispatchDLLName=XMLCallObj.dll
dispatchDLLFunction=_XMLCallObjectDispatch@28
maxNumberOfProcesses=20
numberOfAutoStartProcesses=20
singleThreadedMode=N
ThreadPoolSize=20
ThreadPoolSizeIncrement=5

[JDENET_KERNEL_DEF7]
krnlName=JDBNET KERNEL
dispatchDLLName=jdekrnl.dll
dispatchDLLFunction=_JDEK_DispatchJDBNETMessage@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0
ThreadPoolSize=50
ThreadPoolSizeIncrement=5

[JDENET_KERNEL_DEF9]
krnlName=SAW KERNEL
dispatchDLLName=jdesaw.dll
dispatchDLLFunction=_JDEK_DispatchSAWMessage@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF10]
krnlName=SCHEDULER KERNEL
dispatchDLLName=jdekrnl.dll
dispatchDLLFunction=_JDEK_DispatchScheduler@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF11]
krnlName=PACKAGE BUILD KERNEL
dispatchDLLName=jdekrnl.dll
dispatchDLLFunction=_JDEK_DispatchPkgBuildMessage@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF12]
krnlName=UBE SUBSYSTEM KERNEL
dispatchDLLName=jdekrnl.dll
dispatchDLLFunction=_JDEK_DispatchUBESBSMessage@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF13]
krnlName=WORK FLOW KERNEL
dispatchDLLName=workflow.dll
dispatchDLLFunction=_JDEK_DispatchWFServerProcess@28
maxNumberOfProcesses=5
numberOfAutoStartProcesses=1

[JDENET_KERNEL_DEF14]
krnlName=QUEUE KERNEL
dispatchDLLName=jdekrnl.dll
dispatchDLLFunction=_DispatchQueueMessage@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=1

[JDENET_KERNEL_DEF15]
krnlName=XML TRANS KERNEL
dispatchDLLName=XMLTransactions.dll
dispatchDLLFunction=_XMLTransactionDispatch@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF16]
krnlName=XML List Kernel
dispatchDLLName=xmllist.dll
dispatchDLLFunction=_XMLListDispatch@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF19]
krnlName=EVN KERNEL
dispatchDLLName=jdeie.dll
dispatchDLLFunction=_JDEK_DispatchITMessage@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF20]
krnlName=IEO KERNEL
dispatchDLLName=jdeieo.dll
dispatchDLLFunction=_JDEK_DispatchIEOMessage@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF21]
krnlName=OPE KERNEL
dispatchDLLName=jdeope.dll
dispatchDLLFunction=_BBOPE_DispatchMessage@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF22]
krnlName=XML DISPATCH KERNEL
dispatchDLLName=xmldispatch.dll
dispatchDLLFunction=_XMLDispatch@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF23]
krnlName=XTS KERNEL
dispatchDLLName=xtskrnl.dll
dispatchDLLFunction=_JDEK_DispatchXTSMessage@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF24]
krnlName=XML SERVICE KERNEL
dispatchDLLName=xmlservice.dll
dispatchDLLFunction=_XMLServiceDispatch@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=0

[JDENET_KERNEL_DEF30]
krnlName=METADATA KERNEL
dispatchDLLName=mdserializer.dll
dispatchDLLFunction=_MetadataDispatch@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=1
ThreadPoolSize=50
ThreadPoolSizeIncrement=5
MaxNumberOfConcurrentThreads=50

........ other sections here.........

[CLUSTER]
PrimaryNode=

[JDENET_KERNEL_DEF32]
krnlName=MANAGEMENT KERNEL
dispatchDLLName=jdemgmt.dll
dispatchDLLFunction=_ManagementDispatch@28
maxNumberOfProcesses=1
numberOfAutoStartProcesses=1

[JDENET_KERNEL_DEF31]
krnlName=XMLPUBLISHER KERNEL
dispatchDLLName=xmlpublisher.dll
dispatchDLLFunction=_XMLPDispatch@28
maxNumberOfProcesses=5
numberOfAutoStartProcesses=0
--------------------------------

Vasoo,
Do you think our JDE.INI is good?
We still have scheduler jobs failure every week.

Thanks, --Harry

Why some scheduled jobs were not launched?

Harry Chen

Active Member

peterbruce

Legendary Poster

DSauve

Legendary Poster

gerd_renz3

VIP Member

msouterblight1

VIP Member

gregglarkin

Legendary Poster

vasoov

Active Member

Harry Chen

Active Member

vasoov

Active Member

bwr2006

Member

Harry Chen

Active Member

Harry Chen

Active Member

Harry Chen

Active Member

vasoov

Active Member

Harry Chen

Active Member

Harry Chen

Active Member

Similar threads

We value your privacy