How long package deployment will keep trying if a job is running?

[rolimn]

During my life with OneWorld / EnterpriseOne I have realized that when a package is deployed to an Application Server it will first hold all job queues, then wait for whatever is running to finish, then deploy the package and then release all job queues. Sometimes it keeps trying for 30, 60, 90 minutes and if a job running (in the path code being affected by the package) doesn't finish, it'll give up deploying to this server and release all job queues. I have two question about it:

1) Where is it I can control how long to wait for a job to finish before giving up deployment of the package to that particular server?

2) I have just gone through something that has never happenned before: it has kept trying for 4 hours (!!!) then it moved to the next server but job queues were not released.... I see number of Outstanding Requests on Package Build Kernel growing even after deployment to this server was abandoned!

Does anyone have more information on it? has anyone gone through this also?

OneWorld Xe, SP23_I1/I2, iSeries V5R3, Windows 2000 Advanced Server, Windows 2003, Citrix, HTML.

 

[Shrikanthn]

We have a similar problem deploying packages. During the day 80% of the time the deployment takes seconds. but sometimes for inexplicable reasons it "hangs". That is the deployment process takes a long time, during which time the system performance deteriorates rapidly. reports start getting into the "Wait" status and programmes take forever to launch. We end up killing this deployment. But if we try deploying anything after that it will invariable hang. hence we have to reboot the enterprise server after which deployment goes back to a few seconds. This makes us do deployment at all odd hours (lunch time, post 2300 hrs etc). No solutions in this post sorry, just a description of our problem.

best regards
shrikanth

 

[rolimn]

I can understand what you mean. By dropping the UBE on the Deployment Server, you'll not stop the deployment. The UBE just sends a message to the Enterprise Server's kernel and keeps waiting for a response. This kernel will keep trying to deploy even if you drop the UBE on the Deployment Server. That's why a second attempt will fail. When you bounce services you'll be starting with a brand new kernel so deployment may succeed.

When the kernel gives up deploying (for time-out -- notice I did not end the UBE on the Deployment Server) it should release all job queues and this did not take place as I reported previously.

I still hope someone out there can explain why this happenned and, most importantly, where is it we can control how long it will take for the kernel to give up deploying.

 

[brother_of_karamazov]

If a UBE is processing on the target enterprise server your deployment will wait until it is finished to complete the deployment. During that time, the deployment will hold all jobs submitted in a W status. If you kill the deployment you will destabilize the UBE kernel(s) and submitted jobs afterward will behave erratically.

The inexplicable reason it hangs is actually the deployment process working properly and protecting you from changing specs on a job currently processing.

Simply get your deployment ready then wait until you see no processing jobs, then deploy.



[ QUOTE ]
We have a similar problem deploying packages. During the day 80% of the time the deployment takes seconds. but sometimes for inexplicable reasons it "hangs". That is the deployment process takes a long time, during which time the system performance deteriorates rapidly. reports start getting into the "Wait" status and programmes take forever to launch. We end up killing this deployment. But if we try deploying anything after that it will invariable hang. hence we have to reboot the enterprise server after which deployment goes back to a few seconds. This makes us do deployment at all odd hours (lunch time, post 2300 hrs etc). No solutions in this post sorry, just a description of our problem.

best regards
shrikanth

[/ QUOTE ]

 

[brother_of_karamazov]

I had no idea that there was a time limit, I thought the deployment would just wait forever.

The trick is to not try to deploy when jobs are running.



[ QUOTE ]
During my life with OneWorld / EnterpriseOne I have realized that when a package is deployed to an Application Server it will first hold all job queues, then wait for whatever is running to finish, then deploy the package and then release all job queues. Sometimes it keeps trying for 30, 60, 90 minutes and if a job running (in the path code being affected by the package) doesn't finish, it'll give up deploying to this server and release all job queues. I have two question about it:

1) Where is it I can control how long to wait for a job to finish before giving up deployment of the package to that particular server?

2) I have just gone through something that has never happenned before: it has kept trying for 4 hours (!!!) then it moved to the next server but job queues were not released.... I see number of Outstanding Requests on Package Build Kernel growing even after deployment to this server was abandoned!

Does anyone have more information on it? has anyone gone through this also?

OneWorld Xe, SP23_I1/I2, iSeries V5R3, Windows 2000 Advanced Server, Windows 2003, Citrix, HTML.

[/ QUOTE ]

 

[Shrikanthn]

thanks brother, how do bigger companies work with this small window of opportunity when no jobs are processing. even in my medium sized company people are running jobs all the time. even during lunch they run long running jobs. hence we need to wait till people go home, which makes the life of a jde support guy tough. anyway, if thats how it is we gotta accept it. thanks again.

 

[Nancy_x]

Perhaps the bigger companies do not use WindowOS. But I really want to know the performance difference between Unix and Windows during month end with multiple locations.

 

[brother_of_karamazov]

Late night deployments I suppose. If I really had to solve the problem, really had to deploy packages in the middle of the day I would do the following:

Assuming I had multiple batch servers (and who doesn't these days?) I think I might get the deployment staged and ready, OCM map all jobs to the second batch server, flush the OCM cache (because all your clients are web-based right?), wait until the jobs on the first server finish, deploy the package, go have a beer.

Hmmmm...that might work huh?


[ QUOTE ]
thanks brother, how do bigger companies work with this small window of opportunity when no jobs are processing. even in my medium sized company people are running jobs all the time. even during lunch they run long running jobs. hence we need to wait till people go home, which makes the life of a jde support guy tough. anyway, if thats how it is we gotta accept it. thanks again.

[/ QUOTE ]

 

[cmanderson]

You could do what we do and wait until Sunday mornings to deploy the package. Even still we often resort to the dastardly "manual" deployments on our UNIX batch server.

When we were running Xe, I wrote a nice little script, borrowing a few pieces from other nice little scripts, and it allowed us to deploy the package automagically with post deployment 'project include' emails and status checks, etc.

Unfortunately for me, 8.9 came along and with it the interupt driven Queue Kernel. There goes my hack job automation script.

It really was a thing of beauty - 12 weeks in a row we deployed the production package between 9am and 11am with not a single hiccup. What the script actually did was monitor for runbatch processes in a loop, checking fairly aggressively but not so much that it would hog CPU resources. When the script returned 0 runbatch processes, the next step was to immediately login to the Oracle database and fire off a package which updated all waiting jobs in the Job Master to hold status, enabled a table trigger which updated all future batch jobs to Hold status. We then checked again for rogue runbatch processes that could have slipped through the fence. If 0 then on the server we stopped scheduled job autosubmission, then removed the target bin32/spec files and 'cp -fr' the package files over to the pathcode. Only effective difference between this and actual deployment is the lack of a 'spec lock' function to prevent spec updates and business function execution.

In our case we weren't mapping any business functions to the UNIX server, not running W environments, and besides, we have throw away Windows servers for that purpose. This made it a lot easier for us. Since runube processes are read-only, not updating the specs means safe "manual" package deployments. Note: this isn't for the faint of heart. You really have to have a lot of guts to go through with something like this!

It can be a real nightmare when you process over the better part of a million UBE's a month.

All of this will be so much easier once they move to XML specs, eliminate TAM and PAK and enable 'just in time' server deployments, right? I guess we'll see.

 

[gregglarkin]

[ QUOTE ]
...how do bigger companies work with this small window of opportunity when no jobs are processing. even in my medium sized company people are running jobs all the time. even during lunch they run long running jobs. hence we need to wait till people go home, which makes the life of a jde support guy tough. anyway...

[/ QUOTE ]

Shrikanthn,

JDE support guys are supposed to have a life? That's news to me! I deploy production packages weekly on Saturday mornings. Our process is this:

Every Thursday evening I submit a full package (client and server).
I deploy the full client to a test terminal server. I run a script that copies the PD7333 directory out to all of my terminal servers. The staged copy is named PD7333.new. On Saturday morning, I lock out the users, and then run another script that renames the current PD7333 directory to PD7333.old, and then renames PD7333.new to PD7333.

On the server side of things, life is more manual. On Friday morning, I create a PD7333.new folder on my primary enterprise server. I copy the entire BIN and SPEC folders from the package directory into that folder. I then copy the PD7333.new folders to my other enterprise servers. On Saturday morning, I stop the Net and Queue services on each of the enterprise and batch servers and do the directory renames.

The overall package deploy process takes about four hours every week. It takes me about two hours every Saturday to do the package maintence. I have a window of time from 6 am to 8 am to get this task done.

Gregg Larkin
JDE System Administrator (CNC) / North America
Praxair, Inc.

 

[CitrixDude]

Gregg,

Would you mind sharing these deployment scripts? Sounds like they make life easier!

Dave

E1 8.10 8.95C1 U1, Win2K3 Cluster, SQL2K, Citrix XP

 

[jolly]

[ QUOTE ]
thanks brother, how do bigger companies work with this small window of opportunity when no jobs are processing. even in my medium sized company people are running jobs all the time. even during lunch they run long running jobs. hence we need to wait till people go home, which makes the life of a jde support guy tough. anyway, if thats how it is we gotta accept it. thanks again.

[/ QUOTE ]

Hi,

On our iSeries enterprise servers, the deployment operation doesn't hold any queues. It takes some locks, then blocks attempting to acquire further locks (spec files?) held by the running UBE. Then once the running UBE ends, the deployment completes. However all this time no further UBE's can start and the callobject kernels all go into a mutex wait as soon as they try to do anything.

Needless to say this is all very bad in a production shop! To avoid, simply hold all the UBE queues until nothing is running, then run your deployment, then release the queues.

This may not be the way things work with Unix or Win32 servers, but if you are in that space you're already at a disadvantage!

Cheers,
JohnO

 

[brother_of_karamazov]

I believe that we are saming the same thing just in different ways. When I say that the deployment holds queues I am referring to OneWorld queues not OS400 queues. The deployment operation keeps OneWorld UBE jobs from running until the deployment finishes. The deployment waits until currently processing UBE's complete.


[ QUOTE ]

Hi,

On our iSeries enterprise servers, the deployment operation doesn't hold any queues.

[/ QUOTE ]

 

[Joel]

Hello list,
For Production we use a similar process that gregglarkin mentioned. Also we have a downtime of half an hour where no UBE's are run and the services are bounced. To prevent jobs from running during this downtime we hold the Queues that run the big jobs approx 2 hours before giving it sufficient time to finish.
However for Dv and PY we have had issues. Sometimes when a UBE runs in QBATCH subsystem we get an "Internal error 16". In such situations we would build the package from scratch and deploy it

 

[mt7608764]

Hi Folks..

It is very easy to create and scheduled a WinTask script on the deployment server that would check F986110, logon on E1 and deploy the package to the server(s)... no big deal!!

Going back to the original question from my brazilian friend (tudo bem por ai Nilson?), the "pathcode lock" is acquired by using IPC calls (Mutex at minimum) . EnterpriseOne on iSeries, Linux and Unix uses native IPC calls. On Windows it uses JDE proprietary version of IPC (jdeipc). For each platform you could have a different behavior (or timeouts) for the package deployment "pathcode lock", it all depends how long it would take for an IPC request to time out. This is platform specific.

Unfortunately, as far as I know, JD Edwards has no control over that on iSeries, Linux and Unix.