Deployment Frequency and Strategy

I am looking to see how often large, mulit-site, global companies deploy code to their ERP environments. Further, is their a separate strategy for managing ESUs - stay fix current or apply only fixes needed due to break fix. For ESUs, it is a slippery slope because eventually, baselines are required which can severely disrupt the business and business resouces due to the impact.

I totally understand this question is rooted and driven by organiational culture, budgets, politics etc. However, I am looking to see what others have either seen or experienced. Is anybody willing to share their thoughts and provide some concrete numbers regarding the frequency of deployments - daily, weekly, monthly, quarterly, emergency based, hybrid (release schedule for large initiatives with monthly deployments for break fix as an example).

Where I am based now, for update builds for DV and TS tend to happen on a daily basis when required, and can be deployed during the working day.

However if a full build is required we deploy out of hours normally over a weekend.

Builds for PD for both update and full, we try to wait until we can batch a few projects together, and we build and deploy out of hours over a weekend.

However special circumstances can change our approach slightly, such as volume of objects, severity, priority and deadlines can mean that we do multiple builds into DV or TS daily, and may need to squeeze a PD build in over night during the week.

ESUs we treat differently, we apply when and where required, and carefully manage the objects that are effected.

A past company took a different approach, and had a set deployment window for an hour each morning in which any PD builds could be deployed. DV and PY builds were also done daily at a set time.

Hope this helps
Aidy

At my last gig we had JDE rolled out in 20+ countries, 2000+ users. For our PD deployments we did them once a week on the weekends but what could be included was limited. For instance, version only changes were included each week, new custom programs were included once a month and changes/customizations to global objects (P01012, P4210, etc) and ESUs were done once a quarter. It started out that anything could be included in the PD update every week but we had been burned too many times by regions saying they tested changes but really didn't do enough testing or didn't test at all.

For us ESUs were primarily applied for break fix needs, however, there were plenty of times localizations forced us to apply ESUs and also baselines as well as a pre req.

PY updates were done twice a week.

You are correct this is all dependants on company culture and how the top wants to drive the business.

I have done it several ways and they all worked because it was driven by the business and IT.

Currently we do package builds twice a week
On Tuesday there is a PY/TEST build, we rarely build DEV packages as our developers are used to testing locally. We do a Full DEV once a Month the 1st week, PY the 2nd week.

On Thursdays we do a PD Build and deployment – this is generally done at night before the scheduler kicks off. We do a Full PD build Monthly the 3rd week. If there is an emergency PROD fix we follow a sign off emergency process to make sure that the business is willing to go to top management for the emergency, this weeds out really what are true emergencies.

For ESU’s one your system is like it is difficult to truly star code current, you need a lot of resources to coordinate testing. Again the business culture will dictate that. For use we only apply ESU’s when they are necessary because no one wants to test and no one wants to mandate testing.

Our package build process works pretty well as the users accepts that if they did not test/validate a process they will have to wait an additional week.

We also follow a strict Change Management Approval Process put in place by out internal audit controls so that helps drive the process so IT/CNC does not get blamed for things not making it to PROD.

thanks for the replies.

anybody else want to chime in?

oh go on then....

we do about 5 PY / DV package deployments per week and we do either 1 or 2 production packages per week out of hours.

They are all update packages, we see a full package deployment as a higher impact / risk than updates and we only build / apply them when necessary (such as new DLLs).

As with soulglo we conform to a change control process and require testing to be signed off before we deploy the modifications.

With 8.98.4 you have the ability to deploy some code whilst the system is up (such as UBE's & int apps) but we haven't done this on live yet.

The tricky bit with a large company spanning multiple countries is to find a slot for downtime. The package deployment itself is not time-consuming but if it doesn't work and you need to back it out that might take some time.

You may also want some confidence testing to sign off the change on production after you've deployed it.

In terms of ESU's I think the strategy that works best is get code current before implementation, upgrade and then be stable, only applying what is necessary to fix business critical issues. If you deploy every new ESU you'll be testing / deploying the whole time.

You might want to consider deploying baselines (say) once a year but this does depend on your company culture and number of mods, not a trivial piece of work to rework/test a number of baselines.

Rich

I will add one comment about ESUs from the dark side. Majority of the replies, and I think companies, use the philosphy of 'if it isn't broken, don't fix it'. The justification is ESU testing is tedious and the business users don't want to do it. While it is not a good practice to apply every ESU as it comes out, be careful about not applying ESUs at all.

We are on ERP 8.0 installed in 2003 with "baseline" Update 1 installed. We have installed a few minor ESUs, but no further baselines.

We installed on SQL 2000 and have since upgraded to SQL 2005 which dramatically improved performance. We have also advanced Service Packs to 24.1.2. After going to SQL 2005 we began having strange errors show up sporadically where UBEs will fail with 'undefined database failure'. We also had a case where Credit checking a large customer would crash 1 out of 10 orders placed. Oracle solved these errors with ESUs that match the performance improvements of the new Service Packs and database. In order to fix, we have to apply baselines as prerequisites. So, we are living with sporadic errors since we cannot get traction to apply baseline ESUs.

Now, with shorter support lifecycles and continuous significant advances in underlying technology, staying up to speed on ESUs is essential. This is particularly true for those of us on Microsoft and Oracle platforms. I realize AS400 and UNIX may not have the same technology churn, but creating the habit I outline below may still be beneficial.

My recommendation, like Rich, is picking a time of the year when everyone expects to help with testing new ESUs. When that time comes, if there are baselines or major ESUs from Oracle to apply you do so. If not, you wait until next year.

The advantages of this process:
- people will develop processes and schedules to handle it as efficiently as possible; it should get easier each year as habits are formed
- documentation will be reviewed and kept current for training new employees
- opportunity to explore new functionality or different ways of using current functionality will help advance the business
- opportunity for people new to their positions to really become experts in their area of functionality
- your system can more easily handle patching a flaw that shows up during the year by installing a smaller, specific ESU
- company is getting value out of the massive maintenance fees being paid to Oracle each year
- can budget for consulting help if needed to augment your staff

Jer