Top 10 reasons why Developers should NEVER be able to promote projects or build packages...

Top 10 reasons why Developers should NEVER be able to promote projects or build packages...

I have a customer - who is currently running OneWorld Xe. Quite large, their implementation has several hundred users and given the amount of customization - they have a large number of full time OneWorld developers.

So, the question being posed is "why shouldn't developers have access to be able to promote projects to PY and build PY packages". Their reason is (as stated) because "waiting for CNC to build packages is delaying their testing".

I'd like to create a list in a public area like this to protect companies from their own ignorance - and so I'll kick this little holy war off with my own reasons.

1. In a public company, SOX requires that changes to production are NOT applied by the developers (segregation of duties)
2. Package builds MUST be performed on a centralized, individual basis. Multiple builds cannot happen in parallel (corruption of packages)
3. If developers have access to promote/demote objects - then this will become abused since developers will "fix" objects without other developers knowledge - the "token" system becomes useless
4. Developers usually have no idea about how to troubleshoot package build issues or promotion issues. They can barely troubleshoot why their own objects can't be checked out...
5. Package builds/promotions require access to the system MUCH higher than development access. This could therefore become abused, even without the developers knowledge.
6. To troubleshoot a build issue on the enterprise server requires access to system code on the enterprise server. Again, this access could result in abuse and security breaches without a developers knowledge.
7. Promotion of certain objects AUTOMATICALLY updates an environment - and needs to be carefully managed otherwise users can be severely impacted.

I'll stop there - but I'm sure that others can enlighten this thread.

Heres my proposition. Developers should ONLY have access to development objects - and should NOT have access to promote projects. Access to PY should be limited, and only if a business analyst (tester) has an issue and it requires troubleshooting. Developers should have the ability to promote from "21" to "22" - or some other "dead" code in OMW to mark a project as being ready for promotion to PY. Someone else (a development manager) should then identify whether all the documentation and individual development testing has been completed by moving this from "22" to "23" - ie, ready for CNC to promote. CNC then picks up all projects at a set status on a known, regular schedule (daily) and promotes from DV to PY.

DV packages should RARELY be built - I say developers should "get" whatever objects they need. You don't know what objects ? Check cross-reference (I always give developers those type of tools). Developer package deployments usually result in a developer losing work, and once is too many times. I build a full development package about once a month or even less regularly.

I await comments(!)

Re: Top 10 reasons why Developers should NEVER be able to promote projects or build packages...

Let the Holy Wars commence!

Here's my contribution to the list:

8. Developers should not be able to do anything which demonstrates that they have a better understanding of CNC than the CNC admin . . .
----------------------
Anyway Jon, as always the real answer is "it depends on the shop". If you're under SOX requirements and/or have a number of developers then yes, I see your point(s). Other, smaller shops with only a couple developers and low customization activity - no big deal then. It just depends.

My 2 cents.

Re: Top 10 reasons why Developers should NEVER be able to promote projects or build packages...

One more to the list..

9. Developer should put all the request for Production build by Wednesday EOD all subsequent request will be taken care in following week.

Jon i will not agree for the point of Developer to promote to some dumb status like 22 and then a Manager will review and move it to 23 or PY. This are good for documentation purpose but how many managers will be acutally knowing what the developers did and keep track of each and every projects?. I think best way to twist it is a desired o/p should be given to a developer and if he acheives it he can attach the o/p to project and move it to PY for build request.
My 2.1 cents

Thanks,
Chan

Re: Top 10 reasons why Developers should NEVER be able to promote projects or build packages...

I just have one comment... for now.

The part where you said

"Developer package deployments usually result in a developer losing work, and once is too many times."

I agree that once is "too many times" and we also do DV packages very rarely. However, we have trained our developers to make sure that they utilize the SAVE button. They should never assume that what they were just working on is going to be there the next day.

We also have all of our developers on a VMWare virtual machine and although they have been assigned an instance it may need to be used by someone else the next day. Then, they would have to use a different one.

Anyway, just my quick thought about the idea.

Re: Top 10 reasons why Developers should NEVER be able to promote projects or build packages...

Yup - you started the Holy Wars!

J - I agree that developers should not be doing the promotion process, for
many reasons... but, you don't treat the issue correctly.

The post should have been 'Why CNC should do their jobs'

In too many shops the CNC put a rigid, and often rediculous, timeline on
non-production package builds. Development builds to PY should occur
ATLEAST once a day - if objects are at the correct status by the requested
time. However, too many shops say that DV/PY builds will only occur on
specific days - in doing so, they curtail the development process. The
developer desires to make the User happy and to have their stuff tested,
tested and tested some more. For the CNC to play gawd and appoint a
schedule that benefits them, alone - is NUTZ.

There are reasons Developers should not build packages - most of it has to
do with Single Point Of Responsibility. If things go bad, it is (usually)
the CNC that has to bail the Package Process out. Additionally, like at one
recent client - there could be issues within the process that brings down
the entire system when a deployment takes place under very comon
circumstances... Since the CNC is the one that would have to fix - it
should be solely thier responsibilty to create the problem, too!

The comment about Developers barely being able to figure out their own
issues was unnecessary... I know it was in fun, but ... there have been far
too many times where the development staff has had to help the CNC staff
resolve issues together (ESUs and Tools - we've all been there).

WE (CNC and Development) are a TEAM - and each member of the team has a
position. The CNC is not responsible for Development and the Developer is
not responsible for bringing down the system

See you next week in Denver!

(db)




--

Re: Top 10 reasons why Developers should NEVER be able to promote projects or build packages...

[ QUOTE ]
The post should have been 'Why CNC should do their jobs' <br><br>In too many shops the CNC put a rigid, and often rediculous, timeline on<br>non-production package builds.

[/ QUOTE ]
Wow. Hitting a sore point with you here ? Got something against CNC people having a night off ?

[ QUOTE ]
Development builds to PY should occur<br>ATLEAST once a day - if objects are at the correct status by the requested<br>time.

[/ QUOTE ]
Correct. I actually state at my customers that 5pm is the cutoff time for a PY package. However, almost every day I get emails at 6pm asking for a promotion - and at 9am a followup questioning "why wasn't my object promoted". I've actually got emails as late as midnight asking for a project to be promoted - and a followup at 8am wondering why I hadn't promoted it yet.

[ QUOTE ]
However, too many shops say that DV/PY builds will only occur on<br>specific days - in doing so, they curtail the development process.


[/ QUOTE ]
PY promotions/package builds should be Monday through Friday at a specific time. CNC should NOT curtail the development/testing process.

The big issue is that the business also tries to then get a daily production promotion schedule - which obviously doesn't work because it requires a certain amount of downtime to production for deployment. Usually I push either a weekly or twice-weekly schedule for production promotions.

[ QUOTE ]
The<br>developer desires to make the User happy and to have their stuff tested,<br>tested and tested some more. For the CNC to play gawd and appoint a<br>schedule that benefits them, alone - is NUTZ.

[/ QUOTE ]
which, of course, never happens when I'm around - but its an interesting wound you're opening here (!)
[ QUOTE ]
<br><br>There are reasons Developers should not build packages - most of it has to<br>do with Single Point Of Responsibility. If things go bad, it is (usually)<br>the CNC that has to bail the Package Process out. Additionally, like at one<br>recent client - there could be issues within the process that brings down<br>the entire system when a deployment takes place under very comon<br>circumstances... Since the CNC is the one that would have to fix - it<br>should be solely thier responsibilty to create the problem, too!


[/ QUOTE ]
Right. If I screw up a deployment / package - I have no issues with troubleshooting/fixing/working through 1am to find the issue and repair my problem. Of course, this is about as rare as condors teeth....
[ QUOTE ]

<br><br>The comment about Developers barely being able to figure out their own<br>issues was unnecessary... I know it was in fun, but ... there have been far<br>too many times where the development staff has had to help the CNC staff<br>resolve issues together (ESUs and Tools - we've all been there).


[/ QUOTE ]
Now, there are some "developers" - who I might agree with, they know the system well - and usually these are not the problem people. They relinquish the responsibility of certain parts of the system to the right individuals in the team.

But, there are times, and sometimes its permanent members of staff, and embarrassingly enough sometimes its a consultant, who is a total megalomaniac. Believe me, you people know who you are. These are people who really don't know what they're doing - yet "claim" to understand CNC, "claim" to understand the package build process, "claim" to fully understand the development cycle. If these people really knew what they "claim" they knew, then they'd be running the entire project themselves.
[ QUOTE ]

<br><br>WE (CNC and Development) are a TEAM - and each member of the team has a<br>position. The CNC is not responsible for Development and the Developer is<br>not responsible for bringing down the system

[/ QUOTE ]
You see, I totally agree with this. So why is it that I've seen a few customers in the last year or so where developers claim they could do a better job than a CNC (with horrible consequences) - hence this is my entire beef and reason for this thread. Trying not to make this personal, but I certainly know the toolset (and have been known to train developers) - but I certainly am no developer...

Re: Top 10 reasons why Developers should NEVER be able to promote projects or build packages...

Jon,
Very glad you posted a "Top 10".
I used worked for a very large publicly traded company is South Florida. Sounds like you were refering to them........
That Development Team has a CNC consultant and they will Build and Deploy DV and PY update packages up to 9 times a day. The PD package deployment and promotion of code is controled by another department.

At my new company, Daily DV and PY package build and deployment, on a schedule works just fine.
We do weekly Full builds for Production and Monthly Full builds for DV and PY.

We are an international company, XE, AS400's etc.
We have 9 CNC's around the world that cover the America's, Europe, middle east, Rusia, China and Australia. We are in production in 3 languages (More to follow) and are licensed for 6,000 users, 2,100 most recent concurent max.

Management and deployment of our E1 code is very important.
Most of the "Top 10" that Jon listed are followed by my team.

Thank you. Hope to see some more additions to the "Top 10".

My Addition(s).
9. Document everything about the deployment process as an SOP.
10. Plan. Plan to Plan. Work the Plan. Update the Plan.

Dave

Re: Top 10 reasons why Developers should NEVER be able to promote projects or build packages...

[ QUOTE ]
...a very large publicly traded company is South Florida. Sounds like you were refering to them........

[/ QUOTE ]
No names, but I am sure there are plenty of companies that seem familiar (hence the popularity of this type of post) - but this wasn't identifying any specific company, organization or team.
[ QUOTE ]

That Development Team has a CNC consultant and they will Build and Deploy DV and PY update packages up to 9 times a day. The PD package deployment and promotion of code is controled by another department.


[/ QUOTE ]
He seems busy and has obviously worked out some tremendous job security. I personally don't work that way myself, instead, I find customers benefit from some sort of ROI if they hire me on to administer their environment - even if that means finding a permanent replacement. Pity not all consultants have the same mentality (though I call those people "temporary employees", not consultants).

Re: Top 10 reasons why Developers should NEVER be able to promote projects or build packages...

I love the notion that all of you seem to have of "one size fits all" for building and deploying packages, along with just about everything else about JDE.

Let me state, for the record, that we are not a billion dollar company, with dozens of developers and instances of JDE in various languages around the globe and throughout the galaxy. And as far as I can tell, neither are quite a number of the other companies who use JDE. Hooray for you, if that's your situation, but your ideas of how to run a JDE shop simply would never fly here. Management would throw you out on your ear in about an hour.

The concept of daily/hourly/minutely PY package builds, or full PD packages every week, or every month don't fit a shop of our size. Now we're not a tiny company, but we have a small IT staff -- 8 people in the department (including myself), with 4 of those being developers. We also have two business analysts who man the help desk and test and approve the work of our developers. We build update packages for DV once a week, and PY and PD update packages every other week. We build full packages about once a year, if that often. This has worked perfectly for us for over 6 years.

The job of IT, in this company (yours may be different), is to keep the systems stable and available for as many hours per day as possible. Almost all of us have other duties, like help desk, document imaging, email, web servers, etc. For example, I'm the IT manager (have to manage personnel), the main CNC (my network engineer also does some CNC work) and lead security buffoon. I also am the technical lead on some external projects. Who has time for daily package builds? Who's going to work that many hours?

Prior to go-live, we did indeed build packages (both full and update) very frequently, sometimes several times a week. (We're currently on our 16th full PD package, and our 27th full DV and PY packages.) But since go-live, we have reduced the package build schedule to once a week. We don't need to impact our production users (who are on the system from about 5:00 am until 10 pm daily) with constant package deployments.

Many of the ideas here scale up well, but they don't scale down well, if at all. I realize lots of you are contractors, and have probably worked in a lot of large shops. But just for a minute, try to understand what it's like to work in a small shop, where everyone has to wear lots of hats, and see if the ideas you're promoting (I won't use the term "best practices," as that will start a different holy war) scale down for smaller shops. Of course, if you never, ever in your life intend to work in a small shop, then you can safely ignore such thoughts.

Re: Top 10 reasons why Developers should NEVER be able to promote projects or build packages...

Bill

I think what you're stating EXACTLY jives with my comments so far. My point is that "No MORE than 1 PY package a day" is for a larger shop - and certainly a smaller shop like yourself, that works too - if you do minimal development, you shouldn't really be doing many package builds at all - I know customers who now only do one update package a week, if at all - because everything they've created is so stable. Of course we'd all like our customers to be in that situation.

The holy war is whether DEVELOPERS (and by plural, I'm referring to a larger shop with multiple developers) should have access to create packages and do their own promotions. You might have a very small shop with one guy that does both development AND CNC duties - which is fine, but I'm referring to a larger shop with multiple developers and at least one CNC administrator. I've noticed that some "big-5" consultants and others have recently been pushing customers for developers to do their own promotions/package builds - and wanted to start a mini holy tirade against this type of mentality. I'm also interested in any stories where such a situation has actually been implemented - and the results of such an implementation (!)

By the way, I am going to disagree with one statement. I think there is a "one size fits all" solution. Its called OneWorld. It seems to fit quite a varied size of corporation....

Re: Top 10 reasons why Developers should NEVER be able to promote projects or build packages...

Everyone except John - know that John and I are friends, otherwise we
wouldn't have had so much fun with this thread!

John - I totally agree with your statements.

One other thing to put into the mix:
* only the CNC and the Project Owner should be able to Drop Tokens from a
project. I can't tell you how many times someone impersonating a developer
has asked me to drop the tokens in someone else's project - SCARY...

(db)




--

Re: Top 10 reasons why Developers should NEVER be able to promote projects or build packages...

Oh goody, lots of opportunity for argument here.

Developers develop, CNCs CNC, if you're a big shop and have both, some places don't, and there the argument is moot.
Having someone who is disinterested in the change (as opposed to uninterested) provides the opportunity to put a check in that all parts of the change (documentation, test packs) are ready; not just the code.
You really don't want two simultaneous builds going on; or two Jas Gens if like us that's the end game.
If your Developers don't understand how the CNC layer works you can educate them, if you need to, if there's any value in doing so; but that just goes back to my first point; size matters.
When you build packages, updates or full, how quickly you deploy them are all local issues and will vary with your business process; why would you want to tie up a code coder with that proceduralisation.

Re: Top 10 reasons why Developers should NEVER be able to promote projects or build packages...

[ QUOTE ]
I have a customer - who is currently running OneWorld Xe. Quite large, their implementation has several hundred users and given the amount of customization - they have a large number of full time OneWorld developers.

So, the question being posed is "why shouldn't developers have access to be able to promote projects to PY and build PY packages". Their reason is (as stated) because "waiting for CNC to build packages is delaying their testing".


[/ QUOTE ]

I can't add anything to the list. However, as a BSA, I'd opine that this is a textbook example of slapping a band-aid on a symptom, rather than doing root cause analysis to solve the problem.

I have experience at one company that was trying to implement Xe as cheaply as possible (running HTML version for instance). The technical/developers were also trying to build packages with no CNC. It was a disaster to say the least.

Re: Top 10 reasons why Developers should NEVER be able to promote projects or build packages...

Hi Jon,
I have lots to say on this topic but sorry I don't have time to type it out today. I'm in pre-Collaborate-panic-mode as I leave tomorrow morning.
If you have access to the Collaborate archives my old presentations probably have some points that you can use.
2005 - 102070 - Putting your SOx on one code change at a time
2007 - 20960 - E1 Tech Chg Mgmt

Other than that - find me at the conference and I would be happy to talk your ear off.

Sue Shaw
Shell Canada Ltd
Xe SP23J1 Coexistent

Re: Top 10 reasons why Developers should NEVER be able to promote projects or build packages...

As a programmer/analyst, I was able to perform my own builds for DEV and PY at one company I worked for. Except for the after hours work required to perform the updates, the arrangement seemed to work out for everyone. Production builds were on a schedule, and DEV/PY builds were done every evening as needed. The risk was that I also had access to update production due to the nature of that shop’s set up. At another company, we used an ASP to maintain our servers and perform CNC functions. I initiated a request to perform DEV and PY builds as a cost savings measure since we were charged for each build request made. The ASP developed instructions for us to create the package, but still maintained control over submitting the builds and installs.

Now, I feel compelled to respond to how this issue is being presented (*launching rockets in 3-2-1…*).

This post is a good example of how negative CNC folks can be towards their counterparts, without good merit. The interaction of these 2 disciplines, development and CNC, should be defined by a company’s needs, and not arbitrary policies that one group feels should be in place to control the other. If CNC support is competent, and able to find a happy medium between meeting the needs of end users, developers, and themselves, then you’ll find little debate over the separation of duties. I for one am more than relieved to have a good CNC take on the system responsibilities so I can stay focused on my project work. But when developer and end user needs are not reasonably met, I’ll be the first to push the envelope on CNC vs. Developer responsibilities.

As for the question whether programmers should have access to create packages and do their own promotions, I’ve pushed for this before when I was unhappy with the CNC quality of work. When you request a package build, either by object or project, you expect that all the objects requested will be included in your build, and installed everywhere it needs to go for the environment requested. In a couple of shops I’ve worked in, you would have thought I was asking the CNC for world peace… Missing objects in a build, or a build that was installed on fat clients, but not pushed to the citrix server used by the testers, or a build that just wasn’t even done … makes for a frustrating situation. And when you add to that a build schedule that only allows for weekly or bi-weekly development builds, and a CNC who will not waver from the schedule even if he screwed up the build …

How about, instead of focusing on a “top ten reasons why developers should NEVER do builds”, we talk about CNC and developers working together to keep their change management processes running smooth and efficient.

Re: Top 10 reasons why Developers should NEVER be able to promote projects or build packages...

[ QUOTE ]
How about, instead of focusing on a “top ten reasons why developers should NEVER do builds”, we talk about CNC and developers working together to keep their change management processes running smooth and efficient.

[/ QUOTE ]

Come on, that's just crazy talk!

Re: Top 10 reasons why Developers should NEVER be able to promote projects or build packages...

I would not consider many of the folks classified as "CNC" as "true CNC's". Many of them are nothing more than System Administrators who dabble in CNC. Quite a few of them inherited the job function, they didn't seek it out. Many of those folks, not all, help give CNC a bad name. That is all I'll say about that for now.

I've seen all of this before, and of course I have formed my own opinion about "how things can be run more efficiently" and most likely "better" than they are typically run. Rather than go on and on and on about it, I'd like to offer up (perhaps retread on) a few simple ways those processes can be streamlined. I'll try to eliminate one of the "ons" and only go on and on about it.

First things first, if you put the data where it is needed, you can reduce the number of builds required in order to test a change. With the exception of large projects like installs and upgrades, keeping the lower environments refreshed with production data during the maintenance phase can reduce the number of object promotions necessary and thus reduce the number of builds. I realize this isn't always possible due to the enormous data sets some companies have in production, projects holding data hostage, the lack of available disk space, DBA & CNC time, etc. But, for productivities sake, a business case can often be made for the necessary resources in order help straighten things out. There are many software tools out there that can simplify these processes.

In a development environment, most things can and probably should be run locally unless it absolutely has to run on a server. In those instance where things can run locally (and even in a lot those where they must run on a server), a unit test environment (or two or more) can assist the developer with their changes and testing processes without the need for a CNC. In the case of the latest web enabled releases of JDE, the web instance can be run locally on the fat client, eliminating the need for a CNC to gen the JAS objects provided the local instance is mapped to a personal database for F989999/F989998. Think of how the business analysts or end-users can work side by side with the developers without much interaction at all with the "infrastructure types".

If a server is needed, give the developers a server for crying out loud. Lock the deployment server and databases down so that if you let them build packages for their development environment, they can only hurt themselves, not others. If you're worried about tying up the deployment server, add another one and connect the two with OMW rules. It can be done and it really doesn't make things "too complicated". Effeciency gained....

Then the CNC can worry about more important things than trying not to be the bottleneck in the development lifecycle. They might even have enough time to learn to be a better CNC, then maybe they can be of more assistance to the developers with their problems (maybe even make friends, *gasp*) when they no doubt occur as a result of a hardware issue, Tools update, etc.

I used to work for a company that still had JDE Demo data loaded in DV 8 years after their initial go-live. Only a few select tables were manually copied down from CRP to DV by the developers. They pretty much designed their code and then waited for the promotion to CRP before they ran their first "true test". Sometimes the developers didn't even bother to hide the fact they wanted to go straight to QA (their custom environment between CRP and PROD) skipping over CRP for that first true test. Of course the CNC lead at the time insisted there would be no "fast track" to production. But the result is that all of those unnnecessary promotions led to unnecessary builds, irritated staff members, and most likely errors in builds as a result of the highly stressed "package builder"....and yes even developers sometimes forget to put an object in a build ("but it worked on my machine!"

The same thing goes for integration points. There were of course some integrations that weren't enabled in one or two of the lower environments, thus "necessitating" the extra hope and the delay in the build. Guess what happens when the code doesn't work once it is in QA? Yes, all the way back down to prod and then the same dance begins all over again. I saw the development environment of a custom Order to Cash application hooked up to the CRP environment in JDE. Of course that meant the obvious - JDE developers were always one step behind the OTC developers or testers as they 'danced' their way from DV to PY in order to run their tests. Of course the CNC was not properly consulted prior to this integration, but again, the end result is a bunch of what I believe to be unnecessary package builds.

In my mind it isn't about making the CNC or the developer happy - it is getting the process optimized so you can get the job done.

On that note, sort of off the topic but not entirely, I read the other day that Microsoft is building a huge data center in the Northlake, Illinois. They'll have what is essentially a skeleton crew managing thousands upon thousands of servers. The theory goes that less IT workers means less chance for human error and thus less downtime as a result of those IT workers trying to fix problems that occur when the servers fail in some way or another. By building in tons of redundancy and treating the servers as commodity resources, when servers fail, the others just pick up the slack and things just keep on keeping on. Over time the dead units are replaced and life is good.

It makes me wonder what would happen if more folks started to worry about becoming just another commodity resource and instead started looking for ways to innovate, perhaps by working together for a change?

Re: Top 10 reasons why Developers should NEVER be able to promote projects or build packages...

As a developer, I must go on record as saying: Go Developers!

Seriously, a lot of good quibble here guys. As a consultant, I have been to different sized companies with varying degrees of standards and access to 'CNC-related' tasks. I'll qualify my following remarks by stating they pertain to larger shops. For the most part, I am all in favor of strict standards which separate the duties of the two disciplines. I, personally, have no desire to want to build packages...I like to leave work no later than 2PM (remember, I am a consultant ). I have never really had issue with following whatever guidelines are in place at the respective customer site. I also don't see any reason that developers would need to promote projects. Using the 'Advanced Get' can be a powerful work-around.

There is only 1 thing I have had issue with that can curtail my productivity: Not being able to demote projects from PY to DV. I abhor having to submit a request for project demotion so I can rework an issue with an object(s). Such requests usually have low priority (in my experience). Having a 'save' location could be a work-around, but I've been to sites that do not have them and they can be 'unsafe' in that the object in the save location can be overwritten by another user.