Enterprise One Deployment Server - Full Client Package of WAN Very Slow

Technology
Product: *EnterpriseOne* - E811 SP1 Update1
Service Pack/Tools Release: 8.96.D1
Platform and OS level: Solaris 10 (global zone)
Database and version/level: Oracle 10.2.0.1
Web Server platform and OS level: Win2003 SP1

Layout - Production - Auckland NZ.
Application Server 1: Domain on a E25K (4 * CPU IV+ 32 GB memory - EMC storage) - Solaris 10
Application Server 2: Domain on a E25K (4 * CPU IV+ 32 GB memory - EMC storage) - Solaris 10
Applications Server 3 and Database Server: Domain on a E25K (8 * CPU IV+ 64 GB memory - EMC storage) - Solaris 10.
4 * Win2003 SP1 - Oracle Applications Server - JAS

Layout - Dev/QA - Wellington NZ (600 KM South of Auckland).
Application Server 1: Domain on a E25K (8 * CPU IV 64 GB memory - EMC storage) - Solaris 10
Database Server: Domain on a E25K (8 * CPU IV 64 GB memory - EMC storage) - Solaris 10
4 * Win2003 SP1 - Oracle Applications Server - JAS
Deployment Server - Win2003 SP1.

Network:
All server connected at 1000FD - set on both server and switch.
100Mbit WAN link between Auckland and Wellington ( 12ms latency).


Problem:
Doing a full client package build to QA Database in Wellington - from the development server in Wellington.
The build log below:
Fri Feb 09 17:01:55 - Building TAM and/or Pack file(s) for fdaspec.
Fri Feb 09 17:01:55 - fdaspec will not be packed. It is not necessary for the server.
Fri Feb 09 17:01:57 - Spec count from Central Objects: 400976
Fri Feb 09 17:25:58 - Total Number of spec records read: 400976.
Fri Feb 09 17:25:58 - Total Number of spec records written: 400976.
Fri Feb 09 17:25:58 - Creating index for fdaspec.
Fri Feb 09 17:28:05 - Index created.
Fri Feb 09 17:28:05 - fdaspec built.

The (LAN) build completed in approximately 30 minutes.

Doing a full client package build to Production Database in Auckland - from the development server in Wellington (approximately 600KM away).

Thu Feb 15 17:50:41 - Building TAM and/or Pack file(s) for fdaspec.
Thu Feb 15 17:50:43 - Spec count from Central Objects: 400456
Fri Feb 16 00:21:03 - Total Number of spec records read: 400456.
Fri Feb 16 00:21:03 - Total Number of spec records written: 400456.
Fri Feb 16 00:21:03 - Creating index for fdaspec.
Fri Feb 16 00:23:11 - Index created.
Fri Feb 16 00:23:11 - fdaspec built.

The (wan) build completed in approximately 7 hours.

Questions:
1. Is a build over that wan expected to take approximately 14 times slower then the build in the same data centre?
2. Do people normally do client builds over the WAN?

Thanks in advance for your response.
Regards
Mark

Hi Mark,

Yes, it's normal.
PSE1 package building is network voracious, it reads
~3 Gb data from Central Objects to the Deployment where
it builds Spec files, which in turn will be copied back
to ES when you will deploy the Server Package.
It's convenient to have Central Objects data sources on
the same LAN of your Deployment.
Regards,

[ QUOTE ]
1. Is a build over that wan expected to take approximately 14 times slower then the build in the same data centre?


[/ QUOTE ]
Yes. Latency is the big issue - although you'll probably have bandwidth problems as well. I'm surprised that it ONLY took 7 hours to complete across a WAN !
[ QUOTE ]

2. Do people normally do client builds over the WAN?


[/ QUOTE ]
No. Its a waste of WAN resources and time ! Your deployment server should be as close as possible to wherever central objects is located - and either you should use the deployment server itself to run the build or a FAT Client local to the deployment server to do the build. During the 7 hours of doing this build, you probably dramatically altered performance for any users trying to access oneworld across the WAN - which is a big no-no !

Hope that helped.

Thanks the replies.. but these two replies seem to be opposite.

"Yes, it's normal." vs "No. Its a waste of WAN resources and time !".

Maybe I got the question wrong?
Should the deployment server me in Auckland with Producton, or in Wellington with Dev/QA?

If in Auckland, then the package build should be quick.
If in Wellington, then should the package build be done on a FAT client, and copy the package to the Development server in Auckland.

Am I missing something here... this sounds like a very normal sitution to have Dev/QA in one data centre and Production in another data centre seperated by some distance (650KM for me) - connected over a WAN (100Mbit)?

Thanks
Regards
Mark

Hi Mark,

Yep... "normal" is not the best adjective to describe
your situation. Sorry for my English!
What I meant is ... it's "normal" for packages to be so
slow when they're built across a WAN.
As Jon and I said before, you should have your
Deployment on the same LAN of your Central
Objects datasources to achieve a faster building,
reduce latency and leave more WAN resources for your users.

Mark,

I would disagree that the two replies oppose each other. Adding a conjunction between them may make it more clear:

Yes, it's normal (slow package builds over the WAN) *AND* it's a waste of WAN resources and time.

I don't know what else is going over your WAN link but during your remote package build the build is using a good portion of the link.

Not only is there a large amount of data being pulled down from the database to the deployment server but there is an equally large amount of data being piped back to the Enterprise server. About 3GB of data is being pulled down, somewhat less data is piped to the enterprise server in a compressed format. The 3GB of data that is pulled down is not done by some streaming protocol. It is pulled down via the database client software. There is a lot of back and forth communications in the db client protocol. This is one of the places the latency mentioned by Jon really comes in.

In the days B732, B733 LANs were typically 10Mbit with 100Mbit being a luxury. Build times were long in great part due to the bandwidth issues (minus the WAN latency) that you are experiencing. Depending on platform I used to consider a 16 hour package build average. By trying to build across the WAN you are stepping back in time to the days of 100MBit LANs.

As for whether having QA in one data centre and Prod in another is a common layout I would say it isn't. At least when talking about an E1 implementation a split configuration like yours would not be something I would do if I had a choice. However, I have built and supported many distributed E1 architectures including some that incorporated trans-continental links as slow as a T1. In each of these cases I have installed multiple deployment servers, one to manage each location/path code(s) combination. In your case I would suggest a Production Deployment server and a non-Production deployment server. In this way the only time you will have to go across the WAN is during OMW promotion from your last non-Production path code to your Production path-code.

Your fat client approach to package builds could be made to work. Out-of-the-box the fat client is going to point at the deployment server as the location where the package is built. A fat client doesn't actually build the package on its own disks. You can play games with LMHOST files and make a server local to the fat client look like the deployment server. However you would then need to copy business functon source files ahead of the package build and keep other pieces synchronised. It would all be very messy and I think would end up costing as much in extra labour as a second deployment server.

In my opinion the dual deployment server approach will give you the least hassle with consistent performance.

[ QUOTE ]

Am I missing something here... this sounds like a very normal sitution to have Dev/QA in one data centre and Production in another data centre seperated by some distance (650KM for me) - connected over a WAN (100Mbit)?


[/ QUOTE ]

You are missing something. The architecture.

Its not "normal" to have development working in a different data center from where the servers are located - but if it does occur, then the right architecture should be designed with that in mind. Obviously in your situation, the "architect" just stuck in a standard oneworld design and completely left out where development was going to be located.

Secondly, we're not really talking about development here - we're talking about package builds, the step BETWEEN development and actually deploying the software.

You should never place the deployment server (the primary one) away from Central Objects (the database). However, you have a number of options. Either you come up with a horrible multi-tiered deployment server solution with replicated central objects (warning, this is NOT for the faint of heart) or you do your development using Citrix, centralizing ALL of your OneWorld implementation in one data center - development clients included.

Since you have a relatively fast 100Mb WAN connection - I would certainly go with the latter - create a citrix development solution (I've published a white paper on how to create this on my website) - and have the developers (who don't really matter after all) develop remotely. I've done this with 40 developers in India and a single citrix development box in Kentucky, so I know it'll work fine.

That was a joke about developers not really mattering by the way (!)

There are several options, but replicated central objects is not required if they add another deployment server (basically a new installation of JDE) in to their production data center, migrate the production pathcode to it, reinstall the fat clients which point to prod, and then use OMW rules to connect the two installations and perform transfers between QA and PROD Central Objects over the WAN.

Granted, this is a more complicated setup than a single deployment server - but it works and was utilized for several years by a very large JDE shop.

A byproduct of the type of configuration I mentioned is a true production instance of the deployment server, physically separate from DEV, and this benefits an organization in many ways - from disaster recovery to security audits, etc.