Slow Server Package Build (GenNER) with Tools 9.2

Hi List,

This is my config

Apps 9.0 , Tools 9.2.0.3
DB - SQL 2014 std on Win 2012 R2
ES - Win 2012 R2 Standard
Visual Studio 2013

Fully Virtual Environment on VMWare (don't have the specifics of the version and hardware platform)

When building a full package , client full package finishes fine in 2-3 hours. Server Package seems to take ever , specifically the GenNER process. This process alone takes more than 10 hours . CPU utilization is extremely low on Enterprise Server and the DB server during this time. So that does not appear to be a bottle neck . When comparing with a system where this does work fine , we can see a reasonable amount of CPU and Network activity (35-40%)

When looking at the GenNER.exe in task manager on the Enterprise Server it seems to be waiting on Network I/O.

I have a ticket open with Oracle , and they have not been able to reproduce the issue , so they don't believe it to be a code or tools issue. They think its a network or IO issue

I have also not experienced this issue at other customer sites with similar configs (Virtual , Windows / SQL , same tools , the only difference being it was Apps 9.1)

The Infrastructure team says that they don't see anything on their end. But we are going to have to really dig deeper and do some real time tracing.

But before we go down that path , I wanted to check if anyone on here has experienced anything similar ?

Thanks in Advance !!

Check if anti-virus is running on the ES and exclude the E1 folders. Try turning on jdedebug during the NER process briefly. The log should help you identify where the bottleneck is.

Thanks Hari , the AV has been turned off completely. From the debug log it looks like it is just waiting on the DB read from the Central Object tables. Already tried re-indexing central object tables.

What puzzles me is that the Client build completes fine with no issues. The Same central object tables are read there and the response is fine

In tools 9.2 client package build part doesn't include Ner generation (only BSFNs are built locally). so it is normal when it takes less time than server package build.
I would advise to run statistic on your database. 10 hours sounds like a LOT for server part . I get <2 hrs for server part on very similar config with MS-SQL

Correct , there is only a BusBuild process in the Client Build , but the same Central Object Tables are read , only difference being it was from the build machine which was the deployment server and during the server build those tables are accessed from the Enterprise Server.

We seemed to have narrowed down the issue to being something with the database IO and did some tests using third party utilities like SQLIO and found that the average disk latency is much higher when compared to other systems running the same tools and platform (SQL 2014 / Win 2012 R2 and VMWare). The Infrastructure team is investigating further and I am hoping they are able to find and correct the issue.