OneWorld Xe Application Freeze - Hang - Lockup

[Jotto]

Hello,

We are having an odd issue, and I wanted to throw this out there to see if any of you have experienced anything similar, and may have ideas or light to shed on it...

We are running OneWorld Xe SP 23 Y1 on Windows Server 2003 R2 Enterprise SP2 and SQL Server 2000 sp3a. Our users connect using the Fat Client on Virtual Citrix Servers that are also Windows Server 2003 R2 Enterprise SP2. We also have one Citrix Server that is Physical. The production database server is physical using SAN storage, and we have 3 application servers – two physical and one virtual, and one physical deployment server, all Windows Server 2003 R2 Enterprise SP2. (I know we are completely obsolete, that’s a bigger fish to fry.)

We have users experiencing an issue where the first time (for the most part) they are working through an application such as Sales Order Entry, where they will be tabbing/clicking along and then the whole application will just freeze. Some times for a few seconds, sometimes a few minutes, sometimes they have been nice enough to wait 15 minutes until some timeout appears to go into effect and the application comes back. This problem has happened on both the Physical and Virtual Citrix Servers, so we have removed focus from the Virtual Infrastructure.

This has happened to many users, in many applications, and doesn’t seem to match any kind of reproducible process. It does seem to focus around clicking the OK button, and it seems to happen in the mornings. It also appears to happen, for the most part, only once per Citrix Server per day.

So, to me, this seems like some kind of cache issue, either in OneWorld, or with Windows/Citrix, but I cannot seem to pin it down. We have watched CPU usage, Network usage, Disk usage and latencies, etc. and nothing sticks its head out.

This has happened to users on weekends, so we are thinking we can rule out high system usage as part of the issue. But even then it was the first time the users used an application for the day.

Our next step is we are going to have all of the users try to use the PD7333 environment instead of the WPD7333 environment to see if we have been overloading our Logic server.

Any thoughts or help on this would be greatly appreciated.

Thanks!

 

[babs]

John

Unfortunately I don't have an answer for you. We also experience strange hanging/freezing issues intermittently on Citrix.

Sometimes when the user clicks OK, sometimes when right clicking in a field to obtain Alias, sometimes when tabbing one field to the next. When the user is hung Activconsole.exe cpu usage is at 99% for that user and other users experience poor performance. An additional button appears on the task bar at the bottom of the screen with the Citrix icon that says "remote...app name...". Sometimes the blue banner at the top of the screen with app name goes blank/white.

At my previous company we had similar issues which went away when we upgraded TR 8.94 to 8.97. We also had issues with SQL database blocking (known issue) that went away with application of SQL Server 2003 SP2 (I think).

We are planning to upgrade TR 8.94 to 8.98 and I am hoping the problem goes away! I will post after our TR upgrade and update on whether our problem was addressed by the TR.

Is this a new issue for you and if so what changed?

Sorry I can't offer more but I am interested in the cause/solution if you are lucky enough to figure it out!

babs

 

[Jotto]

Babs,

Thank you for the response, at least I know we are not alone. And, we also experience the 99% CPU active console from time to time. We think ours is related to times when a user exports a grid to Excel, and something goes wrong. But, the high CPU usage does not always happen when a user locks up.

One follow up question I have for you... Are your Citrix Servers virtual? That is the only "change" (we have been that way for a couple years now) that seems to have made this issue more noticeable.

I will continue to post any findings here.

Thanks,

-John

This is not a new issue for us, just one that's become a sore point with our Users because the rest of the system is running fairly well.

The closes we have come to

 

[babs]

Hi John

Yes virtual, however, the previous life scenario was physical not virtual but had the same type of issues.

We never have the problem when exporting to Excel though!

And yes all is fine except this lovely intermittent problem = very annoying to all.

babs

 

[Jotto]

Another question on this topic:

There are times when the users freezes.. and they sit there for exactly 15 minutes, and then it "un-freezes" and appears to the user to continue on like nothing happened.

Sometimes I will see a Timeout error message in their logs, sometimes I don't.

I cannot find any settings for a 15 minute / 900 second timeout. Is anyone out there aware of such a beast?

Thanks,

-John

EDIT / UPDATE: I believe we have found that this is a 15-minute OLEDB / ODBC for SQL Server timeout.

 

[Jotto]

Good morning,

I just wanted to have status update to this post, as we actually did come to a resolution. Our issues were two-fold:

1) We had a custom program that had some BSFNs OCM mapped to the enterprise server. Every so often, there was a timing conflict where the application would run functions both locally and up on the server asynchronously, and the server would not complete the process before the local one needed it to. The local process moved on to it's next logic, but could not continue because is did not have the information it needed from the server process. We resolved this by re-mapping all associated functions to run locally.

2) We found that doing "large" grid exports to Excel (1000+ lines) would kill our single-CPU virtual servers. On Multi-CPU physical servers, the process would take 10-20 seconds, whereas on the virtual it was taking many minutes, and use most, if not all, of the resources on that server -- which impacted all of its other users. Since this process cannot be changed, we are looking at other ways to get the data to the users, such as CSV output and BI reports.

Hopefully, this may give some ideas of where to look if anyone else is running into issues like this.