FDASPEC in use by other user on Citrix

Hi,

starting last week I am getting this error that the fdaspec file is in use by another user on some of our Citrix servers. When I take all users off that server it fixes the problem.

The strange part is that the error always accurs around 3:30 pm, never in the morning or and never at night.

We installed a new full package on last Friday. That day we did not see this error and thought it was resolved until it hit again today, at 3:30pm . The problem also appears on only on 3 or 4 of our 15 Citrix servers at a time.


TamMultiUserOn is set to 1 in the JDE.INI files.

We run on B7334, Oracle 10, HP-UX servers.

Does anybody have any suggestions ?

Thanks, Gerd

Boa tarde Gerd!

Como é você hoje? The last time I started to see a cluster of FDASPEC locks, it turned out to be a user that was breaking our policies. The user was logging on, from multiple computers, using the same lan and JDE id. The user was then running the same report from multiple sessions. In this case, it was a user in a retail store running invoicing. What happened is that when the user submitted the report to the batch server while another report with the same user name was running, JDE got confused and screwed up both sets of reports and corrupted the specs. Take a look at 3:30 and see if you can find any users in multiple times. Our users claimed they weren't doing that, but when I caught them red handed, they confessed.

- Gregg

Thanks Gregg, good point.

I checked with our Citrix admin. According to him it is not possible for a user to connect more then once. But I will keep an eye on it tomorrow, you never know. Good thing is that I know the time it happens.

Cheers, Gerd

[ QUOTE ]
I checked with our Citrix admin. According to him it is not possible for a user to connect more then once.

[/ QUOTE ]

I thought that too, until I had a user with access to three computers log in simultaneously. Try it, it's a loophole. Even if the user logs on with different lan IDs, there's nothing to then stop them from having multiple JDE sessions. Give it a test meu amigo.

- Gregg

[ QUOTE ]
Try it, it's a loophole. Even if the user logs on with different lan IDs, there's nothing to then stop them from having multiple JDE sessions.

[/ QUOTE ]

Sorry, but I'm going to have to call this one. There is NO technical issue having different LAN users running the SAME JDE session over and over. Its obviously not a good idea, because you don't know who does what in the system - but as long as the LAN users are different on each citrix box, then theres no TECHNICAL issue. (I'm stressing the word TECHNICAL - ie, it won't cause grief to the spec files)

If you have multiple citrix boxes - having the same LAN user on different boxes isn't an issue (technically) either.

The only issue is where you have the SAME Network ID logging into the SAME citrix server. However, if you have the WTSLogs all correctly configured in the INI it is STILL session based, so in theory it shouldn't be too much of an issue there either.

No, your issue is more than likely a JITI of a specific application. You see, having multiple users READ the specs (TAM files) isn't an issue - its having more than 1 user WRITE to the TAM file thats an issue - and thats where you get the FDASPEC lock. More than likely, at that time during the day, a specific application is trying to be run by two sessions - and perhaps the user is getting frustrated its taking too long (because its JITI'ing the specs) and therefore they either disconnect OR they open up a second session - and the FDASPEC file locks up between the two JITI's.

What I would suggest is that you look at the last time you ran a full package. More than likely, you need to perform a full package build and clean up your spec files - OR uninstall and reinstall OneWorld on the citrix server. Something is occurring on a specific citrix server, and more than likely just rebuilding it will probably fix the issue (unless, of course, its a hardware failure on the drive !)

Good luck...

Good points. Thanks so far.

We completely re-installed 2 of the servers and booted them. Now the problem happended again, first time before noon, and it happened on one of the two servers that were re-installed.

Now Jon, what would cause a JITI? We have installed a new full package (build last week). We strictly control anything that is advanced to PD pathcode. We build the DD and glbltb files. What could it be? What can I look for? Mind you, we have about 1000 users connected from different countries. Kind of hard o watch everybody closely.

Thanks and keep the good ideas coming.
Gerd

I would say that those 2 servers are specifically causing you issues for some reason. You might want to take them out of the farm temporarily to see what might happen.

I've seen issues where a problem with the OS will cause a spec lockup - you might want to consider rebuilding those two servers. I don't specifically think you're actually seeing a "JITI" per se - because it seems to me that you're seeing this lock occur on a specific server and not on other servers - so its also very unlikely that its due to a specific user doing something (if your users are correctly load-balanced, and not connecting directly to a terminal session).

Couple of things to try..

1. Delete all temporary files on the terminal server causing issues (ie, everything in \TEMP and \WINDOWS\TEMP)

2. Make sure that the OS has been correctly patched and is running at the same patch level that the rest of the farm is running at, ie, run Windows Update. Sometimes being on the wrong patch level causes issues.

3. Check all your database connectivity settings - ie, ODBC for SQL/AS400 or your oracle client. Make sure that everything is set to the right level.

4. Perform a disk repair - or defragment your hard drives (specifically your system disk). Also make sure that your paging file is set up correctly and is on the correct drive.

Make sure that there isn't anything such as WinAT or some sort of weird scheduled process such as a backup that is kicking off at 3.30 - you might also want to check your antivirus settings as well...

If all of the above fail to fix, and the issue is still specific to the server, then you might have to resort to a full refresh of that server (reinstall the OS, Citrix etc etc).

Cumprimentos Gerd,

In addition to Jon's comments, keep an eye out for a user or two double dipping on your system. We had some retail store clerks do that to us, because performance was slow. The performance issue was WAN related, not JDE related as we learned. When they ran the same report at the same time as the same JDE user, they slowed the system down even more by locking up the specs.

Ciao, Gregg

Hi Gerd,

Addition to Jon's and Greg's valuable answers, just a silly answer from me:

Check the Object Management Log table, looking for Get/Check-Out action against your PROD Path Code - and maybe Add/Delete, or other ways all action which is not Transfer.

A question to Jon and Greg:
Wheter Version Creation (mainly UBEVER) and Check-In/Check-Out can also produce spec problems?

Regards,

Zoltán

Zoltán,

I would hope that there are not a lot of versions being checked in and out of Gerds production environment. That would be messy. Sorry, I don't know any Hungarian, so I couldn't work that into my post.

- Gregg

Hi Greg,
OFF Reply - excuse me.
[ QUOTE ]
Sorry, I don't know any Hungarian, so I couldn't work that into my post.

[/ QUOTE ]

You don't do so many mistake in your Hungarian, than me in my English.

Regards,

Zoltán

Thanks a lot for all the good suggestions. We will check all of them.

One more thing: the weekend prior to the problem occur the first time the Citrix Servers received an Office update to Office 2007. Could that have anything to do with it?

Thanks again, Gerd

[ QUOTE ]
the weekend prior to the problem occur the first time the Citrix Servers received an Office update to Office 2007. Could that have anything to do with it?

[/ QUOTE ]

As I mentioned before, you need to check your patch levels. Yes, this could definitely be an issue - the fact that you have perhaps different security patches on your windows servers - I've seen this occur before. Check to make sure that all patches got installed correctly - if necessary, patch everything and check again.

Yes Jon, the office patch is our prime target now. There is strong resistance from the Citrix/Windows crew, but we made them uninstall it from one of the servers which we are observing now.

We also found a plausible explanation about the 3:30 timig. More about that later.

I´ll keep our results posted.

Thanks to all, Gerd

Ok, we fixed it. I promised to post the results here.

Actually the root cause was worse then suspected. It was not Office 2007 or a patch; the windows team had changed the users´ profiles without our knowledge.
The consequence was that users did not have a Citrix-connection timeout any more and connections stayed alive indefinitely. That somehow caused that some long-time inactive sessions for some reason blocked the fdaspec file. That’s the best explanation I can come up with. Maybe someone can enlighten me further on this.
It is also possible that the Citrix servers were overloaded with too many connections, more then usually expected.
The odd timing of the problem can be explained by shift-changes between 3-4 pm or maybe lunch-hours for our users at -3 hours time difference. And that we did not have any problem on Friday was likely due to the fact that on Fridays people actually power down their machines.

Thanks for all your good help, Gerd