Porttest and login fail on hot standby server

[Crazy_About_JDE]

Hello, list-- I have configured a hot-standby server using JDE Consulting Operations document "Mirroring a OneWorld iSeries Enterprise Server" (written by Michael Guerra and Doug Keckler) and have done my first couple tests of it.

After performing the takeover, NETWORK and JDENET_K jobs start, but not SENTINEL. Porttest fails. Attempted login from client results in SEC00000007 Unable to locate security server. The server INI and the porttest logs are attached.

Thank you in advance for any help you can provide.


Here are some additional details:

All JDE libraries, profiles, and IFS folders have been synced to backup server using Echo2 by iTera [www.iterainc.com]. I have also verified that the EDRSQL service is started. I can log in as ONEWORLD and JDE and query tables.

<ul type="square">Takeover Process
[*]End JDE services on the primary box
[*]disable ethernet adapter on primary (NETSTAT, 1)
[*]repoint network's DNS to backup server
[*]change host name on backup server to match the primary (CFGTCP, 12)
[*]add primary's name and IP address to host table (CFGTCP, 10)
[*]successfully ping the primary server's name, reports back backup server's IP
[*]delete local relational database entry (WRKRDBDIRE)
[*]add a new local relational database entry that matches the primary server
[*]start JDE services
[/list]

<ul type="square">From the porttest checklist (OneWorld Xe Installation for AS/400 Systems, p279)
[*]Am logged in as ONEWORLD. (Have also tried while logged in as QSECOFR.)
[*]"Are all the environment variables set?" WHAT environment variables is it talking about?
[*]C compiler is installed.
[*]Can query the database on the enterprise server.
[*]A PostScript or PCL printer is connected to this machine. (Currently using QGPL/LASER04 as the default.)
[*]"Are the printer drivers for this printer installed?" I don't know.
[*]"Is this printer configured as the default printer?" I don't know.
[*]Server INI is located in B7334SYS library.
[*]Server INI authority is *PUBLIC (*CHANGE) and QSECOFR (*ALL).
[*][Network Queue Settings] DefaultPrinterOUTQ = QGPL/LASER04
[*][DB System Settings] Default Environment = PD7334
[*][DB System Settings] Default Pathcode = PD7334
[*][DB System Settings] Server (Database Server Name) = S106461B
[*][JDENET] serviceNameListen = 6010
[*][JDENET] serviceNameConnect = 6010
[*][INSTALL] B733=
[*]Workstations and server agree on the server's IP address (configured in DNS, ping works fine)
[*]"Have your servers map tables (F98611 and F096101) been edited properly?" No clue as to what 'properly' means.
[*]"Run the Verify OCM application and verify: (1) Do only host databases exist? (2) There are no entries for batch applications." I found verifyocm.exe and ran it in production, but it failed there. Anything else I can do here?
[*]"Are OneWorld tables accessible to the host?" I don't know what this means.
[*]I can query the PRODDTA/F0902 table.
[*]Have tried running as myself, JDE, and ONEWORLD.
[*]JDEnet seems to start and stop properly.
[/list]

 

[jjarnot]

Hmmmm... What about the TCP HOSTS file entries on the enterprise server? (CFGTCP/10)
When services start, and the INI is read, the HOT systems IP will need to be resolved by name via the security processes. Also check the domain name & DNS settings under options 10 & 12 of CFGTCP.

If you PING the machine by name from the green-screen, does the iSeries machine reply back?
Are all the serivces starting? SQLERD etc? STRHOSTSVR *ALL and STRTCPSVR? STRTCP? Lets make sure all your required 400 processes are running.
Turn on DEBUG and attach as well. On the 400, you could check the logs. Try a WRKSPF oneworld.
JJ

 

[AS400Guru]

You might want to manually sync the runtime specs i.e. BXXXSYS and also /Pathcode. Also make sure that the runtime library is updated i.e. PDXXX.
When you ping your Security server, what does it return?
From the iseries ping the security server and ping from the command line on a windows machine, does it return the same ip address?
When you ping your back-up server what does it return?

I will check on my end and also verify your ini with mine.

 

[Crazy_About_JDE]

Thank you for writing, Martin.
<ul type="square">[*]I am continuously mirroring the B7334SYS library from the primary to the target. Both are perfectly in sync (except SQLPKGs are filtered out). No known errors.

[*]The IFS folder /PD7334 has been mirrored and verified by me by hand, so I know they are exactly the same.

[*]I am also continuously mirroring the PD7334 library, except for SQLPKGs. No known errors.

[*]In our environment, the iSeries server is both the security server and the enterprise server.

[*]After I've removed the primary server and swapped in the backup server, all pings to the primary server correctly report the backup server's IP address. (I ping from the iSeries server itself and from the Windows client.)
[/list]

 

[AS400Guru]

Since we are sure that replication is good, there is one more thing that we can try without actually undergoing a roll swap.
Parallel Test-
Not sure if you are familiar with this process. High level overview-
Change the host file on the client locally to point the primary server to the backup server.
Change the local relational database directory to match the primary server
Change the tcp ip so that when you ping from the backup server to the primary server you are successful, note that this only needs to work from the iseries.
Test and check if you are able to bring up services on the back-up server and also check whether it is trying to connect to the primary server.

NOTE:
It is advisable to perform this exercise when you have very few users on the system.

 

[Crazy_About_JDE]

Thank you for writing, JJ. My original post addressed all your TCP concerns and included logs.

However, I just went through and compared all the active jobs between the two servers. Here is a list of jobs *not* running on the backup server:

<ul type="square"> QSYSWRK
[*]QDIRSRV QDIRSRV
[*]QPRFSYNCH QSYS
[*]QTTFT00019 QTFTP
[*]QTTFT00021 QTFTP
[*]QTTFT00027 QTFTP
[*]QVNAVARY QSYS
[/list]
<ul type="square"> QUSRWRK
[*]QZDASOINIT QUSER
[/list]
Also note, from my original email, that the SENTINEL service fails to start on the backup server when I do STRNET at the time of "takeover."

Does any of that shed some light?

 

[Crazy_About_JDE]

Hello, list-- I'm happy to report that Oracle Global Support helped me solve this problem. The answer was tucked away in my session's job log (after running porttest, DSPJOBLOG then press F10 to display details):

Cannot resolve to object QBFCPRCED. Type and Subtype X'0203' Authority X'0000'.

Sure enough, QBFCPRCED exists in the QUSRSYS library on the primary server, but not on the backup server. Copied the object over and badda-bing badda-boom, porttest no longer fails.

Feeling brave, I used the backup server's name in B7334SYS/INI then started JDE services. SENTINEL started up successfully this time.

That paved the way for a full test Saturday night. JDE started fine, login was fine, successfully placed an order. I didn't even have to change the local relational database name. Yay!


Does anybody know where QBFCPRCED comes from? I see it referenced in B7334SYS/INI's [AS400] section:

CRTDBPGM4=BNDSRVPGM(JDEKRNL JDELIB JDEIPC QBFCPRCED) ACTGRP(%s) OPTION(*DUPPROC

and a quick search of IBM's iSeries support pages revealed this blurb in the J.D. Edwards OneWorld Implementation for AS/400 redbook on p551: "There are PTFs that must be applied before installing and configuring OneWorld B73.3. The current PTFs can be obtained by following the Informational APAR for your version of OS/400. Please refer to the site at www.as400.ibm.com/service/bms/jde-support.htm. A quick test to see if you have the PTFs installed is to run the WRKOBJ QUSRSYS/QBFCPRCED command. If the object is not found, you need PTFs from the Informational APAR."

I am familiar with the Informational APAR, but am pretty sure I'm current on everything.

Any ideas?


--------------------
OneWorld B7334 Upd 1 SP23_J1
ES: iSeries V5R3
HA: iTera Echo2 (first rollover 2/10/07 -- almost there!)
DS: Win2003 Std R2 SP1, SQL2000 Std SP4
Citrix Metaframe XPe 1.0 FR3 on Windows 2000 SP4
Fat Clients: Windows 2000 SP4, Windows XP SP2

 

[AS400Guru]

Very Interesting Find. Congratz!!! on success.

 

[jjarnot]

Cool, I see you fixed it by an object copy. I wonder if there must be an associated PTF set to install these type of QBFC OBJs? Or is it an option somewhere an from OS install?

I found reference to a PTF for API wrappers for QBFC. SI13763 from the IBM APAR site for E1. (link below) BUT... It notes the QPFC wrapper ptf is not needed for 8.9 & beyond? I'm a OneWorld XE sp23 person, so maybe that makes sense?

Some sites reference the need for QBFC API wrappers for varuous things. If you did a search from one system to the next for QBFC* I wonder how many you have in total?

Here is my count from a SP23K1 box...
QBFCDBBK *PGM QUSRSYS CLE
QBFCDMPB *PGM QUSRSYS CLE
QBFCLISTEN *PGM QUSRSYS CLE
QBFCRECVR *PGM QUSRSYS CLE
QBFCSETP *PGM QUSRSYS CLE
QBFCSSL *PGM QUSRSYS CLE
QBFCPRCED *SRVPGM QUSRSYS CLE

Jeff.


at: http://www-912.ibm.com/n_dir/nas4apar.NSF/fd9ac656a54b8cf286256e770069b197/ec96e34aeec6293186256ead003c84bf?OpenDocument

 

[Crazy_About_JDE]

Thank you for writing, Jeff-- Indeed, I did not have any of the QBFC* objects you listed, so I mirrored them all over from the Echo2 menu. So obvious, now!

There were two last pieces to the puzzle:
- As Martin (AS400Guru) suggested, I turned off IFS synchronization and simply copied the specs in /PD7334 from one server's IFS to the other using copy-and-paste in Operations Navigator.

- Also discovered that my JDE job queues weren't starting, because there were no job queue entries in the QSYS/QBATCH job description. (STRSBS QSYS/QBATCH, WRKSBS, put 5 on the QBATCH line, menu option 6 to display job queue entries.) To fix, I simply mirrored this object from the primary to the target via the Echo2 menu, and jobs have processed beautifully ever since.


Here's a BIG thank you to both you and Martin for your input fixing these problems. We did a second rollover + rollback this weekend, and JDE worked as it should (as did Echo2)!

I'd be CRAZY without JDELIST!