No FAP slots available

We had an issue where most of our users were hung in JDE 9.0 at the same time. In the JDE log for some of the first users to call I noticed a message about “No FAP slots available”.
Does anyone know what this message means?
More log detail below.
We are on JDE 9.0 with Tools 8.98.4.2. Iseries with V6R1. Web servers are Websphere 7 on Windows 2008 R2.
Jeff
--------------
2875/5 WRK:Call Object Wed Sep 26 12:20:35.240672 netmsg.c427
No FAP slots available, unable to add data packet to message, msgId=1469540, msgHost=10.1.0.140, msgPort=6120, sendingHost=10.1.0.96, msgType=901, msgRange=5, reqNet=62643, resNet=807, reqKrnl=62643, resKrnl=2875, msgFlags=2.

2875/5 WRK:Call Object Wed Sep 26 12:20:35.241024 netmsg.c445
Dumping the metadata in the FAP:

2875/5 WRK:Call Object Wed Sep 26 12:20:35.241072 netmsg.c452
Entry-0, Bytes-1468, PID-677, Type-360

2875/5 WRK:Call Object Wed Sep 26 12:20:35.241112 netmsg.c452
Entry-1, Bytes-1468, PID-807, Type-360
…
2875/5 WRK:Call Object Wed Sep 26 12:20:35.285568 netmsg.c452
Entry-999, Bytes-1093, PID-807, Type-901

2875/5 WRK:Call Object Wed Sep 26 12:23:15.301696 netmsg.c427
No FAP slots available, unable to add data packet to message, msgId=1469640, msgHost=10.1.0.140, msgPort=6120, sendingHost=10.1.0.96, msgType=901, msgRange=5, reqNet=58964, resNet=849, reqKrnl=58964, resKrnl=2875, msgFlags=2.

2875/5 WRK:Call Object Wed Sep 26 12:30:58.342304 netmsg.c427
No FAP slots available, unable to add data packet to message, msgId=1470044, msgHost=10.1.0.140, msgPort=6120, sendingHost=10.1.0.96, msgType=901, msgRange=5, reqNet=63481, resNet=803, reqKrnl=63481, resKrnl=2875, msgFlags=2.
--------------------------------

Bit of a while ago but we also hit this today, E9.0, Tools 9.1 on ISERIES v7 R2. Had to restart services. Did you ever get a root cause or fix?

This is usually an indicator that your IPC msg queues are full and the metadata kernel is not able to perform any operations ( which is needed for all the other kernel processes to function properly) .

Did you have any kernels zombie before the incident happened ?

Has your user load increased recently ? How many concurrent user's do you have.

I had this issue occur for one of my clients , also E 9.0 on AS400 (V6R1). We ultimately had to review and adjust our INI settings based on Oracle support's recommendation and it seems to have resolved the issue. I will post the details shortly

kororia1 said:

Bit of a while ago but we also hit this today, E9.0, Tools 9.1 on ISERIES v7 R2. Had to restart services. Did you ever get a root cause or fix?

Click to expand...

Kororia1,
We did get a fix in an indirect way. Support supplied a POC for the JDEKRNL in the E900SYS library. We had the "No FAP" issue 8 times over 20 months before the POC and have not had it in the 18 months since applying the POC. In addition to the "No FAP" issue, we also had the Metadata kernel go zombie frequently, although the two never happened at the same time. Support thought the two were related, and they must have been, since applying the POC neither has happened again. In our support ticket, they referenced bug 11837112 for the metadata zombie issue.
Note that we are on E1 9.0 with tools 8.98.4.2 on V6R1.
Jeff

See my comments below >>

ice_cube210 said:

This is usually an indicator that your IPC msg queues are full and the metadata kernel is not able to perform any operations ( which is needed for all the other kernel processes to function properly) .

Did you have any kernels zombie before the incident happened ?

>>Not sure TBH as we have 000's when it did happen but we never analysed them for this

Has your user load increased recently ? How many concurrent user's do you have.

>>We've had no real increase in users and this is at a time when we have low users i.e peak users are 08:00 onwards with only a handful >>between 00:00 - 08:00 and yet we hit the issue approx 05:30. So dont think it can be attributed to load


I had this issue occur for one of my clients , also E 9.0 on AS400 (V6R1). We ultimately had to review and adjust our INI settings based on Oracle support's recommendation and it seems to have resolved the issue. I will post the details shortly

Click to expand...

>>Thanks ever so much awaiting your feedback

Hi Jeff,

did they give you a BUG Number for the FAP Issue? I checked that BUG and its offically related to the MD kernel issue with no reference to FAP but I will ask them the question anyhow.

Thanks for the reply

Hi Koroia,

Here are some of the INI values that we changed

maxNetProcesses=5 TO maxNetProcesses=7
maxNetConnections=120 TO maxNetConnections=600
maxNumSocketMsgQueue=200 TO maxNumSocketMsgQueue=400 (maxNumSocketMsgQueue should be at least twice maxIPCQueueMsgs)
maxIPCQueueMsgs=110 TO maxIPCQueueMsgs=200

We also had to review following AS400 system values. The below were the values we already had and did not have to change any of them

QJOBMSGQFL - *PRTWRAP
QJOBMSGQMX - 64 MB ( This is the highest value allowed)
QJOBMSGQSZ - 1024 KB (Job message queue initial size, allowed values 1- 16384)
QJOBMSGQTL - 2048 KB ( Job message queue maximum initial size , allowed values 1- 16384)

You will obviously have to review your INI settings based on the number of concurrent users you have ,number of JVMs and maxpool size of each JVM, no of concurrent UBEs you allow , integration connections coming through the JDE application layer etc.

Here are some docs on the support site for reference

- E1: KER: Troubleshooting Tips for Kernel IPC Issues ( Doc ID 1237423.1 );
- E1: KER: Explanation of JDE.INI Kernel, IPC and JDENET Parameters in the JDE.INI File ( Doc ID 656040.1 );
- E1: KER: JDE.INI Tuning & Recommended JDE.INI Settings ( Doc ID 654975.1 )


Did you say you had many zombie kernels at the time of the incident. I wasn't quite sure what you meant by .
"Not sure TBH as we have 000's when it did happen but we never analysed them for this"

You could also have a perfectly tuned system but have some bad code / process that is filling up the IPCMsgQueue making the kernels crash.

kororia1 said:

Hi Jeff,

did they give you a BUG Number for the FAP Issue? I checked that BUG and its offically related to the MD kernel issue with no reference to FAP but I will ask them the question anyhow.

Thanks for the reply

Click to expand...

Koroia,
They did not give me a separate bug number for the FAP issue. But when we applied the POC for the MD kernel issue, they FAP issue stopped happening.
Jeff