Bean lookup from remote client hangs then gives NoSuchElementException or NamingException saying unable to acquire SerialContextProvider

[Blavo]

Lookup of a remote bean from a remote client appears to hang but after 30-90 minutes finally excepts with a NoSuchElementException or a NamingException.

Environment Details

Glassfish 7.0.12

java version "17.0.7" 2023-04-18 LTS
Java(TM) SE Runtime Environment (build 17.0.7+8-LTS-224)
Java HotSpot(TM) 64-Bit Server VM (build 17.0.7+8-LTS-224, mixed mode, sharing)

MacOS 12.4 Monterey
No database

'

Problem Description

A remote bean lookup (initial context lookup()) that succeeds when run on the same machine as the glassfish instance hosting the bean fails when run on a remote machine. The lookup() call on the local machine returns a bean stub. The lookup() call on the remote machine appears to hang but after 30-90 minutes finally excepts with either a NoSuchElementException or a NamingException that says unable to acquire a SerialContextProvider. The actual cause of the former is attempting to remove the last element of a list that is empty. Since the call at the top of the stack trace is getProvider() it is a fair assumption that the empty list should contain a provider, presumable a SerialContextProvider provider. The other exception presumably reflects a fix to the code to not try to remove the last element of an empty list but instead diagnose that the list is empty ie there aren't any providers.

Presumably the Properties that the initial context instantiation can take as a parameter are intended to specify the source of SerialContextProvider providers. The local call seems to work without this. The remote call seems to need something specified. The properties themselves are well-enough documented but not the values they should take, so there is a bit of guesswork involved in setting them.

There are two problems: that the lookup() from a remote machine fails when it should work; and that it takes 30 minutes or more to fail, giving the false appearance of a hang. A developer with normal patience will cancel the lookup() call long before the exception occurs and will get no clue as to where to start debugging.

Note: the client needs some library JARs to provide the InitialContext class; there are two options: we have opted to put the glassfish embedded JAR from the 7.0.12 install into the client class path. The other option is to use the gf-client JAR. This behaves differently and might be worth the subject of a second issue. We choose the former option because the gf-client JAR isn't portable and you get strange results if you use a copy of it rather than the instance in the glassfish install. We can't use gf-client because in general there will be no glassfish install on the user's client machine.

The problem is reliably reproducible.

I suspect the resolution is to specify a source of SerialContextProvider s when calling from a remote machine, presumably via the initial context properties. Its just not clear how to do that. But in any case it can't take 30 - 90 minutes to fail.

Stack trace

Stack trace.txt

Source

CommentServiceLocal.txt
CommentServiceRemote.txt

CommentBeanLocal.txt
CommentBeanRemote.txt
Initialise.txt

SimpleGF7BeanLookup.txt

Steps to reproduce

• compile CommentServiceLocal.txt and CommentServiceRemote.txt in the presence of jakarta.ejb-api-4.0.1.jar into a single JAR; these are the interfaces
• compile CommentBeanLocal, CommentBeanRemote, and Initialise together, in the presence of the above JAR, into a Web app war file
• compile SimpleGF7BeanLookup in the presence of the interfaces JAR and glassfish-embedded-all-7.0.12.jar
• deploy the war to a glassfish server of your choice
• note the insecure iiop listener port of the glassfish server
• run SimpleGF7BeanLookup with two command line parameters: 127.0.0.1 and the iiop port; add any --add s that require adding; eventually the program will succeed
• take the SimpleGF7BeanLookup distribution directory to another machine that can see the machine hosting the glassfish server; make any necessary firewall adjustments
• run SimpleGF7BeanLookup with two command line parameters: the IP address of the glassfish host and the same iiop port; add any --add s that require adding; eventually the program will run and appear to hang; leave it running until it excepts

Impact of Issue

I am just trying to access a bean on a remote server - an entirely normal Enterprise application setup. This bug prevents all remote accesses to server EJBs.

[Blavo]

I think I have found the reason for the hang. Despite setting up the client to do a lookup against a remote server the lookup is actually being done against a local server. If the local server is up the call works and the beans are found. The log of the local server shows activity consistent with the client have gotten its beans, rather than those of the remote server. If the local server is not up then the lookup call hangs as described. This is consistent with the client trying to get the beans from the local server rather than the remote server.

So why is the lookup being executed against the remote server ? The documentation on just how to set up the initial context for a remote lookup is a bit light, so I can't be sure that how I have set it up is correct. However what I have done follows examples on the web that claim to work so it is probably correct,

So the problem becomes not 'why does the lookup hang' but 'why is the lookup directed to the local server' ?