NullPointerException when using CompletionStage fetched by generated proxy

[coldraydk]

New Play application created through sbt running with the following versions:

• play-soap v. 1.1.3
• Play v. 2.6.5
• Scala v. 2.11.11

My Java-classes are generated from the following internal service:
WsdlKeys.wsdlUrls in Compile += url("http://172.x.x.x/SOAP/PortalService.svc?wsdl")

Below is a simple index method in my controller. I've deliberately avoided the return type CompletionStage<Result> on the method for simplicity's sake:

     public Result index() {
        ObjectFactory factory = new ObjectFactory();

        PortalUserDTO portalUserDTO = new PortalUserDTO();
	portalUserDTO.setUsername(factory.createPortalUserDTOUsername("test"));
        portalUserDTO.setFullName(factory.createPortalUserDTOFullName("Test User"));

        ICitizenEndpoint client = portalService.getPortalServiceCitizenEndpoint();
        
        CompletionStage<PortalCitizenDTO> citizen = client.fetchCitizen(10, portalUserDTO);

        // Thread.sleep(500);
        
        citizen.thenApply(x -> x);

        return ok("Everything is fine.");
    }

The NullPointerException appears to be sporadic (roughly 20% of the time), and can be avoided by sleeping the running thread. Below is a stack trace of when the exception is thrown:

Exception in thread "default-workqueue-1" java.lang.NullPointerException
	at org.apache.cxf.endpoint.ClientImpl$1.onMessage(ClientImpl.java:505)
	at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream$1.run(HTTPConduit.java:1185)
	at org.apache.cxf.workqueue.AutomaticWorkQueueImpl$3.run(AutomaticWorkQueueImpl.java:428)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.cxf.workqueue.AutomaticWorkQueueImpl$AWQThreadFactory$1.run(AutomaticWorkQueueImpl.java:353)
	at java.lang.Thread.run(Thread.java:748)

Wireshark shows that the endpoint does provide it with data.

I'm unsure if this behaviour was introduced when you moved from promises to CompletionStages, but I imagine that it is likely. This does prevent me from using the plugin.

[marcospereira]

Hi @coldraydk, could you provide a small project that reproduces the problem?

Thanks!

[gmethvin]

You're letting client go out of scope at the end of the method while your CompletionStage is still running, so I suspect the client is getting garbage collected.

I would return a CompletionStage<Result> from the method and write the last part as:

citizen.thenApply(x -> ok("Everything is fine"))

[marcospereira]

@coldraydk can you confirm that following @gmethvin suggestion fixed the problem?

[coldraydk]

Hi guys,

Thanks for the feedback. I'm on a tight schedule until Thursday, but I will come back to you at the end of next week.

[ihostage]

@coldraydk Do you have new information?
@marcospereira Maybe close this issue as outdated?

[lapidus79]

I encounter this in my Play tests with plays mock server.
Randomly fails, I would say around 20% of the time as well with same Exception in thread "default-workqueue-1" java.lang.NullPointerException

    @Test
    public void dummySoapCall() throws Exception {

        this.server = Server.forRouter(MOCK_PORT, (components) -> RoutingDsl
                .fromComponents(components)
                .POST("/")
                .routingTo((request) -> {

                    Document input = request.body().asXml();
                    assertThat(getElemVal(input, "applicationId")).isEqualTo("abc");

                    return ok().sendResource("dummySoapResponse.xml");
                })
                .build());
        
        dummyService.call().toCompletableFuture().get(5, TimeUnit.SECONDS);
    }

Exception in thread "default-workqueue-1" java.lang.NullPointerException
	at org.apache.cxf.endpoint.ClientImpl$2.onMessage(ClientImpl.java:528)
	at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream$1.run(HTTPConduit.java:1203)
	at org.apache.cxf.workqueue.AutomaticWorkQueueImpl$3.run(AutomaticWorkQueueImpl.java:421)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.apache.cxf.workqueue.AutomaticWorkQueueImpl$AWQThreadFactory$1.run(AutomaticWorkQueueImpl.java:346)
	at java.base/java.lang.Thread.run(Thread.java:834)

[lapidus79]

After some further investigation this seems to be more common if server is located on same host as client and also seems to occur more frequently on slower machines.

It seems to occur when the server has responded with data and now CXF is about to parse/process this response data. Thus a successful call has been made, we've gotten a response and then when the response data is processed, it dies silently.

It happens somewhere inside Apache CXF. And since a PlaySoap waits for a future to complete this NPE results in the future never completing the our PlaySoap call will eventually result in a TimeoutException with no further info.

When narrowing down using breakpoints at HttpConduit.handleResponseOnWorkqueue and ClientImpl.destroy what seems to happen is that inside handleResponseOnWorkqueue we actually get a "Message received on a Client that has been closed or destroyed." exception. And this is becasue ClientImpl.destroy() is called before handleResponseOnWorkqueue and in the destroy method all resources are cleared.

This seems to be related to https://issues.apache.org/jira/browse/CXF-6940 and this commit https://fisheye.apache.org/changelog/cxf?cs=e0ef66391d2e7e62e63e96214eca95e292271892. More specifically the ClientImpl.java lines

	221         bus = null;
 	222	        conduitSelector = null;
 	223	        outFaultObserver = null;
 	224	        outboundChainCache = null;
 	225	        inboundChainCache = null;
 	226	
 	227	        currentRequestContext = null;
 	228	        requestContext.clear();
 	229	        requestContext = null;
 	230	        responseContext.clear();
 	231	        responseContext = null;
 	232	        executor = null;  

Compiling a new version of CXF with the above lines removed will result in this issue being resolved. So this issue might have been going on for a while. Maybe because of newer JDK / different GC it has become more topical since 2017?

Other CXF ticket which seems to be related to this as well http://cxf.547215.n5.nabble.com/jira-Commented-CXF-7664-Upgrading-to-JDK-1-8-seems-to-cause-concurrency-issues-intermittently-Messagd-td5788109.html

[ennru]

This is most likely fixed with the upgrade of CXF to 3.4.0 in #202 which is now released in Play SOAP 1.2.0 https://github.com/playframework/play-soap/releases/tag/1.2.0

[ennru]

As it turns out, the underlying problem is not fixed in CXF 3.4.0.
A workaround seems to be to make sure the JVM holds on to the client instance and does not garbage collect it.

[lapidus79]

@ennru do

As it turns out, the underlying problem is not fixed in CXF 3.4.0.
A workaround seems to be to make sure the JVM holds on to the client instance and does not garbage collect it.

Do you have any pointer or ideas how this could be achieved?

[ennru]

IIUC, the problem occurs when the client gets garbage collected before the citizen completion stage is completed.

Making the client an instance field, or using it after the stage is completed (eg. log.debug(client.toString());) should help.

  private ICitizenEndpoint client = portalService.getPortalServiceCitizenEndpoint();
   
  void call() {      
        CompletionStage<PortalCitizenDTO> citizen = client.fetchCitizen(10, portalUserDTO);
        
        citizen.thenApply(x -> x);
  }

[picoderen]

Has there been any progress regarding the solution of this problem ?

Scala-PlayFw-ApacheCxf / Exception in thread “default-workqueue-1” java.lang.NullPointerException
https://stackoverflow.com/questions/67278814/scala-playfw-apachecxf-exception-in-thread-default-workqueue-1-java-lang-nul

I would like to ask for opinions from those who are interested and have an opinion on the subject. Thanks.