I have a unit test that attempts to verify a graceful grpc server shutdown by sending some grpc calls to the server, shutting down the server, and verifying the requests completed before the server shutdown.

In the test thread I spin up 5 threads that each execute one client rpc call.

In the service end I decrement a countdown latch (receivedLatch) to ensure the call has been received, and then sleep for a few seconds.

Back in the main test thread I assert that the receivedLatch is 0 by calling receivedLatch.await().

At this point my understanding is all 5 client calls are connected to the server and are processing (sleeping).

After that I call:

grpcServer.shutdown()
grpcServer.awaitTermination(25000, TimeUnit.MILLISECONDS)

The issue is a race condition, in that sometimes the test completes successfully, and other times the test errors out on the rpc call in the test thread with the exception:

io.grpc.StatusRuntimeException: UNAVAILABLE: Connection closed after GOAWAY. HTTP/2 error code: NO_ERROR, debug data: app_requested

Looking into this error I found that:

1. UNAVAILABLE... GOAWAY is sent to clients that are attempting to connect to the server that is currently shutting down.
2. "debug data: app_requested" comes from the close method NettyServerHandler.GracefulShutdown()

So the questions are:

1. How is the client not connected since I have asserted my receivedLatch is 0 and the countdown for that latch is in the service method for the RPC call. If it were CANCELLED it would make more sense.

2. Why is the NettyServerHandler getting called before my in-process RPCs have completed as my understanding is that's what awaitTermination should handle.**

Is the call to close in NettyServiceHandler happening directly upon the call to grpcServer.shutdown(). If so wouldn't that always be a race?

Thank you,