We are observing a very strange behaviour in WSO2 EI where the WSO2 server stop sending HTTP requests although everything within WSO2 is continues to work correctly.

We observe logs like below.

WARN {org.apache.synapse.transport.passthru.TargetHandler} - Connection time out after while in state : REQUEST_DONE Socket Timeout : 600000 correlation_id : null Remote Address : localhost/127.0.0.1:8243
TID: [-1234] [2022-03-09 09:14:42,439]  WARN {org.apache.synapse.FaultHandler} - ERROR_CODE : 101507
TID: [-1234] [2022-03-09 09:14:42,439]  WARN {org.apache.synapse.FaultHandler} - ERROR_MESSAGE : Error in Sender
TID: [-1234] [2022-03-09 09:14:42,439]  WARN {org.apache.synapse.FaultHandler} - ERROR_DETAIL : Error in Sender
TID: [-1234] [2022-03-09 09:14:42,439]  WARN {org.apache.synapse.FaultHandler} - ERROR_EXCEPTION : null
TID: [-1234] [2022-03-09 09:14:42,439]  WARN {org.apache.synapse.FaultHandler} - FaultHandler : Endpoint [wso2-delay-url]
TID: [-1234] [2022-03-09 09:14:42,440]  WARN {org.apache.synapse.FaultHandler} - FaultHandler executing impl: org.apache.synapse.mediators.MediatorFaultHandler
TID: [-1234] [2022-03-09 09:14:42,440]  WARN {org.apache.synapse.mediators.MediatorFaultHandler} - Executing fault handler mediator : org.apache.synapse.mediators.base.SequenceMediator
TID: [-1] [2022-03-09 09:14:42,443]  WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-250 Socket Timeout : 600000 Remote Address : /127.0.0.1:60144

So connection timeout starts happening although application it is trying to connect is running fine.

Some other observation in the logs are as below :

TID: [-1] [2022-03-09 11:55:22,668]  WARN {org.apache.axiom.util.blob.OverflowBlob} - Cleaning up unreleased temporary file /opt/<organisationName>/applications/wso2ei-6.6.0/wso2/tmp/tmp_5701187744473942267.dat
TID: [-1] [2022-03-09 11:55:22,668]  WARN {org.apache.axiom.util.blob.OverflowBlob} - Cleaning up unreleased temporary file /opt/<organisationName>/applications/wso2ei-6.6.0/wso2/tmp/tmp_5286863810950226078.dat
TID: [-1] [2022-03-09 11:55:22,668]  WARN {org.apache.axiom.util.blob.OverflowBlob} - Cleaning up unreleased temporary file /opt/<organisationName>/applications/wso2ei-6.6.0/wso2/tmp/tmp_15656718439953223175.dat
TID: [-1] [2022-03-09 11:55:22,668]  WARN {org.apache.axiom.util.blob.OverflowBlob} - Cleaning up unreleased temporary file /opt/<organisationName>/applications/wso2ei-6.6.0/wso2/tmp/tmp_6161525619097223194.dat
TID: [-1] [2022-03-09 11:55:22,668]  WARN {org.apache.axiom.util.blob.OverflowBlob} - Cleaning up unreleased temporary file /opt/<organisationName>/applications/wso2ei-6.6.0/wso2/tmp/tmp_16101322891943797477.dat
TID: [-1] [2022-03-09 11:55:22,668]  WARN {org.apache.axiom.util.blob.OverflowBlob} - Cleaning up unreleased temporary file /opt/<organisationName>/applications/wso2ei-6.6.0/wso2/tmp/tmp_5181555544199697780.dat
TID: [-1] [2022-03-09 11:55:22,669]  WARN {org.apache.axiom.util.blob.OverflowBlob} - Cleaning up unreleased temporary file /opt/<organisationName>/applications/wso2ei-6.6.0/wso2/tmp/tmp_7115647804012046335.dat
TID: [-1] [2022-03-09 11:55:23,005]  WARN {org.apache.synapse.FaultHandler} - FaultHandler executing impl: org.apache.synapse.endpoints.AddressEndpoint
TID: [-1] [2022-03-09 11:55:23,006]  WARN {org.apache.synapse.FaultHandler} - FaultHandler executing impl: org.apache.synapse.mediators.MediatorFaultHandler
TID: [-1] [2022-03-09 11:55:23,006]  WARN {org.apache.synapse.mediators.MediatorFaultHandler} - Executing fault handler mediator : org.apache.synapse.mediators.base.SequenceMediator
TID: [-1] [2022-03-09 11:55:23,006]  WARN {org.apache.synapse.FaultHandler} - FaultHandler executing impl: org.apache.synapse.endpoints.AddressEndpoint
TID: [-1] [2022-03-09 11:55:23,007]  WARN {org.apache.synapse.FaultHandler} - FaultHandler executing impl: org.apache.synapse.mediators.MediatorFaultHandler
TID: [-1] [2022-03-09 11:55:23,007]  WARN {org.apache.synapse.mediators.MediatorFaultHandler} - Executing fault handler mediator : wso2-service-admin-change-service-state

TID: [-1] [2022-03-09 12:03:35,668]  WARN {org.apache.synapse.core.axis2.SynapseCallbackReceiver} - Synapse received a response for the request with message Id : urn:uuid:0b0cf1bb-b88a-4ce9-b5fe-c132a05dd29c and correlation_id : null But a callback is not registered (anymore) to process this response

After all there error, finally the I see no logs in http_access and Audit logs of WSO2 EI server but I can see that WSO2Carbon logs are still there for other processing within WSO2.

Everything starts working fine after server restart.

We have now observed this behavior twice in an interval of 25 days.

Does anybody have any idea on why this might happen?

Please let me know if more details are needed. Thanks!