Resolved

I can now confirm that the issue was caused by us adding the Dynatrace OneAgent extension to our function app in Azure. How exactly it breaks it is unclear but the process seems to do something that results in illegal awaits in the durable task framework.

Original post below

---

My Durable Functions app has been running fine in Azure for while but has suddenly started to misbehave. Based on the history that I've looked at in the task hub, I've noticed that the instances are completing their entire execution for the orchestrator function, but no longer completing their instance status. The statuses stay as "Running" instead of "Completed".

The DurableTask package version I am using is 2.6.0 and my app is running on .NET Core 3.1.

Below is a simplified example piece of code for my orchestrator function. From this example, I have seen instances finish with Calculate, Publish or ErrorHandler functions completed but not completing or restarting with a ContinueAsNew.

[FunctionName("Orchestrator")]
public async Task RunOrchestrator(
    [OrchestrationTrigger] IDurableOrchestrationContext context,
    ILogger log)
{
    var data = context.GetInput<OrchestratorInputModel>();
    try
    {
        if (data.Something)
        {
            await context.CallActivityAsync("Calculate", data);
            context.ContinueAsNew(data);
            return;
        }
        await context.CallActivityAsync("Publish", data);
    }
    catch (SomeException exception)
    {
        await context.CallActivityAsync("ErrorHandler", exception);
    }
}

This is an example of the end history for an instance that completed correctly.

The next two examples are the end histories of instances that are finishing execution but are stuck in the running state.

It looks as though the application has gotten into some strange state where it doesn't add the ExecutionCompleted event in the instance history.

I have tried redeploying the app pointing to a new task hub. This worked for a while but the issue is back now.

Any actions/suggestions that might help would be much appreciated.