LogicAppWorkflowRuntime multiple WorkflowRunCompleted records for same RunId
Hi,
I'm running the following KQL query in log analytics:
LogicAppWorkflowRuntime
| where TimeGenerated > ago(24h)
| where OperationName == "WorkflowRunCompleted"
| order by RunId
I'm seeing duplicate records for workflows that are Failed.
👁 User's image
These are all the same RunId with the same StartTime but the TimeGenerated and EndTime are all different times from Today. Also, the StartTime is from 3 months ago but the TimeGenerated is today. There are dozens of these records. Why does this happen? Thanks for your answers.
1 answer
-
Rakesh Mishra 9,695 Reputation points • Microsoft External Staff • Moderator
Hi Davie,
Thank you for reaching out to Microsoft Q&A.
What you are seeing is not expected behavior for normal workflow runs. While Logic Apps (Standard) relies on the underlying Azure Durable Task framework—which uses event sourcing to replay state—a workflow that has already reached a terminal state (like Failed) should not be continuously replayed months later.
As per official Microsoft documentation:
"The Standard logic app and workflow is powered by the redesigned single-tenant Azure Logic Apps runtime. This runtime uses the Azure Functions extensibility model and is hosted as an extension on the Azure Functions runtime."
Because of this architecture, Logic Apps Standard relies heavily on Azure Storage Queues (Control Queues) to dispatch and track workflow execution steps. The behavior you are observing is caused by a stuck orchestration state or a poison message in your Logic App's storage account.
Here is what is happening under the hood:
- For this specific failed run from 3 months ago, the control queue message responsible for finalizing its state failed to be successfully checkpointed or deleted from the queue (possibly due to a transient storage timeout, locking issue, or memory spike at the time).
- Because the message was never deleted, it becomes visible again in the queue after its visibility timeout expires.
- The runtime picks up the message, replays the workflow to its failed state, emits the
WorkflowRunCompletedtelemetry to Log Analytics (which generates today'sTimeGeneratedandEndTime), but then repeatedly fails to finalize the storage transaction.
This creates an infinite loop where the same old
RunIdkeeps emitting new completion logs on a daily or hourly basis. To resolve this, you need to clear the stuck message from your storage account.Resolution Steps:
Step 1: Identify the underlying Storage Account
- Navigate to your Standard Logic App in the Azure Portal.
- Under Settings, select Environment variables (or Configuration depending on portal view).
- Look for the application setting named
AzureWebJobsStorage. Note the storage account name provided in this connection string.
Step 2: Inspect and clear the Control Queues (Option A)
- Navigate to the identified Azure Storage Account in the Azure Portal.
- Go to Storage browser > Queues.
- Look for queues prefixed with your app's hub name (e.g.,
logicapps-control-00,logicapps-control-01, etc.). - Check the Message count. If you see messages lingering in these queues that correspond to the timeframe of your ghost runs, these are the stuck messages causing your loops.
- To stop the loop, you can select the queue and use the Clear queue action. (Warning: Clearing the queue will drop pending execution steps for any currently active workflows in that specific partition. Only do this during a maintenance window or when there are no active running workflows).
Step 3: Purge Instance History (Option B - Targeted Fix) If you want a more targeted fix without clearing entire queues, you can purge the stuck instance data using the Storage Tables to forcibly sever the loop:
- In the Storage browser, go to Tables.
- Locate the instances and history tables (typically named
logicappsInstancesandlogicappsHistoryor similar hub prefixes). - Query the
logicappsInstancestable for thePartitionKeythat exactly matches your stuckRunId. - Delete the entities associated with that
RunId. This completely removes its historical state, forcing the runtime to drop the poison message upon its next attempted replay since it will no longer find the orchestration history to rehydrate.
Please follow above steps and let me know in comments if it works or any other issue.
Note: This response is drafted with the help of AI systems.
-
Deleted
This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
-
Davie Lau 6 Reputation points
Hi @Rakesh Mishra ,
Thanks for your reply. I looked in the storage account but do not see any queues or tables similar to the names you mentioned. My queues look like "flowa83145f3d8de31916124bed419520dtriggerdispatching" and my tables look like "flowa83145f3d8de31916124bed419520d20260313t000000zactions"
-
Rakesh Mishra 9,695 Reputation points • Microsoft External Staff • Moderator
Hi Davie,
You are looking in the exact right place.
I apologize for the confusion while
logicapps-control-xxqueues are used by default Azure Durable Functions, Logic Apps Standard uses a specialized storage provider that generates isolated tables and queues per workflow to optimize performance.The
flowa83145f3d8de31916124bed419520dprefix you are seeing is a hashed identifier (specifically a MurmurHash64 algorithm) of your Logic App and Workflow name. This is completely normal architecture and confirms you have found the correct backend storage for your specific workflow.Since you've located the correct storage structure, you can proceed with clearing the stuck state to resolve the infinite replay loop.
-
Davie Lau 6 Reputation points
Hi @Rakesh Mishra ,
I wasn't sure which queue to clear since there are so many of them with similar naming. I ended up deleting all the queues but I'm still seeing old completion logs being emitted every few minutes. Should I delete all the tables? Is this safe to do?
-
Davie Lau 6 Reputation points
Hi @Rakesh Mishra ,
Just following up on my last question. Were you able to look into this? Thank so much.
Sign in to comment
