Service Hooks stuck in "Queued" state indefinitely with no delivery attempts to target server

TI Wattwise 0 Reputation points

Hi,

I am experiencing a critical issue with Service Hooks (Web Hooks) in our Azure DevOps organization. Periodically, several "Pull Request Created" events are getting stuck in the "Queued" state for hours or even days, while others succeed normally.

Key observations:

  1. Intermittency: Some webhooks are delivered successfully (Succeeded), but many others remain "Queued" indefinitely.
  2. No Server Logs: We have checked the logs on our target server, and for the events stuck in "Queued", there are absolutely no records of any incoming connection attempts from Azure DevOps.
  3. Detailed Logging: We have detailed logging enabled, but the status remains "Queued" without any HTTP response code or error message.
  4. Target Server Status: The target server is healthy and responding correctly to manual POST requests and to the successful webhooks that do manage to go through.

It seems like the events are being held back within the Azure DevOps internal queue processing before even attempting an external connection.

Could you please investigate if there is any throttling, circuit breaker, or known service degradation affecting the Service Hooks processing for our region?

Organization: TIWattwise (https://dev.azure.com/TIWattwise )

Project: WattWise

Thank you for your help.

  1. Pravallika KV 17,025 Reputation points • Microsoft External Staff • Moderator

    @TI Wattwise ,Following up to see if the provided answer was helpful. If this answers your query, do click Accept Answer =>Yes, and upvote it. If you have any further queries do let us know.

Sign in to comment

1 answer

  1. Pravallika KV 17,025 Reputation points • Microsoft External Staff • Moderator

    Hi @TI Wattwise ,

    Here are the most relevant things to check and the likely patterns that match what you’re seeing (many events staying in Queued with no inbound requests hitting the target).

    Confirm the subscription is enabled

    • Service hook delivery can be affected if the subscription is in a restricted/probation state or disabled due to repeated failures.

    Confirm the subscription settings are correct

    • Double-check event filters (for “Pull Request Created”) and the action/consumer configuration so the right events are triggering and routed.

    Review Service Hooks activity + History (last 7 days)

    • Go to the Service Hooks page in Azure DevOps Services.
    • Select the problematic subscription and open History to see the request/response details and how Azure DevOps is categorizing the failures (if any).
    • The official guidance specifically calls out using History to debug problematic subscriptions. Even though you’re not seeing requests arrive at your server for the stuck items, the troubleshooting docs explain that Azure DevOps can enter states like:
    • Enabled (restricted) = essentially on probation
    • When subscriptions are on probation, new events are lost, and the system makes a limited number of attempts to resend the failed notification.
    • There are also clearly defined buckets of failure types:
      • Transient failures (e.g., 408/502/503/504) get retries.
      • Enduring failures (most other HTTP errors like 404/500) put the subscription on probation.
      • Terminal failure 410 (Gone) disables the subscription immediately.

    Also, there’s a relevant known behavior described in the provided “Service Hooks - Throttling” incident summary: if a large number of subscriptions repeatedly fail (for example due to endpoint/auth problems), it can throttle or overload the notification pipeline, impacting other healthy webhooks. This can manifest as delays/queuing rather than immediate delivery to the endpoint.

    Make sure there’s no current incident affecting Service Hooks delivery: Check Azure DevOps Services Status at https://status.dev.azure.com

    If your webhook payloads can be large:

    • Service hook payload limit is 2 MB.
    • Larger payloads can cause degradation in performance and reliability.

    The service hook system’s “delivery lifecycle” (queued/retry/probation/disable) is managed by Azure DevOps, and retries/probation can occur based on notification outcomes. Since you’re not seeing any inbound traffic at all for the stuck events, the most useful next step is to look in Service Hooks History for those specific queued events/subscriptions to determine whether they’re being retried, failing authentication/authorization, or being placed into probation/restricted states.

    Hope this helps!

    If the resolution was helpful, kindly take a moment to click on 👁 User's image
    and click on Yes for was this answer helpful. And, if you have any further query do let us know.

    0 comments No comments
    Sign in to comment
Sign in to answer

Your answer