Data Flow source test connection to Cosmos DB fails with InvalidSubscriptionId error via Managed VNet Private Endpoint

Yeshwanth Reddy Sangam 0 Reputation points • Microsoft Employee

Azure Data Factory Data Flow fails to connect to Azure Cosmos DB (SQL API) when using Managed Virtual Network with a Managed Private Endpoint. The Copy Activity works correctly with the same linked service and private endpoint configuration.

Error Details:

  • Status code: 400
  • Error: {"error":{"code":"InvalidSubscriptionId","message":"The provided subscription identifier 'resourceGroups' is malformed or invalid."}}
  • This error occurs when testing connection from a Data Flow source, and during Data Flow debug/execution.
  • Tested after publishing as well

Behavior:

  1. Linked service "Test Connection" → SUCCEEDS ✓
  2. Copy Activity using same linked service → SUCCEEDS ✓
  3. Data Flow source using same linked service → FAILS ✗ (400 InvalidSubscriptionId)

This appears to be a bug in how the Data Flow Spark runtime resolves Cosmos DB resource IDs through Managed VNet private endpoints. The error suggests the resource path is being incorrectly parsed, with "resourceGroups" being interpreted as the subscription identifier.

  1. Pilladi Padma Sai Manisha 10,190 Reputation points • Microsoft External Staff • Moderator

    Hi @Yeshwanth Reddy Sangam
    Thankyou for reaching microsoft Q&A!
    we will reaching you in teams for further troubleshooting.

  2. Yeshwanth Reddy Sangam 0 Reputation points • Microsoft Employee

    Hi, it looks like the backend has been refreshed and the connection is now successful. Here are the steps I followed:

    • Create a new linked service in ADF and make sure the following is included in the advanced JSON of linked service.
    {
 "properties": {
 "typeProperties": {
 "subscriptionId": "<subscription-id>",
 "tenantId": "<tenant-id>",
 "resourceGroup": "<resource-group>"
 }
 }
}
Sign in to comment

1 answer

  1. Yeshwanth Reddy Sangam 0 Reputation points • Microsoft Employee

    It looks like the backend has been refreshed and the connection is now successful. Here are the steps I followed:

    1. Create a new linked service in ADF and make sure the following is included in the advanced JSON of linked service. 
       {
 "properties": {
 "typeProperties": {
 "subscriptionId": "<subscription-id>",
 "tenantId": "<tenant-id>",
 "resourceGroup": "<resource-group>"
 }
 }

    }
    0 comments No comments
    Sign in to comment
Sign in to answer

Your answer