Azure AI Foundary Evals not working

Hello @Prasun Aggarwal

Thank you for reaching out to Microsoft Q&A.

Based on your description, the issue appears to occur specifically during evaluation execution, while evaluation creation and other Azure AI Foundry portal operations complete successfully. The combination of:

• HTTP 400 with underlying 403 responses

"Local authentication/API key disabled" messages

References to VNet/network restrictions

Usage of Managed Identity and DefaultAzureCredential

suggests that the evaluation runtime may be encountering either an authorization issue, a network isolation issue, or a combination of both.

Since you are able to access the portal, create evaluations, and perform other project operations successfully, this points more toward a problem affecting the evaluation execution path (data plane) rather than the project management path (control plane).

1. Verify the identity used by the evaluation run

Even when using Managed Identity or DefaultAzureCredential, the evaluation job itself must have sufficient permissions to access the underlying Azure AI resources.

For 401/403-related evaluation failures, please verify:

The Managed Identity used by the evaluation run is the intended identity.

The identity has the required RBAC permissions on the Azure AI Foundry Project and associated resources.

The project endpoint being used is correct and references the appropriate Hub and Project.

At a minimum, please verify whether the identity has:

Azure AI Foundry User

Azure AI Developer (if applicable)

Cognitive Services User

Cognitive Services OpenAI User (if Azure OpenAI is involved)

A common scenario is that the signed-in user can create evaluations successfully, but the runtime identity executing the evaluation lacks the necessary permissions.

2. Verify Local Authentication settings

You mentioned that browser traces show:

"Local authentication/API key disabled"

Please confirm:

Whether Local Authentication is disabled on the Azure AI resource.

Whether the evaluation workflow is fully configured to use Microsoft Entra ID authentication.

Whether any underlying service involved in the evaluation still expects API-key-based authentication.

In some scenarios, evaluation services may fail if the configured authentication method does not match the resource configuration.

3. Review network isolation and VNet configuration

The references to "VNet is disabled" suggest the evaluation runtime may be unable to reach one or more dependent services.

Please verify:

Whether Public Network Access is enabled or disabled.

Whether Private Endpoints are configured.

Whether Bring Your Own VNet (BYO-VNet) is being used for evaluation execution.

Whether required Azure services are reachable from the evaluation runtime.

Whether "Allow trusted Microsoft services" is enabled where applicable.

A common pattern is that portal access works normally, but the evaluation execution service cannot access Storage, Azure OpenAI, or other dependent resources due to firewall or network restrictions.

4. Confirm the evaluation runtime endpoint

For evaluation runs executed through SDKs or pipelines, please verify:

The project endpoint URL is correct.

The evaluation is targeting the intended Azure AI Foundry Project.

No stale or incorrect endpoint configuration is being used.

Authentication failures can sometimes manifest as generic 400 errors while the underlying service returns a 403 Forbidden response.

Please refer this https://learn.microsoft.com/azure/foundry/how-to/develop/cloud-evaluation?wt.mc_id=knowledgesearch_inproduct_azure-cxp-community-insider#troubleshooting

I Hope this helps. Do let me know if you have any further queries.

Thank you!