Arc-enabled Kubernetes Pod kube-aad-proxy failing to start

Desmond Kirrane 40 Reputation points

I'm seeing the Arc Agent kube-aad-proxy Pod failing with x509: certificate signed by unknown authority

It's using container version: mcr.microsoft.com/azurearck8s/kube-aad-proxy:1.18.3

From the Kubernetes Node itself I have no problem curling the endpoint curl -v https://sts.windows.net/<myTenantId>/.well-known/openid-configuration

Full log

time="2024-09-02T12:05:29Z" level=fatal msg="Failed to create authenticator. Error:Get \"https://sts.windows.net/<myTenantId>/.well-known/openid-configuration\": tls: failed to verify certificate: x509: certificate signed by unknown authority
failed to create provider for azure
github.com/azure-core/ClusterConfigurationAgent/kube-aad-proxy/pkg/auth.NewAADAuthenticator
\t/usr/local/ClusterConfigurationAgent/kube-aad-proxy/pkg/auth/aadAuthenticator.go:89
github.com/azure-core/ClusterConfigurationAgent/kube-aad-proxy/pkg/server.(*Server).ListenAndServe
\t/usr/local/ClusterConfigurationAgent/kube-aad-proxy/pkg/server/server.go:137
github.com/azure-core/ClusterConfigurationAgent/kube-aad-proxy/cmd.NewRunCmd.func1
\t/usr/local/ClusterConfigurationAgent/kube-aad-proxy/cmd/run.go:33
github.com/spf13/cobra.(*Command).execute
\t/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987
github.com/spf13/cobra.(*Command).ExecuteC
\t/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115
github.com/spf13/cobra.(*Command).Execute
\t/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
main.main\n\t/usr/local/ClusterConfigurationAgent/kube-aad-proxy/main.go:32
runtime.main
\t/usr/local/go/src/runtime/proc.go:271\nruntime.goexit
\t/usr/local/go/src/runtime/asm_amd64.s:1695"
0 comments No comments
Sign in to comment

1 answer

  1. Amira Bedhiafi 42,941 Reputation points • MVP • Volunteer Moderator

    Hello Desmond !

    Thank you for posting on MS Learn Q&A.

    This usually means the problem is in the pod TLS trust chain, not basic connectivity.

    Since kube-aad-proxy is failing with that error when reaching https://sts.windows.net/..., I think that the Arc agent pod is likely seeing a certificate chain that its container trust store does not recognize.

    The fact that curl works from the Kubernetes node does not fully rule this out because the node and the container can have different trust stores and may even take different proxy/egress paths.

    For Azure Arc enabled Kubernetes, sts.windows.net, login.microsoftonline.com, and login.windows.net are required outbound endpoints and Arc expects HTTPS with publicly verifiable certificates. In proxy environments, a trusted proxy certificate may need to be injected into Arc agent pods via proxy certificate configuration.

    You can resolve this by bypassing TLS inspection for the required Arc/Microsoft identity endpoints or

    configuring Arc with the correct proxy settings and trusted proxy CA certificate so the agent pods trust the inspected certificate chain.

    0 comments No comments
    Sign in to comment
Sign in to answer

Your answer