AI Proxy Advanced

The AI Proxy Advanced plugin lets you transform and proxy requests to multiple AI providers and models at the same time. This lets you set up load balancing between targets.

AI Proxy Advanced plugin accepts requests in one of a few defined and standardized OpenAI formats, translates them to the configured target format, and then transforms the response back into a standard format.

v3.10+ To use AI Proxy Advanced with non-OpenAI format without conversion, see section below for more details.

AI Proxy Advanced plugin supports capabilities across batch processing, multimodal embeddings, agents, audio, image, streaming, and more, spanning multiple providers.

For Kong Gateway versions 3.6 or earlier:

• Chat Completions APIs: Multi-turn conversations with system/user/assistant roles.

• Completions API: Generates free-form text from a prompt.

  OpenAI has marked this endpoint as legacy and recommends using the Chat Completions API for developing new applications.

See the following table for capabilities supported in AI Gateway:

The following providers are supported by the legacy Completions API:

• OpenAI
• Azure OpenAI
• Cohere
• Llama2
• Amazon Bedrock
• Gemini
• Hugging Face

AI Gateway supports proxying requests to the following AI providers. Each provider page documents supported capabilities, configuration requirements, and provider-specific details.

For detailed capability support, configuration requirements, and provider-specific limitations, see the individual provider reference pages.

The AI Proxy Advanced plugin will mediate the following for you:

• Request and response formats appropriate for the configured config.targets[].model.provider and config.targets[].route_type
• The following service request coordinates (unless the model is self-hosted):
  • Protocol
  • Host name
  • Port
  • Path
  • HTTP method
• Authentication on behalf of the Kong API consumer
• Decorating the request with parameters from the config.targets.model[].options block, appropriate for the chosen provider
• Recording of usage statistics of the configured LLM provider and model into your selected Kong log plugin output
• Optionally, additionally recording all post-transformation request and response messages from users, to and from the configured LLM
• Fulfillment of requests to self-hosted models, based on select supported format transformations

Flattening all of the provider formats allows you to standardize the manipulation of the data before and after transmission. It also allows your to provide a choice of LLMs to the Kong Gateway Consumers, using consistent request and response formats, regardless of the backend provider or model.

v3.11+ AI Proxy Advanced supports REST-based full-text responses, including RESTful endpoints such as llm/v1/responses, llm/v1/files, llm/v1/assisstants and llm/v1/batches. RESTful endpoints support CRUD operations— you can POST to create a response, GET to retrieve it, or DELETE to remove it.

AI Gateway transforms requests and responses according to the configured config.targets[].model.provider and config.targets[].route_type, using the OpenAI format by default. v3.10+ To use a provider’s native format instead, set config.llm_format to a value other than openai. The plugin then passes requests upstream without transformation. See Supported native LLM formats for available options.

The following table maps each route type to its OpenAI API reference and generative AI category. See the AI provider reference pages for provider-specific details.

Provider-specific parameters can be passed using the extra_body field in your request. See the sample OpenAPI specification for detailed format examples.

If you use a provider’s native SDK, AI Gateway v3.10+ can proxy the request and return the upstream response without payload format conversion. Set config.llm_format to a value other than openai to preserve the provider’s native request and response formats.

In this mode, AI Gateway will still provide analytics, logging, and cost calculation. When config.llm_format is set to a native format, only the corresponding provider is supported with its specific APIs.

AI Proxy Advanced supports several load balancing algorithms for distributing requests across AI models:

• Round-robin: Weighted traffic distribution.
• Consistent-hashing: Sticky sessions based on header values.
• Least-connections: Route to backends with spare capacity.
• Lowest-latency: Route to fastest-responding models.
• Lowest-usage: Route based on token counts or cost.
• Semantic: Route based on prompt-to-model similarity.
• Priority: Tiered failover across model groups.

For detailed algorithm descriptions and selection guidance, see Load balancing algorithms.

For load balancing across Gateway Upstreams and Targets instead of LLMs, see load balancing with Kong Gateway.

The AI load balancer supports configurable retries, timeouts, and failover to different models when a target is unavailable.

v3.10+ Fallback works across targets with any supported format. You can mix providers freely, for example OpenAI and Mistral. Earlier versions require compatible formats between fallback targets. For configuration details, see Retry and fallback configuration.

Client errors don’t trigger failover. To failover on additional error types, set config.balancer.failover_criteria to include HTTP codes like http_429 or http_502, and non_idempotent for POST requests.

The AI load balancer supports circuit breakers to improve reliability. If a target reaches the failure threshold defined by config.balancer.max_fails, the load balancer stops routing requests to it until the timeout period (config.balancer.fail_timeout) elapses.

For configuration details and behavior examples, see Circuit breaker.

The plugin allows you to substitute values in the config.targets[].model.name and any parameter under config.targets.model[].options with specific placeholders, similar to those in the Request Transformer Advanced plugin.

The following templated parameters are available:

• $(headers.header_name): The value of a specific request header.
• $(uri_captures.path_parameter_name): The value of a captured URI path parameter.
• $(query_params.query_parameter_name): The value of a query string parameter.

You can combine these parameters with an OpenAI-compatible SDK in multiple ways using the AI Proxy and AI Proxy Advanced plugins, depending on your specific use case:

A vector database can be used to store vector embeddings, or numerical representations, of data items. For example, a response would be converted to a numerical representation and stored in the vector database so that it can compare new requests against the stored vectors to find relevant cached items.

The AI Proxy Advanced plugin supports the following vector databases:

• Using config.vectordb.strategy: redis and parameters in config.vectordb.redis:
  • Redis with Vector Similarity Search (VSS)
  • Redis Cloud
  • Valkey v3.14+: When you configure vectordb.strategy: redis, Kong Gateway queries the server and checks the server name field. If it detects Valkey request, it automatically uses the Valkey-specific driver.
  • Managed Redis with cloud authentication:
    • AWS ElastiCache (auth_provider: aws)
    • Azure Managed Redis (auth_provider: azure)
    • Google Cloud Memorystore (auth_provider: gcp)

    For configuration details, see Using cloud authentication with Redis.

• Using config.vectordb.strategy: pgvector and parameters in config.vectordb.pgvector:
  • PostgreSQL with pgvector v3.10+

To learn more about vector databases in AI Gateway, see Embedding-based similarity matching in Kong AI gateway plugins.

This plugin supports all three AI Partial types, which let you define shared configuration once and reuse it across multiple AI Gateway plugins.

A model Partial applies to each entry in the config.targets array, so you can share one provider configuration across multiple targets.

For setup instructions, see AI plugin Partials.

Using cloud authentication with Redis v3.13+

If your plugin uses a Redis datastore, you can authenticate to it with a cloud Redis provider. This allows you to seamlessly rotate credentials without relying on static passwords.

The following providers are supported:

• AWS ElastiCache
• Azure Managed Redis
• Google Cloud Memorystore (with or without Valkey)

config:
 vectordb:
 strategy: redis
 redis:
 host: $INSTANCE_ADDRESS
 username: $INSTANCE_USERNAME
 port: 6379
 cloud_authentication:
 auth_provider: aws
 aws_cache_name: $AWS_CACHE_NAME
 aws_is_serverless: false
 aws_region: $AWS_REGION
 aws_access_key_id: $AWS_ACCESS_KEY_ID
 aws_secret_access_key: $AWS_ACCESS_SECRET_KEY

config:
 vectordb:
 strategy: redis
 redis:
 cluster_nodes:
 - ip: $CLUSTER_ADDRESS
 port: 6379
 username: $CLUSTER_USERNAME
 port: 6379
 cloud_authentication:
 auth_provider: aws
 aws_cache_name: $AWS_CACHE_NAME
 aws_is_serverless: false
 aws_region: $AWS_REGION
 aws_access_key_id: $AWS_ACCESS_KEY_ID
 aws_secret_access_key: $AWS_ACCESS_SECRET_KEY

config:
 vectordb:
 strategy: redis
 redis:
 host: $INSTANCE_ADDRESS
 username: $INSTANCE_USERNAME
 port: 10000
 cloud_authentication:
 auth_provider: azure
 azure_client_id: $AZURE_CLIENT_ID
 azure_client_secret: $AZURE_CLIENT_SECRET
 azure_tenant_id: $AZURE_TENANT_ID

config:
 vectordb:
 strategy: redis
 redis:
 cluster_nodes:
 - ip: $CLUSTER_ADDRESS
 port: 10000
 username: $CLUSTER_USERNAME
 port: 10000
 cloud_authentication:
 auth_provider: azure
 azure_client_id: $AZURE_CLIENT_ID
 azure_client_secret: $AZURE_CLIENT_SECRET
 azure_tenant_id: $AZURE_TENANT_ID

config:
 vectordb:
 strategy: redis
 redis:
 host: $INSTANCE_ADDRESS
 port: 6379
 cloud_authentication:
 auth_provider: gcp
 gcp_service_account_json: $GCP_SERVICE_ACCOUNT

config:
 vectordb:
 strategy: redis
 redis:
 cluster_nodes:
 - ip: $CLUSTER_ADDRESS
 port: 6379
 port: 6379
 cloud_authentication:
 auth_provider: gcp
 gcp_service_account_json: $GCP_SERVICE_ACCOUNT