AI Gateway

Introducing AI Gateway

As AI adoption accelerates, applications are evolving beyond basic LLM calls into complex, multi-actor systems-including user apps, agents, orchestration layers, and context servers that interact with foundation models in real time.

To support this shift, developers are adopting protocols like Model Context Protocol (MCP) and Agent2Agent (A2A) to standardize how components exchange tools, data, and decisions.

But infrastructure often falls behind, with challenges around authentication, rate limiting, data security, observability, and constant provider changes.

AI Gateway addresses these challenges with a high-performance control plane that secures, governs, and observes AI-native systems end to end. Whether serving LLM traffic, exposing structured context via MCP, or coordinating agents through A2A, AI Gateway ensures scalable, secure, and reliable AI infrastructure.

Quickstart

Sign up for Konnect to get started with AI Gateway.

Or, launch a demo instance of AI Gateway running on-prem:

curl -Ls https://get.konghq.com/ai | bash

AI Gateway in Konnect

Konnect provides a unified control plane to create, manage, and monitor LLMs using the Konnect platform.

Key features include:

• Routing and load balancing: Assign Gateway Services and define how traffic is distributed across models.
• Streaming and authentication: Enable streaming responses and manage authentication through the AI Gateway.
• Access control: Create and apply access tiers to control how clients interact with LLMs.
• Usage analytics: Monitor request and token volumes, track error rates, and measure average latency with historical comparisons.
• Visual traffic maps: Explore interactive maps that show how requests flow between clients and models in real time.

Deployment checklist

• AI Gateway resource sizing guidelines: Review recommended resource allocation guidelines for AI Gateway.
• Deployment topologies: Learn about the different ways to deploy Kong Gateway.
• Hosting options: Decide where you want to host your Data Plane nodes, and whether you want Kong to host them or host them yourself.

Tools to manage AI Gateway

• AI Gateway editor: GUI for managing all your AI Gateway resources in one place.
• decK: Manage AI Gateway and Kong Gateway configuration through declarative state files.
• Terraform: Manage infrastructure as code and automated deployments to streamline setup and configuration of Konnect and Kong Gateway.
• KIC: Manage ingress traffic and routing rules for your services.
• Kong Gateway Admin API: Manage on-prem Kong Gateway entities via an API.
• Control Plane Config API: Manage Kong Gateway entities within Konnect Control Planes via an API.

Universal API

Kong’s AI Gateway Universal API, delivered through the AI Proxy and AI Proxy Advanced plugins, simplifies AI model integration by providing a single, standardized interface for interacting with models across multiple providers.

• Easy to use: Configure once and access any AI model with minimal integration effort.

• Load balancing: Automatically distribute AI requests across multiple models or providers for optimal performance and cost efficiency.

• Retry and fallback: Optimize AI requests based on model performance, cost, or other factors.

• Cross-plugin integration: Leverage AI in non-AI API workflows through other Kong Gateway plugins.

As AI technologies see broader adoption, developers and organizations face new risks: the risk of sensitive data leaking to AI providers, which exposes businesses and their customers to potential breaches and security threats.

Managing how data flows to and from AI models has become critical not just for security, but also for compliance and reliability. Without the right controls in place, organizations risk losing visibility into how AI is used across their systems.

AI Gateway helps mitigate these challenges by offering a suite of plugins that extend beyond basic AI traffic management.

• Data governance: Control how sensitive information is handled and shared with AI models.
• Prompt engineering: Customize and optimize prompts to deliver consistent, high-quality AI outputs.
• Guardrails and content safety: Enforce policies to prevent inappropriate, unsafe, or non-compliant responses.
• Automated RAG injection: Seamlessly inject relevant, vetted data into AI prompts without manual RAG implementations.
• Load balancing: Distribute AI traffic efficiently across multiple model endpoints to ensure performance and reliability.
• LLM cost control: Use the AI Compressor, RAG Injector, and Prompt Decorator to compress and structure prompts efficiently. Combine with AI Proxy Advanced to route requests across OpenAI models by semantic similarity—optimizing for cost and performance.

LLMs are only as reliable as the data they can access. When faced with incomplete information, they often produce confident yet incorrect responses known as “hallucinations.” These hallucinations occur when LLMs lack the necessary domain knowledge. To address this, developers use the Retrieval-augmented Generation (RAG) approach, which enriches models with relevant data pulled from vector databases.

While standard RAG workflows are resource-heavy, as they require teams to generate embeddings and manually curate them in vector databases, Kong’s AI RAG Injector plugin automates this entire process. Instead of embedding RAG logic into every application individually, platform teams can inject vetted data into prompts directly at the gateway layer without any manual interventions.

• Authenticate OpenAI SDK clients with Key Authentication in Kong AI Gateway View →
• Send batch requests to Azure OpenAI LLMs View →
• Control accuracy of LLM models using the AI LLM as judge plugin View →
• Control prompt size with the AI Compressor plugin View →
• Guide survey classification behavior using the AI Prompt Decorator plugin View →
• View More →