 |
VOOZH | about |
|
VOOZH | about |
We’re so glad you’re here. You can expect all the best TNS content to arrive Monday through Friday to keep you on top of the news and at the top of your game.
Check your inbox for a confirmation email where you can adjust your preferences and even join additional groups.
Follow TNS on your favorite social media networks.
Become a TNS follower on LinkedIn.
Check out the latest featured and trending stories while you wait for your first TNS newsletter.
Deploying an AI-powered application is more than just calling a model. Developers must wrangle inference infrastructure, version data pipelines and integrate external tools, while also finding ways to monitor or govern outputs that are more likely to hallucinate. The moment a team tries to move beyond a basic prototype, it’s suddenly forced to develop expertise in orchestration, compliance and AI architecture.
As AI capabilities explode across modalities (think: text to image to audio), the developer experience hasn’t kept pace. Teams are duct-taping solutions together across cloud providers, large language models (LLMs) APIs, vector databases and brittle control loops. Even companies with strong engineering muscle struggle to maintain velocity.
What’s missing is a platform-level solution that abstracts these AI concerns the same way traditional Platform as a Service (PaaS) abstracts infrastructure.
This is the space AI Platform as a Service (AI PaaS) aims to fill. It brings the core PaaS principles of simplicity, scalability and developer-first tooling to modern AI building blocks.
Let’s explore what an AI PaaS is and how it lets you ship production-grade AI applications without reinventing your entire stack.
An AI PaaS does exactly what it says: It’s a platform that helps developers build, deploy and operate AI-powered applications in the cloud without needing to manage models, orchestration, pipelines or infrastructure themselves. It builds on the foundation of traditional PaaS but extends it with AI-native features like model access, retrieval pipelines, agent orchestration and evaluation tools.
These platforms fill a critical gap, as many AI projects never make it to production. Gartner predicts that up to 40% of agentic AI initiatives will fail by 2027, often due to integration costs, lack of observability or deployment complexity. An AI PaaS addresses these challenges by providing opinionated, scalable defaults.
So what makes up an AI PaaS? You begin with the foundation of a PaaS and then add AI-specific features.
Every PaaS needs to get a few core things right, whether you’re building a CRUD app or a conversational agent. They are:
These are the table stakes. But building with AI introduces a new layer of complexity. Let’s look at specific features required for an AI PaaS.
To begin building an AI PaaS, the minimal tools required include model inference, retrieval pipelines and Model Context Protocol (MCP) scaffolding.
AI-powered features are centered around LLMs. An LLM offers conversational generative AI, which has become commonplace since the launch of ChatGPT in 2022. An AI PaaS should provide seamless access to various machine learning (ML) models. Models all have different strengths and weaknesses, so having access to multiple models provides the most flexibility for teams building AI agents.
This diversity can also be used to mitigate cost, where some services require complex (and expensive) models, while less complex services can use smaller, less expensive models.
When an LLM provides a response, a control loop should be in place to monitor the responses and verify their quality. Developers can create customer-defined heuristics and rules that will be used to evaluate the response. This could involve hard-coded guardrails or comparing the results of multiple LLMs to achieve consensus.
If the response does not meet the quality standard, the query may be reformulated and queried again. If the response passes the evaluation, the control loop will pass the response on to the next step of the model.
How a closed loop monitors responses.
LLMs are powerful tools and can converse with users on many disparate topics. To power a generative AI that is useful for an organization, additional data must be constantly supplied to ensure timely and accurate responses.
MCP is a standardized approach to connect external tools into an AI system to provide additional data or knowledge. MCP servers make it easy to securely connect existing data tools (both internal and external) to incorporate new data.
MCPs may provide connectivity to an API for frequently changing data (“What is the current traffic in Queens, N.Y.?”) or to a database with enterprise data (“How many deals were signed in Q2 2021?”). These data stores support and enhance the model’s output.
Additionally, the MCP acts as a service directory. When a query is sent to the AI, it formulates its response based on knowing where the data is located and how it can be retrieved and formatted into a response. This allows existing applications and agents to connect to the MCP.
MCPs process requests from applications and large language models, then feed in data from external sources. (Source: Heroku)
The MCP can also be used to expose the AI application as a tool to be utilized by other agentic systems, allowing other agents to use the AI system to complete tasks.
For example, Audata built Aura (an AI support agent) to leverage real-time data from Heroku Postgres and enterprise data from Salesforce Agentforce to answer routine questions. If a case is escalated to the support team, a synopsis of the existing chat is provided to the representative, resulting in faster ticket resolution.
A credible AI PaaS goes beyond inference. It helps teams build responsibly, iterate quickly and scale with confidence. Here’s what to expect from platforms that can support long-term, production-grade AI use:
One common data storage tool for external knowledge is retrieval augmented generation (RAG). The RAG database is generally a vector database containing enterprise data encoded specifically to interact quickly with the LLM. For example, Heroku’s Postgres pgvector provides seamless vector database support without the need for additional database tooling.
When a query is made to the AI model, relevant data from the database is provided by the LLM to formulate a response. RAG architecture allows organizations to insert customized data to influence the LLM’s response.
For instance, loan processing and approvals at 1West were a slow and manual process. After training a machine learning model using Heroku’s AI PaaS to work with a vast number of data sources, loan processing was cut from days to minutes.
A simplified RAG architecture, including data pipelines for contextual data.
Just as LLMs can quickly become outdated and provide incorrect or stale responses on their own, the same can happen with data in the RAG database. To maintain accuracy in the AI application, the RAG database must be continually refreshed to reflect new or changing data. This requires automated workflows for document processing. These workflows should integrate seamlessly with existing systems and handle all processing steps efficiently.
For example, within the Heroku ecosystem, Heroku Scheduler can regularly run workflows to access documents and insert the processed data into the pgvector database. All of the processing occurs in a secure environment, protecting enterprise data.
As companies integrate AI-powered tools into their stacks, many development teams lack the MLOps, governance and orchestration skills necessary to deploy AI in production. Using Heroku’s AI PaaS jumpstarts the process of building, deploying, operating and scaling AI-powered applications.
Drawing on Heroku’s experience and developer-first approach to building cloud architectures means that the enterprise team can focus on building the service, rather than managing servers, networking, security and building orchestration tools.
Heroku Vibes AI code generation allows you to create and deploy to Heroku with natural language. Heroku’s Managed Inference and Agents provides curated AI models to build upon. The Heroku MCP Server makes it straightforward for agents to access Heroku resources like logs, provision add-ons and scale applications. A custom MCP server deployed on Heroku can give access to existing systems to your AI service.
Deploying AI apps should be as easy as pushing a web app. With opinionated defaults and managed services, Heroku continues to evolve alongside developers, providing a streamlined, integrated platform experience.
Heroku is bringing its decades of expertise in deploying applications in the cloud to help developers quickly launch AI technologies. To learn more about Heroku and AI PaaS, watch the demo on YouTube or follow on LinkedIn for updates.
