Scaling AI Agents in the Enterprise: The Hard Problems and How to Solve Them

AI agents are evolving beyond simple chat-based interactions into systems that execute workflows, manage state, and make decisions across long-running processes. These architectures are being adopted in use cases ranging from fully autonomous AI interviewers (Mercor) to financial risk analysis (Robinhood) and data pipeline automation (Databricks). However, moving from prototype to production introduces a new set of challenges:

1. State persistence: AI agents need memory beyond a single prompt-response loop.
2. Reliable execution: If an agent fails during a task, it needs to recover gracefully.
3. Multi-agent coordination: Agents need to interact, share knowledge, and delegate tasks.

Each of these challenges (and of course, security, authentication, and authorization for agents) requires careful architectural decisions, as well as a shift from treating LLMs as simple function calls to designing AI agents as robust, distributed systems.

1. State Persistence: AI Agents Need To Persist State Beyond a Single Prompt-Response Loop

The Problem: Stateless LLMs Fail in Long-Running Workflows

Most AI applications today use LLMs in a stateless fashion: each query is treated independently with no recall of prior interactions. This works for simple queries but fails in complex workflows where agents must remember prior steps, decisions, or user inputs.

Example: Stateful AI for Technical Interviews (Mercor)

Take Mercor, for example, a $2B+ company backed by Benchmark, General Catalyst, and Felicis. They’re building a fully autonomous AI interviewer that adapts in real-time to how a candidate performs. The system could initiate role- and level-specific interview questions, then adaptively generate follow-up questions in real-time based on the candidate’s responses and performance signals. It would conclude by synthesizing performance data to deliver a rigorous, data-driven, and unbiased evaluation of candidate aptitude and fit.

To ask questions that logically follow a candidate’s responses, Mercor’s system needs to be able to persist state, including:

• A candidate’s technical responses and code submissions
• Areas they struggled with or excelled in
• Reference points from similar interviews

Without a persistent state layer, the experience would feel disjointed and repetitive, and worse, the system wouldn’t be able to make a fair or informed evaluation. Real-time, adaptive agents don’t just benefit from memory, they depend on it. Agentic memory in its various flavors and design patterns ensures long-running context for LLMs.

Solution: Architecting State for AI Agents With Embeddings

Persisting state in AI systems isn’t a one-size-fits-all problem. Different workflows require different types of recall: some are semantic and fuzzy, while others are exact and structured. For agents to operate effectively in real-world environments, memory must be purpose-built, composable, and optimized for both speed and relevance.

• Vector Databases (e.g., Pinecone, Weaviate, Supabase pgvector): For tasks like summarization, knowledge retrieval, or referencing prior conversations, try using a vector database like Pinecone, Weaviate, or Supabase’s pgvector extension.
• Structured Storage (e.g., Letta/MemGPT): In contrast, when workflows require exact tracking, think multistep processes, form completion, or reasoning over previous decisions, structured memory is essential. Tools like Letta shine here.
• Agentic Memory Layers: The most advanced memory architectures dynamically integrate both fuzzy and structured memory into a single runtime. Letta, for instance, enables LLMs to operate beyond the fixed-token context window by layering long-term memory directly into the agent architecture.

A fundamental breakthrough in this area is the use of embeddings. The process involves indexing and storing embeddings, then employing search and retrieval techniques to manage long-term state effectively.

Embeddings
Embeddings convert segments of text, such as memories or conversation fragments, into high-dimensional numerical vectors that encapsulate their semantic content. A neural network, trained to recognize linguistic patterns, transforms text into vectors that reflect meaning, context, and the relationships between words or phrases. The resulting high-dimensional space allows for the establishment of contextual similarity, where proximity between vectors indicates relatedness.

Interplay with Vector Databases
Vector databases store these high-dimensional vectors along with associated metadata (e.g., tags, timestamps) and enable similarity searches through various methods:

• Approximate Nearest Neighbors (ANN): Techniques like Hierarchical Navigable Small World (HNSW), Locality-Sensitive Hashing (LSH), and Annoy (which uses random projection trees) quickly narrow down the search to approximate nearest neighbors.
• Inverted Indexing (IVF): This method clusters the vector space, limiting searches to relevant dataset segments.
• Quantization: Optimized product quantization compresses vectors, accelerating distance computations.
• Hybrid Approaches: Often, a combination of these techniques is employed to balance speed and accuracy.

Enhancing Agentic Memory
During retrieval, new inputs are transformed into vectors, and the vector database is queried to find semantically similar vectors. This process enables the agent to maintain continuity over extended interactions, ensuring smoother transitions and coherent decision-making while minimizing latency. As new tasks are completed or interactions are finished, their embeddings are generated and added to the database. This continuous update mechanism allows the agent to dynamically refine its memory, enhancing its long-term contextual awareness and overall learning capabilities.

2. Reliable Execution: If an Agent Fails Midtask, It Needs To Recover Gracefully

The Problem: Failures in Multistep AI Workflows

LLM-based agents rarely operate in a vacuum; they frequently interact with APIs, databases, and various external systems. When an agent encounters an error during execution, it must handle the issue gracefully instead of starting over entirely.

A significant challenge in productionizing AI agents is durability; if an LLM generates responses in a long-running workflow and the process fails midway, does the entire session reset? Or does it recover where it left off? Unlike traditional web applications, which rely on database-backed statefulness, AI agents often operate in stateless environments unless they are explicitly designed for fault tolerance.

Example: Robinhood’s AI-Powered Trading Agent

Robinhood leverages AI agents to function as market analysts, assisting users in constructing trades. These agents integrate high-fidelity market data with real-time trading information, historical trade patterns, and proprietary insights regarding retail trading behavior, enabling the formulation of a robust stock thesis. Operating within a high-stakes, low-latency framework, the system is engineered to prevent incorrect answers or failures that could lead to financial loss or regulatory challenges.

To guarantee reliable trade execution, Robinhood deploys a multilayered AI model fallback architecture as described below:

• Primary High-Performance LLM for Critical Decision-Making:
   A compute-intensive large language model (LLM) processes complex market conditions using chain-of-thought reasoning to generate detailed market insights. A dedicated subsystem interfaces with this model to mitigate hallucinations and ensure the reliability of outputs.
• Secondary Lightweight LLM for Summarization:
   The detailed insights from the primary LLM are then routed to a more cost-efficient, lower-latency model that produces concise summaries. This dual-model approach balances performance with operational cost efficiency.
• Failover and Redundancy Mechanism:
   In the event of a primary LLM failure, the system automatically fails over to the secondary model or retrieves cached responses from a historical vector database. This design ensures operational continuity under adverse conditions.
• Event-Driven Asynchronous Execution:
   The AI-generated insights are presented to the user, who then specifies trade parameters such as price and time targets. These inputs are asynchronously queued and processed to decouple execution stages, preventing error propagation. In case of a failure during any execution step, the process is designed to roll back and retry, rather than initiating a complete system reset.

This architecture enables Robinhood’s AI-driven insights and trade construction platform to maintain near-100% uptime, significantly reducing order failures while effectively managing AI inference costs.

Solution: Ensuring Reliability in AI Agent Workflows

For developers, reliability isn’t just about uptime. It’s about trust. If an agent drops state halfway through a task, fails silently, or returns inconsistent outputs, it breaks the entire user experience. Building for reliability means thinking like a systems engineer: handling retries, surfacing errors cleanly, and making sure agents can recover without starting from scratch.

• Orchestration frameworks (Temporal.io, Crew.ai, Langchain): Provide stateful execution, retries, and recovery mechanisms
• Multi-LLM routing: Intelligent load balancing between foundation models to optimize availability and optimize for model feature use in addition to giving levers for cost and latency tradeoffs.
• Versioning & Rollback:
  AI agents should regularly save checkpointed execution states to enable recovery in case of failures. Moreover, using a model registry for version control provides a robust rollback mechanism, ensuring that if a new model update does not meet expectations, the system can quickly revert to a previous stable version.
• Blue/Green Deployments for Models:
  In production, blue/green deployments are employed by routing traffic between two model versions. For instance, the “blue” model might handle 95% of the traffic while the “green” model handles 5%. Once the green model demonstrates the required accuracy, availability, and stability, it is promoted to blue, and a new improved green model is introduced. This strategy minimizes risk and ensures continuous, reliable performance.

3. Multi-Agent Systems: Distributed AI Coordination

The Problem: Single LLMs Are Inefficient for Complex Tasks

Monolithic LLMs fall short when executing complex, multistep workflows. Instead, different tasks require specialized AI models — some tailored for reasoning, others for retrieval, and still others for execution. For example, consider a system where agents process multimodal inputs (such as text, voice, and video) and generate not only multimodal outputs but also trigger actions, whether through direct computer interaction, function calls, or web-based operations.

👁 Image

Single-agent systems are inherently limited by their sequential processing and single-threaded decision-making, restricting their ability to execute tasks in parallel and distribute workloads effectively. This constraint makes them less suitable for complex, real-world applications, prompting the development of multi-agent systems that can handle distributed tasks more efficiently.

This evolution reflects a broader trend in agentic AI toward enhanced planning, execution, and self-optimization. Even within a single-agent framework, advancements in these areas have paved the way for more robust and resilient AI systems.

Example: AI Agents in Enterprise Security (Palo Alto Networks)

Enterprise security requires AI agents that perform distinct functions:

1. Threat detection agent: Monitors logs and flags anomalies
2. Risk assessment agent: Uses ML-based models to evaluate threats
3. Remediation agent: Automates security responses

Each of these components requires specialized AI, rather than a single LLM attempting to handle the entire workflow.

Solution: Architecting Multi-Agent Systems

Decentralized agent architectures, such as mixtures of experts, distribute tasks among specialized models, thereby reducing inference overhead. Several popular multi-agent architectures include:

• Supervisory Agent: All agents communicate with a central supervisor for coordinated plan execution.
• Networked Agents: Each agent can interact directly with others.
• Hierarchical Systems: A layered structure where supervisors coordinate other supervisors to tackle complex tasks.
• Custom Architectures: User-defined setups that enable coordination among only a subset of agents for specific task execution.

Event-Driven Communication

Agents exchange state updates via message queues or event buses. This event-driven approach allows agents to operate on specific triggers by reading and writing messages to distributed queues, eliminating the need for constant polling or direct connections, reducing latency, and enhancing scalability.

Cross-Agent Memory

AI agents maintain a shared knowledge base to facilitate smooth task hand-offs. They store and retrieve data from shared memory to maintain a unified context around shared goals. This data is encoded into rich, context-aware representations using LLM-friendly embeddings. A key challenge here is ensuring that shared memory implements proper locking and version control to prevent issues from concurrent updates.

Longer-Term Retrieval and Interplay With Microservices

A key unlock for longer-term reterivals is the implementation of a unified data access layer to ensure agents retrieve the correct information. One effective approach combines GraphQL with the Model Context Protocol (MCP) for consistent data delivery.

GraphQL serves as a versatile API layer by providing a single endpoint that allows clients to fetch only the data they need, thereby avoiding issues related to overfetching or underfetching. The Model Context Protocol standardizes how contextual data is packaged and delivered to AI models, ensuring they have the precise context required for accurate decision-making. When these technologies are integrated for agentic AI systems, they dynamically supply the necessary context to autonomous agents, boosting both their adaptability and overall efficiency in data retrieval and consistency.

Modern agentic systems in real-world applications typically comprise:

1. Plan, Execute, Decide loops
   
2. Self-improvement via in-context learning or fine-tuning
3. Tools and plan optimization
   
4. Continuous evaluations
   
5. Robust observability

👁 Image

Solving for Continuous Improvements, Learning and a Mixture of Experts

These systems can be further improved through trajectory optimization, using expert tools and communicators, to generate the desired task output. Supervised finetuning can be done over Planner, Tool user, or communicator data to build expert bots that are good at specific tasks while being lower in memory footprint and latency.

👁 Image

Designing for the Hard Path Is the Only Path

The next generation of enterprise AI will be driven by systems thinking.

AI agents that succeed at scale don’t just take in prompts; they will remember, recover, and collaborate. Building for state persistence, reliable execution, and multi-agent coordination isn’t optional. It’s foundational. They are the difference between a prototype that demos well and a system that delivers every single day in production.

For teams serious about deploying AI agents in high-stakes, high-complexity environments, the call to action is clear: treat agents like distributed systems. Invest in infrastructure, not just inference. Build a persistent state as a core service, not an afterthought. Embrace redundancy, modularity, and the architectural rigor that the enterprise demands.