AI Agents Unite: Conference Reveals Next-Gen Frameworks

NEW YORK — At the first-ever AI Agent Conference here last week, larger players lined up with startups to talk about their innovations in the booming AI agent software and services market.

Program chair Ben Lorica, principal at Gradient Flow, told The New Stack that the idea behind the new conference is to get people together in the same room, compare notes and meet people from other places.

“For any new technology trend, it’s important for early adopters to talk with each other and compare notes, and talk with people to validate your solutions,” he said.

Conference sponsorship “Secret Agent” Omer Trajman told The New Stack, “Our goal was to bring everyone together — so much is changing every day and every week — the larger players can move quickly, but it’s hard to beat the agility of a small university research team. We wanted to get an overall sense of what’s viable across the board.”

Major GenAI Trends

Robert Nishihara, co-founder of Anyscale, said in his opening keynote that he sees a “really big change in the mental model regarding where innovation needs to go” in generative AI (GenAI).

“Model architectures are now more standardized, and learning algorithms are standardized. It’s all going to be about getting higher quality data, and putting more compute into the data,” he said.

In contrast, Christina Huang, member of technical staff at OpenAI, stressed in her talk the “exponential improvements” in models as driving the acceleration of interest in AI agents. As models become more capable, so do agents, she said.

Jeff Boes, a go-to-market pro at Anthropic, stressed the need for careful adoption and collaboration with customers to ensure agents deliver as promised and reliably.

In the “startup space around our technology … the opportunity is to work with the customer, develop deep trust and solve hard problems with our tooling. We want to be fit for purpose, but we also want to be the highest trust and strongest partner for enterprises.”

AG2, AutoGen and Microsoft

Qingyun Wu, creator and founder of AG2, described in her talk how AG2 evolved from the popular Autogen open source framework for building multiagent workflows.

Wu told The New Stack that the ideas behind AutoGen and AG2 came from research she and her colleagues conducted at Penn State, where she is still an assistant professor.

“AI agents have been around for a long time,” she said. “An AI agent is an entity that can act on the environment and react, take action and respond.

“The difference now is the ability of LLMs [large language models] to reason,” she added. “The capability of AI agents has increased due to the increased capability of LLMs, in particular the ability to reason.”

After Microsoft started taking the AutoGen project in a different direction with release 0.4, Wu founded AG2 to continue along the original AutoGen path.

GenAI Agent Frameworks

Like AutoGen, AG2 composes and scales multiagent systems using simple conversations, such as agent to agent, group chats, sequential chats and nested conversations.

AI agents can be compute-intensive, data-intensive or LLM inference-intensive, said Wu. Increasingly, new AI systems are being built from compound systems. Agents are a natural way to build such systems, she added.

First, build conversable agents with customizable capabilities. Next, get the agents to talk with each other using different types of chats and tools. Then define the multiagent orchestration patterns and combine these patterns into sophisticated chats, Wu said.

Another AI agent framework coming out of a university is the Arklex AI Agent Framework from Arklex.AI, founded by Columbia University professor Zhou Yu.

Yu told The New Stack that Arklex agents learn over time, constantly fine-tuning their models with new information from interactions with humans and other agents.

For example, an Arklex agent deployed on a shopping app may learn that some people react more positively to a discount than others, that some customers prefer to use WhatsApp or SMS to receive offers or prefer certain flavors or perfume, she said.

“The AI agent learns by answering questions about products, such as ‘Why is this one more expensive? How long does the battery last?’ etc.,” she added.

Meanwhile, Trajman, who is also a technology advisor and investor, told The New Stack that what’s important to him is to find the gaps in an organization where an agent would be a good fit.

He would like to “have AI build my agent,” he said, and “Build an agent that helps identify relevant data as it executes its tasks.”

BAML

Vaibhav Gupta, CEO of Boundary, discussed in his session how to reliably develop LLM-based applications.

Boundary sponsors BAML, an open source, domain-specific language to generate structured outputs from LLMs. Boundary monitors BAML traffic to help developers improve the performance and reliability of LLM output.

“With BAML you can build reliable agents, chatbots with RAG [retrieval-augmented generation], extract data from PDFs and more,” Gupta told The New Stack.

“Developers create functions for LLMs, such as taking a user query and producing a list of changes to make to the UI,” he added.

“Our goal was to build tooling that allows a traditional developer to iterate fast and use the tooling they use for other applications, thus creating the intersection, or boundary, between old and new worlds.”

Boundary hosts a SaaS platform to collect application data for developers, Gupta added. “You want to observe different parts of the application and find where the chatbot is failing,” he said.

Evaluating AI Agents

Among the often-repeated messages at the conference is that an AI agent needs to be trained and evaluated, iterating the training and evaluation process to ensure it performs as expected.

Agent evaluation can also help resolve GenAI issues of hallucinations, incorrect results and poor quality replies, said Ilan Kadar, co-founder and CEO of Plurai.

To put it in context, creating an agent based on OpenAI, “is like bringing in the equivalent of a PhD student,” Kadar told The New Stack.

“It’s not enough to hire the student, though, you have to get them to understand the organization and the business,” he added.

Plurai’s IntellAgent platform evaluates AI agents by first training them and then generating evaluation scenarios to test the effectiveness of the training.

“The idea is to have the PhD student eventually become a member of the senior staff,” Kadar said.

IntellAgent generates synthetic data to train the model and then generates evaluation scenarios to determine the effectiveness of the training. “Other tools don’t generate the scenarios,” Kadar said.

IntellAgent iterates the evaluation process to confirm training results, find errors and optimize agents for high performance and reliability.

Text to SQL for Agents

AI agents require access to data for training purposes, and accessing data is one of the major functions of an AI agent.

Databases add vector data types to store training results and offer text-to-SQL generative capabilities to support the second.

Mike Freedman, co-founder and CTO of Timescale, spoke at the conference on Timescale’s new text-to-SQL capability.

Freedman told The New Stack that the fundamental issue for AI agents is not how to generate accurate SQL queries from text, but generally “how to support human language for data analysis.” In that context, SQL represents an intermediate form, he said.

Freedman talked about Timescale’s new text-to-SQL capability, which helps create more deterministic queries on nondeterministic data (i.e., LLM data).

“The interesting thing is the end-to-end pipeline,” Freedman told The New Stack. “It’s a broad problem generally — we want to ask questions of data in human language” while ensuring accuracy.

Freedman sees two parts to successful text-to-SQL generation:

1. A semantic catalog that adds context to the questions and also enables semantic search. The catalog helps the LLM understand the data available to it.
2. Evaluation, which uses the query planner part of the query optimizer to evaluate the generated SQL.

Timescale is a PostgreSQL-compatible native database, so the generated SQL should work with any PostgreSQL-compatible database, Freedman said.

Agents Need More Than Vector Search

Access to data, and many types and varieties of data, is essential for an AI agent to take appropriate action and return meaningful results.

LanceDB is an open source database that supports the Lance Format, which is a multimodal format designed for LLM training. Program chair Lorica called it the “Parquet” of AI.

LanceDB CEO Chang She said the problem they are solving is the multimodal problem, where AI data is taken from multiple sources, such as audio, video, text, images and so on.

She told The New Stack that “AI agents must handle complex data types at large scale — from agent state and vector embeddings to documents, images, audio and video. Traditional databases aren’t designed for this kind of multimodal workload at scale.

“To solve this issue, LanceDB offers a unified platform to store, search, analyze, train and process all your AI data,” She said.

For example, if you’re building an LLM-powered data pipeline to power an agent for a school, you need a lot of different types of information from the school, he added, such as the class schedule, student and teacher data, assignments, images and classroom videos.

The problem is especially applicable when you’re building agents and you don’t want to spend your time managing low-level details of how to access and combine all these different types of data, She said.

A LanceDB table provides one format for all types of data, so you can search across the types of data and train models with it.

“We’ve replaced four things with one, and it scales to zero,” She added.

“Accessing data is simple until you get hallucinations and need an evaluation loop. We are therefore starting to manage a lot more metrics and history,” She said.