Finding the Right Data Architecture for RAG Pipelines

It’s become a ubiquitous belief in the tech community that pooling data into a central repository like a data lake is the best approach for data management, and it’s true that this architecture helps to eliminate data silos and makes governance less complex. However, it doesn’t address the main obstacle when building generative AI pipelines: data movement.

Many of the most powerful enterprise use cases for generative AI require timely access to internal company data. Retrieval-augmented generation (RAG) and vector databases have gotten most of the attention when it comes to enabling these applications, but these aren’t the only requirements. It’s also critical to have access to timely and accurate data.

Imagine a travel website that provides a generative AI assistant that can help customers plan holidays and book hotels and flights. This virtual assistant requires up-to-date information from a variety of sources to provide accurate information. And it needs to respond in real time, because information like hotel availability and special offers change frequently.

Data Lakes vs. SaaS Apps vs. Data Streams 

Most data lakes and other centralized data repositories work on the principle that data “lands” somewhere and is then prepared for use, but this process is too slow for intelligent assistants like the travel example above. Many generative AI applications need data delivered to an LLM model prompt in real time, and data warehouses and data lakes are not suited for this.

Data stored in SaaS applications is even less shovel-ready for generative AI. For example, the leading customer relationship management (CRM) vendors are competing to “own” the entire customer journey. But for many generative AI use cases, this data will need to be combined with non-customer data to answer questions about the business. Storing data in a SaaS platform doesn’t allow for the rapid data movement and unification that’s required for generative AI.

It’s not that these architectures aren’t useful. Cloud data lakes and SaaS applications clearly can yield significant business benefits. But by themselves, they aren’t suited for feeding the RAG pipelines that feed generative AI applications the contextual data they need to produce accurate, timely results.

The approach that works best for these applications is event streaming, because it’s designed to circulate feeds of data around a company in real time. In the travel example above, the system needs to retrieve data from a variety of systems to arrive at the right answer, including proprietary apps, SaaS applications, databases and other repositories. The data in these systems may be normalized or not well-contextualized, and the problem is that these systems are meant to be queried interactively in real time and the data can’t easily be consolidated. Other use cases include multiplayer video games, personalized shopping recommendations, ride-sharing applications and stock trades, where providing real-time information and updates based on context is critical to the user experience.

An event-streaming architecture addresses this problem because it’s designed precisely for this type of work. It constantly gathers updates from individual systems and presents them as real-time data feeds that can be delivered to an application. When all these data sources are presented as a unified view, they can be connected to each prompt and help the generative AI interface provide the right answer.

How To Get Started

As generative AI use cases expand, more AI startups will be integrating real-time enterprise data into their offerings. To help startups learn the skills needed to integrate real-time data streaming technology into their AI applications, Confluent is launching an AI accelerator program. This 10-week virtual program includes courses, early access to Confluent technology and mentorship, Confluent Cloud credits, and other support.

As chatbots and other virtual assistants become more widespread, this need to combine data in real time from multiple sources will increase. RAG supplies the pattern for feeding contextual data into an AI application, but an event-streaming platform ensures that all of the most valuable data can be supplied to LLMs and deliver compelling experiences.