Beyond Prompt Engineering: Governing Prompts and AI Models

The practical realities of securing the language models supporting textual applications of generative artificial intelligence are formidable. In addition to implementing, fine-tuning, and prompt engineering these models to yield optimal results, savvy organizations are struggling to govern their interactions with users to preserve data privacy, regulatory compliance, and data security protocols.

Anonymizing and securing data while confining models to internal sources, via constructs like Retrieval-Augmented Generation (RAG), is only half the problem. The larger concern is the ability to govern not only what information users insert in prompts when deploying those models, but also the responses the models elicit. Numerous other factors, including the amount of language models organizations use, the range of applications they support, and the real-time nature in which employees or customers interact with them, seemingly confound traditional governance efforts.

However, with an artful combination of centralized data security governance administered through a decentralized framework inside sources, users can achieve remarkable results while adhering to privacy, compliance, and access control mandates.

According to Piet Loubser, Privacera SVP of marketing, contemporary solutions “declare you will have a library of all your applications and the derivative models underneath that. We can know all the policies that are created upon them, and then we will start tracking all of the interactions and behaviors that you find for the users.”

Moreover, when the governance system detects abuses of privileges, policy violations, and inappropriate content in prompts or model responses, it can modify them — in real-time — to uphold the governance rules that organizations have specified.

Foundation Model Library Governance

The crux of governing user prompts and model outputs across a diversity of foundation models for any number of enterprise applications is the capacity to embed governance tooling within model libraries. From an architectural perspective, organizations might have an array of chatbots, similarity search mechanisms, or other such textual deployments of generative AI accessed by a distributed user base. However, they’ll “utilize LangChain, or [Amazon] Bedrock, or whatever of the other open libraries that connect the chatbot to the backend model,” Loubser stipulated. Users can insert a Privacera agent into their foundation model application development framework of choice (including resources like OpenAI) to propel its data security governance capabilities into that system.

Subsequently, “whatever the prompt is that gets typed in, [meaning] the question, there is an agent in the backend that basically comes to us and says is this okay or not, and is there some treatment that needs to go in,” Loubser revealed. “We literally do a quick peak at the prompt, and the same happens on the way out.” Depending on what policies users have in place, sensitive or regulatory-applicable data can be masked or redacted. It’s also possible to have models deny a request.

Unstructured Text Scanning

Implicit to this paradigm is the ability of the underlying governance system to scan the natural language prompts and language model outputs, and evaluate them for policy violations. Organizations are still responsible for creating policies about how to govern their data, but the enforcement is predicated on the natural language capabilities of the governance engine to understand these unstructured text exchanges. Central to this endeavor is a “pretty sophisticated engine around that, that connects the policy with what we are scanning and identifying from the patterns in the text, as well as the context of whatever questions and prompts you had before,” Loubser added. There’s also a degree of contextual understanding that’s necessary for governing user interactions with language models.

While role-based access controls (RBAC) or attribute-based access controls (ABAC) may be involved in the policy enforcement process, the engine must contextualize the results of what it scans and apply it to the policy and the particular access request. Loubser referenced a hypothetical use case in which credit bureau employees might ask an intelligent chatbot for a customer’s contact information. “If I’m in customer success, I can say what is Kimberley’s cell phone number, and I can get that information,” Loubser commented. “As a marketer, you can ask that question but it’s like sorry, you can’t see it because it’s not in your database. PII isn’t appropriate and the system needs to tell you that.”

Real-Time Applicability

It’s imperative that the engine scanning language model prompts and the ensuing responses operate with low latency. Inordinate delays could result in a lack of adoption and squandered Generative AI investments. According to Loubser, Privacera can scan prompts, summarizations of texts, and other outputs of models “in real-time without any system overhead.” What there are notable limitations for, however, is the scale of the unstructured text the engine can parse. “In this context, you’re not throwing it like four terabytes of a prompt and deciphering it in real-time and doing something about it,” Loubser mentioned. “That’s not the way this works. It’s typically questions, answers, questions, answers, in small snippets. Like, a couple [of] hundred characters.”

The engine responsible for such operations typically harvests metadata in data catalog and data governance solutions. The key is applying it to fully unstructured textual data — as opposed to semi-structured data — and how much context it gleans. In this respect, the engine can evaluate model outputs and incoming prompts according to hundreds of pre-built expressions the solution will “identify based on contextualizing what we see within the question,” Loubser explained. “It will try to identify that you’re talking about a person. It will identify whether you’re asking a question about PII or a social security number.”

Reverberating Effects 

The overall significance of Privacera’s governance capabilities for language models is multifaceted. The solution centralizes secure access controls for the entire gamut of models that organizations might need by embedding itself within application development frameworks that models connect to.

Moreover, it utilizes its own understanding of unstructured text to gauge the context of prompts and model outputs, evaluate them according to policies and classifications, and implement controls in real-time to preserve governance conventions. The result reduces the risk of implementing the most in-demand technology of our time while further broadening its utility to make it more suitable for a wider array of use cases.