OpenAI’s ChatGPT Now Formats Output to Developer Queries

OpenAI, the company behind the popular ChatGPT Large Language Model Generative AI service, has released a tool that will return query results in the exact format specified by users.

This has been the most-asked for request among developers using the AI technology, according to the OpenAI CEO Sam Altman.

Since its debut in November 2022, the ChatGPT service has been harnessed by countless application developers for agent-driven and other apps. But while ChatGPT uses troves of unstructured data to deliver its results (via API-driven function calling), application developers would far prefer to have these results delivered as structured data, so they can be better parsed by their respective apps.

Last year, OpenAI expanded its API to include results delivered in JSON. This week, the company goes one step farther, revealing a new API feature called Structured Outputs, which will deliver the results in whatever JSON schema the developer specifies in the query.

“Developers have long been working around the limitations of LLMs […] via open source tooling, prompting, and retrying requests repeatedly to ensure that model outputs match the formats needed to interoperate with their systems,” wrote Michelle Pokrass, OpenAI technical staff member, in a blog item posted Tuesday.

“Structured Outputs solves this problem by constraining OpenAI models to match developer-supplied schemas and by training our models to better understand complicated schemas.”

How Structured Outputs Improves LLM Quality

Structured outputs differ from those generated by plain user prompts in that they are constrained as to what information they can provide, a technique known as “constrained sampling” or “constrained decoding.”

“In order to force valid outputs, we constrain our models to only tokens that would be valid according to the supplied schema, rather than all available tokens,” Pokrass explained.

For ChatGPT, this additional step of filling in the schema improves its accuracy.

The OpenAI developers found in tests that ChgatGPT was able to fill a pre-defined schema correctly 100% of the time, but would only provide the correct responses 85% of the time through plain prompts.

How to Produce Structured Outputs

Now, when their apps submit a request, developers supply a JSON schema, via json_schema, which is a new option for the response_format parameter. The strict value must be set as “true” within the function definition. The model’s output would then best match its answers to the schema (this works both for AI tools and for direct user inquiries).

The responses will still abide by OpenAI’s safety requirements — which blocks potentially harmful content — and will return a refusal string value to questions it won’t answer. And there are other limitations as well: It supports only a subset of the JSON schema. It won’t prevent mistakes in model definition, and there is an additional latency in the first response, as the ChatGPT sorts out the developer’s schema.

Both the Node.js and Python software development kits from OpenAI have been updated with the new response_format parameter.

The biggest possible use case is, of course, formatting unstructured data into structured data, so it can be ingested and analyzed by a relational database system. which has long been a challenge for organizations with lots of information stashed away in office documents.

But Pokrass elaborates on some possible advanced uses of the technology, including creating user interfaces on the fly based on user input and providing a single answer without the supporting material.

This is a “great feature and much needed,” machine learning researcher Elvis Saravia noted on X. Saravia worked up a tutorial for YouTube on how to use Structured Outputs: