An AI feature can return a perfectly formatted record containing the wrong delivery date.
The software accepts it. The customer doesn’t.
That distinction matters after OpenAI’s August 6, 2024 launch of Structured Outputs.1 The feature improves the reliability of responses conforming to supported developer-supplied JSON Schemas. It is a useful improvement over simply asking nicely for the right shape.
The shape is only part of the job. A beautifully labeled suitcase can still belong to somebody else.
Separate a readable record from a correct one
JSON is a structured data format applications commonly exchange. A schema describes expected fields, types and permitted structures.
For a fictional equipment-hire inquiry, the application might need a requested item, date, location and contact details extracted from an ordinary message.
Valid JSON doesn’t establish the agreed structure
OpenAI’s November 2023 DevDay announcement introduced JSON mode.3 That helped with syntactically valid JSON, but valid syntax alone doesn’t guarantee the structure an application expects.
A response can be valid JSON while omitting a required field or using the wrong type. Structured Outputs addresses schema conformance through the supported API mechanisms and schema capabilities described in the launch material.
That can remove a frustrating class of integration work. The application no longer has to treat every response as a potential surprise arrangement of brackets.
Check the actual supported model and mechanism rather than assuming every earlier JSON feature provides the same behavior.
A conforming date can still be the wrong date
The customer might write next Thursday without enough context to determine the intended calendar date confidently. A schema can require a date-shaped value. It cannot supply the missing business fact.
Design an explicit way to represent uncertainty or request clarification. Do not force the model to fill every field when the source doesn’t contain enough information.
We build AI agents with clear jobs and sensible limits. See how.
Likewise, requested equipment and confirmed available equipment are different facts. Extracting a request should not quietly turn it into a reservation.
A neat record can make an assumption look more authoritative than it was in the original message. Preserve the distinction before the assumption gets a booking number and starts making travel plans.
Validate the business action separately
The receiving application still needs to apply its rules. Schema conformance is not authorization, inventory validation or approval.
That separation is useful even when the extraction performs very well.
Check against authoritative information
Verify that an item exists in the catalog, that relevant dates are permitted and that the user has authority for the intended action.
If the application uses extracted information to prepare a draft inquiry, say that. If it can commit a booking, the confirmation and control requirements are more substantial.
For the equipment example, I would first let staff inspect the proposed record alongside the source message. That makes disagreement visible and helps reveal the cases the design missed.
A person should be able to correct a value without losing the rest of the useful extraction.
Related read: Changing AI providers takes more than swapping the key.
Treat unusual responses as ordinary operating conditions
A request may be refused or a response may be incomplete. The application needs to recognize those outcomes instead of assuming every call returns a complete business record.
Do not retry indefinitely until something fits the parser. A response can become structurally acceptable without becoming more justified.
Hamel Husain’s domain-specific evaluation approach is relevant here.2 Judge the task the business is trying to complete, not only whether the machine produced the requested format.
For this feature, I would include checks for:
- Missing or ambiguous dates and locations.
- An item name that matches several catalog entries.
- Contradictory information inside the same request.
- A record that conforms to the schema but misstates the source.
- Refused or incomplete responses and the resulting recovery path.
That gives the team evidence about both the extraction and the workflow surrounding it.
Structured Outputs is a useful building block. Pretty cool when it removes a recurring formatting headache. We should just avoid thanking it for responsibilities it hasn’t taken on.
Start with an explicit schema, a clear uncertainty path and a small set of representative requests. Then compare the resulting records with the original messages and the authoritative business data.
The feature has earned more trust when the awkward cases are handled clearly, not merely when the pleasant ones return valid JSON.
Let the schema check the suitcase label. Before sending it to the customer, we should still look inside.