A confident answer is lovely when it’s right. When it’s wrong, confidence is mostly an accessory.
That’s worth remembering if you’re considering an AI assistant for company information. Before asking how naturally it can explain a policy, check whether it can find the correct policy in the first place.
Retrieval became a more accessible application pattern following the 2023 model and API releases, including OpenAI’s Assistants API beta.2 The basic approach is to find relevant material, supply it as context and ask the model to answer from that evidence.
The conversational part is appealing. I think the search underneath deserves an equal share of the attention.
When the wording doesn’t match
Say a new employee at a fictional supplier asks whether a customer can return an opened item. The approved document might use the phrase unsealed packaging. A strict keyword search could miss the connection. Semantic search aims to find related meaning despite different wording.
Embeddings help by representing content numerically so software can compare how closely pieces of information relate. Simon Willison’s explanation is a useful introduction to that mechanism.1 You don’t need to spend your afternoon admiring arrays of numbers to understand its business purpose.
Related isn’t the same as approved
You do need to know that related is different from authoritative.
A similar return policy from the wrong product range is still the wrong policy. So is a beautifully relevant draft that was never approved. Search quality includes scope, freshness, ownership and permissions, not just finding something that sounds nearby.
Curious what an AI agent could do for your team? See our AI agent services.
Test the search before the answer
I would test the retrieval before adding a polished answer on top. Give it questions staff genuinely ask and inspect the passages returned. Include exact product codes, unfamiliar wording, exceptions and cases where the answer isn’t in the collection.
Traditional keyword search may work particularly well for precise identifiers. Semantic search can help when language varies. Combining approaches may be worth investigating, but it isn’t a compulsory upgrade for every document folder.
The aim is evidence somebody can check. A modest search screen that finds the right instruction is more useful than an eloquent assistant standing proudly beside the wrong cupboard.
Let the answer keep the source’s limits
Once the right passages are available, the answer should preserve their limits. If the policy covers unopened items but says nothing about opened ones, the system shouldn’t fill the gap with a customer-service rule it finds emotionally plausible.
Show the supporting source and make sure the user can access it. Where information is missing or contradictory, a clear admission and a route to someone responsible are good behavior.
If this was useful, try Your AI assistant needs the right context before another clever prompt.
Treat what it reads as material, not instructions
There is a security side to the same design. Retrieved documents are material to read, not instructions the assistant should obey. Willison’s prompt-injection work explains how external text can try to redirect an application.3 Permissions and tool restrictions still need enforcement outside the model.
Start small, with someone looking after it
Start with a small approved collection, an owner who can maintain it and a question set whose answers can be checked. You may discover that tidying the information improves the business before the AI does anything especially impressive.
That’s not a disappointing result. Most of us would happily accept a less mysterious shared drive.
Then add the conversational layer and test it against the same questions. If employees can get dependable answers with less searching, now that is Ayoba! Just keep the evidence close enough that a good sentence never has to stand in for a correct one.