A client asks a sales agent something about a product. The answer is almost always sitting in our Salesforce knowledge base. Getting to it takes longer than the question is worth, so the agent asks a person instead. We were asked to build the thing that does the looking.
Nothing existed. No app, no pipeline, no cleaned up copy of the content. We had the products, we had the knowledge base, and we had a chat window to build. An agent types a question, gets an answer, and can see where it came from before forwarding it on.
That last part shaped a lot of what follows. An agent is going to paste this to a client. A confident wrong answer is worse than no answer.
Knowledge Exchange is our Salesforce knowledge base. Per product there are a few seed articles. Each of those links out to more articles, and those link further. Hanging off the articles are attachments, and the attachments are usually where the real answer is.
For the proof of concept we were told to crawl it. So there is a crawler, driven by Playwright, that signs in with its own role. It only sees what that role is allowed to see, which keeps the crawler inside the same content boundary the app should have.
From each seed it walks the links down to a set depth. Page content becomes markdown. Each attachment type gets its own parser. Then a small model tidies the output into consistent markdown, because a table lifted out of a pdf and a table lifted out of a deck do not come out looking anything alike.
Each product has subject matter experts, and they are the people who know when the bot is wrong. So the ingestion side got its own UI rather than staying a set of scripts I ran.
An SME can see everything the bot currently knows for their product as a tree, add a page we missed, remove one that should not be in there, ask a test question against it, and leave feedback in the same place. The cron lives on that page too: what ran last night, how far it got, what changed, what failed, the logs.
Without this page every correction would have arrived as an email to our team, and we would have been the bottleneck for content we do not understand as well as they do.
A small model reads the conversation first and decides whether the question is even ours. If it is not, we say so. That check is cheap and it stops the expensive model from having a go at something it has no business answering.
If it is in scope, the same step rewrites the question into one or more precise ones. Agents ask things like "what about for the second one", and the rest of the pipeline needs that turned back into a full question.
The first version was ordinary retrieval. Embed the question, pull the top matching chunks out of an Azure AI index, hand them to a model, get an answer. It worked for a good share of questions and fell over on the rest in a way that took us a while to see clearly.
| Question | Why top k could not do it |
|---|---|
| How many rows in this sheet match X | Retrieval brings back the rows that look most like the question. It has no idea whether it missed any, and neither do you. |
| Anything spanning two sheets | The answer only exists once you put both together, so it is never sitting in one chunk waiting to be found. |
| Anything spread across a deck or a long doc | A slide out of order loses the thing that made it mean something. |
| Does this product do X at all | Nothing written down looks identical to a bad search. Both come back empty. |
The pattern underneath all of it is the same. Top k gives you the chunks that most resemble the question. That is exactly right for "what does the policy say about X", and exactly wrong for anything that has to see every row before it can answer.
Two changes fixed most of this, and they are easy to confuse with each other, so it is worth separating them.
We moved the curated side to OKF, the Open Knowledge Format, an open spec Google Cloud published in June 2026 at v0.1. A bundle is a directory of markdown files, one per concept, with a few structured fields in YAML frontmatter and ordinary markdown links between files, so the bundle reads as a graph.
okf/ claims-adjudication.md eligibility-window.md formulary-tier.md --- one concept file --- --- type: concept title: Formulary tier description: How a drug is placed into a cost tier. resource: kx://products/pharmacy/formulary tags: [pharmacy, benefits, pricing] timestamp: 2026-07-14 --- A formulary tier determines member cost share for a drug ...
Plain files are easy to live with. Grep them, diff them, keep them in git, fix a wrong concept by hand in a pull request. The crawled corpus can only ever say what the source pages say, so the bundle is where we write down what is actually correct.
Worth being precise about one thing, because I got it wrong in my own head for a while. OKF does not retrieve anything. It is a format, not a retrieval system. It made the raw material clean and consistent, which helps, but on its own it does not answer a question.
The part that actually changed the answers was giving a thinking model tools and letting it go and get things itself. Much like an assistant in an IDE, which does not embed your whole repository and hope. It greps, opens a file, runs something, reads the result and tries again.
So the model can list and open concept files, follow their links, run a chunk search when a plain lookup is all that is needed, and query a spreadsheet. That last one matters most. Sheets stay as sheets rather than being flattened into markdown. The model writes a query, a real query engine runs it across every row, and the number that comes back is the actual number. The model does the understanding, the query does the counting. Those are two different jobs and they were going wrong together.
Chunk search did not go away. It stopped being the whole architecture and became one tool among several.
An agentic loop is more model calls than a single retrieval pass, which shows up in both latency and the bill. The thing that took the most out of that is prompt caching. The stable part of the prompt sits at the front, the tool definitions, the system prompt and the concept index, and the question goes at the end. A cached read is far cheaper than a fresh input token and the first token comes back sooner.
It has sharp edges. A cache write costs more than a normal token, so it only pays if the prefix gets reused before it expires. The window is short. Matching is on the exact prefix, so reordering anything near the front throws away the rest. Keeping that prefix genuinely stable is the whole trick, and it is a real constraint on how you assemble prompts.
We could not settle which model to use by arguing about it. Every answer carries a rating and a comment box, so I built a small framework around that. Run one model for a window, watch the feedback along with latency and cost, then switch and watch the same thing again.
The useful side effect is that going back is normal. If a swap makes things worse we return to the previous setting the same way we moved off it, rather than treating it as an incident.
This is what we are building now, and it is the part I am most interested in. The same idea, but the model is not the only thing you can turn.
The knobs are the model, the retrieval approach, the ranking, and where the thing runs, including moving off OpenShift to a cloud or back on premises. Set them, feed in real feedback over a period, watch quality and latency and cost, then keep the setting or drop back to the last one.
The reason to build it properly rather than keep it as our own scripts is that this field moves faster than any one team can follow. Something better lands every few months. If moving to it is a setting rather than a rewrite, every application can pick it up quickly instead of one team at a time.
Sign in is Okta. The app runs on OpenShift. Model calls go through our internal AICoE gateway. Conversations and feedback sit in Postgres, chunks in Azure AI Search.
My side of it was sizing the pods, the rate limiting at the gateway so bursts stay inside the model quota, the caching, and the dashboards. Two sets of those, because the application and the ingestion fail in completely different ways. An app failure is loud. An ingestion failure is silent, and you find out weeks later when a question that should have been easy cannot be answered.