Sales agents get client questions they need to answer the same day. The answer is usually somewhere in the internal knowledge base, but finding it takes time they do not have.
Sales agents needed answers without going to a product expert or another team. They forward these to clients, so they also had to be able to check the answer before sending it. Every answer comes with a link to the source it came from.
After signing in the agent gets a chat screen with suggested questions. No product selector, the classifier works that out from the question. Answers come back with links to the articles behind them, and follow ups keep context, since "what about for the second one" needs the earlier turns.
If an SME or team is listed for that domain, the answer names them too.
The model calls for what it wants to read instead of getting everything in one prompt, so it can judge whether it has enough before answering. If not it retries with different content, up to three times, then says it does not have enough information. I wanted it to be able to give up.
The same questions come up constantly. Two caches sit on this path, mainly so the same question gives the same answer twice.
The field that matters is the corpus version. An ingestion run bumps it and retires everything cached for that product. Without that the app would keep citing a page that has since changed.
The key is an exact hash, so differently worded versions of the same question both miss. Matching on meaning is the next step and I have not done it.
Plain chunk search handled most lookups but failed on two kinds of question: answers spread across a whole document, and questions where you need to know a document exists before you can find anything in it. So I ran three searches together, each strong where the others are weak.
| Retrieval | Good at | Falls over on |
|---|---|---|
| A. Article routing | Whole document questions, and "does this product do X at all" | Cost and speed, you pay for full files |
| B. Chunk search | Precise lookups where the answer is one paragraph. Cheap, fast | Answers split across sections, wording unlike the source |
| C. OKF concepts | Definitions and stable facts. Curated, so it is clean | Anything not yet written up as a concept |
OKF is an open specification Google Cloud published in June 2026, currently v0.1. It standardises the pattern of keeping a curated wiki that an LLM reads.
type, title,
description, resource, tags,
timestamp.okf/ claims-adjudication.md eligibility-window.md formulary-tier.md --- one concept file --- --- type: concept title: Formulary tier description: How a drug is placed into a cost tier. resource: kx://products/pharmacy/formulary tags: [pharmacy, benefits, pricing] timestamp: 2026-07-14 --- A formulary tier determines member cost share for a drug ...
Plain files are easy to work with. Grep them, diff them, keep them in git, fix a wrong concept by hand in a pull request. The crawled corpus only says what the source pages say, so the bundle is where we record what is correct.
Most of the retrieval quality comes down to ingestion. Each product is a tree of linked articles with spreadsheets and decks attached, and crawling it naively loses that structure.
I was given access to the sales agent mailbox. A few months of real client questions, each already paired with the answer an agent had sent back.
Accuracy tells me whether a change helped. The gap analysis has been more useful day to day, since it shows what the corpus cannot answer at all: a section never crawled, a parser dropping content, a concept nobody wrote up.
The ground truth is whatever an agent wrote at the time, usually in a hurry. When the judge disagrees, sometimes the agent was wrong. A low score is something to go and check.
The first judge was not repeatable. Same set twice, two different accuracy numbers, which makes the harness useless. Getting it stable took longer than building the rest of the test bed.
Temperature is zero throughout, which helps but does not settle it. Most of the remaining drift came from batching and from scoring in free text.
Failures here are usually silent, so I built admin pages to show what state things are in.
| Page | What it is for |
|---|---|
| Feedback review | Shows each comment along with the conversation it came from, so you can see what actually went wrong instead of just a rating. |
| Ingested content | What has been crawled and when, with a tree view for the folder layout and a graph view for how articles link to each other. Missed and orphaned branches show up in the graph view. |
| Test bench | Runs a question against a chosen part of the corpus and shows what each retrieval returned, without going through the chat UI. |
| Analytics | Conversation counts, active users, feedback volume, and the same figures plotted over time. |
Each page has its own permission rather than one admin flag. The feedback page shows real client questions, so it needed tighter access than analytics.
Two setups. The first runs today for internal testing, roughly 20 users with 10 active at once. The second is where it goes as usage grows. Model calls in both go through Evernorth's internal AICoE gateway.
One question is not one call, so the numbers above come from the fan out rather than from traffic at the front door. 8k daily active agents asking about 4 questions each is ~32k a day, which lands in an 8 hour window at roughly 1 req/s average and 3 at peak. The answer cache absorbs a bit over half, so ~1.5 req/s reach the pipeline. Each of those is a classifier call, a routing call and a thinking pass that can retry, so about 5 model calls a second.
The pods are the easy part. At ~18s per uncached answer that is under 30 concurrent requests, which two pods would carry, so 6 is really about headroom and restarts. The number that actually constrains this is the token rate, around 3.5M a minute at peak. That is a quota to agree with the AICoE team before launch, and it is the reason the cache and the cheaper classifier model matter more than pod count.