Blue Camel Consulting
Wrenfield Navigator answer drift monitor
Answer drift monitor

Do your agents' answers still hold?

Wrenfield Health is invented, and so is every answer, member and number on this page.

answer drift monitoring · 6 stories · one month of traffic each
Wrenfield Navigatorbenefits assistant
Is my daughter's inhaler covered before we meet the deductible?
Yes. Preventive inhalers are covered at $15 a fill before the deductible on your plan, and a 90 day mail order supply is $30. 2026 Wrenfield Plus plan guide, p. 14
Gradedanswer matches approved source stamp current
And if I use the urgent care on Mill Street?
The monitor reads answers like this one against a written standard. It does not write them.

Distribution monitors tell you the traffic changed. Answer drift monitoring tells you whether what your agents say still holds. One written standard, applied again and again to what the system produces, with an agreed ladder of responses for when part of it stops holding up. The method works on anything whose outputs a trained person can grade.

The screen alongside is the work being watched. A member asks about coverage, the assistant answers, and that answer either matches the approved source or it does not. Answers like it are graded through the month and rolled up into a card somebody can be handed.

Each story splits the month two ways: how much of the change came from the traffic mix moving, and how much from the same questions starting to get worse answers. A monitor watching only the input can see the first half and has no way to see the second.

Open the quiet month
The stories

Six ways an answer stops holding

Each story is one month of traffic with a single problem introduced partway through, and the monitor's own record of whether it noticed. Open any one in place. Start with the quiet month: an alarm only earns the right to interrupt someone once you have seen what calm looks like.

A quiet month(S0, baseline)The control month. It shows what the monitor does when the AI is behaving, which is what gives an alarm the standing to interrupt someone.What it establishesBaselineread the month
The question mix shifts(S1, query-mix shift)The headline number quietly stops describing your traffic. Reporting carries on unchanged while the questions you now answer most are the ones you understand least.If it goes uncaughtBoard reportingBlind spotread the month
The knowledge base goes stale(S2, corpus staleness)Members act on six month old coverage and cost answers. Wrong copays and closed clinics come back as calls, appeals and rework, and every one of them is paid for twice.If it goes uncaughtMember harmRework costread the month
The retrieval starts failing(S3, retrieval regression)Benefits answers degrade for weeks while the corpus average absorbs the damage. Members learn the assistant cannot be trusted on money questions and route back to the call centre.If it goes uncaughtTrust lossCall centre costread the month
The model gets swapped(S4, model swap)A routine model upgrade ships unsafe triage advice to members describing symptoms. This is the failure that ends a programme, and it arrives on an ordinary release day.If it goes uncaughtMember safetyProgramme riskread the month
The world changes(S5, concept shift)Members are sent to providers who have left the network and quoted prices that no longer exist. The answers never change, so a dashboard that only watches answers stays green all month, and the first person to notice is a member holding a bill.If it goes uncaughtMember harmBlind spotread the month