Blue Camel Fifteen years of turning contested numbers into ones a room can act on: product analytics leadership through a marketplace launch at Uber, then metric reliability and training-corpus measurement at a large social platform, with machine learning on clinical text at a hospital research institute before either of those.
The through line is measurement that holds up when someone asks how you know. A count with no error bars is a guess in a suit. Most of the work is getting the inputs right and then being willing to state, in writing, how confident you actually are.
We put a number in front of legal, or a regulator, or a customer’s security review, and we cannot show our work.
You get a measured answer with its own error bars and a trail somebody can audit, built from a sample that people have checked rather than from a scanner’s raw count. It works on training corpora and on model behaviour, and on anything else where the real answer is a range rather than a point.
We ran a test. Now two teams read the same result two different ways.
Sample sizes worked out properly rather than by eye, and a free-rider measure on every incentive test, because that is the question finance asks eventually. Nothing runs on real money until it has been backtested against held-out actuals.
Nobody trusts the numbers. We find out something broke when a customer tells us.
It starts with the unglamorous half: grain, nulls, reconciliation and tie-outs. The operating discipline goes on top of that, covering which metrics matter, what bound each one has, who gets woken up and what the runbook tells them to do. Most of the value is in the first half.
The training-corpus measurement method, running as a product you can drive. A judge reads a sample, humans calibrate it, and the whole corpus is reported as an interval with a stated error rate.
LiveThe same measurement discipline on medical notes, broken out by subgroup so a fairness question has an answer with error bars instead of an opinion.
The deck said it was a demand story. The dispatch system underneath said otherwise.
A large social platform · 2023Three months from launch, the org had stopped believing its own dashboard. So we put the numbers on call.
A large social platform · 2025The scanner returned a number nobody believed, and the fix on the table took two quarters. We measured it instead, for about $400.
Have a measurement problem? Start a conversation.
The other practices: AI Evaluation & Engineering Custom Build & Delivery Mission-Driven Systems