Measuring the personal data in a training corpus, for a large social platform

The problem

A large social platform was about to train on a corpus nobody had characterised. Legal needed to know how much personal information was in it. The off-the-shelf scanner returned a number, and nobody believed it, because scanners flag things that are not what they look like and miss things that are.

The proposal on the table was to build a classifier. Months of work, a training set labelled by hand, and at the end of it a number that would still need defending.

Meanwhile six other teams were waiting, because none of them could move until somebody answered the question.

What we did

We stopped trying to read the whole corpus and measured it instead.

A model reads a sample and flags what looks like personal information, and writes down a short reason for each call, so every flag can be checked by a person rather than taken on trust. Human raters then score a slice of those calls. That tells you how often the model is right and how often it is wrong, in both directions. Once you know that, you can correct the machine’s raw count for its own error rate and report a range for the whole corpus rather than a single number nobody can stand behind.

The important part is not the model. It is that the answer arrives with its own error bars and with a trail of reasons a person can audit.

What happened

The measurement cost about $400 per audit cycle. That is roughly one percent of what the proposed classifier build would have cost, and the projected saving against reviewing the corpus by hand was more than $10M a year.

It took a week rather than two quarters, and it arrived with a stated confidence interval rather than a bare count.

The teams that had been waiting stopped waiting.

What it cost them to find out

Six months, if they had built the classifier first. That is the real number. Every week the question went unanswered was a week six other teams could not plan.

What we got wrong, and where this does not apply

A score that says a developer disclosed something about their training data is a measurement of their paperwork, not of their corpus. Stanford’s transparency index scored ten major developers against 100 indicators in 2023 and the average came in at 37. The 2024 round widened to fourteen developers and the average rose to 58. Both are precise measurements of reporting. Neither tells you what is in the data.

Measurement has an answer. It just does not look like a checkbox.

LLM-as-judge for PII prevalence: the economics, on Jeff’s personal site.