The last twenty percent

Grit is what you have left when the interesting part is over. I did not believe that until CrisisEcho spent several weeks refusing to work for reasons I could not see.

The setup

CrisisEcho watches eight public data sources and turns them into a map of verified emergencies. A post about a fire becomes a dot on a map in under three minutes — and only if six independent gates agree it is real. There is no human reviewer anywhere in the loop.

The design requirement that shaped everything was not detection. It was trust. A crisis map is only useful if people act on it, and people only act on it if it has never lied to them. A handful of false alarms teaches users to ignore the map, and then it fails precisely when it matters.

So: aggressive enough to catch events in minutes, conservative enough never to invent one, with nobody checking its work.

Where it got stuck

Six gates, each with a threshold. The pipeline was dropping real events and I could not tell which gate was dropping them. I was tuning against intuition and spot checks — change a number, re-run, look at the output, form a theory. It was slow, it was unglamorous, and nobody would ever see any of it.

This is the part of a project that does not appear in the demo, the write-up, or the interview. It is also where the project actually lives or dies.

The fix was embarrassing

I instrumented every gate so that each dropped post wrote down why it was dropped, with the score and the threshold it failed against.

That is it. That is the whole fix. It took an afternoon, it produced no visible feature, and within a day I could see that one gate was responsible for the overwhelming majority of the false negatives and that its threshold was wrong in a specific, correctable direction.

The fix was obvious in hindsight and invisible in the product. That is most of what grit actually is: staying in the boring part long enough to find the thing that makes it not boring.

The generalisation

I now think the single highest-leverage move in a system with any kind of scoring or filtering is: make every rejection explain itself. Not sampled. Not aggregated. Every one, with the number and the bar.

• A pipeline that only logs what it accepted is a pipeline you cannot debug, because the interesting cases are the ones that left.

• Aggregate metrics tell you the gate is dropping 40% of traffic. They never tell you it is the same 40% every time for one reason.

• The cost is a log line. The alternative is weeks of guessing, which is what I paid.

Where else this bit me

The same shape appears in the location resolution: a post is geocoded, or it is not, and for a long time “not” was a single unresolved flag with no reason attached. Splitting it into “NLP found nothing”, “NLP found something ambiguous” and “no geotag and no text signal” turned an opaque failure into three tractable ones.

The uncomfortable conclusion

Everyone can do the first eighty percent. It is architecture diagrams and new tools and the feeling of a thing coming together. The last twenty is an offline reconciliation that works nineteen times out of twenty, a race condition that only appears under real load, and a threshold that is wrong by 0.05 and you do not know which one.

The last twenty percent is also the only part that determines whether the thing works. I have not found a way around that, and I have stopped looking for one.