Case studies in measuring whether systems work.
Quality, assurance, security, reliability and integration of the systems people depend on, AI systems included, with the numbers that mattered and the ones that didn’t.
Read the latest case studyPrecision of one anomaly detector under two alert rules
Flag a fixed percentile 0.12
Flag the top 20 (alert budget) 0.60
Recent writing
All posts- The decision gate for agent-proposed code AI reviewers leave comments and policy engines return deny, but neither reads a diff and decides on it. A gate that routes sixty agent-proposed changes into applied, refused, or one reviewer item — with the model kept strictly advisory.
- The tests never reached the bug: evaluating a legacy migration done by AI A pipeline scored a perfect parity match and failed the artifact anyway. Here is why, and why a confidence score should be arithmetic a person can re-run.
- A small model against a large one on a narrow task: what the mistakes showed A fine-tuned small model against a large one on the same governance task. The headline gap was inside the error bars. The useful signal was which way each model got things wrong.
Topics
All writing- Quality Testing, evaluation, metrics and data quality: whether a system does what it should. 1 post
- Assurance Evidence, verification, governance and change control: showing that a system can be trusted. 2 posts
- Security Data protection, access, credentials and the boundaries between components. 1 post
- Reliability Performance, diagnosis, incidents and resilience under real load. Coming up
- Integration Wiring systems into the systems around them, and scaling past the demo. Coming up
- AI systems The same questions asked of AI: model evaluation, agents, context engineering and open models. 4 posts