Adrian Cockcroft on AI-Powered Performance Tools and Why P99 Metrics Fall Short

Adrian Cockcroft has spent four decades building and optimizing systems at companies including Sun Microsystems, Netflix, eBay, and Amazon. At P99 CONF, the performance-focused conference now in its fifth year, he offered a pointed critique of the industry's reliance on P99 metrics while exploring how artificial intelligence is reshaping performance engineering.

From kernel deep dives to AI-assisted tooling

https://www.youtube.com/embed/b4w69RjGF8I?feature=oembed

During his time at Sun, Cockcroft tackled performance problems by examining system metrics directly. He recalled the challenges of that era: "Back in the old days with Sun, people would look at the output of system metrics in vmstat or whatever, and they'd be guessing what the numbers meant. There was a very vague understanding of what these things meant. The manual page wasn't very clear."

Rather than accept incomplete documentation, Cockcroft read the kernel source code itself. "I went and read all the kernel source code and figured out exactly where these numbers came from, exactly what they meant, which ones were approximating what, and wrote all that down." This work resulted in two books: Sun Performance and Tuning and Resource Management.

Modern systems offer far more sophisticated observability tools for end-to-end tracing, yet Cockcroft remains focused on what those tools miss. "Everything sort of looks okay in the tools – but the system isn't behaving well. I usually come in and try to find a new way of looking at the data. A new type of analysis, or go a little bit deeper or finer grain, or stop looking at averages and start looking at distributions, and find all kinds of interesting things that nobody knew were happening."

Today, Cockcroft uses what he calls "vibe coding" to build custom analysis tools, leveraging large language models to accelerate development. The productivity gains are substantial. "My speedup is infinite, because this code would never exist without these tools. I wouldn't have the time to build them," he said.

The case against percentiles

Cockcroft has spent over a decade questioning the utility of percentile-based metrics. He argues that percentiles obscure rather than illuminate the behavior of modern web services. "Percentiles don't work when trying to understand the latency and performance of modern web services," he stated.

A single percentile value like P99 cannot reveal whether response times follow one distribution or multiple distinct patterns. When multiple peaks exist—a common scenario in production systems—traditional statistics lose their explanatory power. Consider a service with two response time modes: rapid responses from cached data and slower responses requiring computation. As the cache hit rate fluctuates, each mode's latency remains constant, but the peaks shift in height. "Your averages and your P99 are changing all over the place, but all that's really happening is your cache hit rate is changing," Cockcroft explained.

Cockcroft developed a statistical approach to analyze response time distributions more effectively, identifying multiple peaks and tracking their movement over time. When ChatGPT became available, he used it to implement the method in R, a language he had not used recently. The resulting tool is open source. More details appear in his article "Percentiles Don't Work" and his talk "A Tale of Two Histograms," which opens with the line: "It was the best of response times, it was the worst of response times…"

Advice for performance teams

When asked what guidance he would offer teams building high-performance systems, Cockcroft recommended starting with a broad view to identify areas of concern, then progressively examining finer details. "Remember the microscope that you got when you were a kid," he said. "First, you have to focus it using the lowest resolution, at 10x, and then you can click it to 100x and adjust that, looking at just one speck now. Once you get that in focus, you click it to 1,000x."

Cockcroft's career exemplifies the value of building specialized tools to surface hidden performance problems. As agentic AI tooling becomes more accessible, similar approaches may help other teams identify and resolve latency issues. P99 CONF 2026 will take place October 21-22 as a free virtual event.