Insights from Goodeye Labs

In November 2025, AI coding tools went from halting and clumsy to surprisingly capable. We gave 22 models the same prompt and ran five replicates each to make the shift visible.

Generating code is easy now, but more code has never meant better products. As production costs fall toward zero, the bottleneck shifts to judgment and taste. This is data science.

The Anthropic Economic Index shows that the quality of what you put into AI almost perfectly predicts the quality of what you get out. If AI mirrors expertise, so must your AI evaluations.

Learn how to build reliable AI systems using LLM evaluations. This talk covers why traditional testing breaks with stochastic systems, how generic LLM-as-Judge approaches miss domain nuance, and practical steps to implement contextual evaluations that actually work.

In 2025, we discovered we'd been measuring memorization, not intelligence. Models scored 80-90% on static benchmarks but dropped to 60-70% on truly novel problems. This year exposed the fundamental crisis in AI evaluation, and taught us what to build instead.