The Bitter Lesson Scale Beats Cleverness
Rich Sutton's "Bitter Lesson" observes that in AI research, methods leveraging computation consistently outperform methods leveraging human domain knowledge. The lesson is bitter because clever engineering feels productive, but scale always wins in the end.
"We've observed a historical pattern: AI models that leverage domain knowledge consistently get overtaken by those that leverage compute. Today's AI products show striking parallels to this pattern." Lukas Petersson
The pattern repeats across every generation of AI. In computer vision, hand-crafted feature detectors gave way to deep learning. In NLP, parse trees and linguistic rules gave way to transformers trained on raw text. In game-playing, expert systems gave way to self-play reinforcement learning. Each time, researchers invested years encoding human knowledge into systems, only to be surpassed by methods that simply threw more data and compute at the problem.
This lesson extends directly to AI products. Vertical workflows carefully engineered pipelines with guardrails, tool chains, and domain-specific logic initially outperform general-purpose models. But as the underlying models improve, the value of all that engineering effort diminishes. As one YC partner noted, "that first wave of LLM apps [vertical workflows] mostly did get crushed by the next wave of GPTs." The bias-variance tradeoff explains why: rigid, domain-specific systems may be more reliable today, but flexible, general systems have more room to improve as compute scales.
The implication for builders is uncomfortable. The features you are painstakingly engineering around model weaknesses the custom parsers, the retrieval pipelines, the multi-step validation chains are temporary scaffolding. When the next model generation arrives, much of that scaffolding becomes unnecessary. The winners will be those who built for flexibility rather than precision.
Takeaway: Bet on approaches that improve with more compute rather than more engineering, because the compute will always arrive faster than you expect.
See also: Text Is the Universal Interface | Compound AI Systems Beat Monolithic Models | Dennard Scaling Ended and Everything Changed