Agents can spend test-time compute by trying, observing, and revising. We derive an Elo reference for repeated sampling, then show that in a 2022 two-week coding marathon, current agents plateau within 24 hours while top humans keep improving.
The derived Elo is a great tool to isolate whether agent loops are actually “reasoning” or just brute-forcing the search space.
It clearly proves that current agent scaling (via basic try-observe-reflect loops) quickly plateaus because it lacks the human capacity for abstract conceptual shifts and structural refactoring over long-horizon tasks.
I believe the future of test-time compute in the agent era shouldn’t just be about scaling trials or running more iterations; it should be about building architectures capable of hierarchical planning that can dynamically pivot their entire strategy when stuck.
The derived Elo is a great tool to isolate whether agent loops are actually “reasoning” or just brute-forcing the search space. It clearly proves that current agent scaling (via basic try-observe-reflect loops) quickly plateaus because it lacks the human capacity for abstract conceptual shifts and structural refactoring over long-horizon tasks. I believe the future of test-time compute in the agent era shouldn’t just be about scaling trials or running more iterations; it should be about building architectures capable of hierarchical planning that can dynamically pivot their entire strategy when stuck.