Agents on Rails: the first benchmark report

TL;DR We ran 8 models against 21 atomic Rails tasks, 3 runs each. Every task runs against Writebook : a bug report, a security finding, a feature request, each written the way you’d actually file it. Read our announcement post for more information about the project. So, as of August 2026, which model is best? Most accurate: Claude Opus 5, 92% of runs solved (58 of 63) * . Cheapest: GPT-5.6 Luna, 73% at its default medium reasoning effort, and all 63 of its runs cost 91 cents combined. Not a t...