The Agents on Rails Files: designing black-box evals for coding agents

Authors: Vladimir Dementyev, Principal Backend Engineer, and Travis Turner, Tech Editor Topics: AI, Rails, DX, LLM benchmarking, Agentic development, LLMs Tips and tricks on writing robust evals for coding agents that test apps whose code we've never seen: different UI/UX contract strategies, UI mutation testing, browser-based E2E verification gotchas. The Agents on Rails is an LLM benchmarking project governed by the Rails Foundation. They commissioned Evil Martians to design and implement t...