The Agents on Rails Files: enter lemans, a Ruby harness for fair agent races

Authors: Vladimir Dementyev, Principal Backend Engineer, Artur Petrov, Backend Engineer, and Travis Turner, Tech Editor Topics: Rails, AI, Open Source, LLM benchmarking, Ruby, LLMs A tour of lemans, the open source Ruby harness behind Agents on Rails: eval task anatomy, how it runs and grades coding agents for hours, and how cheaters shaped its guardrails. Benchmarking coding agents in one sentence: give a model a task, let it work, check the result. The hard part is keeping the race fair. Ev...