Agents on Rails: Stage 2. Can a model ship a feature?

Today we’re expanding Agents on Rails with Stage 2, a new set of benchmark tasks that exercise a model’s ability to deliver real-world features, more in line with how a developer would actually work. Before we go deep, here are the results: First results for Stage 2: Most accurate: GPT-6 Astra, with 35% of runs solved (21 of 60). Cheapest: GLM 5.3 Flash, 13% for $18.78 across all 60 runs. (Luna’s $1.69 would win, but it finished 0 of 60 tasks.) Fastest: GPT-6 Astra again, 9 minutes median per...