Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers
We treat agents like senior engineers, so why evaluate them like junior engineers?
Senior engineers build features without over-specified requirements
Senior SWE-Bench feature tasks have realistic instructions that read like natural language messages rather than over-specified requirements. To reliably evaluate these tasks, we introduce a validation agent which uses expert-designed recipes to write behavioral tests that adapt to submitted solutions.
Senior engineers solve bugs that require runtime investigation from behavioral reports
Senior SWE-Bench bug tasks reflect tricky user reports and focus on investigation, from starting services to debugging subtle runtime issues. They are sourced from PRs that needed significant runtime investigation to solve (e.g. logs, profiling data, reproduction steps).
Senior engineers ship the right code without being told to