Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
Ranked #3 on Hacker News with 38 points and 27 comments.
Benchmarking frontier AI models on private, real-world, enterprise codebases.
Today we are releasing Real-SWE, a benchmark that evaluates frontier AI models on private, real-world, enterprise codebases. Each task comes from a private production codebase that we licensed from a real-world company. These are problems their engineers work on, with all the context and complexity that comes with an existing product.
Can a coding agent actually do the work of a software engineer in the real world?
Expert-generated or synthetic tasks can be well designed, but they aren’t the verbatim, actual tasks that engineers in real companies need to do. Our tasks differ on two axes: the underlying coding artifact and specificity of the instruction. Both add complexities that challenge today’s frontier models.
We use native harnesses to reflect how enterprise engineers work in practice, evaluating model-and-harness combinations rather than models in isolation.
Real company tasks require company-specific context