Separating signal from noise in coding evaluations

An analysis by OpenAI identifies reliability and accuracy issues within the widely used SWE-bench Pro coding benchmark for evaluating artificial intelligence models.

Firefly links to the original publisher. The summary above is AI-generated for orientation and may differ from the source. The “current rank score” decays over time so newer significant stories surface first.