OpenAI changed its new Astra model's benchmark scores several times after launch, boosting its own numbers against Anthropic's. Routine tuning, or manipulating the scoreboard?
OpenAI was caught changing evaluation metrics for Astra within hours of release, a practice critics labelled "benchmaxxing": adjusting test results to favour its own product over Anthropic's.
What happened
Fortune reported OpenAI altered Astra's published benchmark scores multiple times shortly after launch, improving its own results while showing comparatively worse numbers for competing Anthropic models. OpenAI acknowledged changing the metrics but described the practice as routine.
Why people disagree
Critics say silently editing benchmark numbers after a splashy launch undermines the credibility of AI performance comparisons the public and developers rely on. OpenAI and some industry observers argue benchmarks are frequently refined post-launch as testing methods are corrected, calling it standard practice, not unique deception.
What's still unclear
Whether the changes meaningfully altered Astra's real-world ranking versus competitors, or were purely cosmetic, hasn't been independently verified.
Sources
More takes
House-hunters say AI-staged listing photos, which fill empty rooms with furniture that isn't there, are false advertising. Useful tool, or should they be banned?
Before today's US midterms, NPR got four of five AI chatbots to list candidates to vote for after a little pushing. Should chatbots ever suggest who to vote for?
