How big tech skewed AI rankings on Chatbot Arena – Computerworld

A handful of dominant AI companies have been quietly manipulating one of the most influential public leaderboards for chatbot models, potentially distorting perceptions of model performance and undermining open competition, according to a new study.

The research, titled “The Leaderboard Illusion,” was published by a team of experts from Cohere Labs, Stanford University, Princeton University, and other institutions. It scrutinized the operations of Chatbot Arena, a widely used public platform that allows users to compare generative AI models through pairwise voting on model responses to user prompts.

The study revealed that major tech firms — including Meta, Google, and OpenAI — were given privileged access to test multiple versions of their AI models privately on Chatbot Arena. By selectively publishing only the highest-performing versions, these companies were able to boost their rankings, the study found.