What Happened
There's a growing concern in the AI community about how we benchmark open models against closed ones, like GPT or Claude. Typically, when these comparisons are made, closed models often outperform open alternatives such as glm-5.2 or DeepSeek on various tasks. However, this leads to the assumption that the underlying technology of the closed models is inherently superior, which might not be the case.
Why It Matters
This issue is significant because it influences how users perceive the value of different AI products. If benchmarks are skewed, we might be overestimating the capabilities of closed models, which could lead to misallocated resources and investments. The real differentiation might not be in the models themselves, but rather in the additional tools and enhancements that closed models utilize behind the scenes, which are often invisible to users.
Context
Traditionally, benchmarking has been seen as a way to gauge performance based on raw inference capabilities of AI models. However, closed models often incorporate various hidden processes—like specialized prompts, internal tool calls, and preprocessing steps—that enhance their outputs. This means that when we see a closed model outperforming an open one in a benchmark, we might not be comparing apples to apples. It’s akin to testing a car engine in a lab versus evaluating a car on the road with advanced driving aids.
What It Means
This raises critical questions about the actual quality and performance gap between these models. If the perceived superiority of closed models is largely due to the extra tools and context they employ, then open models could be far more competitive than benchmarks suggest. As the industry evolves, the tools and enhancements around AI models may become easier to replicate than the models themselves, potentially leveling the playing field. Therefore, when discussing AI model performance, it’s essential to consider whether we are truly comparing the models or the systems built around them, and whether this distinction is becoming irrelevant in today’s fast-paced AI landscape.
