What Happened
JetBrains conducted testing on its new Caveman skill, designed for agents like Claude Code. This tool transforms text responses into a simpler, more concise style by removing unnecessary words. However, the anticipated token savings from using Caveman turned out to be significantly lower than the claimed 65%, coming in at just 8.5%.
Why It Matters
For developers and companies utilizing AI agents, token savings are a crucial factor, as they directly affect the cost of using these technologies. The testing results may raise doubts about Caveman's effectiveness and impact its popularity among users, as many were hoping for more noticeable results.
Context
Caveman was tested on the SkillsBench platform, where the performance of 87 tasks was compared. Agents were evaluated in two modes: without the skill and with Caveman forcibly enabled. Despite having 85,000 stars on GitHub, indicating its popularity and potential usefulness, the results showed that expectations were not met.
What This Means
The lack of significant token savings may decrease interest in Caveman among developers seeking ways to optimize costs when using AI technologies. While the skill may be beneficial for simplifying responses, its effectiveness in resource savings remains questionable. This also underscores the importance of thorough testing and evaluation of new technologies before widespread adoption.
