Not even Pokémon is immune to the controversies surrounding AI benchmarking.
Recently, an online debate erupted after claims spread widely across social media, asserting that Google’s new Gemini AI model had outperformed Anthropic’s flagship Claude AI in playing through the original Pokémon video game trilogy. Specifically, Gemini had reportedly navigated its way successfully to Lavender Town, as viewers observed on a live-streaming Twitch channel maintained by a developer. Claude, on the other hand, had stalled at Mount Moon as recently as February.
However, closer examination revealed Gemini enjoyed a significant advantage: the presence of a custom-built minimap that assisted the AI by identifying game tiles, such as trees that could be cut. This minimap considerably simplified Gemini’s gameplay, reducing the analytical burden that Claude faced by having to interpret screenshots directly and independently.
Few would argue that Pokémon adventures represent a rigorous AI benchmark comparable to specialized assessments like coding evaluations. Nonetheless, the dispute underscores an increasingly acknowledged issue in the AI community: that slight deviations or customizations in benchmark methodologies can meaningfully influence—and sometimes distort—performance outcomes.
Such discrepancies have been highlighted previously. Anthropic itself recently disclosed variable results from its “Claude 3.7 Sonnet” model depending on how the benchmark was structured. When evaluated with the SWE-bench Verified protocol, intended specifically to measure coding abilities, Claude’s performance significantly rose—from 62.3% accuracy under standard conditions to 70.3% with the aid of Anthropic’s own infrastructure or “custom scaffold.”
Meta has similarly faced scrutiny after releasing metrics for its latest model, Llama 4 Maverick. The perceived progress came under question when it was discovered that the version of Maverick making headlines had been specifically fine-tuned for one benchmark (LM Arena) and recorded markedly lower scores when tested without customized adjustments.
The Pokémon case, while whimsical, serves as an illustrative cautionary tale. AI benchmarks are inherently flawed barometers of model capability, providing limited, context-sensitive snapshots rather than definitive judgments. With an increasing trend towards personalized tests and ad-hoc evaluation solutions, reliable and consistent comparisons among competing AI models could become more elusive than ever.
