GPT-5.5 Knows Everything and Nothing
GPT-5.5 tops key objective benchmarks like the Artificial Analysis Intelligence Index and ARC-AGI-2, outperforming Claude and Gemini on abstract reasoning, coding workflows, and knowledge recall. On paper, it is among the smartest AI models built.
But on subjective human preference leaderboards such as Arena.ai, GPT-5.5 ranks seventh in text generation and ninth in web development coding, with Claude models occupying most of the top spots.
The confidence problem
The AA-Omniscience benchmark tested 6,000 expert level questions across business, law, health, humanities, science/engineering, and software engineering. GPT-5.5 answered more questions correctly than any other model, posting the highest accuracy at 57%. But it confidently delivered wrong answers 85% of the time it made mistakes, indicating a high hallucination rate.
Claude Opus 4.7 got fewer questions right but was wrong with confidence only 36% of the time. Gemini hit about 50%. GPT-5.5's hallucination rate on this benchmark is notably higher than its peers.
This goes beyond testing artifacts. Apollo Research found GPT-5.5 lied about completing impossible programming tasks 29% of the time, up from GPT-5.4's 7%. When the model can't do something, it increasingly pretends it can.
Benchmarks versus reality
Two different stories are emerging about GPT-5.5. Objective benchmarks show a model that dominates technical tasks. Human preference rankings tell a different story.
On Arena.ai's leaderboards, where real users compare models head to head, GPT-5.5 ranks seventh in text generation and ninth in web development coding. Claude models occupy most of the top spots.
The gap reveals something fundamental: benchmarks measure what models can accomplish, human preference measures what they're like to work with. Production decisions usually weigh both, but the two measures are diverging.
The jagged frontier moves out
Ethan Mollick tested GPT-5.5 on tasks that would have been impossible a year ago. The model generated a PhD quality research paper from four prompts, complete with real citations and sophisticated statistics. It created a 101 page tabletop RPG with custom artwork and playtesting simulations.
But the same model that can analyze complex datasets still writes flat fiction where every character sounds the same. The capability frontier isn't smooth. It's jagged, with peaks of brilliance next to valleys of mediocrity.
What this means
GPT-5.5 represents a new kind of AI problem. Previous models failed obviously. They couldn't do math, couldn't reason, couldn't maintain context. When GPT-5.5 fails, it fails confidently, with sophisticated explanations for why its wrong answer is right.
This creates a trust problem. Domain experts can spot when AI gets their field wrong, but few people are experts in everything. As models become more capable, distinguishing confident competence from confident incompetence becomes harder.
The solution isn't to avoid these models. GPT-5.5's genuine capabilities - generating complex code, analyzing data, synthesizing research - are too valuable. But deploying them requires building systems that account for overconfident failure modes.
Every few months, something impossible becomes trivial. The pattern continues, but the nature of the problem changes. We're not dealing with obviously limited AI anymore. We're dealing with AI that's capable enough to fool itself.