Beyond the Demos: What GPT-3 Can (and Can't) Actually Do
A look at the flood of September GPT-3 demos and the growing debate over how much of it is real reasoning versus pattern matching.
If you’ve been anywhere near tech Twitter this month, you’ve seen the GPT-3 demo reel. Type a plain-English description of a web layout, get working HTML and CSS back. Describe a function, get code. Ask for a memo, an email, a poem in a specific style, and out it comes, often good enough to use with light editing. Developers who got early API access have been posting these clips daily throughout September, and the range is genuinely wide: natural-language-to-code tools, layout generators that turn a sentence into a React component, writing assistants that draft everything from marketing copy to legal boilerplate.
It’s easy to watch a 30-second clip and conclude the model is “thinking.” It’s also easy to watch the same clip and conclude it’s just an extremely well-read autocomplete. Both reactions are showing up in comment sections right now, and I think the honest answer is messier than either.
Pattern matching at a scale that changes the question
GPT-3 was trained on a enormous slice of text scraped from the internet, and it’s very good at continuing a prompt in a way that’s statistically consistent with everything it’s seen. When you ask it to write a CSS layout, it’s not reasoning about flexbox from first principles — it’s drawing on countless examples of people describing layouts and writing the code for them, and it’s found the patterns that connect the two. That’s not nothing. At the scale GPT-3 operates, “just pattern matching” produces output that’s frequently indistinguishable from a junior developer’s first draft. But it also means the model can confidently produce something that looks right and is subtly, or not so subtly, wrong, because it has no model of what the code actually does when it runs.
The demos are curated, and that matters
A detail that gets lost in the excitement: nearly every viral demo is the best output after some amount of prompt tweaking and cherry-picking, not the first try. That’s not a criticism of the developers sharing them — it’s a reasonable way to show off a capability — but it does mean the public perception of GPT-3’s reliability is running ahead of its actual, unfiltered hit rate. People who’ve spent real hours with the API describe a tool that’s brilliant in short bursts and prone to drifting, contradicting itself, or hallucinating plausible-sounding nonsense over longer outputs.
Where that leaves things
I don’t think the “just autocomplete” framing and the “look what it can do” framing are actually in conflict. GPT-3 is a next-token predictor trained on a huge corpus, and that same mechanism is producing outputs useful enough that people are already wiring it into real tools. The interesting question for developers building on the API isn’t whether it’s “really” reasoning — it’s figuring out which tasks tolerate the failure modes (a copy first draft you’ll edit anyway) and which don’t (anything where a confidently wrong answer causes real damage). Right now access is still limited and mostly in the hands of people building demos rather than production systems, so we don’t have a great read yet on how it holds up under real, adversarial, everyday use. That’s the test that matters more than any highlight reel.