Please Stop Comparing LLMs With One-Shot Prompts

I keep reading these articles. New model drops, and within a day someone’s already “comparing” it to the others — with one prompt. A single shot. Very few details, and then the LLM has to figure it out.

Personally, I believe this is the worst technique ever to evaluate a model. And spoiler: not because the LLMs are bad. Because we are.

”Build Whatever You Want”

Think of it like when I was a PM. Imagine you’re a PM, and you go to your engineer and say: hey, please develop whatever you want. Period. No spec, no context, nothing. That’s a one-shot prompt. That’s literally what it is. A very generic ask.

If you’re not giving the specs, what do you expect? You can’t even judge the results afterward. It’s completely unuseful.

I Know Why People Do It

I know it’s useful for one thing: hype. And right now, AI is all about hype. A one-shot comparison is easy to write, easy to screenshot, easy to go viral with.

But there’s more than this.

AI is the best tool, the biggest revolution IT has ever seen. I’d say since IT was even created. I’ve been in this industry for the last twenty-five years, now even more, and this is the revolution we’re living through.

Stop the Clickbait

So please, stop this. I’m going to say it plainly: very generic, very clickbait articles about “Model A vs Model B” on a single throwaway prompt.

Give users details. Give readers information. Because if you don’t, AI doesn’t get to evolve — it just gets used to farm clicks.

I’m not saying you need to hand over a full end-to-end test, though by the way, you should. But at least give it the spec — what you expect it to build, how you expect it to behave. Or, at the very least, let it iterate. Let it ask you for information before it starts producing the content.

Please stop this waste. I’m sorry, but I don’t think these comparisons are useful in any real way.

Every day I see fewer websites giving real data, and more generic websites giving shit.