When the model keeps asking for the same thing

I have a tool that turns a sentence into pixel art. Not by generating an image: a model writes a small data file describing the sprite, and a renderer I wrote draws it. The format is deliberately narrow. Eighteen shapes, a fixed set of colour roles, and a validator that rejects anything outside that with a message naming what was allowed.

Narrow is the point. A model cannot invent a shape that renders wrong, because a shape it invents does not render at all.

The cost of narrow is that when the model wants something the format does not have, you find out as a failure. I had a failure rate of about one draw in ten, and I spent longer than I should have treating that as a model problem.

What the failures had in common

I built a bench that runs the same ten prompts through every model, prices the tokens each one actually burned, renders every result and puts them side by side. It scores nothing subjective. It exists because I kept making decisions about quality from memory.

The first full run said one outright failure in ten, plus two draws that needed a repair round. Every failure was the model naming something the format did not have. The commonest by a distance was a colour role called white, in a kit that never declared one.

There is a repair round for exactly this. The spec is rendered server side and thrown away, and if it throws, the error goes back to the model naming what was allowed. It fixes most things.

It did not fix this one. Shown the error and the full list of legal roles, the model asked for white again.

The second attempt produced the same request. That is the part worth noticing.

Reading that correctly took me too long

A model repeating itself after being corrected looks like the model being stupid. It is easy to reach for a firmer instruction, and I did.

The right reading is the other one. A specular highlight is something nearly every sprite wants. Asking for white is not a mistake, it is the correct request, arriving at a format that had no way to receive it. The kits each declared their own palette, and most of them had simply never thought to include plain white, because a human writing a kit by hand does not think about the highlight until they are drawing one.

So the fix was not to stop it asking. It was to make the request legal.

Every kit now carries white, black, none and outline whether it declares them or not. A kit's own definition still wins, so nothing that already existed changed by a single pixel.

What it was worth

Same ten prompts, three runs each, against the ten of the first bench:

Cheaper, faster and slightly more detailed, from adding four entries to a table.

The honest caveat: zero failures in thirty runs puts the true rate somewhere under ten percent with 95% confidence, not at zero. I believe the real number is much lower than that bound, but the reason is the mechanism rather than the sample. That specific failure is now impossible by construction, which is a stronger argument than thirty clean runs.

Then it happened again

A barrel kept coming back as a rounded box. I rewrote the sentence three times and each version was worse than the plain one I started with.

The vocabulary had no shape that bulges. Every shape in it was a rectangle, a frame or a ring, so a barrel could only be approximated by a box with its corners taken off, and the model was doing the best available thing.

I added one shape. It has a waist, a widest point, and optional bands drawn by the same loop that already knows how wide each row is, so a band cannot overhang the body it is wrapped around. Then I deleted all three prompt rewrites and went back to the original plain sentence.

The model picked the new shape first time.

The rule I would now apply first

When the output is wrong, check whether the format can express the right answer before deciding the model chose badly.

This is uncomfortable because prompt text is the cheapest thing to change and the vocabulary is the most expensive, so the incentive runs the wrong way. You will always try the rewrite first. But a prompt cannot conjure a shape that does not exist, and no amount of firmness makes an impossible request possible. Both of my failures were the format's fault and both of them presented as the model being careless.

There is a tell, and it is the thing I nearly missed. A model that has been corrected and repeats itself is usually right. If it asks for the same thing twice, having been shown exactly why it cannot have it, that is evidence the request is reasonable and the receiving end is wrong. A model that was merely confused takes the correction.

The vocabulary is the quality lever. Not the prompt, and not the model.