Teaching a computer to paint

One painting engine, built an episode at a time, every episode painting the same sunset so any two can sit side by side. Then six AI models took turns as art director.

What is real and what is not. The sunset is generated in code, not a photo. Every image below is the program's real output, re-rendered from saved settings. Numbers are measured, not estimated. The model comparison is one run per model, so it is an experiment, not a benchmark. Read the case study.

Episode 1

Random shapes

Propose a random shape, give it the best colour, keep it only if the picture gets closer. 6,000 strokes, error from 24,690 to 1.68. Accurate, and it looks like a filter.

The target sunset beside the hill-climbing result: a close but soft, blurred copy.
Target and result. Hill climbing in the style of Geometrize and primitive.

Episode 2

Brushstrokes that follow the light

Hertzmann's 1998 painterly method: five brushes from big to small, strokes steering along the edges, plus bristle texture and paint thickness lit from the top left. 20,229 strokes.

Three panels: the target, episode 1's soft result, and episode 2's painting with visible curved strokes.
Target, episode 1, episode 2.
Close-up of the painted sun and ridges, showing circular strokes, bristle marks and raised paint.
Up close: the sun is painted in circles.
Three attempts side by side: v1 smears colour across the ridges, v2 has a squiggly sun, v3 paints the sun in circles.
How it got there: v1 smeared across the ridges, v2 had a squiggly sun, v3 fixed both. Each version re-renders exactly.

Episode 3

Paint that mixes like paint

Screens average light, so yellow and blue make a muddy sage. Paint absorbs light, so they make green. Kubelka-Munk mixing, eight real paints, a recipe for every stroke.

The same yellow and blue strokes crossing twice: screen mixing gives a grey-green overlap, pigment mixing gives a clear green.
Same two strokes, mixed both ways.
Three panels: the target, episode 2, and episode 3 painted only with colours mixable from eight paints.
Target, episode 2, episode 3. The deepest navy is out of reach for these eight paints, so the bottom strip goes grey.

Episode 4

The model as art director

The model never places a stroke. Each round it looks at the target and the canvas on a grid and gives up to three notes: where, which brush, and why. The painter does the rest. It never sees the error score.

What the director sees: the target on the left and the half-finished canvas on the right, both with an A to J by 1 to 10 grid.
What the director sees, round 4.
Seven final paintings in a grid, ranked by error: the fixed schedule first, then GPT-6.1-Sol, GPT-6.1 Terra, Claude Opus 5.5, Claude Haiku 5.5, Claude Sonnet 5.5 and GPT-6-Luna. They look very similar.
Same brief, same canvas, one run each. By eye they are hard to tell apart.
Line chart of error against strokes for the fixed schedule and six AI directors. The fixed schedule's line is lowest at the finish.
Error against strokes. Lower is closer.
Error is the mean squared difference from what the eight paints can reach. One run per model.
DirectorStrokesError
Fixed schedule (Hertzmann 1998)9,94919.6
GPT-6.1-Sol (Codex)11,21521.0
GPT-6.1 Terra (Codex)10,99424.4
Claude Opus 5.57,87427.2
Claude Haiku 5.59,68031.4
Claude Sonnet 5.57,19434.3
GPT-6-Luna (Codex)9,18935.6

No model beat the fixed schedule at the finish. The models directed like painters, mostly big shapes first and the sun next, and their notes named real problems. But they were judging by eye, and the schedule was checking every cell.