Are AI labs pelicanmaxxing? – Dylan Castillo
Someone finally ran the experiment to test whether AI labs are gaming Simon Willison's viral "pelican on a bicycle" benchmark - generating 1,008 SVGs across a grid of animals and vehicles to see if pelicans or bicycles score suspiciously well.
Read Original Summary used for search
TLDR
• Tested 7 frontier models on 8 animals × 6 vehicles (1,008 SVGs total) to see if the famous "pelican on a bicycle" prompt gets special treatment
• Five lines of evidence all say no: pelicans rank 6th of 8 animals, bicycles are second-to-last in vehicles, the combination ranks #42 of 48 overall
• Regression analysis with lab-specific interaction terms found zero significant boosts for pelican, bicycle, or the specific combination (except one false positive at p=0.022 that doesn't survive correction)
• The one anomaly - all 21 pelican-bicycle images face right - turns out unremarkable since 60% of all images face right and both pelicans and bicycles individually skew that way
• Labs are probably doing "SVGmaxxing" (optimizing SVG generation broadly) rather than gaming specific prompts, which this experiment can't detect since it lifts all combinations equally
In Detail
Dylan Castillo designed an experiment to test whether AI labs are optimizing specifically for Simon Willison's famous "pelican on a bicycle" benchmark. The setup is elegant: instead of just testing the viral prompt, he created an 8×6 grid of animal-vehicle combinations (pelican/flamingo/heron/otter/raccoon/antelope/whale/cat × bicycle/unicycle/skateboard/scooter/plane/boat), making the famous case just one cell among 48. He generated 3 samples per combination across 7 frontier models, scored them with an LLM judge on animal accuracy, vehicle accuracy, and action coherence, then ran regression analysis to detect lab-specific boosts.
The results are consistent across five different tests. Pelicans rank 6th of 8 animals (behind cats, whales, raccoons, herons, and antelopes). Bicycles rank second-to-last among vehicles. The specific pelican-bicycle combination ranks #42 of 48 overall. A fixed-effects regression with per-lab interaction terms for pelican, bicycle, and the pelican-bicycle cell found no significant effects - every confidence interval includes zero except one (Gemini on bicycles at p=0.022), which is exactly what you'd expect from chance alone given 21 tests. The one compositional anomaly - all 21 pelican-bicycle images facing right - dissolves when you check base rates: 60% of all images face right, and both pelicans and bicycles individually tend that direction.
The more plausible explanation is "SVGmaxxing" - labs optimizing SVG generation broadly rather than gaming specific prompts. Google/DeepMind does this openly. This experiment can't detect that because it lifts all combinations equally. But the data clearly shows labs aren't producing terabytes of pelicans on bicycles to game one viral benchmark. The methodology here - creating a grid around a suspected gaming target to test for specificity - is a template for testing any benchmark gaming hypothesis rigorously.