Findings
What we have shown, what did not hold, and what we have not tested. Each result says how it was measured and what to watch for.
23 results: 9 hold, 7 partly, 6 did not hold, 1 not tested yet.
How to read this
Most of what we have is a screen by a model, not proof that people prefer something. So every result carries a mark for how it was measured, and we list the claims that did not go our way next to the ones that did.
- People
- Choices made by people, from public data or a blind rating.
- Distance between ideas in a text embedding. It says ideas differ, not that they are good.
- AI judge
- A Claude model’s verdict. It moves with its role, the order of options and length. A screen, not evidence people prefer something.
- Literature
- Someone else’s published result.
- Record
- Saved files, hashes and test runs. They show that something was made and checked as described, not that it is good.
Show
Measured by
Do model ideas converge?
The starting problem. If a model gives the same few ideas, a person asking it for ideas gets the same few ideas.
Plain prompting gives few distinct ideas
Asked for 40 concepts, plain prompting gave about 3 distinct ideas on the Helsinki museum brief (2.7) and about 4 on the eVolo skyscraper brief (3.8). Forty real entries hold 10.8 and 16.2.
Watch for. Distinct is counted by embedding distance, so it says ideas differ, not that they are good. On New Yorker captions plain prompting does not collapse.
A different seed or framework per idea restores the spread
Seeded sets hold 7 to 14 distinct ideas in 40. Forty invented people’s frameworks give 9.2 on Helsinki, and 92% of their concepts land in the real field’s far band.
Watch for. Random seeds spread ideas as well as a real person’s life writing, so the spread comes from the seed, not the person. Spread is not quality.
Revising an idea pulls it back toward the usual
One AI revision closed about half the distance between 30 real entries and the model’s usual answer (13.1 to 6.4 units). Eight revisions closed about 70%.
Watch for. Thirty entries and one brief. Fresh outside material in every round keeps ideas apart.
Other researchers find the same convergence
Across more than 70 models, different models give strikingly similar answers to open-ended questions. Writers helped by an AI wrote stories that were more similar to each other as a set.
Watch for. Measured on text and stories, not on designs.
Can the distance from the usual be set?
The machine places each concept on one scale of how far it sits from the usual answers, and has to be able to aim.
The dial sets distance from the usual on request
Asking for unusualness from level 1 to 5 tracks distance from real entries (Spearman 0.89 on Helsinki, 0.68 on eVolo). Forty frameworks put 92% of concepts in the real field’s far band, and the far-tail engine 52%.
Watch for. Weak on captions (0.22). On Helsinki even level 5 stays inside the field’s normal range, and on eVolo levels 2 to 4 are flat.
Is far good, and is it new?
The strangest ideas are the ones worth finding, if they hold up. These are the tests of that.
Concepts far from real entries are judged better
The further a concept sits from real entries, the less an AI judge prefers it (rho −0.21 on Helsinki, −0.28 on eVolo). In Framework’s Ideas dial check, fit falls from 78 to 53 as unusualness rises from 25 to 66.
Watch for. One AI judge, which also picks the real entry as the human one 98% of the time on Helsinki. Design research on analogies finds the same trade between novelty and quality.
Concepts far from the field are new against the built world
An AI checker with web search rated 0 of 20 far-tail engine concepts new against built and published precedent. Real entries scored 5% to 10%.
Watch for. One brief. The checker is a model: on a retest it gave the same new-or-not call on 25 of 28 concepts and the same precedent on 13 of 30.
Fresh outside material keeps ideas far from the usual
After four rounds with a real research abstract each round, 109 of 138 variants were still far from the field, against 40 of 141 in a closed loop.
Watch for. A random fragment did nearly as well (104 of 141), so the effect comes from fresh material, not real material. One brief.
More rounds of development make a concept judged better
The AI judge liked the concept less after four rounds in every arm, from 21% against a real upper-tier entry at round 0 to 2% to 4% at round 4. Selecting on preference first stopped the slide but did not make it climb.
Watch for. One brief, an AI judge, and round 0 was itself selected on a screen.
Is taste personal, and can a selector learn it?
The product is the chooser. These results say how personal taste is and how much a selector can gain.
Most of a person’s rating is personal
Across 1,759 people rating art, songs and jokes, about 44% of a rating is stable and personal, 23% is shared and 33% is noise.
Watch for. Public data, not architecture. Studies of aesthetic taste find strong individual differences for architecture and artworks, and high agreement for faces and landscapes.
People lean toward or away from the unusual
In 45,136 joke raters the lean repeats. Net of length it is small (0.11 to 0.16), and people split about evenly.
Watch for. Jokes, with unusualness measured by an embedding.
Each person has a favourite distance from the usual
No reliable personal sweet spot among 33,360 joke raters, or among 549 raters of 11,723 artworks. A planted sweet spot worth 1% to 2% of rating variance would have shown.
Watch for. Jokes and art, not designs. The far tail and unusualness as people judge it are untested, so the idea is not dead, only unsupported where we could look.
A selector that learns one person beats a shared judge
After 20 ratings a selector learned from one person picks their favourite of three 2.9 points more often than a shared judge, and 6.4 points more after 80 ratings (3,000 people, jokes).
Watch for. Jokes, and it uses other people’s ratings of the same items, which a new design lacks. No selector exists for designs yet.
Juries reward the unusual
At Helsinki 47% of the most unusual-looking fifth of boards landed in the jury’s bottom two tiers, against 17% of the most typical fifth. New Yorker voters favour typical captions. Reedsy juries slightly favour atypical stories.
Watch for. Some bottom-tier boards may be weak or unfinished rather than unusual. Contests differ.
Do frameworks carry a way of thinking?
A framework writes down how a person reasons so a model has to use it. What it does and does not carry.
A framework carries a way of thinking
A judge picked the right framework out of four 88% of the time, and 97% on design briefs for four invented practitioners.
Watch for. The judge reads the framework text. Much of the signal was the person’s world, not their method.
A method-only framework makes designs a critic prefers
A model critic picked method-only designs 44% of the time (invented people) and 36% (famous people), against 11% for plain and 9% for motif-heavy designs. Chance is 25%.
Watch for. An AI critic, four briefs, no human rating.
A framework predicts what a person does next
Written from an architect’s work up to 2008, it made a model sound like him but not predict him. His real later work sits about halfway between plain ideas and the framework’s (0.55), and hits on his central move were no better than plain (19% against 24%).
Watch for. One architect and one cutoff.
Can an idea be developed, and can a model judge it?
From a chosen concept to drawn boards, and how far a model critic can stand in for a jury.
A concept can be taken from brief to drawn boards
One museum concept went from brief to four drawn boards for about $13, and a skyscraper design was developed by hand into a native model.
Watch for. A demonstration, not a test of excellence. Read from its boards the museum concept sits inside the field, not beyond it, and an AI director chose it.
A team of AI roles can take a concept to a reviewed native model
Four versions of one skyscraper question were published on September 27 and 28, each with a saved Rhino model, drawings and a written critique. The reviewer advocated the fourth at concept stage, with a qualification. Later passes produced local studies and clearer boards, and the studio’s own record says no whole was adopted and the last push fell short of its target.
Watch for. The reviewer is an AI role that has seen the lead’s reasoning, and no architect outside the team has rated any of it blind. The models are concept geometry with dimensions chosen by the studio, not engineered.
The studio’s tools do what they say
In its last full recorded run (October 26), 404 of the studio’s 413 automated tests passed and 9 were skipped, and each of the four published versions passes its verification command, which recomputes a hash for every file.
Watch for. The 9 skips are host limits on creating file links, not passing checks. Tests show that the tools behave as written. They do not show that a design is good, and the Rhino agent has been run on one machine.
A drawing can be rebuilt as an editable model, with a render
The studio rebuilt its own drawing, The Unfinished Earth, as a Rhino model of 2,771 parts with a Grasshopper file and five sliders, and rendered it in D5. Changing tower spacing from 1.00 to 1.08 changed 1,971 parts, added 15 and kept 800, and setting it back restored the original. A small canopy test followed a span change from 6,000 to 7,200 mm.
Watch for. The model is cruder than the drawing (more regular rock, repeating rooms), and structure and circulation are unresolved. A later check found 45 saved surfaces differing by under 0.025 mm, with the cause not established, and no separate D5 project file was saved. It is a separate prototype from the Ideas demo: no idea has been passed from one to the other automatically.
An AI judge can stand in for a jury
The judge tracks a real jury moderately (Spearman 0.56 from boards at Helsinki, 0.47 with the New Yorker crowd). On captions it agrees with the crowd about as often as one random voter.
Watch for. Good for screening weak ideas, not for choosing among good ones, which is why every preference claim waits for people.
Do people prefer what the machine makes?
The test that decides the product. It has not run.
Architects prefer what the machine makes
No architect has rated our concepts blind. The blind round has about 10 architects rate 7 real upper-tier entries and 7 of our concepts, all written up and drawn in one style, without knowing which is which. It is built and has not run.
Watch for. Our one human test so far, the founder’s blind ratings of 24 text ideas, went against the recipe (2.62 against 3.25, not significant), while the AI judge preferred our ideas 72% of the time.
What decides it
The blind round with about ten architects is the first test of whether people prefer what the process makes. Seven real upper-tier competition entries and seven of our concepts are written up in the same words and drawn in one style, and each rater says whether they would take a concept forward, what it would be like to stand in, how unusual it is and whether they have seen it before. The analysis is written down before any rating comes in.
If you design for a living and would like to take part, write to us. The raters are not told which concepts are ours.