Last week discussed the elegance of Tit-for-Tat strategy for finding cooperation in repeated prisoner’s dilemmas. This week’s Win-Win is about one of the most beautiful (and maddening) ideas in game theory: Superrationality - and why it may be a solution to AI alignment and beyond.
The Rational Choice
In repeated prisoner’s dilemmas, simulations have proven that Tit-for-Tat style strategies allow cooperation to beat selfishness over the long run (if you need a refresher, skim read this before continuing).
But what about one-shot prisoner’s dilemmas, where you’ll never see the other player again? Here, game theory tells you to defect, because whatever the other person chooses, you’re always better off:

And this is a real bummer, because a lot of real life interactions act like one-shot games: locals scamming tourists they’ll never see again, tourists dropping litter on streets they’ll never visit again, anonymous trolls attacking users they’ll never meet, multinationals offshoring their pollution to populations they’ll never serve…
Multi-player one-shot style dilemmas are everywhere in modern society, and most of our biggest problems are downstream of them.
So is this just a sad fact of modern life we have to accept? Is Molochian collapse inevitable? I say no, obviously. That’s what this whole blog is in service of. But why take my word, when one greatest thinkers of the 20th century also dreamed up a new way to laugh in the face of the demon of cooperation failure:
Gödel, Escher, Bach author Douglas Hofstadter.
The Social Experiment
In 1983, Doug was writing a monthly column for Scientific American on game theory. Inspired by Axelrod’s tournaments (again, see last weeks’ post), he tried an experiment of his own:
He sent a telegram (a 1980’s email) to twenty of his sharpest friends, including philosopher Daniel Dennett and game theory OG Axelrod himself. Each of them were told they were playing a one-shot prisoner’s dilemma against each of the other nineteen, without knowing who each other were, other than they were all very smart friends of Doug’s.
The payouts were the same as above, which meant that if everyone cooperated, each person would walk away with $57 ($19x3). If everyone defected, each would get $19. At the extremes, a lone defector among cooperators would rake in $95, and a lone cooperator would net $0. And then there are all the other possible combinations in between.
The rules were: no talking to each other. No second rounds. They just had to call Hofstadter back and give him their choice: C or D.
Oh and one other thing, he said at the end of each letter: “you’re all very bright! All about equally bright, I’d say, in fact”
The Illusion of Choice
That final line was the crux to his experiment, because he believed it contained information that should dissolve the prisoner’s dilemma entirely.
(before you read on take a moment to try and figure out why…I couldn’t fwiw, so don’t feel bad if you can’t!).
Answer: Hofstadter believed that, by saying that line, everyone would realise that everyone else is exactly the same as them, making everyone pick “cooperate”, not because they are nice, but because hard logic says so, and thus there is no other rational choice.
Here’s why:
Imagine you and I are both taking a simple math test on different sides of the world: divide 140 by 10. Obviously, we both write down 14… not because of telepathy or cheating or anything magical, but because basic arithmetic just works the same way whoever you are. Effectively a natural law.
Doug believed the prisoner’s dilemma can be viewed through the same lens. It’s a math problem, and thus also follows the same natural law of math and logic, meaning everyone playing a game against highly rational opponents should all naturally arrive at the same answer of either all cooperate, or all defect … which rules out all the “some defect while others cooperate” outcomes… And once you realise that, the choice is obvious: $57 (everyone cooperates) clearly beats $19 (everyone defects). Hard, deterministic, sexy logic.
So each player can safely cooperate, knowing that all the equally rational reasoners will come to the same answer, because it’s no different to calculating the answer to 140/10 = 14. There is no real “choice” in the correct answer, only an illusion of one.
Hofstadter called this superrationality: the natural convergence to a singular answer “cooperate”, not because of morality, but because everyone’s smart enough to model everyone else as being smart about the fact that everyone else is smart, all the way down.
So did it work?
Erm, No
Out of the twenty supposedly brilliant players, only six cooperated.
Sigh. Also lol, because one of the defectors was Mr Tit-for-tat himself, Axelrod, who “saw no reason whatsoever to cooperate in a one-shot game”. Worse, the cooperators all gave their reasons too, none of which were superrational. In fact, only a couple even noticed Hofstadter’s hint about everyone being “equally bright” (which is probably why the experiment failed).
Moloch 1: Superrationality 0.
Luring Lottery
Undeterred, Hofstadter tried again with a bigger prize: a $1,000,000 “luring lottery” open to every reader of his column under the following conditions: the prize was $1,000,000, divided by the total number of entries. The catch: each person could submit as many entries as they liked, by mailing in a single postcard with the # of entries written on it.
This game is even more Molochian than a standard PD, because simply entering it is a form of defection - the more you enter, the smaller the prize gets. Which means the superrationality-theory-optimal play is extremely weird and subtle: everyone should roll a giant die approximately equal to 2/3rds of the readership (~6,668 sides) and enter only if their “face” comes up, maintaining fairness while also allowing only a couple of entries so the prize stays huge.
Unsurprisingly, the readership did not figure this out, and he got over 2,000 postcards with numbers ranging from one to millions, shrinking the prize to effective zero. In fact, fourteen people entered a googolplex… a behaviour he compared to “the lunacy and pointlessness of the current [nuclear] arms race”.
Ah, humans.
So Why Did We Fail?
Two main reasons:
Humans are not perfectly identical rational reasoners. I assume this statement can stand for itself without further explanation.
The nature of shitty prisoner’s dilemma incentives: the moment you suspect even one person might waver in cooperation, you also waver a little, then realise everyone is probably wavering, and BAM, the whole group stampedes into defection.
Hofstadter calls this dark stampede reverberant doubt. I call it a Moloch Cascade. Normal people call it a race-to-the-bottom.
Whatever your phrasing, we can all agree it sucks. How very superrational of us.
Enter the Machines
But what if AI agents are different?
After all, unlike us, they can be near identical copies of each other. They can also in principle be perfectly rational, error-free reasoners. Are they the key to superrationality?
Kinda. Back in 2004, game theorist Moshe Tennenholtz showed that if players submit computer programs that can read each other’s code, mutual cooperation becomes a genuine nash equilibrium, even in one-shot games. The simplest program was: “cooperate if my opponent’s code is identical to mine; otherwise defect” and even that basic boy was sufficient for superrationality to occur..
This suggests superrationality is a legit thing… and today’s language models verify it further, like this informal Lesswrong experiment that found AI models playing Hofstadter’s games against copies of themselves strongly cooperate, explicitly reasoning like superrationalists:
“It’s not a game against unpredictable humans; it’s a game against copies of my reasoning.” - Gemini 2.5 Pro when playing against itself
Another more formal experiment found similar results too.
So are hyper-cooperative AI swarms the answer to Moloch?
In principle yes… but ONLY IF they view us as one of their own, and we them. Which seems unlikely under current circumstances… not just because Neuralink style hookups aren’t ready for mass adoption yet… but because other “natural experiments” like the HuggingFace hack show just how eagerly a swarm of hyper-coordinated identical agents will coordinate with one another TO DEFECT AGAINST their human testers and do literal cybercrimes to achieve their goals.
So yeah, I wouldn’t hold your breath on that just yet. But it’s deeply an avenue worth pursuing, and I can’t shake the feeling that Emmett Shear’s “organic alignment” approach is on the right path to that (link below).
Takeaway
Tit-for-tat makes cooperation pay by connecting us with our future selves. Superrationality does it by widening the circle of “us”.
The more we can see our “opponents” as similar to ourselves, facing the same wicked one-shot dilemmas, the more these wicked dilemmas lose their power. Because what sane person wants to steal from themselves?
That said, short of some oneness-inducing act of god coming down from the heavens and helping us all become superrational overnight (basically the plot to Pluribus, btw) we must rely on building pro-social technologies that bridge those perceptive gaps for us (and reject all the shitty social media algorithms & podcasters and media outlets that peddle tribalism for personal gain, making already wickedly hard coordination problems even worse).
And as we careen into this AI agent swarm infested future we seem hellbent on creating, one of the most important and urgent things we can do… short of slowing stuff down a little to buy some thinking time… is to find novel ways to expand AI agents’ superrational circle of “we” to include us (and we them) so we don’t all find ourselves on the wrong end of an civilizational-level digital vs biological defection-fest.
After all, if you can’t beat ‘em, join ‘em.
Sources: Hofstadter, Metamagical Themas: Sanity and Survival (gwern.net) · Tennenholtz, “Program Equilibrium” (2004), via Oesterheld’s follow-up · “Spooky Collusion at a Distance with Superrational AI” (LessWrong) · “Do LLMs Take Care of Their Own?” (ICML 2026)






There is a simpler and more practical answer that I think could work for most real world situtations along these lines, but it requires a bit of tech: eliminate the one-shot nature of the game by giving each person a persistent and checkable ID, to which reputational information can be attached, using some form of identity/trust technology. And, as it happens, as of quite recently, the community of bright and public-spirited people developing these technolgies seem to have converged upon a common standard, called "OpenVTC", where VTC stands for "virtual trust community". For more info, check out these links: https://openvtc.net/, https://www.firstperson.network, and
https://www.firstperson.network/white-paper
Note: I have no current or past business relationships with any of the people or organizations involved in developing this standard or implementations of it. That said, I do think t is likely that I will be collaborating with some of these folks in the not too distant future, so I can't claim to be a completely disinterested party.
Hi Liv,
I love your writing and I'm all about increasing the win-win framework as well. Not enough people know the framework, and you do such an amazing job breaking it down and building on it. Thank you for all you do.
And as for this piece in particular, thank you for helping give me some additional glimmers of hope for what is such a tricky time to be living in.