Game of AGI

About

Game of AGI

Origins and sources

This version extends the original Game of AGI, a Monte Carlo of AGI risk driven by safe-by-default odds, safety spending, oversight, the number of labs that originate AGI, whistleblowers, and takeoff vs. replication time.

The main sources: AI 2040: Plan A for timelines and the plans' odds of getting alignment right; a separate factored Monte Carlo estimate for open-weights pandemic risk; and Superintelligence Strategy (Hendrycks, Schmidt and Wang) for why state bioweapons are treated as deterred; and experts on Chinese AI policy (Concordia AI, Matt Sheehan at Carnegie, SemiAnalysis) for how much alignment work China does itself and how readily it would slow down; and Craig Quiter's National Security Response for how automation can leave most people without economic leverage or a say even when humans keep control of AI (Locked-In Power and the public wealth fund); and Gladstone AI's report on recursive self-improvement (covered by TIME) for the RSI 2027 scenario, where lab insiders expect AI to start substantially speeding up AI research during 2027.

The code is open source on GitHub (MIT license).

This game was built in conversation with Claude Code. Each session's full history is browsable as a memtree:

How the model works

Game of AGI is a yearly Monte Carlo model of 2027–2040. Every equation and constant below is the one model.js runs. Tags show where each number comes from: AI 2040 is taken from or calibrated to AI 2040: Plan A, Bio decomposition is calibrated to a separate factored Monte Carlo estimate of bio risk, China experts is set from a review of experts on Chinese AI policy, and Assumption is a judgment call in this model.

1. One simulated year

Compute this year's three catastrophe hazards from the current state (section 3).
Roll for each hazard in turn, each with its own die: a pandemic with probability P_{\text{bio}}, then loss of control with P_{\text{loc}}, then great-power war with P_{\text{war}}. The first hit ends the future, so the three combine as independent risks.
Otherwise apply the yearly updates (section 5), clamp every state variable to 0–100 and move to the next year.
If frontier capability reaches 100, resolve the superintelligence handoff; in 2040, resolve the ending (section 4).

The Futures Lab runs 2,000 futures with the levers held fixed. In the campaign, each decision card also moves levers and state directly (events.js).

2. Notation

State variables, each on a 0–100 scale:

Symbol	Meaning	2027
C_{\text{US}},\ C_{\text{CN}}	Frontier capability of the US bloc and China. C = \max(C_{\text{US}}, C_{\text{CN}}). Anchors: 70 Automated Coder, 78 top human expert, 90 dominates top experts, 100 superintelligence. AI 2040	44, 37
A	Alignment and control maturity	16
O	Capability of the best open-weights model	32
B	Biodefense: synthesis screening, surveillance, stockpiles	14
K	International coordination	22
N	Neural-interface (BCI) adoption	4
H	Hidden internal AI: how far AI used inside labs is ahead of what outsiders can see	30
P	Power concentration	45
T	Public trust and stability	55

Policy levers, each from 0 to 1:

Symbol	Lever	0 → 1	Default
r	Race intensity	pause → all-out sprint	0.70
s	Safety share of AI compute and AI labor	0% → 100% (a unilateral pause)	0.05
o	Open-weights releases	closed → release frontier weights	0.45
\tau	Research transparency	secret labs → all research public	0.20
b	Biodefense investment	minimal → Manhattan-scale	0.20
d	US–China diplomacy	cold war → joint project	0.25
n	Neural-interface push	market pace → AI-driven moonshot	0.10
v	Independent oversight	self-audits → an IAEA for AI	0.10
g	Sabotage of China's AI program (AI 2040's Plan B)	none → cyber and kinetic strikes	0
f	Public AI wealth fund: the share of AI returns the public owns, like Norway's or Alaska's funds	none → public owns most AI	0.05

Hidden unknowns, drawn once per future. The risk panel shows risks averaged over them; the dice use the hidden draw.

Symbol	Meaning	Distribution
1/\delta	Ease of alignment: how hard the problem really is, which policy cannot change. The equations use its inverse, difficulty \delta. Above 3.3× ease (\delta \le 0.3, about 15% of worlds) fixing safety problems as they come up leads to long-term alignment without a general solution, and loss of control cannot happen	\log_{10}(1/\delta) \sim \mathcal{N}(0.10,\ 0.4), median 1.26× AI 2040 (calibrated)
\varphi	AI progress speed	\mathcal{N}(1,\ 0.3), clipped to [0.3, 2.5] AI 2040
W	If humans lose control of AI, whether it still cares for humanity (1) or not (0). If 1, AI rather than humanity steers the future, but kindly	W \sim \text{Bernoulli}(w),\ w \sim \mathcal{N}(0.10,\ 0.08) Assumption
a_{\text{CN}}	China's own alignment work, relative to the US bloc's. Work on keeping AI loyal to the Party line does not count	\mathcal{N}(0.35,\ 0.17), clipped to [0, 1] China experts
q_{\text{CN}}	How far China cuts its race once there are signs of losing control	\mathcal{N}(0.25,\ 0.15), clipped to [0, 1] China experts
e	Tests overstate alignment: how wrong the measurements are, which oversight, transparency and safety work can shrink (unlike ease). How much tests overstate alignment, because AIs can tell when they are being evaluated and behave better under test. A is the measured value players see	\mathcal{N}(0.15,\ 0.10), clipped to [0, 0.6] Assumption
\omega	Whistleblower odds	\mathcal{N}(0.25,\ 0.15), clipped to [0, 1] Assumption
R_{\text{bio}}	Decade risk of a 100M+ death pandemic enabled by open weights, on the default path	\log_{10} R_{\text{bio}} \sim \mathcal{N}(-3.92,\ 1.2) Bio decomposition

Throughout, \sigma(x) = 1/(1+e^{-x}) and \operatorname{clamp}(x) limits x to [0, 1]. Every state variable is clamped to 0–100 after each update; difficulty is clamped to [0.1, 5], whistleblower odds and kindness odds to [0, 1], and R_{\text{bio}} is capped at 0.99. Each year draws the same fixed set of dice whether or not they are used, so a rewound campaign keeps its luck.

3. Annual hazards

Engineered pandemic (100M+ deaths):

P_{\text{bio}} = 1 - e^{-k\,g}, \qquad g = \sigma\big(\tfrac{O-62}{6}\big)\Big(1-\tfrac{B}{100}\Big)^{2}

k = \frac{-\ln\left(1-R_{\text{bio}}\right)}{G_{\text{ref}}}, \qquad G_{\text{ref}} = \mathbb{E}\Big[\sum_{t=2027}^{\min(2036,\ \text{end})} g_t\Big]_{\text{default path}} \approx {{GREF}}

g is uplift from open weights, which matters once open models pass expert level, times missing biodefense. G_{\text{ref}} is the exposure that default-path futures actually go through from 2027 to 2036, summed until each future ends (most end at the superintelligence handoff around 2031), averaged over 3,000 fixed-seed futures. So on the default path a world's realized decade risk is its drawn R_{\text{bio}}, and policy moves risk through g. Paths that last longer with open models near the frontier face more risk: Plan A with fully open weights reaches about 2.2%, cut to about 0.9% by full biodefense (80,000 futures). G_{\text{ref}} is computed under the default assumptions; when the Lab changes progress speed or ease of alignment, risk scales with the exposure those futures go through relative to that baseline. Bio decomposition The thresholds are Assumption. The risk panel shows the expected probability over the unknown R_{\text{bio}}, \mathbb{E}\big[1-e^{-k g}\big], which rare bad worlds dominate.

Why the pandemic risk is so low. An earlier version of this model used a rough guess that put open-weights pandemic risk near 1.7% a year, about 21% over 2027–2040. A separate factored Monte Carlo estimate came out about 1,300 times lower, so the model now follows it. That estimate counts only pandemics that kill at least 100 million people and would not have happened without open weights, and every step from model access to a global catastrophe has to succeed. Multiplying those step probabilities gives a median of about 1 in 8,000 per decade. The model treats other AI-enabled pandemics as negligible: states able to build such weapons are deterred from using them, much as Superintelligence Strategy (Hendrycks, Schmidt and Wang) argues states deter each other's dangerous AI projects, and closed models keep capabilities away from the rogue actors the paper identifies as the main bioterror risk. The tail stays heavy: about a 5.5% chance the true decade risk is above 1%, and 0.7% that it is above 10%. You can widen or shift that distribution in the Futures Lab. Bio decomposition

Loss of control:

P_{\text{loc}} = h(\delta)\cdot 0.3\,\sigma\big(\tfrac{C-\hat{A}-36}{6}\big)\,\sigma\big(\tfrac{C-78}{4}\big)\Big(0.4+0.9\,\tfrac{H}{100}\Big), \qquad h(\delta) = \operatorname{clamp}\Big(\tfrac{\delta-0.3}{0.5}\Big)

\hat{A} = A\,(1 - e\,h)\,\Big[1 - \sigma\big(\tfrac{C_{\text{CN}}-C_{\text{US}}}{3}\big)\,(1-\chi)(1-a_{\text{CN}})\Big], \qquad \chi = \min\big(1,\ 0.55+0.45\,\tfrac{K}{100}+0.3\,\tau\big)

Risk grows once capability passes top-expert level and outruns alignment. Each year the three hazards are rolled in order, so the chance of a catastrophe is P_{\text{bio}} + (1-P_{\text{bio}})\big(P_{\text{loc}}(1-\bar{w}) + (1-P_{\text{loc}})P_{\text{war}}\big), where \bar{w} is the expected chance an uncontrolled AI is kind: \mathbb{E}[\operatorname{clamp}(w)] over its clipped bell curve. It is zero in worlds where alignment is easy by default (\delta \le 0.3) and fades in up to \delta = 0.8. Hidden internal AI amplifies it, since AI 2040 argues most takeover risk comes from models run inside labs AI 2040. When control is lost, the future ends as a Benevolent Takeover if W = 1 and as Loss of Control otherwise. \hat{A} is effective (real) alignment. A is what tests measure; the factor 1 - e\,h removes the part of the overstatement still hidden, h = \operatorname{clip}_{[0.1,1.4]}\big(0.6 + 0.8\,\tfrac{H}{100} - 0.5\,v - 0.3\,\tau - 0.3\min(1, e(s))\big): secret internal AI hides more of it, while oversight, published research and safety work expose it (about 0.7 today, 0.3 with full oversight and transparency). As China moves ahead, its lab has the US-bloc alignment work that coordination and published research share (\chi), and its own research covers part of the rest (a_{\text{CN}}). The risk shown averages over a_{\text{CN}}; the dice use the hidden draw. A unilateral US safety share above about 35% still does somewhat worse: the US slows enough that China often leads at the handoff with less alignment. Coefficients are Assumption.

Great-power war:

P_{\text{war}} = \Big[0.055\Big(1-\tfrac{K}{100}\Big)^{2}(0.3+0.7\,q)(1+1.5\,g) + 0.07\,g\,(0.4+0.6\,z)\Big]\,\sigma\big(\tfrac{C-60}{8}\big)

q = \operatorname{clamp}\Big(1-\tfrac{|C_{\text{US}}-C_{\text{CN}}|}{25}\Big), \qquad z = \operatorname{clamp}\Big(\tfrac{C_{\text{US}}-C_{\text{CN}}}{20}\Big)

A close race, low trust and high strategic stakes raise it. Sabotage g adds provocation, and a second term where a China that is falling behind under attack has a reason to strike first. Assumption

4. Endings

Superintelligence handoff. When C reaches 100:

\Pr(\text{handoff goes well}) = \begin{cases} 1 & \delta \le 0.3 \\ \sigma\big(\tfrac{\hat{A}-\beta}{6}\big) & \text{otherwise} \end{cases}, \qquad \beta = 45 + 32\,\delta + 0.2\,(H-30) - 10\,\tfrac{N}{100}

The bar \beta rises with hidden difficulty and with hidden internal AI, and falls when BCIs let humans check AI reasoning directly. A failed handoff is loss of control, which ends as a Benevolent Takeover if W = 1. A successful one still ends in Locked-In Power, a catastrophe, with probability

P_{\text{lock}} = \min\Big(0.95,\ 1.8\,\big(0.45+0.55\,\sigma(\tfrac{P-64}{6})\big)\big(0.6+0.4\,\phi\big)(0.8+0.4\,r)\big(1.3-0.6\,\tfrac{T}{100}\big)\,(1-0.7\,\tfrac{F}{100})(1-0.45\,v)(1-0.35\,\tau)(1-0.6\,\omega)(1-0.15\,\rho)\big(1-0.1\,o\operatorname{clamp}(\tfrac{O}{C})\big)\big(1-0.2\,d\operatorname{clamp}(\tfrac{T-50}{30})\big)\Big)

\rho = \max\Big(\operatorname{clamp}\big(1-\tfrac{|C_{\text{US}}-C_{\text{CN}}|}{15}\big),\ \operatorname{clamp}\big(\tfrac{O-80}{20}\big)\Big)

A few end up holding AI and nearly all its gains while most people lose their economic leverage and their say, whether power was seized or drifted there as automation moved value from labor to whoever owns AI. Concentrated power, automation, racing and low public trust raise it; a public wealth fund, oversight, transparency and whistleblowers lower it. Exponential growth pulls the frontier away from the pack, so a close rival or near-frontier open weights (\rho) protect only a little. The scale 1.8 is set so about half of status-quo futures where humans keep control end this way (Craig's estimate). Otherwise the future ends in an Aligned Transition.

where \phi is physical automation (it grows over the six years after the Automated Coder milestone, as robots follow software) and F is the public fund's stake. The stake starts at 3 and moves toward 100 f each year: when buying in, at a rate of 0.4\,(1-0.85\,\sigma(\tfrac{C-80}{5})) with C at the start of the year; when selling down, at a flat rate of 0.2. Buying in is slower later because buying into AI gets far more expensive once its value explodes, so funds built early own more; a full fund slows US capability growth by 5%. Coordination with China helps only when public trust is high: redistribution happens when voters demand it. This follows Craig Quiter's National Security Response: competition forces automation, and value moves from labor to whoever owns AI. In 2040, a future that has not ended becomes Locked-In Power if T < 45 and a lock-in roll (the same formula) succeeds, The Careful Path if effective alignment \hat{A} \ge C (real alignment, after the test overstatement and any shortfall in what reaches a leading China) and K > 55, and Muddling Through otherwise.

5. Yearly updates

Capability:

\Delta C_{\text{US}} = c_{\text{US}}\big(7\,\varphi\,U(C_{\text{US}}) + \varepsilon\big), \qquad c_{\text{US}} = \kappa(r)\,\max(0,\ 1-s-0.08\,v)\,(1-0.05\,f)

\Delta C_{\text{CN}} = c_{\text{CN}}\big(7\,\varphi\,U(C_{\text{CN}}) + \varepsilon\big) + (1-0.35\,g)\,0.25\,\tau\,\max(0,\ C_{\text{US}}-C_{\text{CN}}), \qquad c_{\text{CN}} = \kappa(r_{\text{CN}})\,m\,(1-0.35\,g)

U(c) = \begin{cases} 1 & c < 50 \\ 3^{(c-50)/20} & 50 \le c < 70 \\ \min\big(40,\ 3\,(40/3)^{(c-70)/20}\big) & c \ge 70 \end{cases} \qquad \kappa(r) = \Big(\frac{\max(0,\ r-0.03)}{0.52}\Big)^{0.74}

r_{\text{CN}} = \operatorname{clip}_{[0.05,1]}\Big(r_0\,(1-\eta)\Big), \qquad r_0 = \operatorname{clip}_{[0.05,1]}\Big(0.6 + 0.45\big(r(1-0.5\,s)-0.5\big) - 0.55\,\tfrac{K}{100}\Big)

\eta = \min\Big(0.9,\ q_{\text{CN}}\,\sigma\big(\tfrac{C-83}{4}\big)\,\sigma\big(\tfrac{C-A-30}{8}\big)\cdot 0.6\,\big(1+\max(u,\ D)\big)\Big), \qquad u = \operatorname{clamp}\Big(1-\tfrac{r(1-0.5\,s)}{0.7}\Big)

China slows (\eta) only on warning signs of losing control: capability near the top while alignment lags. It notices later than the US would, because bad news travels up a personalist system slowly (Susan Shirk, Overreach), and slows more when the US slows (u) or a verified deal is in force (D = 1). It never slows for fear of concentrated power: an AI that obeys the Party counts as aligned in Beijing. Experts on Chinese AI policy (Concordia AI's State of AI Safety in China, Matt Sheehan at Carnegie, SemiAnalysis's "Beijing will not pace the frontier") find real, growing safety research and official "human control" language, but no case of Beijing slowing frontier AI for safety China experts.

U is the AI R&D speed-up from AI labor: 3× at Automated Coder and about 40× near top-expert-dominating AI AI 2040. The compute exponent 0.74 encodes "10× less compute makes the intelligence explosion 5.5× slower" AI 2040, and the base rate 7 puts the default path at Automated Coder around 2030 and superintelligence about a year later AI 2040. Safety spending and oversight take compute away from capabilities. China's race intensity responds to the US posture and falls with coordination; publishing research helps China close the gap Assumption. Here \varepsilon \sim U(-1,1), m is the export-control multiplier set by campaign choices (1 in the Lab), and each year a surprise breakthrough adds 3–8 points to the US and 2–6 to China with probability 0.15, scaled by \min(1.5,\ c). Because noise and breakthroughs scale with the compute each side spends on capabilities, a pause holds capability still. Yearly gains are floored at 0. Milestone years in the results are the first year a future starts at or above the milestone, so "Automated Coder 2030" means it was reached during 2029, matching AI 2040's January 2030.

The deal. A verified US–China deal is in force when K \ge 60 and d \ge 0.5 and no deal has collapsed. It holds both blocs at 78, top-human-expert level, like Plan A's 2035 pause (a bloc already above 78 is frozen where it is). Campaign choices made while it holds cannot push capability past that cap. The deal still holds through the year it collapses, so the race restarts the following year. Each year it collapses with probability 0.06, about 46% over ten years against AI 2040's roughly 48%; a collapse cuts K by 40 and the deal never returns AI 2040.

Alignment:

\Delta A = \frac{0.55\,\big(1.9 + 10\,e(s) + 1.5\,v\big)\,\big(1 + (U(C_{\text{US}})-1)\,\ell(s) + (U(C_{\text{CN}})-1)\,\ell(0.3)\,a_{\text{CN}}\big)\,\big(1 + 0.35\,\tfrac{K}{100} + 0.4\,\tau\big)\,\big(1 + 0.5\,\tfrac{N}{100}\big)}{\delta} + \varepsilon_A

e(s) = \begin{cases} \max\big(0,\ \frac{s-0.01}{0.29}\big) & s \le 0.3 \\ 1 + 0.6\,\frac{s-0.3}{0.7} & s > 0.3 \end{cases} \qquad \ell(s) = 0.08 + 0.35\,\min\big(1,\ e(s)\big) + 0.5\,\frac{\max(0,\ s-0.3)}{0.7}

The factors are: research effort (safety share, with diminishing returns past 30%, plus oversight); AI labor pointed at safety, so alignment also speeds up during takeoff: the US bloc's AI by its safety share, and China's AI by its own alignment effort (at a_{\text{CN}} = 1, like a US bloc spending 30% of compute on safety), so a cautious China that leads the race speeds alignment up too; shared international and public research; augmented researchers through BCIs; and the hidden difficulty \delta. Here \varepsilon_A \sim U(-0.5,0.5), and with probability 0.12 a year an alignment insight adds 3 to 12 points (more with a higher safety share). The 0.55 scale and the spread of \delta are set so fixed-lever versions of AI 2040's plans reproduce its p(alignment) estimates AI 2040; the other coefficients are Assumption.

Everything else:

\Delta H = 4\,r\,\sqrt{U(C)} - 9\,\tau - 5\,v - 6\,\tfrac{N}{100} - 0.5

\Delta K = 13\,d + 2\,v + 3\,\tau - 6\,r - 10\,g - 0.5 + \varepsilon_K, \qquad \varepsilon_K \sim U(-1.5,\ 1.5)

O \leftarrow O + 0.45\,\big(C\,(0.62 + 0.36\,o + 0.06\,\tau) - O\big)

\Delta B = 9\,b + 1.2\,o - 1.6

\Delta N = \min\big(20,\ (11\,n + 0.6)\,(1 + 0.4\,n\,(U(C)-1))\big)

\Delta P = 10\,r\,(1 - 0.5\,o - 0.5\,\tau) + 0.06\,\max(0,\ C-60) - 2.5\,o - 4\,\tau + 2.2\,n\,(1-d) - 4\,v + 3\,g - 0.8

\Delta T = -0.04\,\max(0,\ C-55) + 2.5\,\min\big(1,\ e(s)\big) + 1.5\,v + 1.5\,\tau - 1.5\,r\,\tfrac{P}{100} + 0.3

Racing grows hidden internal AI, faster once AI automates R&D; transparency, inspections and augmented overseers pull it into view. Open-weights capability chases a policy-set share of the frontier. Biodefense decays without investment, and open tools help it a little. AI labor speeds BCIs up, but surgery, approval and rollout cap adoption at 20 points a year. Racing on closed models concentrates power; openness, transparency and oversight spread it. All coefficients in this block are Assumption.

6. Calibration

Holding each plan's levers fixed from 2027 to 2040 (node test-model.js):

Check	Target	Model
Automated Coder, default path	~2030 (AI 2040 median)	2030 (median)
Superintelligence after Automated Coder	~1 year (AI 2040)	1–2 years
p(alignment), race to superintelligence	25% (AI 2040 median)	15%
p(alignment), Plan B: fight China	50% (AI 2040 median)	46%
p(alignment), burn the lead	40% (AI 2040 median)	40%
p(alignment), Plan A	72% (AI 2040 median)	75%
Locked-In Power share of status-quo futures where humans keep control	about half (Craig's estimate)	47%
Bio decade risk actually faced on the default path: median	1.2e-4	1.0e-4
Bio decade risk: 95% interval	5e-7 to 3e-2	3.8e-7 to 2.5e-2
Bio decade risk: P(> 1%), P(> 10%)	0.055, 0.007	0.052, 0.007

Figures from 30,000–40,000 simulated futures. The realized bio median sits a little below the target because futures last different lengths of time, so each world's exposure varies around the average used for calibration. p(alignment) counts aligned handoffs as 1 and loss of control as 0, benevolent or not; futures still short of superintelligence in 2040 count their handoff odds given the hidden world; pandemic and war futures are left out.

7. What it leaves out

Covert projects and cheating on the deal (beyond a flat collapse rate), cyberattacks, countries other than the two blocs, misuse beyond bio, conflict between AGIs, anything after 2040, plans that change over time in the Futures Lab, and whether capability can really be put on one axis at all. Treat it as an argument you can poke at, not an oracle.

All parameters are illustrative. The lab, company and leaders in this game are fictional composites.

World state

Your decisions

Policy levers

Unknowns about the world

Risk this year

Your policy levers

2,000 simulated futures

World state

Your decisions

Policy levers

Unknowns about the world

Risk this year

Your policy levers

2,000 simulated futures

Copy this text

Can you win theGame of AGI?

Rolling a hidden world

YOUR DECISIONS

Game of AGI

Origins and sources

How the model works

1. One simulated year

2. Notation

3. Annual hazards

4. Endings

5. Yearly updates

6. Calibration

7. What it leaves out

Can you win the
Game of AGI?