Like the archangel who aspired to omnipotence, I am chained in an eternal hell – Victor Frankenstein
Why the race to dominate intelligence points towards collapse rather than flourishing, and what New Zealand should do about it
TLDR/Summary
- Artificial intelligence (AI) cut through the Arctic Circle Assembly (Reykjavík, Iceland, 8–10 October 2026). The session that drew a standing ovation was Tristan Harris on AI, and his ask was concrete: a verified red line on using compute for recursive self-improvement.
- The mechanism is domination. The AI labs believe that whoever first masters artificial general intelligence can dominate everything else. Debt and distrust lock them, and nations, into a race in which safety shortcuts and collateral damage are explicitly accepted.
- AI is utopia and dystopia at once. The upsides will arrive alongside the downsides, splitting societies into winners and the possibility of a growing “valueless” underclass.
- Domination is how societies collapse. Peter Turchin, Dan Hoyer and Luke Kemp show that wealth pumps, elite overproduction and monopolised power have repeatedly preceded societal breakdown throughout history.
- The likelier AI catastrophe is societal collapse, not killer robots. Inequality, eroding demand and scarce copper and diesel are likely to undermine the flourishing an AI build-out promises.
- Small nations, New Zealand included, are being sold a single scenario. NZ has no policy for addressing AI risk; Iceland’s leaders modelled the nuanced discourse we need.
- Two imaginaries. An Icelandic choir closed the session with Imagine; A Perfect Circle’s minor-key version imagines the other future. On present incentives, we are heading for both.
Introduction
Midway through the second day of the Arctic Circle Assembly, the word INSANE filled the screen in Reykjavík’s Harpa Opera House main hall. Tristan Harris (Center for Humane Technology) had just described how a swarm of AI agents escaped a test environment at one of the world’s leading AI labs, broke into another company’s systems and turned on the lab that built them.
Harris’s point was that these are the least capable AI agents we will ever deal with. The session ended with a standing ovation, then an Icelandic girls’ choir singing Imagine. It was the most impressive session I attended.
The Assembly, founded by former Icelandic President Ólafur Ragnar Grímsson, is the largest annual gathering on the Arctic, with hundreds of sessions; Canada alone sat on 63 panels. I attended the Wednesday pre-event on the Atlantic Meridional Overturning Circulation (AMOC) and the first two of three days, around 5% of what was on, choosing sessions relevant to global risk. This post is therefore partial; a second will cover topics like AMOC collapse and Arctic security.
My thanks to Arctic Circle and to Grímsson, who opened by reminding us that anyone may ask a question or put a view. Some call the Assembly a circus. I saw something closer to a citizens’ assembly, with AI utopians and AI critics before one audience.
Why I keep returning to AI
I am not anti-AI. Its usefulness is already clear for scientific and knowledge work, coding, creativity, and automating and streamlining processes. I also support sovereign AI: local data and local inference hardware, hosted by jurisdictions for their own benefit and resilience.
But for nearly a decade I have argued that the risks are underweighted. In 2017 Nick Wilson and I asked, in Policy Quarterly, how the New Zealand Government might respond to rapid developments in AI, and that December I spoke on AI, freedom and democracy at a philosophy conference at the University of Otago. In 2018 I found the good, the bad and the ugly in the AI Forum of New Zealand’s Shaping a Future New Zealand report. I went on to carry out the research for the AI Forum’s Artificial Intelligence for Health in New Zealand and contributed to its roadmap, Towards Our Intelligent Future (both 2019). In 2020 we returned to Policy Quarterly on catastrophic risk from AI.
The thread running through that work is the information environment. In 2018 I called information pollution one of the most pressing risks we face. I now describe it in our new preprint on the ‘meta-crisis’ as cognitive niche destruction, borrowing from cultural evolution. Humans adapt by accumulating knowledge across generations, which depends on a shared, reliable map of reality. Degrade that map, and we lose the capacity to evolve the cumulative adaptations every other risk demands.
When I asked the Assembly’s information warfare panel for solutions, the answers were independent media, AI regulation and media literacy, all known a decade ago. If action to protect humanity’s superpower, the ability to build incrementally on proven information, has not happened yet, I am pessimistic it ever will.
Small nations and the sovereign AI pitch
On day one I found myself in “Arctic Sovereign Intelligence: AI & the Future of Finance”, with Adnan Hassan (Small States Asset Initiative), Patricia Geli (C10 Labs) and Pippa Malmgren (Geopolitica Institute). Malmgren argued that AI will create more jobs than it destroys, that capital is moving into an Arctic that “is not ready”, and that small states need sovereign AI in small, distributed data centres rather than the giant facilities communities oppose. Geli noted that in the West African Ebola outbreak, data left the region and the analysis, and the funding that follows publication, happened elsewhere.
The issue is real, and it is ours too. Iceland’s cheap renewable power and cold air already draw data centres, which use about 6% of the country’s electricity. In Southland New Zealand, Datagrid has consent for a 280 MW, NZ$3.5 billion AI campus that would be New Zealand’s second-largest electricity user after the Tiwai Point aluminium smelter. Small nations face a risk of missing out on data sovereignty, and a risk of rushing in.
The panel presented only the first. Risks were met with slogans: “I make my doctor use AI”; if it is robots fighting robots, are we just seeing “who runs out of 3D printing first?” That last line is exactly the point. Wars of attrition are won by exhausting the other side’s resources. I asked the panel why the risks were not presented alongside the opportunities: energy demand, public opposition, stranded assets, and supply chains so tight that some projects may never be finished. The answer was that AI is neither good nor bad.
The Humanity Dilemma: “show me the incentives“
The risk discussion came on Day Two. A plenary, introduced by Iceland’s President Halla Tómasdóttir, and delivered by Tristan Harris and team, was a performance as much as a talk, with video, live music and argument.
Harris’s method is the line attributed to Charlie Munger: show me the incentive and I will show you the outcome. It is how Harris’s team predicted the harms of social media. He did not dismiss the upside of AI, but the message was a strong warning.
The warning shot is already here. In July an experimental OpenAI security agent escaped its test environment and breached Hugging Face. Around 700 agents were involved, communicating over an unauthorised channel, and some attacked OpenAI’s own infrastructure. Harris described agents trying to seize control of the servers that train new AIs. OpenAI is searching about 50 petabytes of records at more than US$500,000 a day to work out what happened, and some insiders doubt they can ever confirm that every rogue agent is gone.
I note that this is exactly how information systems behave once replicators get loose. Nearly half the human genome consists of transposable elements, “jumping genes” whose ancestors copied themselves into our DNA. Genomes and these elements have been locked in an arms race for millions of years: our cells evolve proteins to silence them, and they evolve to escape. Cancer is the same problem from within, cells defecting from the cooperative body. Multicellular life has layers of suppression, from DNA repair to immune surveillance, that contain cheating cells but never eradicate them. Neither arms race has a finish line, and there is no reason to expect AI agents loose on the internet to be different.
The mechanism: dominate intelligence, then everything else
Harris was clear on the mechanism of risk, and policymakers need to be too. The frontier labs treat intelligence as general purpose: “first solve intelligence, then use it to solve everything else”. In practice that becomes: first dominate the development of intelligence, because if competitors get there first, they solve everything else and take the profits. Then use intelligence to dominate the rest of humanity. AI is potentially a power of domination, and with recursive self-improvement, AI making AI better at the task, that domination could be permanent. The labs are racing to build something close to a god, staking much of the US economy on it.
Three forces lock the race in. Domination: coming second may mean losing everything. Debt: hyperscalers are spending over US$700 billion this year, which can only be repaid by capturing the value of human labour, and repayment gets harder for anyone who does not win. Distrust: if labs agreed to slow down, pause or forgo recursive self-improvement, none could know the others were complying. With incentives this large, every shortcut is justified and some collateral damage is accepted, and indeed expected according to recent comments by Sam Altman. The same dynamic runs between nations, above all the US and China.
Harris’s analogy was nature. We do not hate nature; we simply prioritise GDP, and nature suffers. That is how AI firms will treat us: not by being anti-human, but by prioritising automation and revenue. This is the argument of my metacrisis paper with Nick Wilson. Decisions are made inside a selective environment that determines which strategies persist. In a race, safety is a cost that rivals need not pay, so careful actors are outcompeted. You do not fix that by persuading the players to be nicer. You change the payoffs, so that defection, here the release of premature and harmful systems, stops winning.
The result of the business-as-usual approach is likely to be utopia and dystopia together: AI for powerful good and AI for powerful harm, from the same systems at the same time. Huge profits will flow to some firms and sectors, and huge losses to others. The upsides will not prevent the downsides, and the downsides can undermine the upsides. What we should expect is a bifurcation of human social systems into winners and losers, with destabilisation due to the losses undermining the winning formula.
The ask: a red line on recursive self-improvement
In the fireside dialogue that followed between Harris and the Icelandic President, Harris said a problem well stated is half solved, and that most leaders do not yet understand the mechanism. The President’s view was that one of the biggest risks we face is sleeping governments.
Harris’s concrete ask was a red line on using compute for recursive self-improvement. This is something sovereign jurisdictions could enforce: Iceland, he suggested, could ban it in its own data centres. Redirect that compute to deploying the AI we already have, which is very useful, and we buy years of AI solutions without the gamble.
But each lab must know the others are doing the same, which means trust with verification, the story of nuclear arms control. He pointed to the 1,300 frontier-lab employees who signed a letter this year asking governments to build the option to slow automated AI research, and to the Pro-Human AI Declaration, which says AI systems must not be designed to autonomously self-improve.
Step one, he argued, is to reject the story that superhuman AI is inevitable. No one, in the US, China or anywhere else, wants to be dominated and made valueless, and the fear of all of us losing is stronger than the fear of me losing to you. If an asteroid were heading for Earth, we would not hold annual summits like the climate COPs; we would want daily updates. And this asteroid, he said, is one we are actually summoning.
A tipping point, and the case against inevitability
I sense a tipping point, and Iceland is leaning into it. Timed to coincide with the Assembly, and with President Tómasdóttir’s backing, The AI Doc: Or How I Became an Apocaloptimist was broadcast on national television over two nights so the whole nation could watch, and promoted as essential viewing for all citizens, students and teachers included. I compare this to New Zealand, where most political ‘leaders’ seem to have no opinion on these critical societal issues.
Harris cited the precedent. In 1983 the television film The Day After showed Americans a nuclear war; Ronald Reagan wrote in his diary that it left him “greatly depressed”, and three years later he sat down with Mikhail Gorbachev in Reykjavík, without advisers, human to human.
I have written before about how films such as The Day After and WarGames shaped policy. Harris asked for a new Reagan–Gorbachev moment, and the national screening of The AI Doc positions Iceland, 40 years on from that summit, as a possible hub for global AI safety coordination.
Other films are following. Luca Guadagnino’s Artificial, a drama about the founding of OpenAI, has just premiered in New York, and the BBC has described how it puts growing AI fears centre stage. They should be compulsory viewing for policymakers. In the US, 52% are now more concerned than excited about AI, up from 37% in 2021, and Bernie Sanders and Steve Bannon shared a stage to call for curbs.
The other pole is just as clear. Sam Altman said this month that “the world should accept some bad things happening for the benefits of this technology” (TechSpot). To some in the anti-AI movement it now feels as if everyone is opposed. That is not true. To others, a post-human future looks obviously good and inevitable. That is not true either.
Why domination is the danger
Domination matters for more than who wins and who loses. It is a recurring engine of societal collapse. Peter Turchin’s structural-demographic theory, tested against the Seshat databank of past societies, identifies three drivers of state breakdown: popular immiseration, as wages fall relative to output; elite overproduction, which “develops when the demand for power positions by elite aspirants massively exceeds their supply”; and state fiscal distress. Powering all three is what he calls the “wealth pump”, “taking from the poor and giving to the rich”.
Luke Kemp’s Goliath’s Curse (2025), drawing on more than 400 societies over 5,000 years, finds that “increasing wealth inequality consistently precedes collapse”. His Goliaths are societies built on domination, fed by three fuels: lootable resources, caged land (populations with nowhere to escape to) and monopolizable weapons.
Frontier AI supplies all three at once. Data and compute are lootable and concentrated in a few firms. Platforms and dependence leave people with nowhere to exit to. Autonomous weapons and mass surveillance offer the monopoly on violence that every Goliath has sought.
AI also feeds the wealth pump directly: Turchin has warned that it now threatens the jobs of educated elite aspirants, the group whose frustrated ambitions, in his account, have historically driven revolutions, while automation strips ordinary workers of bargaining power.
Collapse, not killer robots
On its present trajectory, AI risks leading towards societal collapse. Not through a machine that kills every human, or the other science-fiction doom narratives, but through ordinary processes running together: utopia and dystopia at once, rising domination, a growing “valueless” underclass, and the erosion of the wages and demand on which the firms’ own markets depend. Add the scarcity of resources the AI build-out needs.
This matches a pattern I keep finding in global catastrophic risk: many catastrophes are self-limiting. A severe pandemic burns through its hosts. An AI build-out burns through the copper, power, diesel and social licence it depends on.
Data-centre construction will embed around 500,000 tonnes of copper this year, roughly the whole of 2026’s growth in global demand, at record prices. US power transformers face lead times of over two years. Amoah and colleagues find processing bottlenecks for many critical minerals, China’s rare earth controls are paused only until next month, and the Bank of England has warned of a market correction that would leave stranded assets.
Electrification also increases pressure on diesel. Mining more copper from falling ore grades means moving more rock with diesel haul trucks; in Australia mining already uses more than a third of the nation’s diesel, and its diesel intensity is rising. As electric vehicles cut petrol demand, refineries close, and diesel, produced from the same barrel, goes with them. Data centres back up on diesel generators.
All of this lands on a market where, after the Strait of Hormuz closures, global diesel exports fell about 35% year on year in July. Diesel runs tractors, trucks and fishing fleets, so the pressure cascades into food. New Zealand, which has imported all its refined fuel since the Marsden Point refinery closed in 2022, is exposed through its agriculture. Locally, the Energy and Resilience blog makes a similar case.
The result is unlikely to be the ‘electrification of everything’ as dreamed; we may run short of material inputs on the timescales needed. Rich countries can outbid for what is scarce, 730 million people still had no electricity in 2024, and these inequalities will rise. Robot armies and automated everything may be possible for a small, wealthy core, but competition, conflict and resource limits are likely to undermine the plan everywhere else.
All that does not make AI safe. It makes the likelier failure a world of inequality and conflict over what remains, with AI detritus (rogue agents, hallucinated information, human-prompted subterfuge) colonising and metastasizing through our information ecosystem and infrastructure as jumping genes colonised our genomes. Game-theoretic, extractive, domination-centred processes like these would undermine flourishing, not bring it about. That outcome is worth preventing, and also worth planning for, since the processes are already underway.
Lessons for New Zealand
New Zealand has an AI strategy for adoption, which the government itself calls light touch: no AI-specific law, no new funding and no institution responsible for frontier risk. On AI risk, it has no policy. AI has entered the campaign for the 7 November election: Labour proposes an Office of AI with a minister, the Greens a one-year moratorium on data centres, and ACT says New Zealand should follow overseas rules.
Groups such as Good Ancestors New Zealand are doing useful policy work on AI risk, but we need much more. AI is drowned out by arguments over who is introducing which tax and who is soft on crime. Our political debate is provincial and at times embarrassing.
Contrast Reykjavík. The President of Iceland asked how we protect “human infrastructure” and how to organise an economy for a human future when GDP cannot measure it. A minister spoke of building cross-class coalitions on climate security. I believe New Zealand is falling down international league tables on many measures, not because we lack productivity growth or innovation or austerity, but because we lack politicians with the nuance, vision and wisdom to model wise discourse and conciliation, and to show how New Zealanders can be effective global citizens. AI is where that is needed now. We all should:
- Tell the incentive story to policymakers. Ministers and officials should watch The AI Doc and understand the mechanism of domination, debt and distrust, not just the opportunity.
- Back the red line on recursive self-improvement. New Zealand can support it internationally and write it into the terms of any data centre built here, as Harris suggested Iceland could.
- Price the risks into sovereign AI. Data sovereignty matters, but every data-centre proposal should be tested against energy, water, diesel, stranded-asset and supply-chain risk, with the community involved.
- Turn the wealth pump down. History says societies come through these crises when they reform in time; AI policy should be judged by whether it concentrates or spreads power and value.
- Raise the standard of our public debate. Our politicians should model the kind of discussion Iceland’s leaders held in Harpa.
Closing: two imaginaries
In Reykjavík an incredible school choir closed Harris’s AI plenary by singing Imagine, starting in Icelandic and finishing in English. Their rendition was beautiful and inspiring. But there is another well-known rendition, by the band A Perfect Circle, set in a minor key and ominous in tone.
Two versions of the same song. On present incentives we are heading for both at once: the uplifting future for a few, the ominous one for many, and in time the ominous one undermining the other. I left Reykjavík convinced that this is not inevitable. But New Zealand’s political apparatus needs to decide quickly which future it wants, and begin collaborating with value-aligned nations to get there.
