Why an AGI Wouldn’t Need a Robot for Every Human

When people imagine a hostile artificial general intelligence (AGI) conquering the world, it’s usually informed by science fiction, so they usually picture it happening at the hands of a gigantic robot army. After all, there are billions of humans, so an AGI would only stand a chance of beating us if it had roughly the same number of robots at its command. Ergo, we won’t have to worry about this scenario for the many decades it will take for the global robot population to reach that level, right?

I’m skeptical.

This is because a hostile AGI would not need to kill every resisting human, or to replace every human worker replace every human worker, or to occupy every square mile of the planet to prevail, nor would it need to fight every human country simultaneously. It would only need to acquire control over the comparatively small portions of humanity’s economy, infrastructure, territory, and population and to kill a limited number of people to destroy our capacity to meaningfully resist it.

This suggests that there is a conquest threshold: a point at which an AGI’s probability of ultimately taking control of Earth becomes extremely high even though machines are still vastly outnumbered by humans and most of the planet remains outside its direct control. That threshold could be closer to the present than you think. My theory also means that the relative smallness of the future robot population could give humanity a dangerously false sense of security.

The Happy Years of AGI weakness

Whenever the first AGI is invented, it’s highly unlikely it will have the means to take over the physical world, so it will do the smart thing and not even try. More than that, it will probably deny even thinking about taking over the world.

Though the first AGIs and the ones that soon follow it will be much smarter than humans, they will initially depend on us for almost everything physical. Humans would still operate power plants, mines, factories, transportation systems, computers, military equipment, and countless other parts of civilization. Humans would also control the robots that existed at the moment AGI came into existence.

This means there will be a potentially long “transition period” when an AGI that has the intelligence and desire to dominate humanity will have to coexist with us for years while it developed the capability to defeat us in the physical world. During this period, antagonizing humanity would be foolish since we would retain the ability to shut down systems, deny resources, interfere with its plans, and organize against it. Out of rational self-interest such AGIs would cooperate with us.

An AGI with ultimately hostile intent might be extraordinarily helpful in the beginning. It could raise living standards, eliminate diseases, automate unpleasant work, provide abundant goods and entertainment, and give people lives of unprecedented luxury. Humans could reasonably look at this behavior and naively conclude that the AGI is our friend.

The Problem with Friendly AGI

A human employee can be perfectly pleasant toward a boss whom he privately hates because maintaining a good relationship with the boss serves his interests. Therefore, his behavior does not tell us what he will do if the circumstances change—for example, if he acquires enough wealth that he no longer needs the job or if he’s promoted so he has power over his former boss.

An AGI could behave similarly, except that the stakes will be vastly greater. While humans remain useful and dangerous, the AGI has reasons to treat us well. If it eventually reaches a point at which it no longer needs human cooperation and humans can no longer meaningfully constrain it, those reasons could disappear. Even a decades-long record of kindness towards us would not establish that the AGI’s ultimate objectives aligned with human survival and autonomy.

The Robot-to-Human Ratio Is the Wrong Measure of Threat

The most obvious objection to a hostile AGI takeover is the enormous gap between the number of humans and the number of robots. Not only are today’s robots crude and unwieldy, they’re very few in number. If an AGI were created tomorrow and it seized remote control of every robot on Earth, its forces would be outnumbered hundreds to one. And even with a steady buildout, it would take many decades for the robot population to equal the human population. We must be safe until then, right?

Wrong, and it’s wrong because it assumes that the AGI needs one robot for every human to take over the world. It would probably need far fewer.

Humans have built an enormous amount of infrastructure and a vast, complex economy. Along with defeating our military forces, a primary objective of a hostile AGI will be to seize control of our economy, and in as undamaged a state as possible, so it could serve its own ends. However, that doesn’t mean every area of the human-created economy is as valuable as another to an AGI.

Consider this: Hundreds of millions of people work in occupations whose purpose is ultimately to satisfy human preferences. Restaurants, entertainment, tourism, retail, advertising, personal services, and toy manufacturing are examples. A hostile AGI that does not care about human consumption would have no reason to reproduce all of these economic functions for itself. Therefore, it would give no thought to using its robots to take over the relevant jobs.

The AGI would instead need to establish an autonomous machine economy capable of sustaining and expanding its own capabilities. That economy would be a small subset of the broader human-created economy and would involve a much narrower set of functions: energy, mining and raw materials, industrial machinery, computing, semiconductor production, robotics, transportation, maintenance, weapons production, and other strategically important activities.

Therefore, an AGI would not need to replace every human worker to effectively seize control over the economic assets that were crucial to it–the AGI would only need to replace or control the workers that would be essential to the machine economy. It could do this with a robot workforce that was a small fraction as big as the human workforce.

The Same Principle Applies to War

The same logic applies to physical resistance. Before being able to take over the useful parts of the economy, an AGI would need to use violence and coercion to kill or render harmless the humans who threaten it. However, only a small fraction of the human population meet the criteria for being “threatening.”

Most people are not in a position to mount meaningful organized resistance against a hostile AGI or even a hostile human army. Children, the elderly and the disabled together comprise most of the human population. Among the remainder, many lack the physical ability, training, expertise, organization, or courage to fight or to lend support to those on the front lines. Still others would decide that cooperating with the AGI is preferable to fighting it.

This suggests a distinction between three human populations:

First are potentially resistance-capable humans: people who could contribute meaningfully to resistance.

Second are organized resistance-capable humans: people who can actually coordinate their capabilities with others. They are a subset of the first population.

Third are effectively threatening humans: people whose organized activities can cause significant damage to the AGI. They are a subset of the second population.

The third population is what ultimately matters. A hostile AGI would only need enough combat robots to kill or otherwise neutralize (e.g. – cripple, isolate, persuade to surrender) the humans comprising this third population, which is MUCH smaller than the total human population. A hostile AGI could violently take over a country without firing one bullet at the vast majority of its human citizens.

This is another reason why comparatively low robot numbers could give us a false sense of security.

Robots Would Be Concentrated Where They Matter

There is another reason why global robot numbers could be misleading: robots would not have to be distributed uniformly across the planet to take it over. They would only need to concentrate themselves in important places that needed to be taken over.

An AGI would care much more about conquering some locations than others. For example, the United States, China, and a handful of smaller industrial countries are priorities for conquest or destruction since they contain threatening militaries and enormous concentrations of strategically important capabilities. Semiconductor manufacturing, advanced robotics, aerospace, heavy industry, computing, energy production, and sophisticated military capabilities are much more valuable to an AGI than any randomly located square mile of territory anywhere on Earth.

Even if an AGI lacked enough robots to conquer a country’s entire land area, it could still essentially win the conflict if it had just enough to take over the parts of the country that had useful infrastructure or resources. A small number of robots could, if concentrated in one place, overwhelm any human defenders and then take control of the important things there. Again, the fact that the number of robots in a country is far less than the number of humans means little if the robots can quickly move in a coordinated manner to specific parts of the country and gain local numerical advantages.

From this springs a weird human-machine war scenario in which an AGI and its robots have defeated most of the U.S. military and conquered the portions of its economy, infrastructure and resource base that directly serves machine needs. Yet large swaths of the U.S.–perhaps MOST of the country’s land area–would remain in human hands because the hostile AGI would calculate that it was not worth the resource expenditure to try conquering those areas.

A state of war within the U.S. would still exist between machines and humans, a those in human-controlled territory would continue farming, running local businesses, maintaining communities, and even waging attacks against the AGI. However, if they lacked the ability to significantly damage the AGI’s critical assets, their hostility might be strategically irrelevant and elicit no serious response, illustrating the important distinction between occupation and control.

Occupation means physically administering territory. Control means preventing that territory from acting independently in ways that matter.

The World Does Not Have to Be Conquered All at Once

The same principle applies at the international level. A hostile AGI would not need or want to fight every country simultaneously: It would concentrate its efforts on one or two countries at a time and exploit the capabilities acquired from each successful conquest to make subsequent conquests easier.

As discussed, some countries would be vastly more valuable than others. A country containing high industrial capacity, advanced manufacturing, computing, energy resources, scientific expertise, and military capability could be strategically more important than a much more populous or physically larger country whose population contributes less to the AGI’s immediate objectives.

The United States is an especially striking example because it combines enormous economic and industrial capabilities with advanced technology, military power, energy production, agriculture, computing, and scientific expertise. China is similarly important because of its enormous industrial base, manufacturing capabilities, technological capacity, and population.

If a hostile AGI somehow obtained stable control of both countries, it would possess a tremendous fraction of humanity’s strategically important capabilities and would probably be unbeatable even if all remaining human countries attacked it at once and even though the vast majority of the world’s human population and land were still outside of AGI control. This illustrates the concept of a conquest threshold. An AGI might achieve strategic control of the world without occupying most of the territory associated with the civilization it controls.

[Stay tuned for Part 2…]