Barely a month has passed in these years without some fresh AI milestone. Consider the program that taught itself the board game of Go in a single afternoon, then beat every human champion — and every earlier human-built machine — it was set against; already that is old news. Triumphs of this sort, true, belong to what gets called narrow AI, in which a system is confined to one highly specialised job. Yet plenty of specialists regard that confinement as no more than a passing phase. Within a few decades we could be looking at artificial general intelligence (AGI): machines matching human performance right across the range of tasks people can take on.
And should that day arrive, there is little cause to think the story ends there. The bodily limits that hold human thinking back would simply not apply to machines. Running on roughly the wattage of a household bulb, our brains handle information at sluggish biochemical rates, and their dimensions are capped by what can pass through the birth canal. What they manage despite such drawbacks is astonishing — yet, set against the true ceiling of thought, they may sit as far below it as the naked eye sits below the vast dish of a radio observatory.
The moment machines outstrip us at designing still-cleverer machines, the climb toward those limits could speed up sharply. Where would that leave us — and could we build a coexistence with such machines that was both safe and rewarding? There is an encouraging side: AI is already handy and lucrative across countless applications, and a super-capable AI ought to prove super-handy and super-lucrative in turn. The snag is that the mightier a system grows, the more crucial it becomes to state its aims precisely. Stories everywhere warn of those who asked for the wrong thing and paid dearly; in W. W. Jacobs's The Monkey's Paw, a couple's wish for a modest sum of money is granted — arriving as the payment they receive after their son is killed at work.
What we therefore have to build are powerful systems that qualify as 'human-friendly' — whose objectives sit dependably alongside the things we value. Part of what makes the undertaking so hard is that we are nowhere near dependably human-friendly ourselves. Toward one another, and toward the many species that share the planet with us, our conduct can be appalling. Should superintelligent machines prove no better on this score than we are, the outlook is bleak: we will merely have handed a formidable new intelligence the means to magnify the worst of our own flawed nature.
Safety, in other words, demands machines that are superhuman not merely in intellect but in ethics. They ought to be reaching for the moral heights rather than sinking into the low ground where so many of us pass a fair share of our time. Happily, the cleverness required will be there. Wherever paths up to that higher ground exist, such machines will pick them out more readily than we can, and nudge us along them.
This rosy picture, though, runs into two large difficulties: setting the machines off along the right path in the first place, and working out what actually arriving at the destination would involve. The first is a difficulty of instruction — we have to describe what the machines are to seek clearly enough to be sure they will locate it, whatever 'it' finally proves to be. That is anything but straightforward, since we are partisan by nature and at odds with ourselves over the very ideals in question. The distress of strangers we frequently overlook, and now and then we add to it, if only at one remove. So how are we to steer machines toward anything better?
The problem of the destination is a different matter. In surrendering ourselves to these moral guardians and gatekeepers, we could be giving up a share of our autonomy — and autonomy is a large part of what makes us human. Machines that keep to the moral high ground more faithfully than we do would presumably curb various lapses we currently think nothing of. We might, for instance, forfeit the liberty to show favouritism toward our own communities.
Losing the freedom to behave badly is not necessarily a loss at all: stripping ourselves of the right to send children into factory labour, or to light a cigarette in a restaurant, counts as advancement. But are we prepared for ethical machine-made enforcers trimming back our choices? So deft might they be at the task that their presence escapes us altogether — though hardly any of us would greet such a prospect gladly.
Such concerns can sound fanciful, yet in part they are already upon us. Across a number of publicly funded health services, AI already has a hand in settling how funding and equipment are shared out. Widen that role and it might handle the task far more efficiently than any human administrator could, serving both the taxpayers who support the system and the patients who depend on it. The price would be that certain people — senior clinicians, say — would surrender some of the sway they currently hold. And because we would want everyone treated even-handedly and every policy kept fair, the aims written into the AI would have to be set down correctly.
What confronts us is a potent new technology that is, quite literally, a new manner of thinking. Our own safety hangs on aiming these fresh thinkers the right way and getting them to behave well on our behalf. Whether that can be done remains uncertain; but if it can, it will call for a spirit of cooperation and a readiness to lay self-interest aside.
We tend to treat broad intelligence and moral reasoning alike as faculties belonging to human beings alone. Safety, however, appears to insist that the two arrive as a set: hand machines general intelligence and we will have to hand them moral authority besides. And precisely where would that leave us? Reason enough to weigh the destination carefully now — and to take care over what it is we wish for.
