| Next: | 4: Policy implications | Bookmark | |
| Up: | Creating Friendly AI | Monolithic | |
| Prev: | 3.5: Developmental Friendliness |
The Transition Guide is the hoped-for outcome of Friendly seed AI - a successfully Friendly AI, at or substantially above the transhuman level, with enough intelligence that the potential exists to exert an influence on the world that is different in kind from the capabilities of humans, or any groups of humans. Given multithreaded serial-fast intelligence, there are several "ultratechnologies" (such as nanotechnology) that probably lie within fairly near reach of present-day capabilities, or certainly the capabilities of a few years hence. For example, the cognitive capability to crack the protein folding problem suffices to attain the material capability of nanotechnology in whatever the mininum-turnaround time is between a custom DNA strand synthesis lab and a custom DNA-to-protein synthesis lab, both readily available commercial capabilities in today's world. But such augmented material abilities are probably not truly necessary. A transhuman intelligence, in and of itself, is enormous power with or without the trappings of material technology. We don't know what a transhuman could do because we are not ourselves transhuman. This is the effect that lies at the heart of the Singularity - stepping outside the bounds of the last fifty thousand years.
Ultraspeed intelligence and ultrapowerful material technologies are ultimately only human projections. Given the near-certainty of technical feasibility, we can say that such capabilities represent lower bounds on the Singularity; we cannot, however, state upper bounds. The Singularity begins immediately after the rise of transhuman intelligence. Even if things simply mysteriously halted there, even if superintelligence and nanotechnology (or their successor versions) never came into existence, a Singularity would remain in effect; the world would still be changed in unpredictable ways. We don't know in how many ways humanity is missing the obvious solutions, the obvious ways to do things. If you haven't read any true Singularity material, then I most strongly advise "Staring into the Singularity", or some of Vernor Vinge's books, or something that conveys the idea that moving from human to slightly-transhuman is just as large a change as moving from Homo erectus to Homo sapiens.
If, as seems both likely and desirable, transhumanity first comes into existence as the result of AI, and if that AI is Friendly, then Friendliness is called upon to shoulder the massive burden of deciding, not only the future, but how much of the future should be decided. Transhumanity is almost unlimited power-in-potentia; the question becomes, not just how this power should be wielded, but whether it should be wielded, or even whether that power should be acquired in the first place.
As humans, we fear power because we exist in a world of humans who have an innate tendency to abuse power. Given the amount of evolutionary pressure behind power-abuse behaviors, and given the specificity of those behaviors for evolutionary advantage, I think that fear of minds-in-general is unfounded. I thus think that there is no good reason why a Transition Guide should not race at maximum speed to superintelligence and the greatest obtainable degree of material power. However, that verdict is strictly contingent upon the statement that only humans are corruptible by power. The decision is both real and serious, and a Friendly AI needs to be able to make it.
Friendliness is ultimately called upon to understand the idea of "humanity's destiny", in the way that a human would. Humanity's destiny will very probably fall into the hands of a self-modifying AI, hopefully a Friendly one, because self-modifying AI is the easiest path to transhumanity. Perhaps the Friendly decision is to never acquire that power, but the possibility would still exist in potentia. I think that the Friendly decision is to acquire that power. Either way, it exposes the final design requirement of Friendly AI. A Friendly AI needs to be able to understand all the philosophy that led up to vis creation. A Friendly AI needs to understand the philosophical issues bound up in the idea of "Friendliness" itself.
I've never heard a really good, non-fairy-tale argument for why pain and death are good things. Nonetheless, a lot of people seem to think they are. I don't think that means the argument somehow becomes more plausible; in our world, large chunks of people are often massively wrong. Nonetheless, lots of people asserting a cosmic meaning for death means that there is no excuse for being taken by surprise if that possibility does somehow materialize. That many people warn you about a fairy-tale possibility doesn't increase the actual probability, but it does mean that if the possibility does show up, and you're not ready, after you were warned about it, then you deserve to be drummed out of humanity for sheer absolute incompetence. As best I can tell from the current theory, between shaper/anchor semantics and causal validity semantics, a Friendly AI can handle, or learn to handle - or, at worst, admit that ve can't handle - any possibility that human beings can understand or even admit that we can't understand, from the moral value of death to the existence of psychic powers to the noncomputability of qualia to fairies in the garden. And this is necessary because a Friendly AI is humanity's emissary into an absolute unknown.
I have a visualization of a sample better future that lies on the other side of the Singularity. With the volitional-Friendliness respect for individual wills as structure, and the material technology of superintelligence as power, I expect that the result would be apotheosis - the attainment of the best of all possible worlds consonant with the laws of physics and the maintenance of individual rights. Not just freedom from pain and stress, or a sterile round of endless physical pleasures, but endless growth for every human being and the new beings we create - growth in mind, in intelligence, in strength of personality; life without bound, without end; experiencing everything we've dreamed of experiencing, becoming everything we've ever dreamed of being. Or perhaps embarking on some still greater quest that we can't even conceive.
I don't believe that there are fundamental boundaries, either ethical or physical, that would keep humanity as it is now. There were no fundamental boundaries preventing humanity from crossing over from hunting and gathering to the Internet - for all that almost every local transition along the path was protested ethically and theologically, and predicted to result in failure or terrible consequences. There is nothing holding the present-day world in place; the present-day world, like the worlds before it, is something that "just happened".
I think that humanity's destiny, insofar as there is one, would be fulfilled as completely as possible in a universe that is less hostile and more friendly. Others seem to think that pain is necessary for growth. I, in turn, think such people have been living on Earth too long. Sure, there are some kinds of human adulthood that require massively unpleasant life events, but that's a flaw in our underlying cognitive architecture, not something that carries over into the transhuman spaces. The Transition Guide would need enough of a shaper network - enough real philosophy - to decide whether I'm massively wrong, or the Luddites are massively wrong, or we're all wrong; or whether it doesn't matter who's massively wrong because decisions like that are up to individual sentients.
Greg Egan's Diaspora offers a vision of a future that is, so far, one of the most unique in science fiction. The vast majority of humans are no longer biological; they are what we would call "uploads", minds running on nanocomputers or other ultracomputing hardware in the same way that a present-day human runs on neural wetware. We would call them uploads; in the future of Diaspora, they are merely "citizens". They gather into "polises", one "polis" being a few meters, perhaps, in diameter, and containing trillions or quintillions of intelligent beings; Egan doesn't specify. Human space consists essentially of the Coalition of Polises, an uncountable number of sentients, with a few embodied robots roaming the solar system and still fewer biological humans on Old Earth, the "fleshers", divided into the genetically modified "exuberants" and the "statics", with the "statics" being around the only entities one of today's humans could shake hands with.
Of course, everyone in Diaspora is still far too human. All the entities are still running on essentially human computing power and have essentially human cognitive architectures. There are no superintelligences. None of the protagonists display transhuman intelligence; that, of course, would require a transhuman author. There is a tradeoff, in depictions of the future, between drama and realism; Diaspora already departs so far from our present-day future that even with human-architecture characters it is still difficult to sympathize with the protagonists. There is only one truly tragic figure in Diaspora and it took me over a year to figure out who it was (1).
More importantly, Greg Egan skips over the question of why all of humanity, all the polises, and Old Earth, have suddenly turned peaceful simply because everyone is free, immortal, and rich. It only takes one aggressor to make a war, and during the twentieth century, offensive technology has considerably outrun the defensive. No present-day shield will withstand a direct hit by a nuclear weapon. Whether offensive technology overpowers defensive at the limit of achievable technology is another question, and obviously the answer is "It could go either way."
But even if the various "polises" - different operating systems - in Diaspora were surrounded by utterly impermeable defenses, that would create another moral problem, this one even worse: that of an evil polis, where the rules against coercion don't hold, and some ruling class creates and tortures countless trillions of sentient victims. By hypothesis, if defensive technology beats offensive, there is nothing that anyone can do about this evil polis; nothing that can break the defenses.
It's also hard to see how the Diaspora solution could consistently arise from today's world. Suppose that each seed AI in the twenty-first century, as ve reaches transhumanity, becomes the seed and operating system of a polis - so that everyone gets to pick their own definition of Friendliness and live there. It doesn't seem that the system-as-a-whole would last very long. Good AI , good AI , good AI , good AI , good AI, evil solipsist AI, good AI, good AI, good AI, evil solipsist AI, good AI, good AI, good AI, good AI, evil aggressor AI. At this point, everyone in the Solar System who isn't behind the impregnable defenses of an existing superintelligence gets gobbled up by the evil aggressor superintelligence, after which the sequence ends. Flip through a deck of cards long enough, and sooner or later you'll turn up the ace of spades.
These are some of the factors which, in my opinion, make it likely that the Transition Guide will implement a Sysop Scenario - one underlying operating system for the Solar System; later, for all of human space. It is possible, although anthropomorphic, that the end result will be a Diaspora-like multiplicity of communities with virtual operating systems, or "Sysop skins", existing on top of the underlying operating system. I, for, one, strongly doubt it; it doesn't seem strange enough to represent a real future. But, even in the "operating system skin" case, the "flipping through the deck" and "hell polis" problems do not exist; try and construct a virtual operating system which allows you to create and abuse another sentient, and the underlying operating system will step in. Similarly, even if Earth turns out to be a haven for the Luddite biological humans and their kin, I would expect that the Sysop would maintain a presence - utterly unobtrusive, one hopes, but still there - to ensure that nobody on Earth launches their own hell polis, or tries to assimilate all the other Earthly refugees, or even creates a tormentable AI on their home computer. And so on.
But that is simply my personal opinion. A Friendly AI programmer does not get to decide whether Friendliness manifests in an individual human-level AI trying to do good, or in an AI who becomes the operating system of a polis, or in an AI who becomes the Sysop of human space. A Friendly AI programmer does not even get to decide whether the Sysop Scenario is a good idea; Sysop / nonSysop scenarios are not moral primitives. They are, formally and intuitively, subgoal content: The desirability of a Sysop Scenario is contingent on its predicted outcome. If someone demonstrated that neither the "flipping through the deck" nor the "hell polis" problems existed - or that a Sysop Scenario wouldn't help - then that would remove the underlying reason why I think the Sysop Scenario is a consequence of normative altruism. Similarly, most of the people who come down on the nonSysop side of the issue do so because of testable statements about the consequences of uniformity; that is, their indictment of the Sysop Scenario is contingent upon its predicted outcome. Whether the Transition Guide favors a Sysop Scenario or a "Coalition of Polises" is not a decision made by Friendship programmers. It is a consequence of moral primitives plus facts that may still be unknown to us.
Similarly, the statements "No one entity can be trusted with power", "It is unsafe to put all your eggs in one basket", "It is morally unacceptable to implement any one underlying moral rule, including this one", or "The predicted frustration of people who just plain hate Sysops would outweigh the predicted frustration (2) of people in hell polises", are all factual statements that would force multiple, independent civilizations as a consequence of Friendliness.
It's an interesting question as to whether, as in Diaspora, the post-Singularity world of Old Earth will become a reserve for any pedestrians that remain. ("Pedestrian" is a term, invented by someone who has asked to be called "Debbie the Roboteer", that describes an individual who chooses to remain bounded by humanity and twentieth-century tech when there are other choices available. I like the term because it so neatly summarizes all the implications. Pedestrians are slow. Pedestrians walk through rain or snow or broiling heat. Sensible people travel in air-conditioned cars, if available. But pedestrians still have rights, even if they lack technology. You can't just run them over.) The Old Earth Refuge for the Willfully Human is an anthropomorphic scenario; if it happens, it will happens because people want it. Volition, after all, is probably the dominant force in the Sysop Scenario, playing perhaps the same role as gravity in our mundane Universe. Earth is a tiny fraction of the mass in the Solar System; if at least that fraction of humanity decides to stay behind, then it would seem fair to let Earth be the inheritance of the meek.
One of the oft-raised speculations is that the Singularity will be kind only to the rich; that is, that participation in the Singularity will be expensive. This would inherently require a Slow Singularity, and a non-AI or non-Friendly-AI scenario at that; a superintelligent Friendly AI armed with nanotechnology doesn't care how many green pieces of paper you own. Given a hard takeoff and Sysop Scenario, the Transition Guide would presumably (a) create nanotechnology using the minimum material-time technological trajectory, (b) construct or self-enhance into a Sysop, followed by (c) the Sysop sweeping Old Earth with nanobots, femtobots, Planckbots, or whatever material agency turns out to be the limit of technology, followed by (d) all extant individuals being offered the choice of uploading and a six-billionth share of the Solar System (3), or staying behind on Old Earth, or perhaps staying behind temporarily to think about it. The corresponding event in Greg Egan's Diaspora is called the "Introdus". I like to think of the Singularity as "generating an Introdus wavefront".
Or it may be that all of these speculations are as fundamentally wrong as the best speculations that would have been offered a hundred, or a thousand, or a million years ago. This returns to the fundamental point about the Transition Guide - that ve must be complete, and independent, because we really don't know what lies on the other side of the Singularity.
| Next: | 4: Policy implications |
| Up: | Creating Friendly AI |
| Prev: | 3.5: Developmental Friendliness |