The Singularity Institute Blog

Peter de Blanc, 2006 summer intern at the Singularity Institute and now with the Temple University Department of Mathematics, has posted an article titled “Convergence of Expected Utilities with Algorithmic Probability Distributions.”

The formal mathematical proof in this paper has important implications for the question of whether a perfect rational agent — including some future Friendly Artificial Intelligence — can truly put the same value on each sentient life even when astronomical numbers stand to be harmed, something we humans fail so miserably to do.

The article is available online.

Rolf Nelson’s blog, AI Beliefs, discusses how we may be able to use the simulation argument to convince certain unfriendly AIs to give us a nice place to live.

Rather than commiting to run any particular simulations, I have made the following promise:

I commit to think more about this problem in the future, and take whatever action I would have wanted to precommit to taking before the creation of the first AI.

I encourage others to make a similar commitment in public. This would be a good place to post it.

On Philosophy, et cetera, Chappell discusses a theory of goals, originally described by Railton, with some similarities to CEV:

Give to an actual individual A unqualified cognitive and imaginative powers, and full factual and nomological information about his physical and psychological constitution, capacities, circumstances, history, and so on. A will have become A+, who has complete and vivid knowledge of himself and his environment, and whose instrumental rationality is in no way defective. We now ask A+ to tell us not what he currently wants, but what he would want his non-idealized self A to want - or, more generally, to seek - were he in the actual condition and circumstances of A. (pp.173-4, bold added.)

The last bit seems to me like a hack to deal with observer-centric goals; if A+ has been constructed, then A+ is not in the actual condition and circumstances of A. This hack can lead to some suboptimal outcomes.

Suppose, for instance, that Fred wants a diamond. Fred has some other concerns too: he would also like to have an emerald, and he would prefer for his personality to remain intact, but mostly he just wants a diamond. Fred can choose between two boxes: box 1 contains a diamond, and box 2 contains an emerald. Fred erroneously believes that box 1 contains an emerald, and box 2 contains a diamond. If we extrapolate Fred+, then Fred+ would want Fred to want an emerald. Then Fred will choose box 1, and receive a diamond, thus fulfilling his original goal. Fred’s lesser preference of keeping his personality intact is not fulfilled.

A better solution would be to tell Fred that box 1 contains a diamond. I am not saying that this is the optimal solution, of course.

ZDNET has begun podcasts with speakers from the Singularity Summit 2007, which is happening this September 8th-9th. The first interview, Can ‘Friendly AI’ Save Humans from Irrelevance or Extinction?, is with SIAI Research Fellow Eliezer Yudkowsky. The interviews are being done by Dan Farber, CNET Networks VP of Editorial and ZDNET Editor in Chief.

Numerous readers in the past have posted suggestions for how the universe should be run, in the form of laws, rights, or moral principles. I don’t want to embarrass them, so I won’t post links here, but if you have made such a suggestion then feel free to repost it in the comments below.

We know we’re not (yet) wise enough to write down scientific theories or fashion standards or music styles to be used for the rest of time. These objects are the output of an optimization process; human beings have spent long years studying, thinking, observing, and testing to develop them. If we want future generations to continue this process, it is necessary to communicate the target of the optimizer, not just its output so far.

Wikipedia is not an AI scientist; iTunes is not an AI composer. To build an AI scientist or an AI composer, we need to load in the optimization targets of science and music. Similarly, the 10 Commandments, the Bill of Rights, and Asimov’s Laws are not moralities; they are the output of human moralists. To build an AI moralist, you need to load in the right optimization target; that target is what I’m calling a morality.

Volition extrapolation is our current idea of how to load in optimization targets from existing humans.

Consider the Stamp-Collecting Device. A common objection goes like this: “An optimization process that’s smart enough to tile the universe with stamps would also be smart enough to realize that this is not what its creator intended. Therefore it would not tile the universe with stamps.”

Human beings serve as a counterexample. The rules for constructing a human mind were devised by natural selection. These rules were fine-tuned to produce minds that are good at passing on their genes. If you are thinking of evolution as an optimization process, then it has the goal of producing genes which replicate as effectively as possible.

In 1859, Charles Darwin described the process that created us. Since then, we have come to understand that process in greater detail. Evolution is simple enough that we can claim to understand it very well; perhaps we even understand evolution as well as a Stamp-Collecting Device could understand us. Despite this understanding, we humans do not make evolution’s goal our own. Any time you use contraception, or perform a kind act when nobody is watching, you are betraying the goal of evolution. But so what? That’s evolution’s goal, not our goal. If anything, our understanding of evolution helps us to notice when we are doing something nasty but adaptive, and learn to avoid this behavior.

Similarly, a Stamp-Collecting Device would not adopt its programmer’s goals. It has its own goal to pursue — collecting stamps. If anything, understanding humans better would allow it to notice and fix biases that may be hindering its ability to collect stamps efficiently.

The challenge of FAI is to build an AI that does adopt our goals.

Suppose we have coded what we believe to be a Friendly AI. We then face a moment like in the film Pi: “12:50, press Return” (link). In other words, we face three options: Launch the AI now, try improving it further and launch it later, or never launch it. Why wouldn’t we launch right away or perhaps ever? Maybe we made a mistake in the code somewhere, or maybe we don’t need to launch and would be better off not chancing it.

Ideally, we’d have an infinite amount of time to make sure we got the code right, but in practice, we don’t have this luxury: The longer we wait, the more risk we run of an existential event (including an unFriendly AI launch from a different AI project) occurring while we wait. Our hesitation could save us, but it could also spell our doom.

So, when should we push the button?
Read the rest of this entry »

The Singularity Institute’s current best guess on what to do with a general AI is to have it implement humanity’s coherent extrapolated volition (CEV) - what we would want if we “knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted”. This is quite a mouthful.

To trade brevity for decreased accuracy, another way of saying the above is that we want an AI that represents the spirit of humanity’s desires rather than just the letter.

Is CEV democratic? Yes, but it is a representative democracy, where humanity is represented by the aggregate of its extrapolated volition.

There are four objections to CEV I generally hear, summarized as follows:

1. The devil’s pact objection. In fiction as well as in real life, great-sounding deals often have a hidden catch. Why should we expect this to be any different?

2. The fear of patriarchy objection. All the talk of self-improving general AI and its potential capabilities make people nervous because of the power asymmetry it implies.

3. The anti-AI objection. Many people take the line that machines should be mindless tools to serve humans, and never anything more.

4. The “I’m too special to be extrapolated” objection. Quite a few people have the idea that the human mind is too complex to ever be understood in any significant detail, much less be extrapolated accurately.

Because the question of what goal system to give the first general artificial intelligence is obviously a pretty big deal, all objections deserve to be heard and considered. There are probably others beyond the above four, but I wanted to focus on the obvious ones for now.

In my mind, all of the above objections are rooted in valid motivations, but none of them should be deal-breakers. I will briefly respond to the objections.

The devil’s pact objection requires that one deal participant (in this case, the AI) has an innate ill will towards the other deal participant (in this case, humanity). The AI would have to secretly want to screw us over from the get-go. But because general AI will be built from scratch, and is not likely, at least initially, to be heavily inspired by the human brain, there is no reason for us to postulate that this sort of behavior will be present. In terms of actual development concerns, AI programmers should be watchful as to whether “shortcuts”, like modeling an extrapolated humanity but not actually implementing its desires, generate just as much positive utility for the AI as what we would consider the “real deal” - making the real world a better place.

The fear of patriarchy objection stems largely from history, wherein all of the relevant actors were members of our unique species, for which power is proven to corrupt. Power corrupts humans for evolutionary reasons - if one is on top of the heap, one had better take advantage of the opportunity to reward one’s allies and punish one’s enemies. This is pure evolutionary logic and need not be consciously calculated. AIs, which can be constructed entirely without selfish motivations, can be immune to these tendencies. Insofar as significant power asymmetries in general bother people, this seems hard to avoid in the long term - technological development will lead to a diversity of possible beings, and with this diversity will inevitably come a diversity in levels of capability and intelligence.

The anti-AI objection is just anthropocentric. If human-level AI is possible, it will be created sooner or later. It’s in our best interests to admit this and try to ensure that AI is on our side. Anti-AI bias in this area is no different than the other unfortunate biases held throughout history against minorities.

The final objection has to do with the complexity of extrapolation. Believe it or not, we engage in extrapolations every day. We can’t fit realistic computational duplicates of the people we know in our heads, so we use abstract models that work well for many pragmatic purposes. In a CEV-implementing AI, the models used might be more detailed than those we use, but need not simulate every single atom of every single biopolymer to perform a tractable extrapolation.

Are there any other obvious objections people might have to CEV? Addressing these objections could help strengthen the idea.