The Singularity Institute Blog

ZDNET’s second and third interview with speakers from the Singularity Summit 2007:

“AI, Nanotech, and the Future of the Human Species,” with Steve Jurvetson, Draper Fisher Jurvetson Managing Director

“Pathways to Artificial Intelligence,” with Barney Pell, Powerset CEO

Dr. Ben Goertzel is SIAI’s Director of Research. In this interview, he explains the Singularity Institute’s mission and research objectives. You can download the audio version here.

Solomonoff induction is a general, but uncomputable, solution to the problem of prediction: given the past, what probability should you assign each possible future?

In my previous post, I described a sequence predictor that works by modeling the sequence as generated by a random program in a simple language. I’ve hacked together an implementation of this predictor in python.

Read the rest of this entry »

The problem of prediction is: given a series of past observations, what future observations do you expect? When we are rigorous about expectations we assign probabilities to the different possibilities. For example, given the weather today we assign 50% probability to a rainy day tomorrow, 30% probability to a cloudy day, and 20% probability to a sunny one.

How can we determine the probability of a future given the past? Solomonoff induction is a solution to this problem. Solomonoff induction has a strong performance guarantee: any other method assigns at most a constant factor larger probability to the actual future. This constant is equal to the complexity of that predictor.

Solmononoff induction itself is uncomputable, but there are computable analogs. It serves as a simple method of specifying a device which accurately predicts a series of observations. Were such a device to exist we would think it highly intelligent as it correctly predicted any patternful sequence we entered with little error.

Below the fold I describe some of the machinary behind Solomonoff induction. I describe a computable approximation which can be exactly and efficiently solved. Although this computable predictor is not particularly intelligent, it shares the same structure as Solomonoff induction.
Read the rest of this entry »

There’s no such thing as a free lunch. But many lunches can be purchased at a reasonable price.

I want to talk to you today about that knee-jerk reaction you get when you hear about SIAI supporters and our crazy notions about solving the world’s problems through artificial intelligence. In particular, we talk about artificial general intelligence (AGI), a system that can think usefully about many different things, just like we can — and eventually much better than we can.

“This is too good to be true,” you say. “Should I file this under perpetual motion machines, or get-rich-quick schemes?”

Rest easy! Perpetual motion is not required.
Read the rest of this entry »

The Singularity Institute’s current best guess on what to do with a general AI is to have it implement humanity’s coherent extrapolated volition (CEV) - what we would want if we “knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted”. This is quite a mouthful.

To trade brevity for decreased accuracy, another way of saying the above is that we want an AI that represents the spirit of humanity’s desires rather than just the letter.

Is CEV democratic? Yes, but it is a representative democracy, where humanity is represented by the aggregate of its extrapolated volition.

There are four objections to CEV I generally hear, summarized as follows:

1. The devil’s pact objection. In fiction as well as in real life, great-sounding deals often have a hidden catch. Why should we expect this to be any different?

2. The fear of patriarchy objection. All the talk of self-improving general AI and its potential capabilities make people nervous because of the power asymmetry it implies.

3. The anti-AI objection. Many people take the line that machines should be mindless tools to serve humans, and never anything more.

4. The “I’m too special to be extrapolated” objection. Quite a few people have the idea that the human mind is too complex to ever be understood in any significant detail, much less be extrapolated accurately.

Because the question of what goal system to give the first general artificial intelligence is obviously a pretty big deal, all objections deserve to be heard and considered. There are probably others beyond the above four, but I wanted to focus on the obvious ones for now.

In my mind, all of the above objections are rooted in valid motivations, but none of them should be deal-breakers. I will briefly respond to the objections.

The devil’s pact objection requires that one deal participant (in this case, the AI) has an innate ill will towards the other deal participant (in this case, humanity). The AI would have to secretly want to screw us over from the get-go. But because general AI will be built from scratch, and is not likely, at least initially, to be heavily inspired by the human brain, there is no reason for us to postulate that this sort of behavior will be present. In terms of actual development concerns, AI programmers should be watchful as to whether “shortcuts”, like modeling an extrapolated humanity but not actually implementing its desires, generate just as much positive utility for the AI as what we would consider the “real deal” - making the real world a better place.

The fear of patriarchy objection stems largely from history, wherein all of the relevant actors were members of our unique species, for which power is proven to corrupt. Power corrupts humans for evolutionary reasons - if one is on top of the heap, one had better take advantage of the opportunity to reward one’s allies and punish one’s enemies. This is pure evolutionary logic and need not be consciously calculated. AIs, which can be constructed entirely without selfish motivations, can be immune to these tendencies. Insofar as significant power asymmetries in general bother people, this seems hard to avoid in the long term - technological development will lead to a diversity of possible beings, and with this diversity will inevitably come a diversity in levels of capability and intelligence.

The anti-AI objection is just anthropocentric. If human-level AI is possible, it will be created sooner or later. It’s in our best interests to admit this and try to ensure that AI is on our side. Anti-AI bias in this area is no different than the other unfortunate biases held throughout history against minorities.

The final objection has to do with the complexity of extrapolation. Believe it or not, we engage in extrapolations every day. We can’t fit realistic computational duplicates of the people we know in our heads, so we use abstract models that work well for many pragmatic purposes. In a CEV-implementing AI, the models used might be more detailed than those we use, but need not simulate every single atom of every single biopolymer to perform a tractable extrapolation.

Are there any other obvious objections people might have to CEV? Addressing these objections could help strengthen the idea.

An avid stamp collector, who is also an AI enthusiast, decides to build a stamp collecting device. This margin is too small for the details, but the idea is simple:

1. The device will be active for one year. It is connected to the internet, from which it sends and receives packets.
2. The device has an internal model of the universe. This model captures how likely each state of the world is, can predict future packets received, and can simulate the effect of packets sent.
3. For every possible sequence of packets, the model extrapolates the final state and counts the number of stamps collected.
4. The device outputs the sequence leading to the largest number of stamps.

This is a powerful device. It models every possible course of action to output the best. Outputting a one kilobit packet per second, a single day has 2^86,400,000 = 10^26,000,000 possible packet sequences. By comparison, the number of atoms in the observable universe is about 10⁸⁰, and its volume is only 10⁴²⁶ cubic Planck units. There are a lot of possibilities to consider.

Read the rest of this entry »

If you have not read “Cognitive Biases Potentially Affecting Judgment of Global Risks” and “Artificial Intelligence as a Positive and Negative Factor in Global Risk“, I recommend reading them. They are excellent book chapters from SIAI Research Fellow Eliezer Yudkowsky, forthcoming in the edited volume Global Catastrophic Risks from Oxford University Press (Nick Bostrom and Milan Cirkovic eds.). If you do not have time to read both, I recommend reading the conclusion of the first, repeated below, and reading the second in its entirety.

Conclusion of “Cognitive Biases Potentially Affecting Judgment of Global Risks”:

Why should there be an organized body of thinking about existential risks? Falling asteroids are not like engineered superviruses; physics disasters are not like nanotechnological wars. Why not consider each of these problems separately?

If someone proposes a physics disaster, then the committee convened to analyze the problem must obviously include physicists. But someone on that committee should also know how terribly dangerous it is to have an answer in your mind before you finish asking the question. Someone on that committee should remember the reply of Enrico Fermi to Leo Szilard’s proposal that a fission chain reaction could be used to build nuclear weapons. (The reply was “Nuts!” – Fermi considered the possibility so remote as to not be worth investigating.) Someone should remember the history of errors in physics calculations: the Castle Bravo nuclear test that produced a 15-megaton explosion, instead of 4 to 8, because of an unconsidered reaction in lithium-7: They correctly solved the wrong equation, failed to think of all the terms that needed to be included, and at least one person in the expanded fallout radius died. Someone should remember Lord Kelvin’s careful proof, using multiple, independent quantitative calculations from well-established theories, that the Earth could not possibly have existed for so much as forty million years. Someone should know that when an expert says the probability is “a million to one” without using actuarial data or calculations from a precise, precisely confirmed model, the calibration is probably more like twenty to one (though this is not an exact conversion).

Any existential risk evokes problems that it shares with all other existential risks, in addition to the domain-specific expertise required for the specific existential risk. Someone on the physics-disaster committee should know what the term “existential risk” means; should possess whatever skills the field of existential risk management has accumulated or borrowed. For maximum safety, that person should also be a physicist. The domain-specific expertise and the expertise pertaining to existential risks should combine in one person. I am skeptical that a scholar of heuristics and biases, unable to read physics equations, could check the work of physicists who knew nothing of heuristics and biases.

Once upon a time I made up overly detailed scenarios, without realizing that every additional detail was an extra burden. Once upon a time I really did think that I could say there was a ninety percent chance of Artificial Intelligence being developed between 2005 and 2025, with the peak in 2018. This statement now seems to me like complete gibberish. Why did I ever think I could generate a tight probability distribution over a problem like that? Where did I even get those numbers in the first place?

Skilled practitioners of, say, molecular nanotechnology or Artificial Intelligence, will not automatically know the additional skills needed to address the existential risks of their profession. No one told me, when I addressed myself to the challenge of Artificial Intelligence, that it was needful for such a person as myself to study heuristics and biases. I don’t remember why I first ran across an account of heuristics and biases, but I remember that it was a description of an overconfidence result – a casual description, online, with no references. I was so incredulous that I contacted the author to ask if this was a real experimental result. (He referred me to the edited volume Judgment Under Uncertainty.)

I should not have had to stumble across that reference by accident. Someone should have warned me, as I am warning you, that this is knowledge needful to a student of existential risk. There should be a curriculum for people like ourselves; a list of skills we need in addition to our domain-specific knowledge. I am not a physicist, but I know a little – probably not enough – about the history of errors in physics, and a biologist thinking about superviruses should know it too.

I once met a lawyer who had made up his own theory of physics. I said to the lawyer:
You cannot invent your own physics theories without knowing math and studying for
years; physics is hard. He replied: But if you really understand physics you can explain it to your grandmother, Richard Feynman told me so. And I said to him: “Would you advise
a friend to argue his own court case?” At this he fell silent. He knew abstractly that
physics was difficult, but I think it had honestly never occurred to him that physics might
be as difficult as lawyering.

One of many biases not discussed in this chapter describes the biasing effect of not knowing what we do not know. When a company recruiter evaluates his own skill, he recalls to mind the performance of candidates he hired, many of which subsequently excelled; therefore the recruiter thinks highly of his skill. But the recruiter never sees the work of candidates not hired. Thus I must warn that this paper touches upon only a small subset of heuristics and biases; for when you wonder how much you have already learned, you will recall the few biases this chapter does mention, rather than the many biases it does not. Brief summaries cannot convey a sense of the field, the larger understanding which weaves a set of memorable experiments into a unified interpretation. Many highly relevant biases, such as need for closure, I have not even mentioned. The purpose of this chapter is not to teach the knowledge needful to a student of existential risks, but to intrigue you into learning more.

Thinking about existential risks falls prey to all the same fallacies that prey upon thinking-in-general. But the stakes are much, much higher. A common result in heuristics and biases is that offering money or other incentives does not eliminate the bias. (Kachelmeier and Shehata (1992) offered subjects living in the People’s Republic of China the equivalent of three months’ salary.) The subjects in these experiments don’t make mistakes on purpose; they make mistakes because they don’t know how to do better. Even if you told them the survival of humankind was at stake, they still would not thereby know how to do better. (It might increase their need for closure, causing them to do worse.) It is a terribly frightening thing, but people do not become any smarter, just because the survival of humankind is at stake.

In addition to standard biases, I have personally observed what look like harmful modes of thinking specific to existential risks. The Spanish flu of 1918 killed 25-50 million people. World War II killed 60 million people. 10^7 is the order of the largest catastrophes in humanity’s written history. Substantially larger numbers, such as 500 million deaths, and especially qualitatively different scenarios such as the extinction of the entire human species, seem to trigger a different mode of thinking - enter into a “separate magisterium”. People who would never dream of hurting a child hear of an existential risk, and say, “Well, maybe the human species doesn’t really deserve to survive.”

There is a saying in heuristics and biases that people do not evaluate events, but descriptions of events – what is called non-extensional reasoning. The extension of humanity’s extinction includes the death of yourself, of your friends, of your family, of your loved ones, of your city, of your country, of your political fellows. Yet people who would take great offense at a proposal to wipe the country of Britain from the map, to kill every member of the Democratic Party in the U.S., to turn the city of Paris to glass – who would feel still greater horror on hearing the doctor say that their child had cancer – these people will discuss the extinction of humanity with perfect calm. “Extinction of humanity”, as words on paper, appears in fictional novels, or is discussed in philosophy books – it belongs to a different context than the Spanish flu. We evaluate descriptions of events, not extensions of events. The cliche phrase end of the world invokes the magisterium of myth and dream, of prophecy and apocalypse, of novels and movies. The challenge of existential risks to rationality is that, the catastrophes being so huge, people snap into a different mode of thinking. Human deaths are suddenly no longer bad, and detailed predictions suddenly no longer require any expertise, and whether the story is told with a happy ending or a sad ending is a matter of personal taste in stories.

But that is only an anecdotal observation of mine. I thought it better that this essay should
focus on mistakes well-documented in the literature – the general literature of cognitive
psychology, because there is not yet experimental literature specific to the psychology of
existential risks. There should be.

In the mathematics of Bayesian decision theory there is a concept of information value – the expected utility of knowledge. The value of information emerges from the value of whatever it is information about; if you double the stakes, you double the value of information about the stakes. The value of rational thinking works similarly – the value of performing a computation that integrates the evidence is calculated much the same way as the value of the evidence itself. (Good 1952; Horvitz et. al. 1989.)

No more than Albert Szent-Gyorgyi could multiply the suffering of one human by a hundred million can I truly understand the value of clear thinking about global risks. Scope neglect is the hazard of being a biological human, running on an analog brain; the brain cannot multiply by six billion. And the stakes of existential risk extend beyond even the six billion humans alive today, to all the stars in all the galaxies that humanity and humanity’s descendants may some day touch. All that vast potential hinges on our survival here, now, in the days when the realm of humankind is a single planet orbiting a single star. I can’t feel our future. All I can do is try to defend it.

This is a 5-minute spoken introduction to the Singularity I wrote for a small conference. I had to talk fast, though, so this is probably more like a 6.5 minute intro.

The rise of human intelligence in its modern form reshaped the Earth. Most of the objects you see around you, like these chairs, are byproducts of human intelligence. There’s a popular concept of “intelligence” as book smarts, like calculus or chess, as opposed to say social skills. So people say that “it takes more than intelligence to succeed in human society”. But social skills reside in the brain, not the kidneys. When you think of intelligence, don’t think of a college professor, think of human beings; as opposed to chimpanzees. If you don’t have human intelligence, you’re not even in the game.

Sometime in the next few decades, we’ll start developing technologies that improve on human intelligence. We’ll hack the brain, or interface the brain to computers, or finally crack the problem of Artificial Intelligence. Now, this is not just a pleasant futuristic speculation like soldiers with super-strong bionic arms. Humanity did not rise to prominence on Earth by lifting heavier weights than other species.

Intelligence is the source of technology. If we can use technology to improve intelligence, that closes the loop and potentially creates a positive feedback cycle. Let’s say we invent brain-computer interfaces that substantially improve human intelligence. What might these augmented humans do with their improved intelligence? Well, among other things, they’ll probably design the next generation of brain-computer interfaces. And then, being even smarter, the next generation can do an even better job of designing the third generation. This hypothetical positive feedback cycle was pointed out in the 1960s by I. J. Good, a famous statistician, who called it the “intelligence explosion”. The purest case of an intelligence explosion would be an Artificial Intelligence rewriting its own source code.

The key idea is that if you can improve intelligence even a little, the process accelerates. It’s a tipping point. Like trying to balance a pen on one end - as soon as it tilts even a little, it quickly falls the rest of the way.

The potential impact on our world is enormous. Intelligence is the source of all our technology from agriculture to nuclear weapons. All of that was produced as a side effect of the last great jump in intelligence, the one that took place tens of thousands of years ago with the rise of humanity.

So let’s say you have an Artificial Intelligence that thinks enormously faster than a human. How does that affect our world? Well, hypothetically, the AI solves the protein folding problem. And then emails a DNA string to an online service that sequences the DNA, synthesizes the protein, and fedexes the protein back. The proteins self-assemble into a biological machine that builds a machine that builds a machine and then a few days later the AI has full-blown molecular nanotechnology.

So what might an Artificial Intelligence do with nanotechnology? Feed the hungry? Heal the sick? Help us become smarter? Instantly wipe out the human species? Probably it depends on the specific makeup of the AI. See, human beings all have the same cognitive architecture. We all have a prefrontal cortex and limbic system and so on. If you imagine a space of all possible minds, then all human beings are packed into one small dot in mind design space. And then Artificial Intelligence is literally everything else. “AI” just means “a mind that does not work like we do”. So you can’t ask “What will an AI do?” as if all AIs formed a natural kind. There is more than one possible AI.

The impact, of the intelligence explosion, on our world, depends on exactly what kind of minds go through the tipping point.

I would seriously argue that we are heading for the critical point of all human history. Modifying or improving the human brain, or building strong AI, is huge enough on its own. When you consider the intelligence explosion effect, the next few decades could determine the future of intelligent life.

So this is probably the single most important issue in the world. Right now, almost no one is paying serious attention. And the marginal impact of additional efforts could be huge. My nonprofit, the Singularity Institute, is trying to get things started in this area. My own work deals with the stability of goals in self-modifying AI, so we can build an AI and have some idea of what will happen as a result. There’s more to this issue, but I’m out of time. If you’re interested in any of this, please talk to me, this problem needs your attention. Thank you.