The Singularity Institute Blog

The Uncertain Future, a web application built by Michael Anissimov, Steve Rayhawk, Anna Salamon, Tom McCabe, and Rolf Nelson during the Singularity Institute Summer 2008 Research Program, with helpful discussions with a few others, is now in beta and ready for public announcement.

The Uncertain Future represents a new kind of futurism — futurism with heavy-tailed, high-dimension probability distributions. In fact, that’s the name of the paper presented at the European Conference on Computing and Philosophy that unveiled the project: “Changing the frame of AI futurism: From storytelling to heavy-tailed, high-dimensional probability distributions”.

Most futurism is about telling a story — more like marketing than an honest attempt at uncovering the possible range of what the future may hold. Better than creating a single story is scenario building — but this falls short as well. Scenario building is human nature, but it leaves us susceptible to anchoring effects where we overestimate the probability of vivid scenarios. To quote “Cognitive Biases Potentially Affecting Judgment of Global Risks”, page 6:

The conjunction fallacy similarly applies to futurological forecasts. Two independent sets of professional analysts at the Second International Congress on Forecasting were asked to rate, respectively, the probability of “A complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983″ or “A Russian invasion of Poland, and a complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983″. The second set of analysts responded with significantly higher probabilities. (Tversky and Kahneman 1983.)

The conjunction fallacy means that people overestimate the probability of vivid, detailed scenarios, even though each additional detail necessarily decreases the probability that the event will occur.

To combat against the conjunction fallacy and storytelling fallacies in our particular area of futurism, which includes intelligence enhancement, AI, and global catastrophic risk, we created an interactive system that allows the user to input their own probability distributions for different variables potentially associated with the future of AI and humanity, including a probability distribution of how much computing power would required to create human-level AI, a probability distribution for the likelihood of global thermonuclear war in the next century, and many other variables. Our model includes variables for the creation of AI, the possible success of intelligence amplification technology, and the potential extinction of the human species by technological mishap before either of these occurs.

Our system is built on the assumption that breaking down a challenging prediction task into its constituent parts can be quite beneficial, because it forces us to think about the task in greater detail, and avoid obvious biases associated with specific scenarios we may be anchoring on. Some people may criticize such a view for being excessively reductionist, but many prediction tasks really can be broken down into component pieces. The alternative is making “expert” guesses based on a holistic evaluation of the prediction task, which leaves us open to many well-documented biases.

Here is the opening blurb for the webapp, by Tom McCabe:

The Uncertain Future is a future technology and world-modeling project by the Singularity Institute for Artificial Intelligence. Its goal is to allow those interested in future technology to form their own rigorous, mathematically consistent model of how the development of advanced technologies will affect the evolution of civilization over the next hundred years. To facilitate this, we have gathered data on what experts think is going to happen, in such fields as semiconductor development, biotechnology, global security, Artificial Intelligence and neuroscience. We invite you, the user, to read about the opinions of these experts, and then come to your own conclusion about the likely destiny of mankind.

It’s not perfect, but we think that our system might be a seed for looking at futurism in a different way — providing an alternative to storytelling and scenario building. This sort of “probabilistic futurism” encourages would-be seers to widen their confidence bounds when confronted with uncertainty, instead of irrationally making overconfident guesses to seem like “experts”. The particular issues we focus on are controversial — human-equivalent AI, biotechnology used to select gametes with genes associated with intelligence, the probability of planet-ending catastrophe — but we chose these issues specifically because there is disagreement about what degree of uncertainty is warranted from our present position is evaluating these scenarios.

We visualize this tool being used among futurists to specify their quantitative background assumptions regarding the technologies discussed. This might be used to clear aside straw men and zoom in on the core disagreements. It might also be used to evaluate the degree to which respective futurists have considered the technological prerequisites and other assumptions underlying their scenarios.

If you like the system or find it useful, be sure to post a link to it on Facebook, or suggest it to your friends. The system still has quite a few bugs; we used Java applets for the probability distributions, and designed it so that the Java applet makes calls to the surrounding HTML, which may fail on some combinations of OS and browser. If you use a Mac, you should use Safari, and if you use Linux/Windows, use Opera or Firefox.

Peter de Blanc (Temple University), an SIAI Visiting Fellow, has recently published the final version of his most recent paper, titled “Convergence of Expected Utility for Universal Artificial Intelligence“. The paper, based on earlier work in the field of expected utility theory by Marcus Hutter (Australian National University) and SIAI Research Fellow Eliezer Yudkowsky, proves mathematically that any unbounded, perception determined, computable utility function cannot assign a defined utility to any action, assuming a Solomonoff-like prior. The abstract of the paper is as follows:

“We consider a sequence of repeated interactions between an agent and an environment. Uncertainty about the environment is captured by a probability distribution over a space of hypotheses, which includes all computable functions. Given a utility function, we can evaluate the expected utility of any computational policy for interaction with the environment. After making some plausible assumptions (and maybe one not-so-plausible assumption), we show that if the utility function is unbounded, then the expected utility of any policy is undefined.”

Peter de Blanc has done research with the Singularity Institute since 2006, when he participated in the “Summer of AI” research program along with Nick Hay (UC Berkeley), Marcello Herreshoff (Stanford University) and Eliezer Yudkowsky. During the summer of 2009, he was a Singularity Institute Summer Fellow, helping SIAI researchers work on problems in the fields of Friendly AI, rationality and decision theory. His personal website, Space and Games, can be found at http://www.spaceandgames.com/.

The following message is cross-posted from Less Wrong on behalf of SIAI Research Fellow Anna Salamon:

Last summer, 15 Less Wrongers, under the auspices of SIAI, gathered in a big house in Santa Clara (in the SF bay area), with whiteboards, existential risk-reducing projects, and the ambition to learn and do.

Now, the new and better version has arrived.  We’re taking folks on a rolling basis to come join in our projects, learn and strategize with us, and consider long term life paths.  Working with this crowd transformed my world; it felt like I was learning to think.  I wouldn’t be surprised if it can transform yours.

A representative sample of current projects:

• Research and writing on decision theory, anthropic inference, and other non-dangerous aspects of the foundations of AI;
• The Peter Platzer Popular Book Planning Project;
• Editing and publicizing theuncertainfuture.com;
• Improving the LW wiki, and/or writing good LW posts;
• Getting good popular writing and videos on the web, of sorts that improve AI risks understanding for key groups;
• Writing academic conference/journal papers to seed academic literatures on questions around AI risks (e.g., takeoff speed, economics of AI software engineering, genie problems, what kinds of goal systems can easily arise and what portion of such goal systems would be foreign to human values; theoretical compsci knowledge would be helpful for many of these questions).

Interested, but not sure whether to apply?

Past experience indicates that more than one brilliant, capable person refrained from contacting SIAI, because they weren’t sure they were “good enough”.  That kind of timidity destroys the world, by failing to save it.  So if that’s your situation, send us an email.  Let us be the one to say “no”.  Glancing at an extra application is cheap, and losing out on a capable applicant is expensive.

And if you’re seriously interested in risk reduction but at a later time, or in another capacity — send us an email anyway.  Coordinated groups accomplish more than uncoordinated groups; and if you care about risk reduction, we want to know.

What we’re looking for

At bottom, we’re looking for anyone who:

• Is capable (strong ability to get things done);
• Seriously aspires to rationality; and
• Is passionate about reducing existential risk.

Bonus points for any (you don’t need them all) of the following traits:

• Experience with management, for example in a position of responsibility in a large organization;
• Good interpersonal and social skills;
• Extraversion, or interest in other people, and in forming strong communities;
• Dazzling brilliance at math or philosophy;
• A history of successful academic paper-writing; strategic understanding of journal submission processes, grant application processes, etc.
• Strong general knowledge of science or social science, and the ability to read rapidly and/or to quickly pick up new fields;
• Great writing skills and/or marketing skills;
• Organization, strong ability to keep projects going without much supervision, and the ability to get mundane stuff done in a reliable manner;
• Skill at implementing (non-AI) software projects, such as web apps for interactive technological forecasting, rapidly and reliably;
• Web programming skill, or website design skill;
• Legal background;
• A history of successfully pulling off large projects or events;
• Unusual competence of some other sort, in some domain we need, but haven’t realized we need.
• Cognitive diversity: any respect in which you’re different from the typical LW-er, and in which you’re more likely than average to notice something we’re missing.

If you think this might be you, send a quick email to anna@singinst.org.  Include:

1. Why you’re interested;
2. What particular skills you would bring, and what evidence makes you think you have those skills (you might include a standard resume or c.v.);
3. Optionally, any ideas you have for what sorts of projects you might like to be involved in, or how your skillset could help us improve humanity’s long-term odds.

Our application process is fairly informal, so send us a quick email as initial inquiry and we can decide whether or not to follow up with more application components.

As to logistics: we cover room, board, and, if you need it, airfare, but no other stipend.

Looking forward to hearing from you,
Anna

Just prior to Singularity Summit 2009, Singularity Institute President Michael Vassar did a podcast with “The Skeptics Guide to the Universe”, produced by the New England Skeptical Society in association with the James Randi Educational Foundation (JREF).

Skip to 26:00 to get past the news items. Here’s a funny tidbit of Michael talking about some of the poor thinking seen when people discuss how to make AI friendly:

We have a lot of silliness, such as worst moral of the story ever… Lilo and Stitch. “If you’re just nice enough to the fundamentally evil creature, it will have to love you.”

That’s at 31:50. Also check out 41:00, where Michael explains the whole reason for having a Singularity Summit and Singularity Institute. At 49:30: “So how do we keep it from deciding that it wants to make ice cream out of human brains?”

SIAI was invited to contribute a series of articles to Good.is, the website of GOOD magazine, which focuses on philanthropy and activism. The first article, “What is the Singularity?” is now live, and new articles will appear every Monday through January 26th. Myself and SIAI volunteer Roko Mijic will be alternating posts.

For an idea of how well-exposed GOOD is, see that their Twitter account has almost 170,000 followers.

Check out my blog post, “Hungry Optimizers with Low-Complexity Values” at Accelerating Future.

For an interesting example of the power of lateral thinking solving scientific problems, check out the article “Was our oldest ancestor a proton-powered rock?” at New Scientist.

The videos for Singularity Summit 2009 are now available at Vimeo. The few that are missing are either still awaiting confirmation of permission or the speaker asked for video not to be posted of their talk.

Thanks to all 813 people who attended the Singularity Summit! The event was a huge success. If you want to buy a T-shirt to commemorate the event, you can do so here.

The SIAI cause group on Facebook has recently surpassed 1,000 members. Join now to support our mission.