Please email us with your thoughts
February 5th, 2010 –
New staff bios have been added to SIAI’s team page: short bios for Research Fellows Anna Salamon and Steve Rayhawk, Media Director Michael Anissimov, and Chief Compliance Officer Amy Willey. Check them out, and feel free to ask if you’re interested in knowing more about what each staff member does.
February 5th, 2010 –
Here’s a extended abstract presented at the 2009 Asia-Pacific Conference on Computing and Philosophy by participants in SIAI’s 2009 Visiting Fellows Program that is making the rounds. The point of the paper, which was written by Carl Shulman, Nick Tarleton, and Henrik Jonsson, is that consequentialism as commonly discussed has a number of “free variables” where intuitions disagree about the right values of these variables. Therefore, machine ethics should draw on the emerging field of moral psychology to figure out how to fill in these free variables. This point is plainly put in the title of one of the last sections, “Current moral theories are inadequate for machine ethics”.
A reply from UK philosopher David Pearce has recently been posted by Roko Mijic at Less Wrong.
January 5th, 2010 –
On Sunday, January 3rd, I did an interview on KUSP in Santa Cruz, California, a National Public Radio affiliate. I talked to Rick Kleffel for an hour about the Singularity, the Singularity Institute, what we do, anthropomorphism, Friendly AI, and the like. It was for his “Talk of the Bay” radio program. Here is the audio archive.
January 5th, 2010 –
The Foresight Institute, an organization close to the Singularity Institute, is holding their 2010 conference at the Sheraton Palo Alto Hotel this January 16-17. Here is the blurb from the website of the conference:
Join us in for an exciting conference focused on the Synergy of Molecular Manufacturing and general Artificial Intelligence and celebrate the 20th anniversary of the founding of Foresight. Register online here. The two day conference rate is $175 with discounts for early registration!
Several rapidly-developing technologies have the potential to undergo an exponential takeoff in the next few decades, causing as much of an impact on economy and society as the computer and networking did in the past few. Chief among these are molecular manufacturing and artificial general intelligence (AGI). Key in the takeoff phenomenon will be the establishment of strong positive feedback loops within and between the technologies. Positive feedback loops leading to exponential growth are nothing new to economic systems. At issue is the value of the exponent: since the Industrial Revolution, economies have expanded at rates of up to 7% per year; however, computing capability has been expanding at rates up to 70% per year, in accordance with Moore’s Law. If manufacturing and intellectual work shifted into this mode, the impact on the economy and society would be profound. The purpose of this symposium is to examine the mechanisms by which this might happen, and its likely effects.
I will be giving a talk there on Friendly AI and anthropomorphism. See you in Palo Alto!
December 23rd, 2009 –
Offering unusually good philanthropic returns — meaning greater odds of a positive Singularity and lesser odds of human extinction — the Singularity Institute has launched a new challenge campaign. The sponsors, Edwin Evans, Rolf Nelson, Henrik Jonsson, Jason Joachim, and Robert Lecnik, have generously put up $100,000 of matching funds, so that every donation you make until February 28th will be matched dollar for dollar. If the campaign is successful, it will raise a full $200,000 to fund SIAI’s 2010 activities.
For almost a decade, the Singularity Institute has been asking questions on the future of human civilization: How can we benefit from increasingly powerful technology without succumbing to the risks, up to and including human extinction? What is the best way to handle artificial general intelligence (AGI): programs as smart as humans, or smarter?
Among SIAI’s core aims is to continue studying “Friendly AI”: AI that acts benevolently because it holds goals aligned with human values. This involves drawing on and contributing to fields like decision theory, computer science, cognitive and moral psychology, and technology forecasting.
Creating AI, especially the Friendly kind, is a difficult undertaking. We’re in it for as long as it takes, but we’ve been doing more than laying the groundwork for Friendly AI. We’ve been raising the profile of AI risk and Singularity issues in academia and elsewhere, forming communities around enhancing human rationality, and researching other avenues that promise to reduce the most severe risks the most effectively.
If you make a donation to the Singularity Institute, you can choose which grant proposal your donation should help to fill. Any time a grant proposal is fully funded, it goes into our “active projects” file: it becomes a project that we have money enough to fund, and that we are publicly committed to funding. (Some of the projects will go forward even without earmarked donations, with money from the general fund — but many won’t, and since our work is limited by how much money we have available to support skilled staff and Visiting Fellows, more money allows more total projects to go forward.)
Donate now, and seize a better than usual chance to move our work forward.
December 23rd, 2009 –
2009 has been a year of growth and new horizons for the Singularity Institute for Artificial Intelligence (SIAI). We achieved a number of milestones relevant to our mission — pursuing dialogue, research, and activism to promote a beneficial Singularity. The response we’ve received has been considerable — SIAI is more high-profile and frequently-mentioned now than it has ever been.
Our key accomplishments in 2009 were holding the Singularity Summit in New York, hiring three new employees (Michael Vassar, Michael Anissimov, and Amy Willey), establishing a continuous SIAI Visiting Fellows Program, delivering eight presentations across four conferences, improving cooperation with allied organizations such as the Future of Humanity Institute, and establishing the Less Wrong web community, which receives thousands of visitors per day and fosters many high-quality discussions on philosophical and practical issues related to decision theory and rationality. The Uncertain Future, an interactive web application for quantitatively modeling future possibilities such as human-level AI, human intelligence enhancement, and global catastrophic risk, was also released as a beta version in December.
In April, Eliezer Yudkowsky completed two years of posting sequences on Less Wrong (which will be edited into a book on rationality and Singularity-relevant topics like reductionism and decision theory), drafting strategy documents for improving internal organization and long-term planning. Throughout the year, we continued consolidating SIAI staff, Visiting Fellows, volunteers, and interns in the San Francisco Bay Area. SIAI Visiting Fellow Peter de Blanc revised a paper on unbounded utility functions. The Singularity Institute received media coverage for its work in The New York Times, Popular Mechanics, Popular Science, Forbes, and many other venues. An article by SIAI President Michael Vassar, “Machine Minds”, made it into the Forbes special “The AI Report”.
The Singularity Institute’s long-term mission is to maximize the probability of a beneficial Singularity, through dialogue, research, and activism. All of our activities are ultimately chosen to further this purpose. The Singularity Institute particularly focuses on the possibility of a Singularity through artificial general intelligence, but also analyzes other potential pathways, including whole brain emulation and human cognitive enhancement.
To summarize our major accomplishments over the past year:
1. Singularity Summit 2009 in New York. Our fourth annual Singularity Summit was the first Singularity-focused conference ever held on the East Coast. Held October 3-4, the Singularity Summit featured 25 excellent speakers on topics including biotechnology, futurism, decision theory, artificial intelligence, quantum computing. the scientific method, cognitive ability, philosophy, computer science, and even synthetic neurobiology. Over 800 people attended, and the conference attracted reporters from over two dozen news organizations, including the New York Times. Coverage was provided by Popular Mechanics, Popular Science, Forbes, and many other media venues. Speakers this year included venture capitalist Peter Thiel, Wired magazine contributing editor Gary Wolf, AI researchers Juergen Schmidhuber, Marcus Hutter, and Itamar Arel, SIAI employees Anna Salamon, Ben Goertzel, and Eliezer Yudkowsky, philosopher David Chalmers, futurist Ray Kurzweil, Stephen Wolfram of Mathematica and Wolfram Alpha fame, and many others. Videos from the Summit are online at Vimeo. After the Summit, SIAI held an in-depth workshop, which allowed the speakers and SIAI staff to share ideas and brainstorm about the risks and benefits of a possible Singularity.
2. Hiring of new employees. Early in the year, Executive Director Tyler Emerson departed the Singularity Institute and his role was filled by a new President, Michael Vassar. Mr. Vassar holds a B.S. in biochemistry from Penn State and an MBA from Drexel University, and was previously Founder and Chief Strategist at Sir Groovy, an online music licensing firm. Prior to that, he held positions with Aon, the Peace Corps, and the National Institute of Standards and Technology. Throughout the year, he participated in numerous interviews and podcasts on behalf of SIAI, including interviews at Accelerating Future, The Futurist, Future Blogger, and h+ magazine.
Two new research fellows, Anna Salamon and Steve Rayhawk, were hired by SIAI in late 2008. Salamon and Rayhawk had previously participated in the 2008 SIAI Summer Program, which was led by Salamon. Salamon holds degrees in mathematics from UC Santa Barbara and Great Books from St. John’s, and Rayhawk holds a degree in mathematics from UC Santa Barbara. Salamon and Rayhawk are both focusing on applying computational Bayesian decision theory to problems in technological forecasting, risk management policy, and social epistemology, and form the core of our Visiting Fellows Program, bringing visiting scholars up to speed on the work that SIAI does. In early 2009, SIAI also hired a Media Director, Michael Anissimov, responsible for compiling, distributing, and promoting SIAI media materials including our writing, websites, and videos, and communicating the activities of SIAI to the public. Anissimov is author of Accelerating Future, a popular blog focused on science and futurism. Most recently, in December, SIAI hired Amy Willey, who holds a law degree from New York University, as Chief Compliance Officer.
With the addition of these new employees, SIAI brought its total full-time employee count to six, including Research Fellow Eliezer Yudkowsky, who has worked for SIAI since he co-founded the organization in 2000.
3. In 2009, SIAI established a Visiting Fellows Program, based in Silicon Valley. The program began with SIAI’s 2009 Summer Fellows, brought together to work on challenging projects in decision theory, philosophy, technological forecasting, heuristics and biases, and planning for the Singularity Summit 2009. Primarily graduate students, the Fellows came from educational backgrounds in mathematics, computer science, and physics, with the remainder ranging from philosophy to economics and biochemistry. They attend or hold degrees from universities including Harvard, Stanford, Yale, Cambridge, Carnegie Mellon, Auckland University, Moscow Institute of Physics and Technology, and the University of California-Santa Barbara. Fellows traveled to Silicon Valley from throughout the United States and from Russia, Belgium, the Netherlands, Sweden, Australia, New Zealand, the United Kingdom, and Canada. Some of these researchers stayed on past the summer or joined shortly thereafter to work with SIAI as volunteers or Visiting Fellows on a more extended basis. Some of the work that came out of the Visiting Fellows Program has been presented in papers and talks at venues like the European Conference on Computing and Philosophy, the Asia-Pacific Conference on Computing and Philosophy, and a Santa Fe Institute conference on forecasting. The Visiting Fellows Program has been instrumental in fostering a devoted community of Singularity Institute supporters making useful contributions towards SIAI’s ultimate goal, and SIAI recently put out a fresh call for new SIAI Visiting Fellows.
4. SIAI researchers, volunteers, and Visiting Fellows presented the following nine talks and papers throughout 2009:
* “Changing the frame of AI futurism: From storytelling to heavy-tailed, high-dimensional probability distributions”, by Steve Rayhawk, Anna Salamon, Tom McCabe, Rolf Nelson, and Michael Anissimov. (Presented at the European Conference of Computing and Philosophy in July ‘09 (ECAP))
* “Arms Control and Intelligence Explosions”, by Carl Shulman (Also presented at ECAP)
* “Machine Ethics and Superintelligence”, by Carl Shulman and Henrik Jonsson (Presented at the Asia-Pacific Conference of Computing and Philosophy in October ‘09 (APCAP))
* “Which Consequentialism? Machine Ethics and Moral Divergence”, by Carl Shulman and Nick Tarleton (Also presented at APCAP);
* “Long-term AI forecasting: Building methodologies that work”, an invited presentation by Anna Salamon at the Santa Fe Institute conference on forecasting;
* “Shaping the Intelligence Explosion” and “How much it matters to know what matters: A back of the envelope calculation”, presentations by Anna Salamon at the Singularity Summit 2009 in October;
* “Pathways to Beneficial Artificial General Intelligence: Virtual Pets, Robot Children, Artificial Bioscientists, and Beyond”, a presentation by SIAI Director of Research Ben Goertzel at Singularity Summit 2009;
* “Cognitive Biases and Giant Risks”, a presentation by SIAI Research Fellow Eliezer Yudkowsky at Singularity Summit 2009;
* “Convergence of Expected Utility for Universal Artificial Intelligence”, a paper by Peter de Blanc, an SIAI Visiting Fellow.
Many more talks and papers are in the works for 2010, including a talk by SIAI Media Director Michael Anissimov at the Foresight 2010 conference in January.
5. One of the primary goals of the Singularity Institute in 2009 was to strengthen our ties to academia and allied organizations, which was accomplished through talks, papers, and direct dialogue. SIAI researchers and representatives built closer ties to organizations such as the Future of Humanity Institute at Oxford University, Santa Fe Institute, American Association for Artificial Intelligence, Foresight Institute, and many others. SIAI researcher Anna Salamon was invited to give a talk at an exclusive conference on technological forecasting held by the Santa Fe Institute. The Singularity Institute has been using videoconferencing, blogs, and mailing lists to keep in contact with our supporters and collaborators around the globe. SIAI more than tripled its representatives through the Visiting Fellows program, allowing it to better interface with a larger network.
6. 2009 saw the founding of the Less Wrong web community. Less Wrong was founded as a rationalist community to “systematically improve on the art, craft, and science of human rationality”. Thousands of people visit the site every day, with hundreds participating regularly in the comments sections. Less Wrong grew out of Overcoming Bias, a blog co-authored by SIAI Research Fellow Eliezer Yudkowsky and George Mason University economist Robin Hanson. Yudkowsky wrote extensively on Overcoming Bias from 2007-2009, and his posts have been ported over to Less Wrong, where they are organized into sequences that address topics such as reductionism, determinism, human rationality, metaethics, mathematics, and many others.
Less Wrong is important to the Singularity Institute’s work towards a beneficial Singularity in providing an introduction to issues of cognitive biases and rationality relevant for careful thinking about optimal philanthropy and many of the problems that must be solved in advance of the creation of provably human-friendly powerful artificial intelligence. At the same time, it has gathered a community that can provide rapid feedback and significant progress on such problems. For instance, Less Wrong participants Wei Dai and Vladimir Nesov proposed decision algorithms that can deal with a certain classes of problems where Bayesian updating seems to lead decisionmakers astray. This work was closely related to decision theory work done in-house at SIAI, namely Eliezer Yudkowsky’s timeless decision theory, an algorithm that computes the counterfactual consequences of possible actions using an extension of Judea Pearl’s formalism of causal networks to logical uncertainties, and additional work by Anna Salamon and Steve Rayhawk. These developments have received positive attention from Gary Drescher and philosopher David Chalmers, and will be written up for peer review in the coming year.
Besides providing a home for an intellectual community dialoguing on rationality and decision theory, Less Wrong is also a key venue for SIAI recruitment. Many of the participants in SIAI’s Visiting Fellows Program first discovered the organization through Less Wrong.
7. This year Eliezer Yudkowsky finished his posting sequences on Less Wrong, which attracted thousands of enthusiastic readers and came to serve as the seed of a new community. Yudkowsky used the blogging format to write the substantive content of a book on rationality, enabling that work to be read and receive feedback as it progressed. Throughout the summer, Yudkowsky engaged in Friendly AI research with Marcello Herreshoff, a Stanford mathematics student who previously spent his gap year working for SIAI. Yudkowsky is now converting his blog sequences into the planned rationality book, which he hopes will significantly assist in attracting and inspiring talented individuals to effectively work towards the aims of a beneficial Singularity and reduced existential risk.
8. In December, a subset of SIAI researchers and volunteers finished improving The Uncertain Future web application to officially announce it as a beta version. The Uncertain Future represents a new kind of futurism — futurism with heavy-tailed, high-dimensional probability distributions. The purpose is to provide a tool for use by futurists and the informed public to input probability distributions over quantitative questions like, “how much computing power would be necessary to implement neuromorphic AI?”, combining them into a “picture of the future according to you”. Another goal of the project is to provide an alternative to the futurist methodologies of storytelling and scenario building, which dominate the field even though they often cause futurists to overestimate the probability of precise, vivid stories at the expense of a wider space of neglected possibilities.
December 12th, 2009 –
The Uncertain Future, a web application built by Michael Anissimov, Steve Rayhawk, Anna Salamon, Tom McCabe, and Rolf Nelson during the Singularity Institute Summer 2008 Research Program, with helpful discussions with a few others, is now in beta and ready for public announcement.
The Uncertain Future represents a new kind of futurism — futurism with heavy-tailed, high-dimension probability distributions. In fact, that’s the name of the paper presented at the European Conference on Computing and Philosophy that unveiled the project: “Changing the frame of AI futurism: From storytelling to heavy-tailed, high-dimensional probability distributions”.
Most futurism is about telling a story — more like marketing than an honest attempt at uncovering the possible range of what the future may hold. Better than creating a single story is scenario building — but this falls short as well. Scenario building is human nature, but it leaves us susceptible to anchoring effects where we overestimate the probability of vivid scenarios. To quote “Cognitive Biases Potentially Affecting Judgment of Global Risks”, page 6:
The conjunction fallacy similarly applies to futurological forecasts. Two independent sets of professional analysts at the Second International Congress on Forecasting were asked to rate, respectively, the probability of “A complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983″ or “A Russian invasion of Poland, and a complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983″. The second set of analysts responded with significantly higher probabilities. (Tversky and Kahneman 1983.)
The conjunction fallacy means that people overestimate the probability of vivid, detailed scenarios, even though each additional detail necessarily decreases the probability that the event will occur.
To combat against the conjunction fallacy and storytelling fallacies in our particular area of futurism, which includes intelligence enhancement, AI, and global catastrophic risk, we created an interactive system that allows the user to input their own probability distributions for different variables potentially associated with the future of AI and humanity, including a probability distribution of how much computing power would required to create human-level AI, a probability distribution for the likelihood of global thermonuclear war in the next century, and many other variables. Our model includes variables for the creation of AI, the possible success of intelligence amplification technology, and the potential extinction of the human species by technological mishap before either of these occurs.
Our system is built on the assumption that breaking down a challenging prediction task into its constituent parts can be quite beneficial, because it forces us to think about the task in greater detail, and avoid obvious biases associated with specific scenarios we may be anchoring on. Some people may criticize such a view for being excessively reductionist, but many prediction tasks really can be broken down into component pieces. The alternative is making “expert” guesses based on a holistic evaluation of the prediction task, which leaves us open to many well-documented biases.
Here is the opening blurb for the webapp, by Tom McCabe:
The Uncertain Future is a future technology and world-modeling project by the Singularity Institute for Artificial Intelligence. Its goal is to allow those interested in future technology to form their own rigorous, mathematically consistent model of how the development of advanced technologies will affect the evolution of civilization over the next hundred years. To facilitate this, we have gathered data on what experts think is going to happen, in such fields as semiconductor development, biotechnology, global security, Artificial Intelligence and neuroscience. We invite you, the user, to read about the opinions of these experts, and then come to your own conclusion about the likely destiny of mankind.
It’s not perfect, but we think that our system might be a seed for looking at futurism in a different way — providing an alternative to storytelling and scenario building. This sort of “probabilistic futurism” encourages would-be seers to widen their confidence bounds when confronted with uncertainty, instead of irrationally making overconfident guesses to seem like “experts”. The particular issues we focus on are controversial — human-equivalent AI, biotechnology used to select gametes with genes associated with intelligence, the probability of planet-ending catastrophe — but we chose these issues specifically because there is disagreement about what degree of uncertainty is warranted from our present position is evaluating these scenarios.
We visualize this tool being used among futurists to specify their quantitative background assumptions regarding the technologies discussed. This might be used to clear aside straw men and zoom in on the core disagreements. It might also be used to evaluate the degree to which respective futurists have considered the technological prerequisites and other assumptions underlying their scenarios.
If you like the system or find it useful, be sure to post a link to it on Facebook, or suggest it to your friends. The system still has quite a few bugs; we used Java applets for the probability distributions, and designed it so that the Java applet makes calls to the surrounding HTML, which may fail on some combinations of OS and browser. If you use a Mac, you should use Safari, and if you use Linux/Windows, use Opera or Firefox.
December 3rd, 2009 –
Peter de Blanc (Temple University), an SIAI Visiting Fellow, has recently published the final version of his most recent paper, titled “Convergence of Expected Utility for Universal Artificial Intelligence“. The paper, based on earlier work in the field of expected utility theory by Marcus Hutter (Australian National University) and SIAI Research Fellow Eliezer Yudkowsky, proves mathematically that any unbounded, perception determined, computable utility function cannot assign a defined utility to any action, assuming a Solomonoff-like prior. The abstract of the paper is as follows:
“We consider a sequence of repeated interactions between an agent and an environment. Uncertainty about the environment is captured by a probability distribution over a space of hypotheses, which includes all computable functions. Given a utility function, we can evaluate the expected utility of any computational policy for interaction with the environment. After making some plausible assumptions (and maybe one not-so-plausible assumption), we show that if the utility function is unbounded, then the expected utility of any policy is undefined.”
Peter de Blanc has done research with the Singularity Institute since 2006, when he participated in the “Summer of AI” research program along with Nick Hay (UC Berkeley), Marcello Herreshoff (Stanford University) and Eliezer Yudkowsky. During the summer of 2009, he was a Singularity Institute Summer Fellow, helping SIAI researchers work on problems in the fields of Friendly AI, rationality and decision theory. His personal website, Space and Games, can be found at http://www.spaceandgames.com/.
December 1st, 2009 –
The following message is cross-posted from Less Wrong on behalf of SIAI Research Fellow Anna Salamon:
Last summer, 15 Less Wrongers, under the auspices of SIAI, gathered in a big house in Santa Clara (in the SF bay area), with whiteboards, existential risk-reducing projects, and the ambition to learn and do.
Now, the new and better version has arrived. We’re taking folks on a rolling basis to come join in our projects, learn and strategize with us, and consider long term life paths. Working with this crowd transformed my world; it felt like I was learning to think. I wouldn’t be surprised if it can transform yours.
A representative sample of current projects:
Interested, but not sure whether to apply?
Past experience indicates that more than one brilliant, capable person refrained from contacting SIAI, because they weren’t sure they were “good enough”. That kind of timidity destroys the world, by failing to save it. So if that’s your situation, send us an email. Let us be the one to say “no”. Glancing at an extra application is cheap, and losing out on a capable applicant is expensive.
And if you’re seriously interested in risk reduction but at a later time, or in another capacity — send us an email anyway. Coordinated groups accomplish more than uncoordinated groups; and if you care about risk reduction, we want to know.
What we’re looking for
At bottom, we’re looking for anyone who:
Bonus points for any (you don’t need them all) of the following traits:
If you think this might be you, send a quick email to anna@singinst.org. Include:
Our application process is fairly informal, so send us a quick email as initial inquiry and we can decide whether or not to follow up with more application components.
As to logistics: we cover room, board, and, if you need it, airfare, but no other stipend.
Looking forward to hearing from you,
Anna
November 23rd, 2009 –
Just prior to Singularity Summit 2009, Singularity Institute President Michael Vassar did a podcast with “The Skeptics Guide to the Universe”, produced by the New England Skeptical Society in association with the James Randi Educational Foundation (JREF).
Skip to 26:00 to get past the news items. Here’s a funny tidbit of Michael talking about some of the poor thinking seen when people discuss how to make AI friendly:
We have a lot of silliness, such as worst moral of the story ever… Lilo and Stitch. “If you’re just nice enough to the fundamentally evil creature, it will have to love you.”
That’s at 31:50. Also check out 41:00, where Michael explains the whole reason for having a Singularity Summit and Singularity Institute. At 49:30: “So how do we keep it from deciding that it wants to make ice cream out of human brains?”
