Comments on: Three Major Singularity Schools http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/ The Singularity Institute exists to confront the challenge of powerful AI, both the opportunity and the risk. Sun, 14 Mar 2010 03:35:02 +0000 http://wordpress.org/?v=2.1.3 By: Cosmic Vortex http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-6852 Cosmic Vortex Mon, 01 Oct 2007 01:40:25 +0000 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-6852 Nice essay. I understand what you are saying about the 3 different projections contradicting each other, I think the odds that they do are actually very small. The only way it would be possible for these views to contradict would be if the superintellgence that emerges chooses not to continue on Moore's path. While this is certainly possible, I think its very improbable that it would choose to do so. I think theres at least one reasonably safe assumption we can make about the intelligence beyond the singularity... it will continue to expand its intelligence and power. This will mean continuing on Moore's path at least until it gets a technology that can replace it altogether. If thats a contradiction between views, I think it's a bit too small to count. Nice essay. I understand what you are saying about the 3 different projections contradicting each other, I think the odds that they do are actually very small.
The only way it would be possible for these views to contradict would be if the superintellgence that emerges chooses not to continue on Moore’s path. While this is certainly possible, I think its very improbable that it would choose to do so. I think theres at least one reasonably safe assumption we can make about the intelligence beyond the singularity… it will continue to expand its intelligence and power. This will mean continuing on Moore’s path at least until it gets a technology that can replace it altogether. If thats a contradiction between views, I think it’s a bit too small to count.

]]> By: Education Futures » Three Singularities, three conversations http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-6865 Education Futures » Three Singularities, three conversations Mon, 01 Oct 2007 06:04:55 +0000 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-6865 [...] Yudkowsky, on the SIAI blog, posted his observations of the emergence of three “logically distinct” schools of thought related to the [...] […] Yudkowsky, on the SIAI blog, posted his observations of the emergence of three “logically distinct” schools of thought related to the […]

]]> By: Joshua Fox http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-6872 Joshua Fox Mon, 01 Oct 2007 08:14:29 +0000 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-6872 Great clarification of these concepts. By the way, the word is "Apocalypticism" rather than "Apocalyptism." Not a big deal, but considering the common mangling of "Singularitarianism," a similar mouthful, it's best to get the term right. Great clarification of these concepts.

By the way, the word is “Apocalypticism” rather than “Apocalyptism.” Not a big deal, but considering the common mangling of “Singularitarianism,” a similar mouthful, it’s best to get the term right.

]]> By: Joe Hunkins http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-6887 Joe Hunkins Mon, 01 Oct 2007 15:32:36 +0000 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-6887 A nice post, though I'm not clear on why these are incompatible ideas. I'm guessing Kurzweil would agree that an intelligence explosion is likely, though he seems to think it'll take some years of conscious computing before that happens (you don't?), and most would agree that the future is *very* difficult to predict even without a singularity, so the event horizon is also a reasonable assertion in the other two scenarios. A nice post, though I’m not clear on why these are incompatible ideas. I’m guessing Kurzweil would agree that an intelligence explosion is likely, though he seems to think it’ll take some years of conscious computing before that happens (you don’t?), and most would agree that the future is *very* difficult to predict even without a singularity, so the event horizon is also a reasonable assertion in the other two scenarios.

]]> By: Kaj Sotala http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-6888 Kaj Sotala Mon, 01 Oct 2007 15:55:35 +0000 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-6888 Accelerating Change is a bit different from the two others, but I'm not sure why there's a need to differentiate between Event Horizon and Intelligence Explosion. Intelligence Explosion, at least, seems to lead to Event Horizon (though not necessarily vice versa - but then, I don't think I've heard of an Event Horizon formulation that wouldn't have included Intelligence Explosion at least implictly). The core claim of Accelerating Change supports the two others in the "why should we belive this should happen soon", though its strong claim is more dubious. Accelerating Change is a bit different from the two others, but I’m not sure why there’s a need to differentiate between Event Horizon and Intelligence Explosion. Intelligence Explosion, at least, seems to lead to Event Horizon (though not necessarily vice versa - but then, I don’t think I’ve heard of an Event Horizon formulation that wouldn’t have included Intelligence Explosion at least implictly).

The core claim of Accelerating Change supports the two others in the “why should we belive this should happen soon”, though its strong claim is more dubious.

]]> By: Roko http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-6910 Roko Mon, 01 Oct 2007 20:34:51 +0000 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-6910 Kaj said: "Accelerating Change is a bit different from the two others, but I’m not sure why there’s a need to differentiate between Event Horizon and Intelligence Explosion." I'll second that, Kaj. Kaj said: “Accelerating Change is a bit different from the two others, but I’m not sure why there’s a need to differentiate between Event Horizon and Intelligence Explosion.”

I’ll second that, Kaj.

]]> By: Jeffrey Herrlich http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-6911 Jeffrey Herrlich Mon, 01 Oct 2007 21:16:40 +0000 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-6911 I'm an Intelligence Explosioner too. I don't favor the Accelerating Change much because it's a speculative extrapolation of *human* ability. Whereas I believe that the first Strong AI will be of a huge qualitative difference. And I don't favor the Event Horizon, in the sense that I believe that the "objectives" of the future *can* be predicted and guided - hence the Friendly AI thesis. It's only the means to the ends that *perhaps* can't be reasonably predicted by us humans. But fortunately a Friendly AI will be a super-expert at strategizing ways to effectively avoid violating its stable goals. I’m an Intelligence Explosioner too. I don’t favor the Accelerating Change much because it’s a speculative extrapolation of *human* ability. Whereas I believe that the first Strong AI will be of a huge qualitative difference. And I don’t favor the Event Horizon, in the sense that I believe that the “objectives” of the future *can* be predicted and guided - hence the Friendly AI thesis. It’s only the means to the ends that *perhaps* can’t be reasonably predicted by us humans. But fortunately a Friendly AI will be a super-expert at strategizing ways to effectively avoid violating its stable goals.

]]> By: Robin Hanson http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-6913 Robin Hanson Mon, 01 Oct 2007 21:49:14 +0000 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-6913 OK, for an analogy, we driving down a mountain and: 1) we can see that the road ahead steadily gets steeper, or 2) the road ahead is straight, but then passes into clouds, or 3) we clearly see both a cliff ahead, and a vertical drop below 4) we are just warned "beware of something weird ahead" To which I'd add: 5) we see the road ahead suddenly turns and gets much steeper. OK, for an analogy, we driving down a mountain and:
1) we can see that the road ahead steadily gets steeper, or
2) the road ahead is straight, but then passes into clouds, or
3) we clearly see both a cliff ahead, and a vertical drop below
4) we are just warned “beware of something weird ahead”
To which I’d add:
5) we see the road ahead suddenly turns and gets much steeper.

]]> By: TONY COHEN http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-6924 TONY COHEN Tue, 02 Oct 2007 03:02:18 +0000 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-6924 20 years ago, if you were a 'naked' Singularity, you'd be at a Black Hole"s Event Horizon about to be sucked in. 20 years ago, if you were a ‘naked’ Singularity,

you’d be at a Black Hole”s Event Horizon about to

be sucked in.

]]> By: Tom McCabe http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-6986 Tom McCabe Tue, 02 Oct 2007 22:38:08 +0000 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-6986 "This will mean continuing on Moore’s path at least until it gets a technology that can replace it altogether." What is the probability that a superintelligence would develop chips at *exactly* the same speed that humans do? Not even twice as slow, or twice as fast? Consider that the difference in raw FLOPS is going to be several orders of magnitude and up. “This will mean continuing on Moore’s path at least until it gets a technology that can replace it altogether.”

What is the probability that a superintelligence would develop chips at *exactly* the same speed that humans do? Not even twice as slow, or twice as fast? Consider that the difference in raw FLOPS is going to be several orders of magnitude and up.

]]> By: Zubon http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-8221 Zubon Mon, 15 Oct 2007 14:40:56 +0000 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-8221 Don't the three lead to one another? Accelerating Change is the source of the Intelligence Explosion, as the curve becomes increasingly steep. This leads to the Event Horizon, since we cannot predict beyond an Intelligence Explosion. Intelligence Explosion may wear out the "smooth curve" part of Accelerating Change, but I don't think Mr. Kurzweil is that strongly committed to it. If nothing else, we are usually slightly off the curve, and "slightly off" becomes "way off" pretty quickly when we increase at x^y^z. And unless you believe that you can predict what a superinteligence will do, Eliezer, Intelligence Explosion implies an Event Horizon. Don’t the three lead to one another? Accelerating Change is the source of the Intelligence Explosion, as the curve becomes increasingly steep. This leads to the Event Horizon, since we cannot predict beyond an Intelligence Explosion.

Intelligence Explosion may wear out the “smooth curve” part of Accelerating Change, but I don’t think Mr. Kurzweil is that strongly committed to it. If nothing else, we are usually slightly off the curve, and “slightly off” becomes “way off” pretty quickly when we increase at x^y^z. And unless you believe that you can predict what a superinteligence will do, Eliezer, Intelligence Explosion implies an Event Horizon.

]]> By: Life, the Universe, and Everything » Objections to SIAI/AGI/FAI http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-22949 Life, the Universe, and Everything » Objections to SIAI/AGI/FAI Sat, 29 Dec 2007 15:08:17 +0000 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-22949 [...] There are so many different meanings attached that the term Singularity has ceased to be useful. (Anissimov, Eliezer [...] […] There are so many different meanings attached that the term Singularity has ceased to be useful. (Anissimov, Eliezer […]

]]> By: Accelerating Future » IEEE Spectrum Special Issue on the Singularity http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-54776 Accelerating Future » IEEE Spectrum Special Issue on the Singularity Tue, 03 Jun 2008 07:15:29 +0000 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-54776 [...] robots be conscious? 2. Is technological change accelerating? (Core claim of Accelerating Change school.) 3. Does technological change follow smooth exponential curves? (Strong claim of AC school, by [...] […] robots be conscious? 2. Is technological change accelerating? (Core claim of Accelerating Change school.) 3. Does technological change follow smooth exponential curves? (Strong claim of AC school, by […]

]]> By: Russell123 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-55982 Russell123 Sat, 07 Jun 2008 15:02:16 +0000 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-55982 Interesting article, but its unclear which one leads to robot sword fights in outer space. http://www.youtube.com/watch?v=DDrbbe7lgyE&feature=related My guess would be apocalypticism. Interesting article, but its unclear which one leads to robot sword fights in outer space.

http://www.youtube.com/watch?v=DDrbbe7lgyE&feature=related

My guess would be apocalypticism.

]]> By: The Singularity Institute Blog : Blog Archive : IEEE Spectrum special report on the Singularity http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-56675 The Singularity Institute Blog : Blog Archive : IEEE Spectrum special report on the Singularity Tue, 10 Jun 2008 09:36:50 +0000 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-56675 [...] Russell123 on Three Major Singularity Schools [...] […] Russell123 on Three Major Singularity Schools […]

]]> By: Greg http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-57596 Greg Sat, 14 Jun 2008 23:37:00 +0000 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-57596 I would like to favour the Exploding Intelligence theory for future technological change, with a mixture of both the core claim and the strong claim, If human intelligence could match pace? I don't think it could just by been augmented, if there were other pure super intelligences computer based. For human intelligence to keep pace it would have to go like Kurzweil thinks where humans upload there minds to a computer or android but would the new person recognize themselves as the same person as their former selves. If the former human version still was alive after transfering their mind to an android, wouldn't it be like identical twins. While knowning that eachother are very similar, they both recognize eachother as different persons. Not one and the same, but if a human uploaded their minds to an android and then the instant it was activated the human was killed or at least put into some kind of suspended animation, then the android might be able to recognize themself as the former human. Another way might be to try and keep the knowledge of one version of the person from the other, but how could the original human forget that they had just uploaded. The knowledge of uploading their minds along with all other knowledge would be transfered as well to the android. Unless this information could be deleted from both. It might be possible for the two versions to exist at the same time until the human version died naturally then of course the android version would have evolved a lot different and faster than the human version.and would have evolved into soneone entirely different but still having knowledge from its human past. If humans at that time knew that the uploaded version of themselves became someone entirely different and their own conscious was not transfered then they might not bother to transfer. This type of question will need to be addressed, either starting now or some time in the near future when more knowledge is available. I would like to favour the Exploding Intelligence theory for future technological change, with a mixture of both the core claim and the strong claim, If human intelligence could match pace? I don’t think it could just by been augmented, if there were other pure super intelligences computer based. For human intelligence to keep pace it would have to go like Kurzweil thinks where humans upload there minds to a computer or android but would the new person recognize themselves as the same person as their former selves. If the former human version still was alive after transfering their mind to an android, wouldn’t it be like identical twins. While knowning that eachother are very similar, they both recognize eachother as different persons. Not one and the same, but if a human uploaded their minds to an android and then the instant it was activated the human was killed or at least put into some kind of suspended animation, then the android might be able to recognize themself as the former human.
Another way might be to try and keep the knowledge of one version of the person from the other, but how could the original human forget that they had just uploaded. The knowledge of uploading their minds along with all other knowledge would be transfered as well to the android. Unless this information could be deleted from both. It might be possible for the two versions to exist at the same time until the human version died naturally then of course the android version would have evolved a lot different and faster than the human version.and would have evolved into soneone entirely different but still having knowledge from its human past.
If humans at that time knew that the uploaded version of themselves became someone entirely different and their own conscious was not transfered then they might not bother to transfer.
This type of question will need to be addressed, either starting now or some time in the near future when more knowledge is available.

]]> By: BrainsLab.net » Blog Archive » Eliezer on the singularity http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-63305 BrainsLab.net » Blog Archive » Eliezer on the singularity Fri, 04 Jul 2008 16:02:27 +0000 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-63305 [...] His notion of the singularity involves I. J. Good’s “intelligence explosion.” More here. [...] […] His notion of the singularity involves I. J. Good’s “intelligence explosion.” More here. […]

]]> By: Cesium http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-92104 Cesium Thu, 16 Oct 2008 17:00:56 +0000 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-92104 The core claims of the Event Horizon and Intelligence Explosion schools of thought are hugely flawed. Event Horizon says "technology will advance to the point of improving on human intelligence". Intelligence Explosion says "If technology can significantly improve on human intelligence - create minds smarter than the smartest existing humans". The problem is that superhuman intelligence already exists. Putting a man on the moon required more than one person's mental abilities. Building modern cpu processors consumes more than one person's intelligence. We don't sit around waiting for someone to invent a super human intelligence and then everything goes "foom". We are currently in the rapid feedback cycle where our very modest artificial processing power contribute to coordinating global human processing power to build slightly more powerful artificial processing power. These are not three separate schools. Accelerating Change and Intelligence Explosion are tightly coupled. Event Horizon is completely orthogonal and asks whether or not one can sensibly talk about what the world will look like after a sufficient amount of change. The strong claims are not in conflict. Accelerating Change says that we can confidently predict that new technologies will arrive by certain periods of time. It does not specify what those technologies are. Intelligence Explosion says that one of the areas that will be changed by Accelerating Change is the amount of processing power available to humans. Together, Accelerating Change and Intelligence Explosion state that we can confidently predict that we will someday have a *lot* of processing power available. Event Horizon asks whether or not we can imagine what the world will be like once that much processing power is available. The timing of the Singularity needs to be taken with a great grain of salt. Current timings suggest that when we build a cubic-foot box of processing power that is as powerful as a human brain, things will get quite interesting. However, it doesn't matter if computation is performed on a distributed computer or within a single square foot box. There are enough different problems to work on that distributing the processing of the problems around the world isn't a problem. Thus, the point at which we build cubic-foot boxes that contain human equivalent intelligence is roughly the point at which we build, in one year, an amount of processing power equal to the processing power of existing humans. It's the point at which we start doubling the total processing power available to humanity every couple of years. It will still take an additional 14 years or so to get "orders of magnitude more" processing power than is available to unaugmented humans. We also need to note that today, we are not utilizing much of the available human processing power. Roughly half the world population is well connected. Africa is poorly connected; education rates are low in many parts of the world; rural farmers are poorly connected. Improvements in education and the interconnectedness of people will grow human processing power much faster for most of the next few decades than building faster artificial processing power. The core claims of the Event Horizon and Intelligence Explosion schools of thought are hugely flawed.

Event Horizon says “technology will advance to the point of improving on human intelligence”.

Intelligence Explosion says “If technology can significantly improve on human intelligence - create minds smarter than the smartest existing humans”.

The problem is that superhuman intelligence already exists. Putting a man on the moon required more than one person’s mental abilities. Building modern cpu processors consumes more than one person’s intelligence.

We don’t sit around waiting for someone to invent a super human intelligence and then everything goes “foom”. We are currently in the rapid feedback cycle where our very modest artificial processing power contribute to coordinating global human processing power to build slightly more powerful artificial processing power.

These are not three separate schools. Accelerating Change and Intelligence Explosion are tightly coupled. Event Horizon is completely orthogonal and asks whether or not one can sensibly talk about what the world will look like after a sufficient amount of change.

The strong claims are not in conflict. Accelerating Change says that we can confidently predict that new technologies will arrive by certain periods of time. It does not specify what those technologies are. Intelligence Explosion says that one of the areas that will be changed by Accelerating Change is the amount of processing power available to humans.

Together, Accelerating Change and Intelligence Explosion state that we can confidently predict that we will someday have a *lot* of processing power available.

Event Horizon asks whether or not we can imagine what the world will be like once that much processing power is available.

The timing of the Singularity needs to be taken with a great grain of salt. Current timings suggest that when we build a cubic-foot box of processing power that is as powerful as a human brain, things will get quite interesting. However, it doesn’t matter if computation is performed on a distributed computer or within a single square foot box. There are enough different problems to work on that distributing the processing of the problems around the world isn’t a problem.

Thus, the point at which we build cubic-foot boxes that contain human equivalent intelligence is roughly the point at which we build, in one year, an amount of processing power equal to the processing power of existing humans. It’s the point at which we start doubling the total processing power available to humanity every couple of years. It will still take an additional 14 years or so to get “orders of magnitude more” processing power than is available to unaugmented humans.

We also need to note that today, we are not utilizing much of the available human processing power. Roughly half the world population is well connected. Africa is poorly connected; education rates are low in many parts of the world; rural farmers are poorly connected. Improvements in education and the interconnectedness of people will grow human processing power much faster for most of the next few decades than building faster artificial processing power.

]]> By: Cesium http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-92105 Cesium Thu, 16 Oct 2008 17:04:12 +0000 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-92105 Jeff -- Why? Processing power is processing power. It doesn't matter if it is in vivo or in silico. Distributed processing and shared memory processing are isomorphic. Why will "Strong AI" be qualitatively different? Jeff — Why? Processing power is processing power. It doesn’t matter if it is in vivo or in silico. Distributed processing and shared memory processing are isomorphic. Why will “Strong AI” be qualitatively different?

]]> By: Ambition, Intelligence, and Artificial Intelligence | B Frank http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-96150 Ambition, Intelligence, and Artificial Intelligence | B Frank Sat, 01 Nov 2008 19:38:48 +0000 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-96150 [...] For starters, Yudkowsky pointed out that Lanier was taking “Singularity” to mean at least three different things (elsewhere Yudkowsky has delineated “Three Major Singularity Schools“). [...] […] For starters, Yudkowsky pointed out that Lanier was taking “Singularity” to mean at least three different things (elsewhere Yudkowsky has delineated “Three Major Singularity Schools“). […]

]]> By: Eliezer on the singularity « Brains Lab http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-152107 Eliezer on the singularity « Brains Lab Sun, 05 Apr 2009 16:19:38 +0000 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-152107 [...] His notion of the singularity involves I. J. Good’s “intelligence explosion.” More here. [...] […] His notion of the singularity involves I. J. Good’s “intelligence explosion.” More here. […]

]]> By: The End of Suffering? -Humanity+ UK 2010 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-248997 The End of Suffering? -Humanity+ UK 2010 Wed, 17 Feb 2010 14:12:34 +0000 http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/#comment-248997 [...] risk, radical life-extension and the (contested!) prospects of some kind of technological Singularity. Clearly, transhumanism is a broad church. What ties link such seemingly diverse [...] […] risk, radical life-extension and the (contested!) prospects of some kind of technological Singularity. Clearly, transhumanism is a broad church. What ties link such seemingly diverse […]

]]>