Comments on: The Power of Intelligence http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/ The Singularity Institute exists to confront the challenge of powerful AI, both the opportunity and the risk. Wed, 17 Mar 2010 20:05:06 +0000 http://wordpress.org/?v=2.1.3 By: The Power of Intelligence - Emuforums.com http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-492 The Power of Intelligence - Emuforums.com Wed, 11 Jul 2007 12:15:23 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-492 [...] Power of Intelligence The Singularity Institute Blog : Blog Archive : The Power of Intelligence [...] […] Power of Intelligence The Singularity Institute Blog : Blog Archive : The Power of Intelligence […]

]]> By: James R Hughes, MD,PhD http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-493 James R Hughes, MD,PhD Wed, 11 Jul 2007 12:37:32 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-493 Very well written:) The gray goo in our skull will be critical to the future of AI, and the dexterity of our frail shell will be eventually enhanced rather than replaced. There is an intelligence of dexterity and adaptability of the human form, that AI is still far from duplicating. We will evolve. Very well written:) The gray goo in our skull will be critical to the future of AI, and the dexterity of our frail shell will be eventually enhanced rather than replaced. There is an intelligence of dexterity and adaptability of the human form, that AI is still far from duplicating. We will evolve.

]]> By: Roko http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-512 Roko Wed, 11 Jul 2007 23:07:45 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-512 "People - venture capitalists in particular - sometimes ask how, if the Singularity Institute successfully builds a true AI, the results will be commercialized." I'm sitting here trying to imagine Eliezer et al at a meeting with some hard-nosed venture capitalists. It sounds like it could be hilarious, if only the fate of the world didn't rest on it... ************************************************* Hard-nosed venture capitalist: So tell us, once you've got it working, how will you make people pay for it? Are you thinking of charging for AI subscription on a monthly basis or are you going for a lump-sum payment? SIAI: (pause) um actually we anticipate that a scarcity-based economy will cease to exist soon after the singularity. Money will no longer have value. Hard-nosed venture capitalist: Come again? SIAI: The AI will give people what they want because we've programed it to be friendly. There will be no hunger, no scarcity of housing, clothes, gadgets, etc. Hard-nosed venture capitalist: That doesn't sound so good... I don't think you've thought your *business strategy* through ... Maybe you should put some DRM in your friendliness software! I know, you could call it a pay-as-you-go-friendly AI; it's only friendly to people if they keep paying the monthly subscription fees. I think you could market that really well! Or, better still, why not forget the friendliness software altogether and instead just program the AI to make you as much money as it can, by whatever means it chooses - now that's thinking out of the box, eh! SIAI: (holding their heads in their hands weeping) “People - venture capitalists in particular - sometimes ask how, if the Singularity Institute successfully builds a true AI, the results will be commercialized.”

I’m sitting here trying to imagine Eliezer et al at a meeting with some hard-nosed venture capitalists. It sounds like it could be hilarious, if only the fate of the world didn’t rest on it…

*************************************************

Hard-nosed venture capitalist: So tell us, once you’ve got it working, how will you make people pay for it? Are you thinking of charging for AI subscription on a monthly basis or are you going for a lump-sum payment?

SIAI: (pause) um actually we anticipate that a scarcity-based economy will cease to exist soon after the singularity. Money will no longer have value.

Hard-nosed venture capitalist: Come again?

SIAI: The AI will give people what they want because we’ve programed it to be friendly. There will be no hunger, no scarcity of housing, clothes, gadgets, etc.

Hard-nosed venture capitalist: That doesn’t sound so good… I don’t think you’ve thought your *business strategy* through … Maybe you should put some DRM in your friendliness software! I know, you could call it a pay-as-you-go-friendly AI; it’s only friendly to people if they keep paying the monthly subscription fees. I think you could market that really well! Or, better still, why not forget the friendliness software altogether and instead just program the AI to make you as much money as it can, by whatever means it chooses - now that’s thinking out of the box, eh!

SIAI: (holding their heads in their hands weeping)

]]> By: Roko http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-513 Roko Wed, 11 Jul 2007 23:22:02 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-513 On a more serious note, I don't envy you - the task of trying to sell an invention that will probably be the end of money is not an easy one, and it's probably especially hard to sell to people whose whole lives run on the base assumption that there will always be money and markets. I'd be interested to hear what actually happens in these situations. What do they say? Anyway, great article. :) On a more serious note, I don’t envy you - the task of trying to sell an invention that will probably be the end of money is not an easy one, and it’s probably especially hard to sell to people whose whole lives run on the base assumption that there will always be money and markets. I’d be interested to hear what actually happens in these situations. What do they say?

Anyway, great article. :)

]]> By: Eliezer Yudkowsky http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-518 Eliezer Yudkowsky Thu, 12 Jul 2007 00:22:26 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-518 I bet you think you're joking. I bet you think you’re joking.

]]> By: Tyler Emerson http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-529 Tyler Emerson Thu, 12 Jul 2007 02:25:06 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-529 Hilarious. Yes, if only you were joking. Maybe some time we'll tape an SIAI Salon Dinner. Hilarious. Yes, if only you were joking. Maybe some time we’ll tape an SIAI Salon Dinner.

]]> By: Roko http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-547 Roko Thu, 12 Jul 2007 10:49:17 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-547 I get the impression that my attempts at humor are painfully close to the truth... Well, all I can say is good luck finding some people who have both money and enough sense to realize that there is a better world out there beyond money, marketing and the executive golf club. If I weren't a poor student, I'd be donating a good chunk of cash myself. I get the impression that my attempts at humor are painfully close to the truth… Well, all I can say is good luck finding some people who have both money and enough sense to realize that there is a better world out there beyond money, marketing and the executive golf club. If I weren’t a poor student, I’d be donating a good chunk of cash myself.

]]> By: Stephen Burrows http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-565 Stephen Burrows Thu, 12 Jul 2007 17:14:32 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-565 There is a non zero probability that an artificial general intelligence will first be created by some one, or some group, without affiliations with any formal organization. The first powered aircraft was designed, built, and tested entirely in secret by self made experts, once the technology required became cheap enough for wide access. I predict the same will be true with AGI. The trick here is to get the idea of making the first one "nice" out to where those making the attempt are aware of that crucial requirement. There is a non zero probability that an artificial general intelligence will first be created by some one, or some group, without affiliations with any formal organization. The first powered aircraft was designed, built, and tested entirely in secret by self made experts, once the technology required became cheap enough for wide access. I predict the same will be true with AGI. The trick here is to get the idea of making the first one “nice” out to where those making the attempt are aware of that crucial requirement.

]]> By: Nick Tarleton http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-629 Nick Tarleton Fri, 13 Jul 2007 21:44:19 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-629 You mean someone's actually suggested that? (holds head in hands and weeps) You mean someone’s actually suggested that?

(holds head in hands and weeps)

]]> By: Tom McCabe http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-784 Tom McCabe Tue, 17 Jul 2007 02:32:42 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-784 "I bet you think you’re joking." Maybe if you explain to them that your goal isn't to make lots of money, but to advance human civilization, like if the Romans had started a large project to mechanize the production of clothing. It's not like SIAI is going to show a profit in fiscal terms. “I bet you think you’re joking.”

Maybe if you explain to them that your goal isn’t to make lots of money, but to advance human civilization, like if the Romans had started a large project to mechanize the production of clothing. It’s not like SIAI is going to show a profit in fiscal terms.

]]> By: Jeffrey Herrlich http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-808 Jeffrey Herrlich Tue, 17 Jul 2007 16:04:47 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-808 "People - venture capitalists in particular - sometimes ask how, if the Singularity Institute successfully builds a true AI, the results will be *commercialized*. This is what we call a framing problem." I'm speaking entirely for myself here, and not for SIAI. Perhaps it would be good to explain to the VCs that their money will have no value whatsoever if they and everyone else no longer exists, because Friendly AI never received enough early funding. They need to understand that with a Friendly AI their personal quality-of-life will be unimaginably better than anything they could possibly purchase by traditional means - no matter *how* much money they have. I know that it would probably be difficult for SIAI to make this presentation, entirely tactfully. But sometimes even very smart people are in need of a reality-check. You can show them this post by me, and I'll take the blame for not being tactful. ;-) “People - venture capitalists in particular - sometimes ask how, if the Singularity Institute successfully builds a true AI, the results will be *commercialized*. This is what we call a framing problem.”

I’m speaking entirely for myself here, and not for SIAI.

Perhaps it would be good to explain to the VCs that their money will have no value whatsoever if they and everyone else no longer exists, because Friendly AI never received enough early funding. They need to understand that with a Friendly AI their personal quality-of-life will be unimaginably better than anything they could possibly purchase by traditional means - no matter *how* much money they have.

I know that it would probably be difficult for SIAI to make this presentation, entirely tactfully. But sometimes even very smart people are in need of a reality-check. You can show them this post by me, and I’ll take the blame for not being tactful. ;-)

]]> By: Gully Foyle http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-842 Gully Foyle Wed, 18 Jul 2007 04:47:18 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-842 This is the same problem that Babbage faced when he sought funding for his difference engine. No one believed a single machine could do all manor of things by simply reconfiguring it. This is the same problem that Babbage faced when he sought funding for his difference engine. No one believed a single machine could do all manor of things by simply reconfiguring it.

]]> By: Fredrik http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-884 Fredrik Thu, 19 Jul 2007 07:00:33 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-884 Venture capitalists are survivors. They are not stupid. They will ask a couple of stupid questions. It's their job. But they actually do listen to you and some of them will understand, and some will not. Just like anybody else ... Venture capitalists are survivors. They are not stupid. They will ask a couple of stupid questions. It’s their job. But they actually do listen to you and some of them will understand, and some will not. Just like anybody else …

]]> By: Fredrik http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-885 Fredrik Thu, 19 Jul 2007 07:02:30 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-885 Eliezer: What a FANTASTIC story! Thank you! Eliezer: What a FANTASTIC story! Thank you!

]]> By: Jeffrey Herrlich http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-893 Jeffrey Herrlich Thu, 19 Jul 2007 13:54:32 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-893 No they're not stupid, but it appears that the majority of them *don't* truly understand, or at least they don't accept the situation viscerally. It's definitely an unfortunate situation at this time, hopefully it can be changed. No they’re not stupid, but it appears that the majority of them *don’t* truly understand, or at least they don’t accept the situation viscerally. It’s definitely an unfortunate situation at this time, hopefully it can be changed.

]]> By: Life, the Universe, and Everything » If Singularity Were Like Nanotechnology http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-1323 Life, the Universe, and Everything » If Singularity Were Like Nanotechnology Thu, 26 Jul 2007 20:22:11 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-1323 [...] take into account the need for a reliable FAI brand name… wait, this has apparently already happened [...] […] take into account the need for a reliable FAI brand name… wait, this has apparently already happened […]

]]> By: Grant Czerepak http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-4567 Grant Czerepak Sun, 09 Sep 2007 02:55:23 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-4567 The concept of migrating intelligence from the strata of carbon based organisms to another strata is still hard to conceive. But it is obvious that even though the body or id may change, the soul and spirit or ego and superego, depending on you perspective, are still inseparable at this point. What I am trying to say is we may create an intelligence, but we also have to create an ethical foundation that guides that intelligence before we unleash it. And the more I learn about the relativistic reasoning of the human mind when making ethical judgments the less faith I have in our ability to create what we would deem an ethical AI from our own perspective. Half the world could consider the AI friendly and the other half could consider it an abomination. The concept of migrating intelligence from the strata of carbon based organisms to another strata is still hard to conceive. But it is obvious that even though the body or id may change, the soul and spirit or ego and superego, depending on you perspective, are still inseparable at this point.

What I am trying to say is we may create an intelligence, but we also have to create an ethical foundation that guides that intelligence before we unleash it. And the more I learn about the relativistic reasoning of the human mind when making ethical judgments the less faith I have in our ability to create what we would deem an ethical AI from our own perspective. Half the world could consider the AI friendly and the other half could consider it an abomination.

]]> By: Davy http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-4585 Davy Sun, 09 Sep 2007 06:03:39 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-4585 Perhaps we should first create an ethical foundation that guides our intelligence. Perhaps we should first create an ethical foundation that guides our intelligence.

]]> By: Steven Thomas http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-4592 Steven Thomas Sun, 09 Sep 2007 08:04:42 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-4592 This AI is going to have to be very smart to overcome the problem of limited resources, and smarter still to overcome the conflicts that arise from the other functions of our brains, such as instincts and passions. Our planet can only accommodate so many people. Will this AI teach us how to live simply and fairly? Will this AI solve jealousy, gluttony, xenophobia, religious fundamentalism, recklessness, vengeance, and other basic tensions of primate social structures? Can it solve these problems, and leave us human? This AI is going to have to be very smart to overcome the problem of limited resources, and smarter still to overcome the conflicts that arise from the other functions of our brains, such as instincts and passions. Our planet can only accommodate so many people. Will this AI teach us how to live simply and fairly? Will this AI solve jealousy, gluttony, xenophobia, religious fundamentalism, recklessness, vengeance, and other basic tensions of primate social structures? Can it solve these problems, and leave us human?

]]> By: denis bider http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-15129 denis bider Tue, 20 Nov 2007 21:54:20 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-15129 Ugh. "Ethics". What we're talking about here is no ethics. We don't want the supermachine to be "ethical". We ourselves are not "ethical", especially not towards inferior things. No one (except in limited cases) expects us to be ethical towards inferior things, and no one (except in limited cases) actually enforces our being ethical towards inferior things. Our "ethicism" in this regard is fraught with inconsistency. On the one hand we have PETA and the pet police. On the other hand we slaughter pigs and cows and create chicken who can't stand upright because we've made their breasts grow too big. We go to the supermarket and buy a bag of shrimp that have been slaughtered by the billion. We have no bad thoughts about that. We operate on no "ethics". What you're talking about here with "friendly AI" and "ethical AI" doesn't involve any lofty principles either. The basic principle you wish to imbue your AI with is something like: "thou shalt not harm thy creator". This is pure human selfishness, and our ability to imbue a vastly superior being with that concept may be very limited. After all, nature has imbued us with a desire to have sex in order to reproduce, and so we invented condoms. A superior machine can invent all kinds of workarounds to avoid the inconvenience of our flimsy, built-in "ethics". If someone succeeds in building strong AI, and the creature doesn't kill itself immediately upon coming into existence, then it's pretty much over for the human race. So far we've argued against religious believers that we can't see god. Well, if strong AI does come into existence, there won't be any doubt, we'll all see one. That said, I don't really see any incentive for a VC to fund something like this. A VC is a creature that operates and makes sense within the confines of a certain environment. A VC, as an organization, doesn't necessarily have any incentive to change or revolutionize that environment. Nor does it have the authority to do it. Your best bet for funding would be rich people who can dispense with their own money without having to answer to anyone. Ugh.

“Ethics”. What we’re talking about here is no ethics. We don’t want the supermachine to be “ethical”. We ourselves are not “ethical”, especially not towards inferior things. No one (except in limited cases) expects us to be ethical towards inferior things, and no one (except in limited cases) actually enforces our being ethical towards inferior things. Our “ethicism” in this regard is fraught with inconsistency. On the one hand we have PETA and the pet police. On the other hand we slaughter pigs and cows and create chicken who can’t stand upright because we’ve made their breasts grow too big. We go to the supermarket and buy a bag of shrimp that have been slaughtered by the billion. We have no bad thoughts about that.

We operate on no “ethics”.

What you’re talking about here with “friendly AI” and “ethical AI” doesn’t involve any lofty principles either. The basic principle you wish to imbue your AI with is something like: “thou shalt not harm thy creator”. This is pure human selfishness, and our ability to imbue a vastly superior being with that concept may be very limited. After all, nature has imbued us with a desire to have sex in order to reproduce, and so we invented condoms. A superior machine can invent all kinds of workarounds to avoid the inconvenience of our flimsy, built-in “ethics”.

If someone succeeds in building strong AI, and the creature doesn’t kill itself immediately upon coming into existence, then it’s pretty much over for the human race. So far we’ve argued against religious believers that we can’t see god. Well, if strong AI does come into existence, there won’t be any doubt, we’ll all see one.

That said, I don’t really see any incentive for a VC to fund something like this. A VC is a creature that operates and makes sense within the confines of a certain environment. A VC, as an organization, doesn’t necessarily have any incentive to change or revolutionize that environment. Nor does it have the authority to do it.

Your best bet for funding would be rich people who can dispense with their own money without having to answer to anyone.

]]> By: Richard Hollerith http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-15425 Richard Hollerith Thu, 22 Nov 2007 03:58:21 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-15425 <i>A superior machine can invent all kinds of workarounds to avoid the inconvenience of our flimsy, built-in “ethics”</i> But a correctly built superior machine has no goals or preferences other than what the designers intentially give it, so it will not want to invent a workaround. I hope you will read more of Eliezer's writings on AI. For example, <a href="http://sl4.org/wiki/KnowabilityOfFAI" rel="nofollow"> Knowability of AI</a> A superior machine can invent all kinds of workarounds to avoid the inconvenience of our flimsy, built-in “ethics”

But a correctly built superior machine has no goals or preferences other than what the designers intentially give it, so it will not want to invent a workaround.

I hope you will read more of Eliezer’s writings on AI. For example, Knowability of AI

]]> By: Day of the Squishy Things | Isotropic http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-54355 Day of the Squishy Things | Isotropic Sun, 01 Jun 2008 22:29:40 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-54355 [...] A Premature Word on AI (at Overcoming Bias), Eliezer Yudkowsky referred to an earlier post of his, The Power of Intelligence (at The Singularity Institute), which starts out with: In our skulls we carry around 3 pounds of [...] […] A Premature Word on AI (at Overcoming Bias), Eliezer Yudkowsky referred to an earlier post of his, The Power of Intelligence (at The Singularity Institute), which starts out with: In our skulls we carry around 3 pounds of […]

]]> By: Sam http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-190041 Sam Tue, 09 Jun 2009 15:22:32 +0000 http://www.singinst.org/blog/2007/07/10/the-power-of-intelligence/#comment-190041 Umm...scarcity is an inherent property of the universe. As long as there are sentient beings there will be transactions; as long as there are transactions there will be money. It doesn't matter how much *any* technology improves the world--no technology is going to be omnipotent; no technology can create resources out of wishes. Money is necessary to the survival of intelligent life. Umm…scarcity is an inherent property of the universe. As long as there are sentient beings there will be transactions; as long as there are transactions there will be money. It doesn’t matter how much *any* technology improves the world–no technology is going to be omnipotent; no technology can create resources out of wishes. Money is necessary to the survival of intelligent life.

]]>