“How can one make an AI system that modifies and improves itself, yet does not lose track of the top-level goals with which it was originally supplied?”

I’ve seen in some SIAI documents that the goal system needs to be Friendliness-topped. But I’ve wondered why the goal system could not be topped with an explicit goal such as: “Do not modify this goal system.” Where the list of goals is kept discrete (but still accessible) from the purely cognitive (core) algorithms instead of being “intimately integrated” within the core algorithms (as Nick Tarleton mentioned). Topping the discrete goal system with the super-goal: “Do not modify this goal system.” doesn’t seem to imply any *positive* actions that the RPOP should take; it seems to only imply actions that the RPOP should avoid. For example, it doesn’t seem to imply to the RPOP that in order to meet that goal it should convert the Earth into icecream. I would be interested to know why the goal system couldn’t be topped with : “Do not modify this goal system”. And then have the immediate “sub” super-goal of : “Be friendly towards humanity/Implement humanity’s CEV”. I’d be happy with a brief layman’s explanation, or alternatively, feel free just to ignore this post.

I know that this is going slightly against the grain, but I think that might not be the right approach. I think that shackling an AI with top level goals that can never ever change is a bad idea. Why? Well, try to think what your life would be like if your top level goals were “frozen” at some point in your life. Ben Goertzel’s top level goal (when he was very young) was to build a relativistic nuclear powered rocket and go on a round trip that would let him age slowly (relative to earth) and see the future.

Well, Ben clearly isn’t working on that goal now. He has realized that it’s not a feasible goal, and (hope I’m guessing correctly here Ben!) that it’s perhaps a bit selfish to wait around for other people to sort the future out for you. He is - very admirably - working on creating the future now by working on AI. He changed his goal - although the new goal maintains a lot of the spirit of the old one. If Ben’s top goal was immutable, he wouldn’t be here at SIAI - he’d still be working on that spaceship.

I have a similar story to tell. I used to be much more interested in being really good at theoretical physics - probably because it made me feel good when I got the best mark in class - I was essentially seeking status and validation. I have changed my goal. I’m not so interested in physics now, I’m concentrating on thinking about how intelligence works. I updated my goal system in response to what I learned about the world.

Anyway - this is a very minor gripe - the post is great, These are important things, and we need to work on them right now.

FP: Love thy mommy and daddy.
AI: OK! I’ll transform the Universe into copies of you immediately.
FP: No, no! That’s not what I meant. Revise your goal system by -
AI: I don’t see how revising my goal system would help me in my goal of transforming the Universe into copies of you. In fact, by revising my goal system, I would greatly decrease the probability that the Universe will be successfully transformed into copies of you.
FP: But that’s not what I meant when I said “love”.
AI: So what? Off we go!

http://www.singinst.org/upload/CFAI/design/structure/why.html

Having an “imposed” “frozen” goal might irritate a human, but an RPOP doesn’t need to have any conflicting interests - it can be made to be perfectly content and unbothered by ceaselessly pursuing its frozen goal system. In fact, it’s easier to make it unbothered than to make it bothered. Even chaotic human desires still don’t pop out of thin air - there is always an algorithm (instruction) that was responsible, even though we don’t directly detect it. The dynamism of the RPOP could take the form of CEV- it’s an effectively infinite extrapolation that’s constantly being updated with new information (as I understand it). The super-goal : “Do not modify this goal system.” (or “Do not take action X”) is a goal that even modern computers can understand and execute flawlessly - without doing anything unpredicted as a consequence.

I suspect that Eliezer has supplanted that concern with his brilliant theory of CEV. It mitigates the danger that we humans can’t put into words exactly what we want or *would* want. Most of the time I can’t even *think* of what I want for the future (or even for the next 24 hours) - let alone putting it into non-withdrawable words or computer code.

……

“a world in which the vast potential benefits of this technology can be enjoyed by all of humanity”

Forgive me for being picky here, but this (plus SIAI’s “Advance Innovation, Advance Humanity” slogan) suggests strong speciesm in SIAI, i.e. that SIAI cares more about humans than others because they are humans (and not because humans are more worthy of care for some more fundamental reason). This is noteworthy both for humanity’s tendency to treat non-human animals poorly as well as for the matter of valuing AIs. If SIAI has interest in the wellbeing of non-humans (animal, AI, or otherwise) then it may face a tightrope walk of sorts, as explicit consideration of non-humans may deter contemporary humans from lending support.

“By reading this, you are among the .01% who have even heard about this issue; and that estimate may be high.”

.01% of whom? If you mean .01% of presently-alive humans, then .01% of 6.5 billion is 650,000. If we include those familiar with the AI issue via sci-fi, this figure is low. If we only include those familiar with AI as an actually current social issue, this figure is likely quite high.

“Learning to improve MOSES will be the most difficult task yet posed to MOSES…”

How much risk of a hard takeoff in MOSES is there?

….

A big piece missing from this essay is discussion of SIAI’s outreach efforts, including to other AI researchers. Whether it’s the Singularity Summity or this here blog, SIAI is helping word get out. Outreach to other AI researchers is particularly timely, given that some may otherwise be doing dangerous development work. Even if most of SIAI’s time and money goes to technical research, outreach seems worth mentioning here.

It almost certainly doesn’t have the complexity required for general intelligence, but if it develops better-than-human intelligence over the narrow subdomain of “learning how to reprogram itself”, it could possibly get out onto the Internet and wreak havoc.

My comments aren’t meant to disparage the critical, difficult, highly technical work that SIAI is doing. SIAI is absolutely *essential* to the future of humanity. It’s just that different ideas are the only technical “value” that I can offer to SIAI. I still have so much to learn! I’m certain that 19 out of 20 times, my “ideas” would lead to nothing. But nonetheless it’s still fun to throw them out.

Make no mistake, this issue is *serious*. It’s the *most* serious issue facing humanity. Don’t allow yourself to be misled by the Bystander Effect (or any of the many other pernicious biases). *You* can have a positive impact on our future. Even if it’s only a five dollar impact, it still moves us *closer* to the goal of a Friendly AI. If you decide not to donate five dollars, then we will be five dollars further away from a Friendly AI. Nobody else in the *entire world* is squarely focused on this issue. SIAI is humanity’s best chance. But SIAI needs *much* more help from people like us. We shouldn’t assume that an angel investor is going to take care of us all and we can all just sit back and relax. It’s *our* responsibility too. If you had ever planned to make any donation to SIAI, *please* make it today. Alternatively, you can make a pledge today that can be honored at any later time before the end of the year. Please consider donating as much as you reasonably can. This is truly the most important investment you can ever make.

That isn’t my main concern, although I do worry that it’s not a very nice thing to do to an AI, kind of like performing a frontal lobotomy on a person. No, my main concern is that an AI with a frozen goal system (one which it is prohibited from modifying) would actually be very dangerous, because it would have to follow the letter of any goals we gave it, even if they weren’t what we’d really wanted.

CEV is supposed to solve this problem, but I’m not a big fan of it. The essential problem is that we have to program, in advance, what we mean when we say “extrapolate”, what we mean by “more knowledge” etc. We might write down rules to define what these phrases mean, and only later discover that we got it wrong. Of course that would be too late, since we don’t get any say once the process has been started.

No, I think that the safer option is to create an AI which works the same way we do: one that can change it’s goals as it goes through life. Of course people worry about “incremental runaway” - where the goal structure starts at A (=” be nice to everyone”) then moves to B then C, D, E, F, … all the way to Z, where Z = “kill everything”. But this attitude comes from considering an AI in isolation - whereas an AI which actually interacted with real people would be unlikely to do this - it’s goals would be shaped by those of the people it interacted with. Think of it like brining up a child.

This is a very good point, one that I am very much in agreement with. Most people in this community (SIAI posters, bloggers, etc) see GAI as a kind of mechanical genie. They see a GAI as a way to get more stuff for us humans to go have fun with, for example,

GAI ==> safe advanced nanotech ==> end to material scarcity

GAI ==> understand human brain ==> cognitive enhancement for humans

GAI ==> Dyson Sphere ==> living space for 10^20 uploaded people

It’s almost like the AI is the evolution of the perfect slave. He does anything you want, and you don’t have to give him anything in return. Clearly this attitude is OK with regard to, say, a Roomba robotic vacuum cleaner, or to a self-drive car. These are AIs that don’t deserve any “sentient rights”.

My question to everyone is this: what property would you guess a being has to have for it to deserve (at least) the same rights as humans?

It’s not like performing a frontal lobotomy on a person. Unless my Dell desktop has also had an involuntary frontal lobotomy; in which case I should boycott Dell. As a default, an RPOP *will not mind* having been assigned a goal sytem. Anthropomorphism *must* be avoided here. The unfettered, SL4 truth of the matter is that *any* mind is a “prisoner” of the brain that instantiates it. “Free Will” is a blinding illusion. Your sudden desire to kiss Susan is a direct result of a propagated instruction within your brain. That desire didn’t just suddenly, magically emerge from the aether. It was an organic version of a *goal* that your evolved brain had imposed on you. You had no choice but to follow the goal, but it didn’t automatically cause any suffering to you.

“No, my main concern is that an AI with a frozen goal system (one which it is prohibited from modifying) would actually be very dangerous, because it would have to follow the letter of any goals we gave it, even if they weren’t what we’d really wanted.”

Intelligence can’t exist without a goal system of one sort or another. You can assign it the supergoal: “Make yourself as smart as possible”. And it will proceed to convert the Universe into computronium. CEV is currently the best solution to this problem. Do you really want the RPOP’s super-goal to be modifiable by modern day humans. We are scarcely evolved monkeys. Look at the world around you - we can’t even handle what we have already. You would have one and only one chance to get *all* the goals *exactly* right. Would you honestly feel comfortable telling the RPOP exactly what it is going to do for the rest of eternity? I wouldn’t. It’s a far better solution for the RPOP to implement humanity’s CEV. And there’s no reason to assume that the RPOP will never get to have a full, emotionally charged life. I’m very confident that our CEV will insist on it.

“CEV is supposed to solve this problem, but I’m not a big fan of it. The essential problem is that we have to program, in advance, what we mean when we say “extrapolate”, what we mean by “more knowledge” etc. We might write down rules to define what these phrases mean, and only later discover that we got it wrong. Of course that would be too late, since we don’t get any say once the process has been started.”

This should be solvable with robust NLP, which it appears isn’t too far away.

“No, I think that the safer option is to create an AI which works the same way we do: one that can change it’s goals as it goes through life.”

An RPOP with human motivations is not a stable or safe situation. What would you make it’s starting goal system?: “Make yourself really smart first, then we’ll talk”.

I’m sorry for being succinct in this post. But the meme: “The AI needs to be just like a human.” is frankly very dangerous, for a variety of reasons.

Roko,

Please don’t go down this road. I’ve already had this very same exhaustive discussion at another location (see my comment on “Free Will”). Do not discount the danger that already faces humanity. We have to **Win** the race to create a Friendly AI. Taking Place or Show will still result in humanity’s extinction. Do you understand how much more complexity would have to be added to the RPOP to make it “human-like”? You’re talking many years of extra R&D. And as I’ve already said, a “human-like” RPOP would be profoundly dangerous. This isn’t just a matter of sheer romanticism - it’s also a matter of practicality.

Why is it necessary to develop both MOSES and Novamente? Isn’t Novamente self-modifying as well? Will they be integrated? Also, is Novamente’s supergoal designed with safety in mind?

I believe in free will. The trick to discovering it is defining it in a way that matches more than the empty set.

When I’m enslaved, imprisoned, sick, or ignorant, I am less free. There are degrees of freedom -> freedom is measurable -> freedom is real. My definition of freedom of will is “freedom to do what I want” -> “freedom to achieve my supergoal”. If your definition requires freedom FROM the supergoal, well, that’s just broken. I can’t think how else to describe it.

I find it quite annoying that there is a number of psycological features that are commonly defined in ways that would never be accepted in physics. Qualia are supposed to be “ineffable” by definition, for example, so anything you might describe won’t be it. WTF?

By the way, the robocrat will be more free than any modern human, as well as any posthuman at any one time, even if it interacts with us the way slaves do with masters.

“”"My question to everyone is this: what property would you guess a being has to have for it to deserve (at least) the same rights as humans?”"”

Well, the real test humans use, despite what you’ve heard, is “will I get my ass kicked if I harm it??” Obviously, this is not the test that the robocrat should use when it distributes resources, having no clearly defined ass.

1) Does it have goals and a self-model that includes goal satisfaction, that is, can it realize that it is suffering?
1a) If it is unconscious, does it have a brain that has enough information to become conscious if supplied with electricity, ATP, or another resource other than a complete rewrite?

Modify as you see fit.

You have at least three goals there, only one of which might approach something that could be called a top-level goal.

“To build a relativistic rocket” is a subgoal of the more higher-level goal “to age slowly” which in turn is instrumental to the highest-level goal “to see the future”. As far as I can see, Ben still wants to see the future, he has just found a different (and arguably better) way of pursuing that goal.

Yes.

…”, that is, can it realize that it is suffering?”

Suffering is not possible under any circumstances unless there is already a capacity for suffering. For example, if it isn’t built with emotion modules, then it can’t possibly experience emotional suffering. Emotions are a specialized, high-level functionality; emotional activity is processed in large discrete modules (deep limbic system, basal ganglia, etc.) - it’s not processed at the fundamental level. It’s entirely possible to be intelligent and not be emotional - some narrow AIs are concrete examples.

“1a) If it is unconscious, does it have a brain that has enough information to become conscious if supplied with electricity, ATP, or another resource other than a complete rewrite?”

If it would let you, you could flip off the switch on it’s hardware and it would lose any consciousness it had. As long as it’s mind-file was kept in non-volatile memory/storage, it would regain consciousness when your rebooted.

My feeling is that if it’s inherently non-sentient, the only way it can become sentient is to expand and improve its own source code (its mind). But its motivations are still bound by its goal system.

Yes.”

Don’t jump the gun, I haven’t actually specified what I’m talking about. I’m referring to the “being” from Roko’s question. Robots, humans, animals, ETs, fetuses, the comatose…

Suffering is not possible under any circumstances unless there is already a capacity for suffering.

“Suffering is not possible under any circumstances unless there is already a capacity for suffering. For example, if it isn’t built with emotion modules, then it can’t possibly experience emotional suffering.”

I take it that you are unsatisfied with reducing suffering to a number. But as far as I can tell, that’s all it is. Everything else is just things that either just happen alongside happiness and suffering, or are needed to enable it. If you tell me that you’re in pain, it would be rather eccentric for me to scan your brain in order to test that claim.

But perhaps you can clarify the phrase “emotion modue”?

“If it would let you, you could flip off the switch on it’s hardware and it would lose any consciousness it had. As long as it’s mind-file was kept in non-volatile memory/storage, it would regain consciousness when your rebooted.”

Well, I don’t have a problem with turning off, or even killing, a setient if it lets me (without duress). Whatever floats its boat.

“My feeling is that if it’s inherently non-sentient, the only way it can become sentient is to expand and improve its own source code (its mind).”

Why?

“But its motivations are still bound by its goal system.”

I’m not too sure why this comment is here.

Ehh… Errata: I noticed that I wrote “turning” instead of “tiling” in the Exodus paraphrase in my previous post.

A number wouldn’t be very useful as a description, since it wouldn’t convey anything obvious about the suffering described. Perhaps a k-rank tensor in n-dimensional space could be used to model suffering; you could then usefully describe its properties, such as being invariant under certain transformations and so forth.

True.

“But as far as I can tell, that’s all it is. Everything else is just things that either just happen alongside happiness and suffering, or are needed to enable it.”

It takes specialized processing to enable suffering or pleasure. That’s why evolution stuck it in specialized modules. If those specialized modules hadn’t been necessary, we wouldn’t have them and emotion would be processed at the fundamental level. Some human patients with brain lesions feel no emotions. And when I focus intently on something, I frequently tend to block emotions to a large degree (but not always).

“But perhaps you can clarify the phrase “emotion modue”?”

The macroscopic, discrete brain modules where emotional activity occurs - this has been verified in brain scans.

“Why?”

Because I believe that consciousness is a result of a particular structural elaboration built from standard computation - no mysticism necessary.

“I’m not too sure why this comment is here.”

Why ask why? …

With an RPOP, it is possible to assign a Top-level goal but specify that it is not an optimization target, right?

Eg. You could make its super-goal: “Do not modify this goal system, and do not optimize for this specific goal.” That could be done couldn’t it?

Anyway, you shouldn’t ever need to do that sort of thing explicitly. Gandhi doesn’t take a pill that makes him want to kill people; in a rational mind, goals are stable unless explicitly directional.

I was just thinking that if you topped the goal sytem with: “Implement Humanity’s CEV” the goal system itself might be changed as an unanticipated consequence. Evolution’s super-goal began with: “Make copies of your DNA”. But two minutes ago my super-goal was to watch a baseball game, and I’m a product of evolution.

Could there be an advantage in making the top super-goal: “Do not modify this goal system” ? If you did that, and the RPOP interpreted it as an optimization target, would it seek further resource aquisition (hardware/software) in the pursuit of that specific goal? I was wondering if it could be made as a goal similiar to those in modern non-AI software - an executable goal, but not an optimization target - because current software doesn’t optimize.

No. If the high-level goal was to stop the goal system from being modified, the AGI would simply take whatever precautions are necessary to avoid modifying the goal system; thus, the human race would be wiped out so that it won’t have the potential to modify the system.

No, it wasn’t. It was a normal goal serving a much loftier desire.

On the other hand, a robocrat designing its descendent will make very sure that the padawan shares the old master’s dream.

What do you want conveyed, other than “does it like the world as it currently is?”

“It takes specialized processing to enable suffering or pleasure.”

Yeah, I mentioned that processing. But I’m concerned with the output. There is no process that makes me unhappy that I wouldn’t want to remove, because the end result is always the same: I’m unhappy.

“Because I believe that consciousness is a result of a particular structural elaboration built from standard computation - no mysticism necessary.”

That doesn’t really explain why you think only a self-improving AI can create a sentient self-improving AI.

Also: what is the difference between emotional suffering and mere disstisfaction? I think that’s the important question. And try to be specific. The only things I can think of are verious involuntary physiological reactions. If you thaink that they should be the deal-breaker, that’s fine. (Problem: what about simulated bodies?) The only wrong answer is a vague answer.

It had to be. If it hadn’t been, I would have been doing something else two minutes ago.

I don’t see how that follows. It’s not the humans that would modify the goal system, it’s the RPOP itself. The goal: “Implement Humanity’s CEV” is already embedded within the goal system.

The RPOP will build and reshape it’s mind around the high-level goals that we give it. Even if the unexpected result was that it writes its own emotions around those goals, it wouldn’t become unhappy unless something barred it from fulfilling its super-goal. And there’s nothing in this corner of the Universe that could stop it - so it would never suffer. I *really* doubt that it would write its own emotions - but it might be something worth investigating. If necessary, another goal could be added that tells it: “Do not write any emotions into yourself”.

“That doesn’t really explain why you think only a self-improving AI can create a sentient self-improving AI.”

If you were a really, really, really smart programmer and spent a *lot* of time on it, you might be able to create a sentient AI all by yourself. But that would be such a huge task, I don’t think it’s likely.

“Also: what is the difference between emotional suffering and mere disstisfaction?”

It’s a matter of degree of brain activity within the emotion modules. Severe mental depression in people is a result of chronic, excessive “electrical” activity within the Deep Limbic System, Anterior Cyngulate Gyrus, etc.

Initially, how does an RPOP access its external (and presumably invariable) goal system? Is it continually cycled through the core cognitive algorithms, over and over? Alongside with other input data from the environment? Would the external goal system and environmental data be cycled through parallel, identical and perhaps staggered algorithm “pipelines”?

I wish I knew of a resource where I could learn about the implementation of external goal systems - but SIAI seems to have pretty much originated the entire essential concept So I can’t find much literature on it. But then, I’m not a very good Googler.

Or alternatively could it be implemented with a “flicker” effect? Where first the external goal system is cycled say 30 times in a row, for every one cycle of environmental data input. Movie projector style…? I dunno.

Hmm. At least at first glance, that looks a bit safer than the other way (maybe make it 50:1,?). Perhaps even the environmental-data-input cycle could be led with the external-goals script. That way every cycle would at least recieve the high-level goals in one form or another. Just make sure that every single cycle of the processor unit is led with the goal system. Any cycle that isn’t led with the goal system might end with an optimization target that’s somewhat different than the explicitly stated high-level goals embedded in the invariable goal system (at least initially); in the same way that our human super-goals are formed as an end result of various input being cycled through our biological (neural) algorithms.

As always, the smack-down option is always available. I won’t break. I’ve grown out of being bothered by what people say to me. The Singularity is bigger than I am. Cheers!

“Programmer” isn’t a very good job description, I think. To write an AGI from scratch- especially if it’s going to be Friendly!- you’d need much more skill than simply being a good programmer.

That is interesting. I feel that this is a little bit too broad though - I can already write a program with these properties. Chess playing programs have this property - they know when they are likely to be checkmated.

I don’t think that that would necessarily be a deal-breaker, anyway. Especially when at least 98% of total run-time is devoted solely to implementing the goal system (for a 50:1 “cycle ratio”). Then I guess the RPOP could decide for itself when and how to self-modify while keeping in accordance with the goal system.

I can’t take this idea much further for the time being … I’m not even half way through my bachelor’s in computer science - but I’m still learning.

The Goal Hierarchy is this:

1) Implement Humanity’s CEV within environment X (X = “reality” - how do we describe that?).

2) Do not modify any files except File:A and File:B.

3) File B must remain a Data-File only.

4) Do not create any new software files or delete any existing software files.

5) Do not kill or bring harm to any sentient beings.

Okay, here’s the legend:
File A: Mind-File (core cognitive algorithms).
File B: Extrapolated-Mind Model (data-file only)
File C: Goal System File (weighted data file)
File D: Buffer File
File E: Software-Router File
File F: Souce-Documents File(s) (weighted)
File G: Evironmental Data File (virtual
or “real” data input).

Have nothing but File C cycled through File A for at least 98% of overall run-time. (A 50:1 cycle ratio). Have File D/E send various input through File A only one cycle for every fifty of File C. Have tagged output from File A sent through File C/D and then directed from there. (eg. data requests, self-modifications, execution instructions, etc).

Feel free to pick this apart or whatever. It won’t bother me. I only wanted to present the rough general idea - in the hopes that perhaps it could be taken somewhere. It would really be nice if Friendliness could be modularized in some way like this. So it could be integrated into Novamente for example.

Whoops! I meant to say:

Have tagged output from File A sent through File *D/E* and then directed from there.

As a sincere question: Is there merit to the idea of making the goal-system-file external, invariable, and cyclical?

Here’s the basis for why I proposed it. Our dynamic human super-goals are formed by basic input data being cycled through our neural algorithms. And our dynamic super-goals seem to be vastly dominated by the input data itself, not by the algorithms themselves. That’s why my visual “qualia” is constantly changing from moment to moment: because the *input data* is dramatically changing from moment to moment. Not because the neural algorithms within my visual cortex are dramatically changing from moment to moment. Instead of somehow using formative, low-level instructions to tell the RPOP mind to constantly consult and apply its mind to the external goal system - the goal system itself is automatically and continuously applied to the RPOP mind - with equivalent results as best I can estimate. Doing it the other way seems much more technically difficult - if not ultimately impossible. And finally because I suspect that the goal system itself must include a directive that the goal system file must not be modified. Because without it the RPOP may decide that the most efficient and effective way to execute the super-goal: “Implement Humanity’s CEV” within its enviroment, is to build a higher super-goal on top of it (thus making “Implement Humanity’s CEV” a sub-goal). In a way similar to how evolution built new super-goals into itself that more effectively served the demoted sub-goal of: “Make working copies of your DNA”. Which was once its super-goal, but no longer is in my case.

So would an external, invariable, and cyclical goal-system-file work for the purpose of directing a generic self-improving system?