Transhumanism as Simplified Humanism
June 16th, 2007 –
Frank Sulloway once said: “Ninety-nine per cent of what Darwinian theory says about human behavior is so obviously true that we don’t give Darwin credit for it. Ironically, psychoanalysis has it over Darwinism precisely because its predictions are so outlandish and its explanations are so counterintuitive that we think, Is that really true? How radical! Freud’s ideas are so intriguing that people are willing to pay for them, while one of the great disadvantages of Darwinism is that we feel we know it already, because, in a sense, we do.”
Suppose you find an unconscious six-year-old girl lying on the train tracks of an active railroad. What, morally speaking, ought you to do in this situation? Would it be better to leave her there to get run over, or to try to save her? How about if a 45-year-old man has a debilitating but nonfatal illness that will severely reduce his quality of life - is it better to cure him, or not cure him?
Oh, and by the way: This is not a trick question.
I answer that I would save them if I had the power to do so - both the six-year-old on the train tracks, and the sick 45-year-old. The obvious answer isn’t always the best choice, but sometimes it is.
I won’t be lauded as a brilliant ethicist for my judgments in these two ethical dilemmas. My answers are not surprising enough that people would pay me for them. If you go around proclaiming “What does two plus two equal? Four!” you will not gain a reputation as a deep thinker. But it is still the correct answer.
If a young child falls on the train tracks, it is good to save them, and if a 45-year-old suffers from a debilitating disease, it is good to cure them. If you have a logical turn of mind, you are bound to ask whether this is a special case of a general ethical principle which says “Life is good, death is bad; health is good, sickness is bad.” If so - and here we enter into controversial territory - we can follow this general principle to a surprising new conclusion: If a 95-year-old is threatened by death from old age, it would be good to drag them from those train tracks, if possible. And if a 120-year-old is starting to feel slightly sickly, it would be good to restore them to full vigor, if possible. With current technology it is not possible. But if the technology became available in some future year - given sufficiently advanced medical nanotechnology, or such other contrivances as future minds may devise - would you judge it a good thing, to save that life, and stay that debility?
The important thing to remember, which I think all too many people forget, is that it is not a trick question.
Transhumanism is simpler - requires fewer bits to specify - because it has no special cases. If you believe professional bioethicists (people who get paid to explain ethical judgments) then the rule “Life is good, death is bad; health is good, sickness is bad” holds only until some critical age, and then flips polarity. Why should it flip? Why not just keep on with life-is-good? It would seem that it is good to save a six-year-old girl, but bad to extend the life and health of a 150-year-old. Then at what exact age does the term in the utility function go from positive to negative? Why?
As far as a transhumanist is concerned, if you see someone in danger of dying, you should save them; if you can improve someone’s health, you should. There, you’re done. No special cases. You don’t have to ask anyone’s age.
You also don’t ask whether the remedy will involve only “primitive” technologies (like a stretcher to lift the six-year-old off the railroad tracks); or technologies invented less than a hundred years ago (like penicillin) which nonetheless seem ordinary because they were around when you were a kid; or technologies that seem scary and sexy and futuristic (like gene therapy) because they were invented after you turned 18; or technologies that seem absurd and implausible and sacrilegious (like nanotech) because they haven’t been invented yet. Your ethical dilemma report form doesn’t have a line where you write down the invention year of the technology. Can you save lives? Yes? Okay, go ahead. There, you’re done.
Suppose a boy of 9 years, who has tested at IQ 120 on the Wechsler-Bellvue, is threatened by a lead-heavy environment or a brain disease which will, if unchecked, gradually reduce his IQ to 110. I reply that it is a good thing to save him from this threat. If you have a logical turn of mind, you are bound to ask whether this is a special case of a general ethical principle saying that intelligence is precious. Now the boy’s sister, as it happens, currently has an IQ of 110. If the technology were available to gradually raise her IQ to 120, without negative side effects, would you judge it good to do so?
Well, of course. Why not? It’s not a trick question. Either it’s better to have an IQ of 110 than 120, in which case we should strive to decrease IQs of 120 to 110. Or it’s better to have an IQ of 120 than 110, in which case we should raise the sister’s IQ if possible. As far as I can see, the obvious answer is the correct one.
But - you ask - where does it end? It may seem well and good to talk about extending life and health out to 150 years - but what about 200 years, or 300 years, or 500 years, or more? What about when - in the course of properly integrating all these new life experiences and expanding one’s mind accordingly over time - the equivalent of IQ must go to 140, or 180, or beyond human ranges?
Where does it end? It doesn’t. Why should it? Life is good, health is good, beauty and happiness and fun and laughter and challenge and learning are good. This does not change for arbitrarily large amounts of life and beauty. If there were an upper bound, it would be a special case, and that would be inelegant.
Ultimate physical limits may or may not permit a lifespan of at least length X for some X - just as the medical technology of a particular century may or may not permit it. But physical limitations are questions of simple fact, to be settled strictly by experiment. Transhumanism, as a moral philosophy, deals only with the question of whether a healthy lifespan of length X is desirable if it is physically possible. Transhumanism answers yes for all X. Because, you see, it’s not a trick question.
So that is “transhumanism” - loving life without special exceptions and without upper bound.
Can transhumanism really be that simple? Doesn’t that make the philosophy trivial, if it has no extra ingredients, just common sense? Yes, in the same way that the scientific method is nothing but common sense.
Then why have a complicated special name like “transhumanism”? For the same reason that “scientific method” or “secular humanism” have complicated special names. If you take common sense and rigorously apply it, through multiple inferential steps, to areas outside everyday experience, successfully avoiding many possible distractions and tempting mistakes along the way, then it often ends up as a minority position and people give it a special name.
But a moral philosophy should not have special ingredients. The purpose of a moral philosophy is not to look delightfully strange and counterintuitive, or to provide employment to bioethicists. The purpose is to guide our choices toward life, health, beauty, happiness, fun, laughter, challenge, and learning. If the judgments are simple, that is no black mark against them - morality doesn’t always have to be complicated.
There is nothing in transhumanism but the same common sense that underlies standard humanism, rigorously applied to cases outside our modern-day experience. A million-year lifespan? If it’s possible, why not? The prospect may seem very foreign and strange, relative to our current everyday experience. It may create a sensation of future shock. And yet - is life a bad thing?
Could the moral question really be just that simple?
Yes.
[…] post by Eliezer Yudkowsky Share and Enjoy: These icons link to social bookmarking sites where readers can share and […]
I’m noting that this discussion of morality, although logical and self-consistent, doesn’t take anyone’s volition into account. Have you changed your views on what an AI’s moral system should be, or are you just trying to keep it simple?
Tom, I’m talking about transhumanism as a moral philosophy, not AI.
[…] makes complex moral decisions much simpler: saving a life is always preferable to not saving a life.read more | digg story Popularity: 1% [?] Share and Enjoy: These icons link to social bookmarking sites […]
I think you’re neglecting some things and that it’s really NOT so simple — falling into the same trap as “those people” you’re criticizing that dislike transhumanism because it’s too trivial. You’re giving it value because it’s trivial, and things like this should get value if they’re RIGHT. In particular, you’re assuming linear utility for life, and while that’s elegant, it’s not necessarily true. Perhaps people might get tired of living at some point, after living a time L — however high that may be — and extending their life becomes torture rather than happiness. Perhaps there are other adverse effects to immortality or living disease-free, of a different sort. Assuming that we can overcome such adversities with technology or some such, though, your conclusion holds. But even though it’s not a trick question, when there are humans involved, it’s always more complicated than we hope.
The problem here Mauro, is that your objection seems to assume that if there were some non-linear utility for life, it would be consistant person to person.
It really isn’t. Some people already want to kill themselves in their teens, while others want to keep living in their hundreds.
This isn’t a problem with life extension you have, but essentially one with suicide and/or assisted suicide.
Usually issues are worded positive liky ‘live as long as you wish’ rather than the opposite: ‘death as optional’.
Many people are against that option. Suicide and assisted suicide being illegal it is even written into law.
It is a very odd thing, to assume that wanting death is perfectly fine at one age while perfectly horrible at another. In fact, it even goes beyond that, there’s not that sort of middle ground. It essentially FORCES people to live if they are young, and FORCES people to die if they are old. There’s simply no personal freedom at all. I think we really need to take an absolute stance on this issue that does not reak of illogical ageism.
Either being able to choose to kill yourself is okay, no matter what your age (though I’d have a waiting period on it similar to buying guns, and preliminary psychological assessment), or wanting to die is bad at ANY age. Honestly, the silliness of things like hospices where you can go refuse medication for your disease, while they shove people trying to shoot themselves into psych wards, seems ridiculous to me because they’re essentially the same thing.
I understand why the division exists. People believe suicide when your life is good is bad (in the prime of your youth, or health) while suicide when you are suffering (old, have a disease, being tortured/raped) is a more acceptable thing.
I don’t think it’s acceptable, just understandable, because we’re able to contemplate that sort of desperation, we empathize with it to a great deal that we believe to think it is logical and justifiable. It isn’t though, it’s insanity, and should be treated as such so long as we are against it for any other person (as we certainly are with protecting children from killing themselves).
Suffering and happiness happen on more than just material levels. There’s much more to quality of life than how old you are, how your health is for your age, and such things. People suffer psychological trauma that the best physical pleasures cannot cure, or enjoy aspects of life that it outweighs the negatives of a currently bleak physical existance.
When we understand that, seeing how people divide individuals like this into acceptable and unacceptable death wishers seems utterly archaic. If people still insist to judge suitability they should at least attempt to do so on more specific criteria, and hold off from judging at all until such things are adequately defined and agreed upon.
Mauro, whether life has linear utility is a trickier question than whether it has positive utility. And your point about conditions under which life is not worth living, might apply equally to, say, a debt-ridden peasant a thousand years before cryonics, contemplating suicide or considering whether to bring a child into the world. This debate would turn around *what* makes a life worth living - *why* is life better than death - and it might reveal possible life circumstances under which, regardless of what age you are, and regardless of the technologies involved, you would prefer to die. (Or not - some transhumanists (e.g. BJ Klein) profess that life is strictly preferable to death regardless of life circumstances.)
But even though we’re zooming in on the moral microscope and fine details are appearing and everything looks more complicated, transhumanism is still relatively simpler because it still strips out ages, technology invention dates, and other unexplained inversions of utility.
“Tom, I’m talking about transhumanism as a moral philosophy, not AI.”
Ah, okay. But isn’t the first superintelligent AI supposed to implement our collective extrapolated moral philosophy?
Tom, it seems to me as of June 16th 2007 that this would be a good way to deal with metamoral uncertainty. But my post above was on morality simple - what is good or bad to the best of my current knowledge.
Not every post I write is about the same topic. Discussion of CEV would be appropriate for Michael Anissimov’s post on same.
Suppose a boy of 9 years, who has tested at IQ 230 on the Wechsler-Bellvue, is threatened by a lead-heavy environment or a brain disease which will, if unchecked, gradually reduce his IQ to 100. I reply that it is a good thing to save him from this threat. If you have a logical turn of mind, you are bound to ask whether this is a special case of a general ethical principle saying that intelligence is precious. Now the boy’s sister, as it happens, currently has an IQ of 100. If the technology were available to gradually raise her IQ to 230, without negative side effects, would you judge it good to do so?
It seems to me that there is an implicit claim here, that applied ethics as practiced today is biased toward excessively complex reasoning. The main evidence offered is that many favored Freud over Darwin for psychology. To be convinced that this is a general bias, it would be nice to see more examples, and to see more historical support for this account of Freud versus Marx.
Er, make that Darwin, not Marx.
Hi Eliezer, nice post. Learning about transhumanism and transhumanist culture has been a most pleasant surprise since I began studying ethics about a year ago. I’m also quite comforted in knowing that SIAI has attracted many people who place a high priority on thinking about ethics. Let me see if I can add some helpful thoughts to what you’ve written. My apologies for being a little long-winded here.
1. I see ethics as a mix of logic and intuition. Tension often exists between the two, as your post clearly expresses. (You seem to prefer more logic than those you see opposing transhumanism.) Another good example of the tension between the two can be found in this debate between Richard Posner and Peter Singer.
http://www.slate.com/id/110101/entry/110129/
2. Trying to base ethics exclusively on logic is like asking, Which are the correct ethical views? Others disagree, but I suspect no correct views exist, i.e. I doubt the “is-ought problem” can be solved. Without adding intuition, we get nihilism. So how much intuition do we add?
3. Perhaps even more so than you, I’m extremely minimalist with my use of intuition. All I say is “if it feels good then it is good”. (This is “hedonism”. Note that “feeling good” here is not just crude pleasure but the richer, more sophisticated enjoyment we’re capable of, and of course also even richer enjoyments we’re not capable of.) So I wouldn’t even say “learning is good” unless it helps us enjoy (which of course it does).
4. So, I agree with Peter Singer that species membership should have no moral significance, and I agree with you that existing human physiology should not either, as long as it’s clear in both cases that this “should” that I use is based on my minimalist hedonist intuition. However, I do pay considerable respect to those who disagree with me, especially when they haven’t spent much time reflecting on these matters.
5. Is failing to save someone’s life morally equivalent to killing someone? In both cases, the result is one fewer person alive. The (possibly crucial) difference between the two is in the complex societal consequences. But what if the societal consequences weren’t there? This is a good instance of a situation where logic and intuition conflict. Siding with logic is to say yes, morally equivalent. However, doing so has gotten people like Peter Singer in a lot of controversy. I avoid these matters because they distract attention from more important things.
6. Is neglecting to procreate a new baby morally equivalent to killing someone? Again (assuming the societal consequences are equivalent even though they’re not), siding with logic is to say yes, morally equivalent. This gets to the heart of the ethics of immortality: Should we try to cure aging or should we just achieve immortality through our descendants? Logic would say it doesn’t matter (again ignoring certain details like the cost of raising children). You may enjoy this
http://felicifia.com/showDiary.do?diaryId=98
7. If our goal is for there to be the most of some good, whether it’s utility/happiness/wellbeing/etc or even learning, what should we do? Logic has lead me to conclude that the most important thing is existential risk reduction, given the very long (possibly infinite) period of time we have ahead of us to enjoy. See
http://felicifia.com/showDiary.do?diaryId=84
http://felicifia.com/showDiary.do?diaryId=103
8. So while I don’t strictly oppose the transhumanist goals of immortality and intelligence amplification, I don’t strictly support them either. I ask, Do these goals effective ways of reducing existential risk? Carl Shulman presented an interesting argument connecting immortality to existential risk reduction here
http://felicifia.com/showComment.do?commentId=262
As for intelligence amplification, as long as this intelligence would be used one way or another for risk reduction, then I’m all for it.
Your thoughts?
Eliezer: So, you’re distinguishing the volition-based morality used by the original CEV as a means to the end of a human-perfect morality, rather than as the ultimate moral system in and of itself? We’re getting the CEV to figure out and implement our moral system, because we’re too stupid to do it ourselves?
“Without adding intuition, we get nihilism. So how much intuition do we add?”
Presumably all of it, and then we cancel out the parts that contradict each other. That’s how CEV works, I think.
“All I say is “if it feels good then it is good”.”
This principle seems to get us a perfect morality in universes with zero or only one optimization process. However, in universes with multiple optimization processes (such as ours), you run into the problem of when one processes’ “good” contradicts another processes’ “good”.
“Is failing to save someone’s life morally equivalent to killing someone? In both cases, the result is one fewer person alive.”
You are correct that the *results* are the same (barring grieving relatives and the like). However, the *prior conditions* aren’t the same, and unless you can calculate a mathematical model which both fits our moral judgment in these kinds of problems and cancels out any prior condition terms, I wouldn’t be so quick to discard prior conditions as having any effect on the morality calculation. For instance, a universe that’s empty because we blew it up strikes me as much worse morally than a universe that’s empty because G was 27% too large.
“Should we try to cure aging or should we just achieve immortality through our descendants?”
“Immortality through descendants” is vacuous; the only thing passed on is your genes, and by the thousandth generation they’re so dilute you probably couldn’t even recognize them. It’s like claiming that the Roman Empire was immortal because the atoms in Caesar’s last breath are still with us today.
“I ask, Do these goals effective ways of reducing existential risk?”
Intelligence amplification is absolutely essential to reducing existential risk. Immortality is just huge benefit #61,102 to intelligence amplification.
Tom, if you have a comment about CEV, post it to Anissimov’s thread on CEV, not this thread on transhumanism. I will note that I have various forms of respect for individuality - for people solving their own problems, making their own choices, fulfilling their own goals and selecting new ones. All of this is entirely and absolutely orthogonal to CEV, which is a metamoral solution to a metamoral problem. An extrapolated volition, in particular, is of null moral significance - no person feels it as an urge or enjoys fulfilling it, so it is of no intrinsic utility - however useful it may be for computing metamoral questions. I think this says most of what there is to say about the relevance of CEV to non-meta transhumanist morality. Further discussion should go to CEV threads, not this one!
Seth, I think that if anything you are expressing a far stronger drive toward logical simplicity than I do. I don’t see anything paradoxical, or even morally erroneous, about a hero sacrificing their life so that ten others may live. Since the hero does not get to see the results of their action, and since it is not neurologically realistic to suppose the hero can momentarily experience more hedonic reward than they could expect over their remaining lifespan, I must believe that people can have interests beyond themselves. Indeed, I don’t see anything the slightest bit odd about this - if you are concerned for your own happiness, that is only one special case of our human ability to steer the future in all sorts of directions.
“If it feels good than it is good” is problematic for several reasons, some of which I’m sure you’ve heard before. Human reasoning suffers from many known flaws (cognitive biases) which distort our present subjective expectation of future experiences. By “feels good -> is good” do you mean that the decision which feels best is the right one, or that we should defy our intuitions in order to best cold-bloodedly maximize our future integral of hedonic experience over time? These are incompatible imperatives. If you hold that we should cold-bloodedly maximize our hedonic integral over time, ignoring if necessary any feelings we have about what are good or bad decisions, then how do you justify refusing to inject yourself full of crack - it feels like the wrong decision, but wouldn’t it maximize your integral?
In any case, I simply don’t see why I should be prohibited from having interests outside myself. It sounds like arbitrarily scooping out a large portion of my utility function over possible futures. If anything similar happened on a planetary scale - intelligence arbitrarily losing an important part of its utility function - it would be a realized existential risk because it would destroy a significant fraction of the overwhelmingly huge expected value of posterity. As a builder of AIs, this is not a minor detail unto me.
Robin, I do indeed suspect that modern ethics, as practiced in academia, are biased to be fun to argue - that the end user is no longer the individual faced with a real-life moral dilemma, but rather the academic trying to impress an audience of other academics. In particular, I think that academic ethics are biased to be minimally counterintuitive, sounding just strange enough to be interesting and worthy of argument, but not so strange as to be politically incorrect relative to other philosophers. I’m not sure I have the time myself to verify this by detailed historical studies - this is just a blog post, after all - but I suspect you can think of more examples than I can!
Panu, I would indeed - if I had to make a single choice on the basis of present information - say that it is better to gradually raise the one’s IQ to 230, than to not do so. Yes, we have less information about whether IQ 230 is a fun place to be, than we do about IQ 140 - but if I had to make a guess, I’d guess yes.
Tom:
“Presumably all of it (intuition), and then we cancel out the parts that contradict each other.”
I would be *very* concerned that this would not leave us with anything given the diversity of intuitions.
“multiple optimization processes”
You’re describing rational egoism, not utilitarianism. Utilitarianism is agent-neutral: it recommends doing whatever is best for the total good, not the agent’s own good. Since all have the same goal, there are no conflicts. In the real world where not everyone is a utilitarian, an individual utilitarian still does not have conflicts- she just has a different strategy.
“For instance, a universe that’s empty because we blew it up strikes me as much worse morally than a universe that’s empty because G was 27% too large.”
Assuming that the first universe’s inhabitants enjoyed themselves before blowing their universe up, I’d say the first universe is better. It didn’t last forever, but at least it got something out of the deal.
“the only thing passed on is your genes”
The crucial things passed on are the ability to enjoy life and the ability to help others do so. Genetic similarity is not directly relevant here.
……
Eliezer:
“I don’t see anything paradoxical, or even morally erroneous, about a hero sacrificing their life so that ten others may live.”
You also appear to be confusing rational egoism with utilitarianism.
Re cognitive bias: Given uncertainty, utilitarianism typically recommends maximizing expected utility using subjective estimation. Understanding cognitive bias certainly is important here. In that spirit, I am also glad that the bias community is closely connected to both the SIAI community and the utilitarianism community. (Hi Robin!)
“As a builder of AIs, this is not a minor detail unto me.”
This is off-topic, but the AI problem gives me more pause with my own ethical thinking than anything else, since it may work so well.
“modern ethics, as practiced in academia, are biased to be fun to argue”
The repugnant conclusion may be a good case. Much discussion over what was thought to be strictly a theoretical dilemma. (I may have found a real-world instance in human-insect interaction.)
Blogs may also be biased towards fun to argue, and political discourse, and many other things. If we all agree, what’s left to discuss?
“Further discussion should go to CEV threads, not this one!”
Although I understand you are trying to keep the thread on the original topic, I do ask why you are so against the thread veering off into discussing other things. This is not a social convention I am familiar with.
“I would be *very* concerned that this would not leave us with anything given the diversity of intuitions.”
Comparing one human to another human, we all have pretty much the same brain architecture, and so we tend to make similar moral judgments across a wide variety of situations. For more information on this, I recommend any good introduction to evolutionary psychology. Within a single human brain, I don’t know enough about evolutionary psychology to guess how many contradictory instincts there are, but I suspect it cannot be that many as ethical dilemmas are considered an atypical situation.
“Utilitarianism is agent-neutral: it recommends doing whatever is best for the total good, not the agent’s own good.”
How would you define total good except as some kind of mathematical operation over individual goods? Note that in the universes with zero or one optimization process, the total good and the individual good are identical, making it simple.
“The crucial things passed on are the ability to enjoy life and the ability to help others do so.”
That’s not what we mean by immortality. If you just want people to be able to enjoy life, you could go to war to help preserve quality of life and get shot at age 20, but I hope you would agree that this is not becoming immortal.
Nice post, Eliezer. All I can say is that we will need an extremely large planet to fit all of these “Very Old” Humanoids and their offspring.
[…] article however, Transhumanism as simplified humanism, I think does a good job of explaining things: If you believe professional bioethicists (people who […]
“How would you define total good except as some kind of mathematical operation over individual goods?”
Direct sum of individual goods. This does involve asking agents to sacrifice their own wellbeing for the total good, but some sacrifice is inevitable whenever two or more agents’ interests conflict.
This is the basics of utilitarianism mathematics, aka felicific calculus
http://felicifia.blogspot.com/2006/09/utility-curves.html
I wrote it almost a year ago for a pretty intro audience so it may seem overly simplistic but it should help. It should be well enough written but I haven’t reviewed it recently.
“we tend to make similar moral judgments across a wide variety of situations”
We tend to, but we don’t always. Where do we draw the line between outlier and normal? Also, intuitions regarding violence and war concern me, as do intuitions concerning “proximalism”. (There may be a better word than “proximalism”, but I mean to refer to our tendency to value those who are in some sense close to us more than those who are in that sense far from us, whether it’s by race, nationality, region, time, family, or something else.) …My biggest exposure to psych literature is in the future valuation studies (eg Loewenstein) where our intuition seems pretty all over the map.
For point #1: A direct sum is a kind of mathematical operation, so your reply doesn’t actually contradict anything I said.
For point #2: Even if our intuition is wildly off-base in most situations (there is a great deal of literature documenting where it is off-base and why), that still doesn’t mean our instincts routinely contradict each other.
See Should We Maximize National Happiness? if you haven’t already, for some of the objections to treating the sum of happiness as an object of maximization by an external process. They’re writing about government, and while some of their objections are not applicable to SI/FAI, some of them are - in particular, those having to do with people trying to steer their own lives, and those having to do with quantities such as truth and justice having components in the utility function alongside pleasure.
Suppose I say: “Yes, I want to be happy, but I also want to know what’s true. I express my preference to know an unpleasant truth over believing a pleasant lie - this preference holds in abstract even if I never find out.”
Why is this not a worthy component of my utility function, or a utility function being optimized on my behalf?
Tom:
1. When you said “However, in universes with multiple optimization processes (such as ours), you run into the problem of when one processes’ “good” contradicts another processes’ “good”.”, I thought you meant to point out that if all agents try to maximize their own utility then conflicts occur (unless there are only zero or one agents). Did I misinterpret you?
2. I agree, our instincts often line up. But how do you determine how often is often enough?
…..
Eliezer:
*Thank you* for that paper. It’s an excellent one, and very timely. Do send more you think would be good to look at.
I’m reminded of Toby Ord’s thesis,
http://www.amirrorclear.net/academic/papers/decision-procedures.pdf
I agree with Toby’s basic premise, “I should follow the decision procedure that will lead to the best outcome.” As both papers point out, maximizing any sort of utility function in practice is (for humans at least) both an art and a science.
“Yes, I want to be happy, but I also want to know what’s true.”
Interestingly, Michael Anissimov and I had a very similar exchange:
http://felicifia.com/showComment.do?commentId=261
I’ll ask the same question of you: Do you value knowledge in someone’s mind or just knowledge? For example, does a book count as knowledge if no one’s read it? It sounds like you (as with Michael) mean knowledge in someone’s mind, but I’ll let you speak for yourself.
Regarding justice, I’m reminded of the Greene and Baron study on the diminishing marginal utility in utility
http://www.wjh.harvard.edu/~jgreene/GreeneWJH/Greene-Baron-JBDM-01.pdf
See also Felicifia: “Intuitions and utilitarianism”
http://felicifia.com/showDiary.do?diaryId=28
What both the paper you linked to and your own words seem to be getting at is the point that our words “happiness” and “pleasure” don’t encapsulate all of that which seems good to us. I’ll restate here from 3 in my first comment above:
Note that “feeling good” here is not just crude pleasure but the richer, more sophisticated enjoyment we’re capable of, and of course also even richer enjoyments we’re not capable of.
I perhaps should back away from the word “hedonism”. I interpret it very broadly, I believe to include all that you describe, but perhaps the word’s connotations are a problem.
Here’s another question for you: Is ignorance bliss? For example, suppose I prefer that you don’t eat meat. Suppose you eat meat, but I never find out, not even in the slightest hint. Thus, my brain is never affected by the act of your eating meat. In your view, does this act still count against my utility function?
(In my view, it does not: ignorance is bliss. However, I find thinking that ignorance is bliss to be somewhat disconcerting- eg I strongly doubt I would plug myself back into the Matrix- so, following a better decision procedure, I often drop the matter.)
I’ll ask the same question of you: Do you value knowledge in someone’s mind or just knowledge? For example, does a book count as knowledge if no one’s read it?
All knowledge is in someone’s mind. As the Bayesian master E. T. Jaynes once said of probability, ignorance exists in the mind: if I am ignorant about a phenomenon, that is a fact about my state of mind, not a fact about the phenomenon. To think that probabilities were properties of the things themselves, rather than states of uncertainty in some particular mind, was designated by Jaynes the Mind Projection Fallacy. It generalizes.
Knowledge is a state of mind and needs some particular mind to be a state of.
Is ignorance bliss? For example, suppose I prefer that you don’t eat meat. Suppose you eat meat, but I never find out, not even in the slightest hint. Thus, my brain is never affected by the act of your eating meat. In your view, does this act still count against my utility function?
(Can I really talk about your utility function? But I shall rephrase as though it were of myself.)
I prefer that people don’t die. Suppose someone dies, but I never find out. Does it count against my utility function? Of course. Absolutely. I will work as hard to drain probability out of futures in which someone dies and I never find out, as futures where someone dies and I find out. If I find out, I may become a little unhappy, which would count as a small additional negative utility, but nowhere near the negative utility of the death itself. Death is a far greater problem than the sadness of death.
The reason I’m not using your example of your dislike of my eating meat - presumably apart from your desire to preserve the life of animals per se - is that I don’t think it’s a legitimate interpersonal desire, that is, you might not want Fred to eat meat, but if I have to intervene in your dispute I will come in on Fred’s side (which is how I judge interpersonality in morality). Thus, it’s the sort of thing I wouldn’t want to see built into an FAI.
The negative utility of the sadness of death is a special case of utility functions being able to reference events in general. I care as much about other people being sad about death, as I do myself being sad about death - the sadness of death has negative utility whether or not I know about it. Also note that as part of the (deontological) drive toward truth, I would not attempt to stop someone being sad about a death by preventing them from learning about it; the only acceptable way of working to end that sadness is working to end death. (Which is a larger problem in any case!)
I think it is wise to be rigorous in distinguishing the utility of external events, from the separate and additional utilities of our emotional reactions to those events. Otherwise your utility calculations start to become very confused, and you begin to bake recipe cakes (made from only the most delicious printed recipes).
“I thought you meant to point out that if all agents try to maximize their own utility then conflicts occur (unless there are only zero or one agents).”
Conflicts will occur from the point of view of any given agent. To be specific, each agent will desire the modification of the other’s goal system.
“2. I agree, our instincts often line up. But how do you determine how often is often enough?”
As long as *something Friendly* emerges out of the extrapolation, it’s good enough.
“I prefer that people don’t die. Suppose someone dies, but I never find out. Does it count against my utility function? Of course. Absolutely. I will work as hard to drain probability out of futures in which someone dies and I never find out, as futures where someone dies and I find out. If I find out, I may become a little unhappy, which would count as a small additional negative utility, but nowhere near the negative utility of the death itself. Death is a far greater problem than the sadness of death.”
To paraphrase, Eliezer’s utility function considers events far more important than the knowledge of events, since the latter exists only in Eliezer’s mind anyway. An analogy would be desiring ownership of the land much more than desiring a map of the land.
Eliezer:
First, I agree with your assessment of the nature of knowledge, i.e. that it is a state of mind.
“Can I really talk about your utility function?”
If we cannot discuss others’ utility functions, then how can we recommend anything beyond Pareto optimality? Or do you recommend nothing beyond this?
I continue to find perplexing your views regarding death. Perhaps answers to these will clarify:
1. Whose death concerns you? Humans? Sentient beings? Living things? Non-living things, such as stars?
2. A clarification, please: If someone is going to die anyways, would you prefer knowing about it or not knowing about it?
3. So if someone dies, and no one is ever sad as a result, you would still attribute negative utility to the event? If so, what property of death makes this so?
4. What are your views on birth? Does this event also constitute any sort of special case for you?
5. Do you have any other special cases?
“presumably apart from your desire to preserve the life of animals per se”
Let’s replace eating animals with wearing blue shirts, which is more ethically neutral.
“I don’t think it’s a legitimate interpersonal desire, that is, you might not want Fred to [wear blue shirts], but if I have to intervene in your dispute I will come in on Fred’s side”
We’re touching on a thorny side of preferentialism (the view that what matters is the satisfaction of individuals’ preferences). For example, with whom would you side if I prefer Fred to not have any preferences about other people?
More on this can be found in
http://www.amazon.co.uk/Hedonistic-Utilitarianism-Torbjorn-Tannsjo/dp/0748610421
See in particular “Elimination” in Ch.6, Against Preferentialism. I agree with Tannsjo on this matter that preferentialism is a difficult if even functional view to uphold.
“it’s the sort of thing I wouldn’t want to see built into an FAI.”
But if it happened as a result of CEV?
“as part of the (deontological) drive toward truth, I would not attempt to stop someone being sad about a death by preventing them from learning about it; the only acceptable way of working to end that sadness is working to end death.”
So you give knowledge acquisition lexical (complete) priority over emotional response? If not, how do you draw the line between the two?
Also, what property of knowledge causes you to value it?
“distinguishing the utility of external events, from the separate and additional utilities of our emotional reactions to those events”
I attach zero utility to anything other than our emotional reactions (broadly interpreted). Indeed, I define utility as an emotional phenomenon. Thus, no confusion here. If you do attach utility to non-emotional phenomena, why? What property of these phenomena makes them worth anything?
…..
Tom:
“As long as *something Friendly* emerges out of the extrapolation, it’s good enough.”
Define “something Friendly”.
“To paraphrase, Eliezer’s utility function considers events far more important than the knowledge of events, since the latter exists only in Eliezer’s mind anyway. An analogy would be desiring ownership of the land much more than desiring a map of the land.”
But isn’t desire a phenomenon that exists only in our minds?
And for you, as well: If you do attach utility to non-emotional phenomena, why? What property of these phenomena makes them worth anything?
If we cannot discuss others’ utility functions, then how can we recommend anything beyond Pareto optimality?
I just meant that I’m uncomfortable speaking for your utility function when you’re, like, right here.
Whose death concerns you? Humans? Sentient beings? Living things? Non-living things, such as stars?
Certainly not stars, unless you count Sol to which I have a certain sentimental attachment. I don’t know whose death should concern me - I’d say, things bearing subjective experiences, but I suspect that only names my confusion, and I’m not even sure it should be the main criterion of value! This is one of those cases where my moral theory contains unresolved questions, and hence, I try to assign probabilistic utilities. I don’t think cows are people but I’m not totally sure. Chimpanzees seem like they could very plausibly be people.
If someone is going to die anyways, would you prefer knowing about it or not knowing about it?
I would prefer to know, even though I will be sad about it. That which can be destroyed by the truth should be, including ignorance; the rational emotion is the emotion that fits the facts. See the Twelve Virtues of Rationality if you haven’t already.
So if someone dies, and no one is ever sad as a result, you would still attribute negative utility to the event? If so, what property of death makes this so?
What property of me makes me attribute negative utility to timelines that have the property of containing deaths? My evolved psychology and moral psychology cause me to see people as friends, to mourn their loss, to empathize with the dying human’s fear of dying and will to live, and to empathize with other survivors’ sadness.
Preferences are states of mind, and need minds to be states of - my utility assignment to an event is not a property of that event, it is a property of me.
What are your views on birth? Does this event also constitute any sort of special case for you?
When people are born they can have fun life experiences and thereby bring new positive utilities into the world. But the death of an existing person is very different from having never lived, and much worse; for only living people can fear dying and prefer not to die.
With whom would you side if I prefer Fred to not have any preferences about other people?
Fred. And not just because I, myself, have preferences about other people. But I might side with you if you tried to prevent Fred from imposing by coercion his preferences on other people.
Not every moral that I have is worth enforcing by violence - those that are, like my morality that people not commit murder, have a status distinct from my preference that other people be rational.
So you give knowledge acquisition lexical (complete) priority over emotional response?
I see knowledge acquisition as determining the rational emotional response, and personally, I want people to be rational.
I attach zero utility to anything other than our emotional reactions (broadly interpreted). Indeed, I define utility as an emotional phenomenon.
Utility is an emotional phenomenon that is about external events, in the same way that my knowledge that the sky is blue is an internal state of my mind which is about the external sky. Thus you appear to me to be making recipe cake.
In particular, your question as to why I disvalue deaths I don’t know about, presumes not just that utility is something that I can assign only to emotional phenomena, but that it is something I can assign only to emotional phenomena that go on inside my own head. Otherwise I could assign utilities to the deceased’s fear of dying, and all the good experiences the deceased cannot now have, and the deceased’s goal of having those experiences, whether or not I ever knew about them.
Eliezer, thanks for laying down the gauntlet. I was watching The Fountain (Darren Aronofsky, 2006) last night and was struck by how, in a film in some ways quite sympathetic to the idea of technological progress in medicine, there was still the assumption that it was unseemly and perhaps blasphemous to resist the cycle of life that God, or the Universe, had established. But as you point out, we’re already resisting, and to try and arrange some sort of boundary beyond which acceptance is morally imperative, is especially problematic when it is quickly becoming clear that the “natural” boundary is changing and will continue to change within our lifetimes.
[…] have claimed that transhumanism arises strictly from love of life. A bioconservative humanist says that it is […]
Eliezer - thanks for this post. It’s well written, and thought provoking. It compares two ethical issues - life extension and intelligence enhancement - about which I have very different views (in favour of the first, often against the second). Sorting through that will be a fascinating process.
However, you sometimes seem to make the correspondence mistake - attributing opposition to life extension to dispositional aspects rather than to circumstances. There is a rational position against life extension, based on costs. Saving young lives is generally cheap. Research into life extension of the old is expensive, and is money diverted from other research. If you add to this a preference for saving those who have had less time to enjoy life (ie the young), then you can rationally argue against life extension research (given a choice, who would you save? The girl of 5 or healthy man of 65? Not a trick question).
I don’t agree with that, but it’s definitely rational. And the more speculative and expensive life extension technologies are, the more rational the argument is. Once we have indefinite life extension technologies cheaply available (or on the cusp of being available) then I foresee a major change in people’s attitudes.
[…] Suppose I gave you a control with two buttons, a red button and a green button. The red button destroys the world, and the green button stops the red button from being pressed. Which button would you press? The green one. Anyone who gives a different answer is probably overcomplicating the question. […]
Sorry for the delay in responding…
“I just meant that I’m uncomfortable speaking for your utility function when you’re, like, right here.”
So you’re ok with making cardinal utility comparisons across individuals?
“Certainly not stars, unless you count Sol to which I have a certain sentimental attachment.”
Sounds here like it’s not the death that matters but the sadness caused by the death.
“I don’t think cows are people but I’m not totally sure. Chimpanzees seem like they could very plausibly be people.”
I wouldn’t get too caught up on the definition of “person”. Biologists seem to have the species concept down pretty well. However, they have not yet resolved which species are sentient; a probabilistic approach to handle this uncertainty seems appropriate here. Cows are most likely sentient; fish are probably sentient; lobsters and insects, we’re much less sure of.
“That which can be destroyed by the truth should be”
Including humanity’s existence?
“Preferences are states of mind, and need minds to be states of”
I agree. So, if someone dies, and no one is ever sad as a result, you would not attribute negative utility to the event?
“But the death of an existing person is very different from having never lived, and much worse”
You would rather have never lived in the first place than to live a mortal life? Not me.
“for only living people can fear dying and prefer not to die.”
While some lives may very well be worth not living (tortured livestock animals?), many (most?) are worth living, despite what fears they experience regarding death or anything else.
” “With whom would you side if I prefer Fred to not have any preferences about other people?”
Fred.”
So you prefer here that Fred has preferences about other people, even though you prefer that I do not. I don’t follow this logic. This is of course a paradoxical situation, designed to highlight a problem with preferentialism. I do not have a strong solution to this paradox, but alas, I am no preferentialist.
“my morality that people not commit murder”
Would you kill someone in self-defense, or to prevent someone from killing another, or to prevent someone from killing multiple others, if killing this person was the only means of preventing this person from killing you or other(s)? (Related: the trolley problem.)
“rational emotional response”
Forgive me here, but I thought rationality was supposed to be distinct from emotion. Am I wrong?
“I want people to be rational”
Would you prefer someone (including yourself) to be happy and irrational or unhappy and rational? I prefer happy and irrational, and view rationality as good only to the extent to which it brings happiness.
“Utility is an emotional phenomenon that is about external events”
I would not rule out the possibility that utility can be about internal events too, whatever the distinction