Artificial Intelligence, Cognitive Biases, and Global Risk
June 4th, 2007 –
If you have not read “Cognitive Biases Potentially Affecting Judgment of Global Risks” and “Artificial Intelligence as a Positive and Negative Factor in Global Risk“, I recommend reading them. They are excellent book chapters from SIAI Research Fellow Eliezer Yudkowsky, forthcoming in the edited volume Global Catastrophic Risks from Oxford University Press (Nick Bostrom and Milan Cirkovic eds.). If you do not have time to read both, I recommend reading the conclusion of the first, repeated below, and reading the second in its entirety.
Conclusion of “Cognitive Biases Potentially Affecting Judgment of Global Risks”:
Why should there be an organized body of thinking about existential risks? Falling asteroids are not like engineered superviruses; physics disasters are not like nanotechnological wars. Why not consider each of these problems separately?
If someone proposes a physics disaster, then the committee convened to analyze the problem must obviously include physicists. But someone on that committee should also know how terribly dangerous it is to have an answer in your mind before you finish asking the question. Someone on that committee should remember the reply of Enrico Fermi to Leo Szilard’s proposal that a fission chain reaction could be used to build nuclear weapons. (The reply was “Nuts!” – Fermi considered the possibility so remote as to not be worth investigating.) Someone should remember the history of errors in physics calculations: the Castle Bravo nuclear test that produced a 15-megaton explosion, instead of 4 to 8, because of an unconsidered reaction in lithium-7: They correctly solved the wrong equation, failed to think of all the terms that needed to be included, and at least one person in the expanded fallout radius died. Someone should remember Lord Kelvin’s careful proof, using multiple, independent quantitative calculations from well-established theories, that the Earth could not possibly have existed for so much as forty million years. Someone should know that when an expert says the probability is “a million to one” without using actuarial data or calculations from a precise, precisely confirmed model, the calibration is probably more like twenty to one (though this is not an exact conversion).
Any existential risk evokes problems that it shares with all other existential risks, in addition to the domain-specific expertise required for the specific existential risk. Someone on the physics-disaster committee should know what the term “existential risk” means; should possess whatever skills the field of existential risk management has accumulated or borrowed. For maximum safety, that person should also be a physicist. The domain-specific expertise and the expertise pertaining to existential risks should combine in one person. I am skeptical that a scholar of heuristics and biases, unable to read physics equations, could check the work of physicists who knew nothing of heuristics and biases.
Once upon a time I made up overly detailed scenarios, without realizing that every additional detail was an extra burden. Once upon a time I really did think that I could say there was a ninety percent chance of Artificial Intelligence being developed between 2005 and 2025, with the peak in 2018. This statement now seems to me like complete gibberish. Why did I ever think I could generate a tight probability distribution over a problem like that? Where did I even get those numbers in the first place?
Skilled practitioners of, say, molecular nanotechnology or Artificial Intelligence, will not automatically know the additional skills needed to address the existential risks of their profession. No one told me, when I addressed myself to the challenge of Artificial Intelligence, that it was needful for such a person as myself to study heuristics and biases. I don’t remember why I first ran across an account of heuristics and biases, but I remember that it was a description of an overconfidence result – a casual description, online, with no references. I was so incredulous that I contacted the author to ask if this was a real experimental result. (He referred me to the edited volume Judgment Under Uncertainty.)
I should not have had to stumble across that reference by accident. Someone should have warned me, as I am warning you, that this is knowledge needful to a student of existential risk. There should be a curriculum for people like ourselves; a list of skills we need in addition to our domain-specific knowledge. I am not a physicist, but I know a little – probably not enough – about the history of errors in physics, and a biologist thinking about superviruses should know it too.
I once met a lawyer who had made up his own theory of physics. I said to the lawyer:
You cannot invent your own physics theories without knowing math and studying for
years; physics is hard. He replied: But if you really understand physics you can explain it to your grandmother, Richard Feynman told me so. And I said to him: “Would you advise
a friend to argue his own court case?” At this he fell silent. He knew abstractly that
physics was difficult, but I think it had honestly never occurred to him that physics might
be as difficult as lawyering.
One of many biases not discussed in this chapter describes the biasing effect of not knowing what we do not know. When a company recruiter evaluates his own skill, he recalls to mind the performance of candidates he hired, many of which subsequently excelled; therefore the recruiter thinks highly of his skill. But the recruiter never sees the work of candidates not hired. Thus I must warn that this paper touches upon only a small subset of heuristics and biases; for when you wonder how much you have already learned, you will recall the few biases this chapter does mention, rather than the many biases it does not. Brief summaries cannot convey a sense of the field, the larger understanding which weaves a set of memorable experiments into a unified interpretation. Many highly relevant biases, such as need for closure, I have not even mentioned. The purpose of this chapter is not to teach the knowledge needful to a student of existential risks, but to intrigue you into learning more.
Thinking about existential risks falls prey to all the same fallacies that prey upon thinking-in-general. But the stakes are much, much higher. A common result in heuristics and biases is that offering money or other incentives does not eliminate the bias. (Kachelmeier and Shehata (1992) offered subjects living in the People’s Republic of China the equivalent of three months’ salary.) The subjects in these experiments don’t make mistakes on purpose; they make mistakes because they don’t know how to do better. Even if you told them the survival of humankind was at stake, they still would not thereby know how to do better. (It might increase their need for closure, causing them to do worse.) It is a terribly frightening thing, but people do not become any smarter, just because the survival of humankind is at stake.
In addition to standard biases, I have personally observed what look like harmful modes of thinking specific to existential risks. The Spanish flu of 1918 killed 25-50 million people. World War II killed 60 million people. 10^7 is the order of the largest catastrophes in humanity’s written history. Substantially larger numbers, such as 500 million deaths, and especially qualitatively different scenarios such as the extinction of the entire human species, seem to trigger a different mode of thinking - enter into a “separate magisterium”. People who would never dream of hurting a child hear of an existential risk, and say, “Well, maybe the human species doesn’t really deserve to survive.”
There is a saying in heuristics and biases that people do not evaluate events, but descriptions of events – what is called non-extensional reasoning. The extension of humanity’s extinction includes the death of yourself, of your friends, of your family, of your loved ones, of your city, of your country, of your political fellows. Yet people who would take great offense at a proposal to wipe the country of Britain from the map, to kill every member of the Democratic Party in the U.S., to turn the city of Paris to glass – who would feel still greater horror on hearing the doctor say that their child had cancer – these people will discuss the extinction of humanity with perfect calm. “Extinction of humanity”, as words on paper, appears in fictional novels, or is discussed in philosophy books – it belongs to a different context than the Spanish flu. We evaluate descriptions of events, not extensions of events. The cliche phrase end of the world invokes the magisterium of myth and dream, of prophecy and apocalypse, of novels and movies. The challenge of existential risks to rationality is that, the catastrophes being so huge, people snap into a different mode of thinking. Human deaths are suddenly no longer bad, and detailed predictions suddenly no longer require any expertise, and whether the story is told with a happy ending or a sad ending is a matter of personal taste in stories.
But that is only an anecdotal observation of mine. I thought it better that this essay should
focus on mistakes well-documented in the literature – the general literature of cognitive
psychology, because there is not yet experimental literature specific to the psychology of
existential risks. There should be.
In the mathematics of Bayesian decision theory there is a concept of information value – the expected utility of knowledge. The value of information emerges from the value of whatever it is information about; if you double the stakes, you double the value of information about the stakes. The value of rational thinking works similarly – the value of performing a computation that integrates the evidence is calculated much the same way as the value of the evidence itself. (Good 1952; Horvitz et. al. 1989.)
No more than Albert Szent-Gyorgyi could multiply the suffering of one human by a hundred million can I truly understand the value of clear thinking about global risks. Scope neglect is the hazard of being a biological human, running on an analog brain; the brain cannot multiply by six billion. And the stakes of existential risk extend beyond even the six billion humans alive today, to all the stars in all the galaxies that humanity and humanity’s descendants may some day touch. All that vast potential hinges on our survival here, now, in the days when the realm of humankind is a single planet orbiting a single star. I can’t feel our future. All I can do is try to defend it.
A very well constructed essay and very important. Unfortunately too many of us reach for simple math models or our “all to flawed” gut instinct when it comes to risk. It would be nice to see more collected articles about the dangers of being “exactly wrong” either via individuals or via group think.
I largely agree with going on in this summary, but as a unix geek and artist who has his own theories that dip into the realm of physics, I do take issue with some of the biases the article itself displays – specifically, the dismissive tone of those who would dare to theorize without the proper credentials.
Clearly, I am not a theoretical physicist or an AI specialist, but I have been following advancements in these areas since I was a kid. I’m not suggesting that I know how to create a theory on physics, but I do know how to watch the arc of a scientific field’s development. Just like Kurzweil can take statistical data points and show exponential growth, writers and other creative voices are often able to see patterns and suggestions of possibility from outside the scientific community that are harder to perceive from inside the community. Science Fiction writers have certainly been limited in their success of predicting the future, but they often have had a prescient view into inevitable, upcoming advancements in science and technology.
I think my creative imagination is what allows me to understand, all too well, the possible impact of blindly stomping forward in AI, without regard to the existential risks associated with the changes. As it is, I see businesses, relationships, governmental policies and more that aren’t keeping up with what is happening in technology. Structurally, these systems aren’t designed with the flexibility that will be required to deal with the implications of strong AI.
I think that AI does have the potential for wide ranged devastation. However, even in best-case scenarios, it will fundamentally alter our understanding of civilization – likely over the course of just a few years. Most people, companies, and countries are not aware of how ill prepared they are for a world that complex.
As for the AI community – I applaud the epiphany of understanding just how remarkably complex AI actually is. I also hope that they are aware that sometimes their linear, scientific models unnecessarily limit the possible solutions. I think we’re starting to see that this is way more complicated than pure, hard science can accomplish alone. This is a creative venture at its core, so a certain amount of unconventional creative thought will be required to keep up.
Theorizing without credentials isn’t the problem. The problem is acting without expertise. Putting a device into motion without understanding exactly what it will do.
For most computer programs this isn’t a serious problem. You have a bug, it annoys a few people, you fix it (maybe). But as you mention, with strong AI you have the potential to threaten humanity’s existence. One must be extremely careful as bugs can be fatal.
There are certain classes of mistakes humans reliably make. This is an artifact of how our brains works. We use heuristics to quickly arrive at generally good answers. But these heuristics aren’t perfect, they have systematic biases, as detailed further in the paper. The field of heuristics and biases studies exactly this. As a result it can tell you how to avoid a bunch of predictable mistakes.
If you plan to work with AI you should at least understand the basics of this field to avoid the obvious mistakes.
I can live with doubt and uncertainty and not knowing. I think it is much more interesting to live not knowing than to have answers that might be wrong.
- Richard Feynman
Whether we be “amateur” or expert, when we substitute what we wish to be true for what *is* true (factual and repeatedly verifiable in its nature), we have made a grievous error in judgment.
When we a priori dismiss the work of the amateur for the opinion of the expert who *knows* what is true we may have made a similar error in judgment.
Our biases (our *knowing*) often affect our judgment in ways that blind us to the truth, to the facts, and to the evidence.
[…] Via the SIAI blog. […]
The Survival of Humankind, and Improving the World, Society, and Yourself!
Yet who can the world trust to be idealistic and moral enough to help all of humanity and the environment, and at the same time, be practical enough to make extremely difficult decisions that can and will harm a great deal of people?
Humanitism is a philosophy for the continued survival and perpetuation of the human race. Humanitists (people who believe in humanitism) do not have the luxury of trying again after failing. Humanitists must be more vigilant than environmentalists, because we will not have a second chance at survival.
The survival of humanity is more important than the well being of our environment; however the environment is necessary for humanity to survive. That does not give the right for big businesses to continue doing whatever they want with only minimal or no consideration for the environment, so long as our surroundings support human life. We need to protect the environment for the continued survival and future well being of humanity. Keep in mind that without the human race, there would be no one and no need to protect the environment. Therefore, humanitism is more important than environmentalism.
It seems that in the past 50 years the human race has pursued the money train, that such desire for financial gain has caused society to ignore and abandon honesty, values, morality and candidness etc.
The race to financial gain has caused our leaders and the executives of the corporate world to disregard laws, ethics and the caring for each other and humanity as a whole. Deception, fraud and outright theft are their new motto all for the sake of financial gain, fame and success.
It seems that for the sake of success and profit people will step on anybody, family friends, co-workers and anyone who stands in their way or take advantage of anyone that could help them achieve what they want.
That is not to say that honest and compassionate people who care do not exist, where honesty and integrity is a way of life for them, but they are a very small minority.
As we begin the year 2008, we should all look at the past and decide with determination that everyone will from now on contribute to the betterment of humanity, society and mankind.
We should all learn to live with each other and respect each other for the sustainability of mankind.
Compiled by: Yehuda Draiman – 1/1/2007
The deterioration of family values
Since World War 2 when women were encouraged to join the work force en mass, to replace the men who went to war and keep the economy and the war effort going.
There has been a trend where a mother was not home to take care of her children, monitor their behavior, help with the homework and discipline when and where necessary.
The advancement in technology has harmed family values. The Media and Television has totally destroyed any comprehension of values in our society.
The lack of discipline and total disregard for authority and respect is clear to anyone who has watched the past 50 years and seen our society’s values deteriorate.
One example alone is that 50 years ago a teacher was happy to go to school to teach, a teacher was respected and looked up-to, a teacher could discipline. Today teacher fear for their lives they are petrified by their students.
This scenario caries on to other social interactions of society today, and the situation is getting worse and worse every year.
You will notice that many families who come from other countries have a very strong family values, good education, respect and the children excel in their studies. That is because they have not had the chance to be influenced by our society.
The education of our children begins at home and continues in school – the parents and the school must take a proactive approach to teach our children values and respect.
In today’s society a teacher is not permitted to discipline a student, the teachers will be sued, not to mention that teachers fears for their safety.
Parents in today’s society are also restricted as to how to discipline their children; in many cases parents are getting sued. In many cases children would never dream of treating their parents with such disrespect 50 years ago. Today some parents are afraid of their own children.
Abuse has been and will be with society to eternity that does not give society the right to prohibit discipline; a few acts of abuse should not cause society to prohibit proper discipline.
When an individual or individuals utilize a vehicle to commit a crime cause the death of others, does society prohibit vehicles altogether, no, a vehicle is very important for our everyday life.
Well, the discipline of our children by parents and teachers is extremely important for our society and the preservation of humanity.
It seems that our society is so busy chasing the dollar, fame and glory, that anything goes all values goes out the window. We should be an example of honesty, integrity and respect to our children.
Yehuda Draiman, Northridge, CA
PS
Tell me and I will forget
Show me and I may remember
Involve me and I will understand.
– Chinese Proverb.
American economy in crises - a long time coming
When a country and its society import more than they export for over a quarter of a century, it is bound to erod the economy to its primate state.
We have only ourselves to blame, what goods and products are we exporting, what goods and services are produced in the USA, the answer is very little by comparison.
In the past 50 years as our population has increased, technology advanced, we have become a nation that consumes enormous amounts of resources, we shop for competitive prices. Corporate America is constantly looking to increase the bottom line.
Most of the goods for and by Americans and its companies are produced overseas and in the past decade with the advancement of telecommunications, many of the services sector are also imported.
The increased costs of energy over the past 10 years, has affected the economy to unimaginable comprehension.
This economic activity has eroded our economy to its core. It seems that the situation is getting worse every year. American debts are increasing beyond our wildest dreams, endangering the future economic vitality of our future generation.
I hope it is not too late for our society to recognize the graveness of our economic predicament and its resolve to take appropriate action to stem the tide of our economic downturn.
Americans are a nation of great technology and knowhow. We must utilize that technology and our resources to find new means to regain our economic independence.
We must face and implement fiscal responsibility, both by the government and the population with its infrastructure of corporate America.
It is no longer an option, it is a must if we as a nation want to survive and retain our way of life and economic vitality.
Inflation, recession and financial crises are here. Let us take the bull by the horn, initiate immediate actions to minimize and hopefully reverse our economic crises.
Yehuda Draiman, Northridge, CA. 1-22-2007
PS
The US economy has enormous momentum. Metaphorically speaking, if someone turned off the locomotive that drives the US economy, the economy would go on for miles before anyone would likely notice something was wrong. But something has been wrong for many years. Is there really hope for the future? Maybe. But the terrible truth is that no one really knows. But if there is hope, we’re already on the wrong track. And that has to change.
The very dim outline of just how badly we have handled the the art of rational thinking is coming into view. When fully disclosed, I fear that a worldwide gasp of horror will be heard, nearly too late to matter.
The greater error may be the error of assuming that the twenty-first century ( and indeed the late 20th century also) was or is being guided by insightful, “Nobles”. Hindsight must inform those who are having a difficult time of imagining life beyond 2015, that no Nobles exist. Those who claimed to be so were simply the early ancestors of today’s “mortgage bankers”.
There is a war going on at this very minute that puts humanity on the endangered species list. The combatants seem to be clashing cultures and ill-informed national and religio-cultural leaders all bent on shaping the world in their narrowly defined image(s). The third combatant is the earth itself, as it speaks out with loud voices in many places almost daily. Tornados, Tsnumis, earthquakes, widespread food shortages that we will see developing in America’s flood ravaged heartland.
We cannot afford to not look at everything we do now, the future may still seem bright but errors, biases, and political miscalculation loom larger than ever.
“Can the energy crises be overcome?” – I say yes!
I think that the public, the government and corporate America should treat these energy crises as a danger to our way of life.
During World War II, the America we know unified in a common cause. Everyone rolled their sleeves to chip in and Americans produced an enormous amount of hardware for the war effort. “I see a solution within 36 months”.
If we as a nation can really appreciate and understand the severity and enormity of the energy crises, the catastrophic impact on our economic stability and civilization,
we must unite and do whatever is necessary to produce other forms of energy and overcome this energy and economic crises “by putting all politics and egos aside and look for the good of our nation”.
Jay Draiman, Northridge, CA
PS
ENERGY
Soaring gas prices, record oil company profits, unsustainable trade deficits, soldiers dying in Iraq’s deserts and catastrophic climate change— conservative energy policy is running us toward ruin. We urgently need to stop subsidizing dirty fossil fuels and start investing in a clean energy economy. In 1961, President Kennedy challenged us to send a man to the moon within a decade and launched the Apollo plan to make it happen. Now we need a similar vision—an Apollo initiative for energy independence, mobilizing science and technology and investing in energy efficiency and alternative energy.
The benefits are immense. We can create jobs, capture growing global green energy markets, eliminate our dependence on Persian Gulf oil, reduce our trade deficit, improve our children’s health and end the catastrophic threat of global warming. It’s time to act.
THIS IS MUCH MORE IMPORTANT THAN SENDING A MAN TO THE MOON.
“Determination and perseverance will bring your goal to fruition” – never give-up