"Creating Friendly AI is the most intelligent writing about AI that I've read in many years."
            -- Dr. Ben Goertzel, author of The Structure of Intelligence and CTO of Webmind.

"With Creating Friendly AI, the Singularity Institute has begun to fill in one of the greatest remaining blank spots in the picture of humanity's future."
            -- Dr. K. Eric Drexler, author of Engines of Creation and chairman of the Foresight Institute.

Creating Friendly AI describes the design features and cognitive architecture required to produce a benevolent - "Friendly" - Artificial Intelligence.  Creating Friendly AI also analyzes the ways in which AI and human psychology are likely to differ, and the ways in which those differences are subject to our design decisions.

Multi-page version:		http://singinst.org/CFAI/
Single-page version:		http://singinst.org/CFAI.html
Printable version:		http://singinst.org/printable-CFAI.html

• Preface [4K]
• INIT [1K]
• 1: Challenges of Friendly AI [39K]
• 1.1: Envisioning perfection
• 1.2: Assumptions "conservative" for Friendly AI
• 1.3: Seed AI and the Singularity
• 1.4: Content, acquisition, and structure
• An Introduction to Goal Systems [24K]
• Interlude: The story of a blob [12K]
• 2: Beyond anthropomorphism [82K]
• 2.1: Reinventing retaliation
• 2.2: Selfishness is an evolved trait
• 2.3: Observer-biased beliefs evolve in imperfectly deceptive social organisms
• 2.4: Anthropomorphic political rebellion is absurdity
• Interlude: Movie cliches about AIs
• 2.5: Review of the AI Advantage
• Interlude: Beyond the adversarial attitude [18K]
• 3: Design of Friendship systems [0K]
• 3.1: Cleanly Friendly goal systems [57K]
• 3.1.1: Cleanly causal goal systems
• 3.1.2: Friendliness-derived operating behaviors
• 3.1.3: Programmer affirmations
• 3.1.4: Bayesian reinforcement
• 3.1.5: Cleanliness is an advantage
• 3.2: Generic goal systems [79K]
• 3.2.1: Generic goal system functionality
• 3.2.2: Layered mistake detection
• 3.2.3: FoF:  Non-malicious mistake
• 3.2.4: Injunctions
• 3.2.5: Ethical injunctions
• 3.2.6: FoF:  Subgoal stomp
• 3.2.7: Emergent phenomena in generic goal systems
• 3.3: Seed AI goal systems [101K]
• 3.3.1: Equivalence of self and self-image
• 3.3.2: Coherence and consistency through self-production
• 3.3.3: Unity of will
• 3.3.4: Wisdom tournaments
• 3.3.5: FoF:  Wireheading 2
• 3.3.6: Directed evolution in goal systems
• 3.3.7: FAI hardware:  The flight recorder
• 3.4: Friendship structure [5K]
• Interlude: Why structure matters [7K]
• 3.4.1: External reference semantics [35K]
• 3.4.1.1: Probabilistic supergoal content
• 3.4.1.2: Bayesian affirmed supergoal content
• 3.4.1.3: Semantics of external objects and external referents
• 3.4.1.4: Deriving desirability from supergoal content uncertainty
• Interlude: Philosophical crises [13K]
• Crisis of Bayesian affirmation
• 3.4.2: Shaper/anchor semantics [60K]
• 3.4.2.1: "Travel AI":  Convergence begins to dawn
• 3.4.2.2: Some forces that shape Friendliness:  Moral symmetry, semantics of objectivity
• 3.4.2.3: Beyond rationalization
• 3.4.2.4: Shapers of philosophies
• 3.4.2.5: Anchors
• 3.4.2.6: Useful shaper content requires high intelligence
• 3.4.3: Causal validity semantics [38K]
• 3.4.3.1: Taking the physicalist perspective on Friendly AI
• 3.4.3.2: Causal rewrites and extraneous causes
• 3.4.3.3: The rule of derivative validity
• 3.4.3.4: Truly perfect Friendliness
• 3.4.3.5: The acausal level
• 3.4.3.6: Objective morality, moral relativism, and renormalization
• 3.4.4: The actual definition of Friendliness [8K]
• 3.4.4.1: Requirements for "sufficient" convergence
• 3.5: Developmental Friendliness [50K]
• 3.5.1: Teaching Friendliness content
• 3.5.2: Commercial Friendliness and research Friendliness
• 3.5.3: Singularity-safing ("In case of Singularity, break glass")
• Interlude: Of Transition Guides and Sysops [20K]
• The Transition Guide
• The Sysop Scenario
• 4: Policy implications [36K]
• 4.1: Comparative analyses
• 4.2: Policies and effects
• 4.3: Recommendations
• 5: Miscellaneous [5K]
• 5.1: Relevant literature
• END [1K]
• Appendix A: Friendly AI Guides and References [0K]
• A.1: Indexed FAQ [36K]
• A.2: Complete Table of Contents [0K]
• A.3: Glossary [39K]
• A.4: Version History [3K]

Next:	Preface
Up:	Creating Friendly AI
Prev:	A.4: Version History