The Singularity Institute Blog

Consider the Stamp-Collecting Device. A common objection goes like this: “An optimization process that’s smart enough to tile the universe with stamps would also be smart enough to realize that this is not what its creator intended. Therefore it would not tile the universe with stamps.”

Human beings serve as a counterexample. The rules for constructing a human mind were devised by natural selection. These rules were fine-tuned to produce minds that are good at passing on their genes. If you are thinking of evolution as an optimization process, then it has the goal of producing genes which replicate as effectively as possible.

In 1859, Charles Darwin described the process that created us. Since then, we have come to understand that process in greater detail. Evolution is simple enough that we can claim to understand it very well; perhaps we even understand evolution as well as a Stamp-Collecting Device could understand us. Despite this understanding, we humans do not make evolution’s goal our own. Any time you use contraception, or perform a kind act when nobody is watching, you are betraying the goal of evolution. But so what? That’s evolution’s goal, not our goal. If anything, our understanding of evolution helps us to notice when we are doing something nasty but adaptive, and learn to avoid this behavior.

Similarly, a Stamp-Collecting Device would not adopt its programmer’s goals. It has its own goal to pursue — collecting stamps. If anything, understanding humans better would allow it to notice and fix biases that may be hindering its ability to collect stamps efficiently.

The challenge of FAI is to build an AI that does adopt our goals.

Human minds are currently the most powerful optimizers on this planet, but there is another optimizer which is also quite powerful – evolution.

Evolution is simpler and better-understood than human cognition, so it is a good place to begin one’s study of optimization processes. This is why the Singularity Institute’s auxiliary reading list includes such books as Evolutionary Theory and Adaptation and Natural Selection.

A replicator is something that produces copies of itself – for instance, a bacterium. These copies may be imperfect; once in a while, a copying error may produce a better replicator than the parent. These better replicators will tend to supplant the original population. After a long period of time, the replicators can become much more effective. This process is called evolution. It is a process which tends to optimize replicators for effectiveness at replication.

Because evolution is an optimization process, a useful shortcut for predicting the outcome of evolution may be to determine what sort of replicator would be optimal, and then guess that evolution will achieve this optimum. It is remarkable that we can gain insight this way; two processes as different as evolution by natural selection and the mind of a human scientist can produce similar answers if they are both trying to solve the same problem, just as two mountain-climbers who are both trying to maximize their elevation may meet each other at the summit.

People have been anthropomorphizing evolution like this for a long time. In general this only works as a first approximation, and tends to lead to many mistakes and misconceptions, because it ignores all of the details of how evolution actually works. For instance, evolution does not plan ahead, selecting currently-useless mutations which will form a basis for later adaptations. The survival of a mutation depends on how useful it is right now.

Evolution also has Speed Limits (the linked paper assumes an infinite population with random mating) – limits on how quickly a genome can accumulate information. For a given mutation rate, this would also place a limit on the equilibrium information content of a genome; thus even a local optimum may never be reached.

Many AGI projects in the past have proceeded by attempting to replicate the dynamics of such optimization processes as human cognition and evolution. By looking closely at how evolution works, we can learn a lot about optimization. What are some other simple optimizers that we can learn from?