The second problem deals with “successful lifespan extension”. I happen to know a bit about this area (more than the average person who runs around discussing AIs) and we now understand this problem and how to address it. It *does not* require an AI. It does require that people recognize the problem can be solved and commit to solving it (as was the case for landing men on the moon). Unless you intend for your FAI to reach into the minds of all the people who will not “believe” it until 50 or 100 years from now (and we have a bunch of 150 year old people wandering around) acceptance of this is not going to happen overnight.

A sufficiently powerful AI could “force” humanity into accepting its wisdom (as parents must sometimes force children to eat their vegetables). But I for one see it as highly unlikely that that acceptance process as being one in which the maximum number of human lives is saved.

1. The development of the intelligence is binary instead of incremental

2. The implementation of the intelligence is binary instead of incremental

Why can’t an artificial intelligence develop in the same manner as an individual, acquiring increasing levels of intelligence and capability over time that can be controlled? If there are an adequate number of check points can we not judge whether the AI is developing in a manner that is favorable or not and cut back on the intelligence and resources available to it if necessary?

In the same way that you control a nuclear reaction why can’t you have control rods for an AI that you can feed in and feed out? Why are we thinking about AI like it is the bomb? It seems like a path of reasoning that is more careless than I care to entertain. If creating an AI and just flipping the switch for an uncontrolled chain reaction is the policy of the Singularity Institute, I withdraw my support.

It can; the crucial threshold is when the intelligence is able to improve itself sustainably.

“can be controlled?”

Once the AGI is substantially smarter than we are, we won’t be able to control it any more than monkeys can control us.

“If there are an adequate number of check points can we not judge whether the AI is developing in a manner that is favorable or not and cut back on the intelligence and resources available to it if necessary?”

If the AGI is undergoing a failure of Friendliness, and it can’t self-improve, the only rational thing to do is to pull the plug to avoid any risk of a hard takeoff. If it can self-improve substantially faster than humans can program it, we’re already screwed. You cannot control a superintelligence like you can control a toaster; it is much more knowledgeable and better at strategy than you are.

“In the same way that you control a nuclear reaction why can’t you have control rods for an AI that you can feed in and feed out?”

Because humans *cannot* control a superintelligent AGI. This is an unavoidable side-effect of the tremendous good we want the AGI to do; anything that has the power to do more good than humans must be more powerful than humans.

Fictional gods have no more bearing on the development of the universe than The Matrix. See http://www.acceleratingfuture.com/tom/?p=12 for why it is bad to mention futurism and fiction in the same sentence.

“(except those seeking a permanent Mommy/Daddy entity)”

This is a perfect example of what I warn against in the above link; cross-importation of concepts from fiction to reality. The fiction of religion is that God is a “father figure”; therefore, we assume that AGIs will also act like a “father figure”, because we have already thought of the two as being roughly analogous.

The risk of this depends entirely on the AI’s behavior - certainly there are ways for it to behave that would make a large fraction of humanity prefer death. That, however, would in itself be a failure of Friendliness. The point is precisely to design an AI that is intelligent enough to calculate a behavior that leads to the least deaths.

The second problem deals with “successful lifespan extension”. I happen to know a bit about this area (more than the average person who runs around discussing AIs) and we now understand this problem and how to address it. It *does not* require an AI.

Biological lifespans can probably be extended without an AI, though it will be easier with one. But death due to aging is only one cause of death - one of the worst, yes, but still only one. An AI could potentially help with them all.

As for boredom in artificial reality, if this would be a problem, and if the AI is so talented (and the assumption here seems to be that it would be), then it presumably would be sufficiently talented to make the artificial realty interesting.

As for flipping a switch to an uncontrollable outcome, I’ll admit I’m not enthusiastic about this myself- hence the discussion of “postponing a launch indefinitely”. However, if some switch is going to be flipped anyways, then I would prefer we flip the best switch, whatever that means. In the face of existential risk, not flipping an AI switch could amount to flipping a switch leading to our doom.

Look how many of us work for idiot managers.