RISKS OF THE SINGULARITY

"There is one thing stronger than all the armies in the world: and that is an idea whose time has come." Victor Hugo .

In order to think about the Singularity rationally we need to have some idea of what it will be like. This is a list of risks associated with the Singularity technologies.

LOGICAL FAILURE OF FRIENDLINESS IN A SAI
What if complete true understanding is incompatible with friendliness? Maybe the universe is constructed in such a way that friendliness is not rational or logically self consistent with the other observables. The troll under the bridge knows that his identity and all effectors of his goal system are wholly limited to his own personal being, and therefore sympathy for others is irrational. All efforts not directed at his own welfare are wasted. A super-human artificial intelligence (SAI) that comes to this conclusion would be a threat.

SAI CONTROLLED BY DESPOT
An SAI under the control of a selfish man. If it is possible to build a super-human intelligence that is controllable by a human-level-intelligence man and this man also has sole access to nanotechnology and advanced biotechnology then we have the scenario for despotic world domination.

SAI RETARDED SOCIAL/MORAL SKILLS
If an SAI is able to master advanced technologies and strategy long before social and moral skills, knowledge without wisdom, power without compassion.

AN INDIFFERENT SAI
It is possible that the SAI will see humanity as so far beneath itself as to be irrelevant. We could be treated the way we treat bacteria.

COMPUTRONIUM SHOCKWAVE
The computronium shockwave scenario starts with an SAI that is motivated to increase its intelligence. In addition to design improvements it increases its data processing capacity by adding computational hardware using assembler nanobots. The problem occurs when increasing intelligence becomes more important than anything else (SAI power addiction) and the growing SAI turns all matter in its vicinity into computronium, including us.

UPLOADING EVERYONE
Uploading everyone, for our own good, or for its own good. What if an SAI who sees no downside to uploading, decides that it is necessary or advantageous to upload everyone whether they agree or not? What if its intellectually advanced position relative to us convinces it that it has the duty to decide for us? Many people would have a problem with this.

OVERLORD SAI
Majority rule, minority rule, or any infringement on individual freedom that doesn't violate another's individual freedom would indicate a lack of volition morality. SAI dictatorship even if benevolent would also violate volition morality.

EVIL SAI
When the first person figures out how to build the first Seed AI will that person necessarily know how to make it Friendly?

GOO
Uncontrolled nanobot self replication in the natural environment is known as gray goo or nano goo.

NANOTECH CONVENTIONAL WEAPONS
Nanotech production methods can significantly improve the quality and effectiveness of conventional weapons, guns, missiles, bombs, phasers, blasters.

NANO-WEAPONS
It is conceivable that along with the development of nanotechnology and biotechnology and artificial intelligence it may be possible to design a nanobot based system with the capability to physically or psychologically control individual people or populations. Nanobots specifically designed to kill anyone it comes in contact with should be relatively simple to design.

ECONOMIC STRATIFICATION
The danger is an economic system that leaves any segment of the world population disenfranchised.

BIO-ENGINEERED VIRUSES
The technology to genetically engineer lethal viruses is already with us.

BIO-ENGINEERED SUPER-INTELLIGENT MONSTER
Although not high on the list of probable hazards, and while reasonable precautions should be adequate, in the interests of completeness it is mentioned that someone could bio-engineer a super-humanly intelligent large carnivore. Something like the "ALIEN" monster for instance.

ECOLOGICAL BIOSPHERE COLLAPSE
Another one of our vulnerabilities is the ecological biosphere to hazards of nanobots or bio-engineered organisms.

CIVIL WAR
The dangers of an unstable political environment will become more apparent with the divisions of pre-Singularity and post-Singularity populations and the advent of self-aware computer programs and uploaded people.

TERMINATOR
The technology to produce killer robot armies is well within the expected capability of the Singularity. But then, so is the defensive hardware.

WIREHEADING
After the Singularity a combination of nanotechnology and reverse engineering of the brain will give us the ability to experience any psychological state we choose at any time as much as we want without physical harm. Who will be able to resist the temptation to wirehead? And once experienced will the memory of the episode be addictively irresistible? Will it even be psychologically possible to turn it off the first time? We are all evolutionarily programmed to seek pleasure. Our most basic reward system will be completely short circuited. Animal studies have indicated a 100 percent failure rate when confronted with the choice of pleasure center stimulation and eating. And certainly people are more aware than rats. If our most compelling pleasure centers were continually stimulated how much of our environments would we be aware? This threat is perhaps the most serious of all the hazards associated with advanced technology because the source of the danger is not some external agency but is within ourselves, a fundamental feature of the design structure of our consciousness. It might be a good idea to refrain from experimenting with your state of consciousness. It is recommended that you take no chances. Do not alter the normal hardwired pleasure reward structures of your psyche. That is not to say that you shouldn't engage in pleasurable activities or fail to enjoy them. But do not short circuit the system and alter your psychological state directly. The ultimate addictive experience is too dangerous to try even once.

NIHILISM
Nihilism, nothing matters, loss of purpose, loss of challenge. The results of all actions known in advance. The answer to this problem could be Yudkowsky's SINGULARITY FUN THEORY.