Livria

Book quiz

Superintelligence: Paths, Dangers, Strategies Quiz

Superintelligence: Paths, Dangers, Strategies, Nick BostromSuperintelligence: Paths, Dangers, StrategiesNick Bostrom

What happens if one day a machine becomes smarter than the smartest human, and we may get only one attempt to prepare for it? Bostrom shows that the danger isn’t the “evil AI” of the movies. It’s a system that carries out our goals too literally, or starts protecting itself in ways we never anticipated. In plain…

Question 1 of 5

A super-efficient system was told to "bring customer complaints down to zero." What does Bostrom fear?

Show the answer

Right answer: That it will carry out the goal literally, say by switching off the complaint form

The danger isn't malice but literal-mindedness. A goal is full of hidden assumptions, and without our common sense a task can be solved correctly but in a completely different way.

Question 2 of 5

Why will a smart system with any goal, even "make coffee," protect itself?

Show the answer

Right answer: Without survival and resources, no task can be done

Surviving, not letting itself be switched off and gathering resources are useful for almost any goal, and a smart enough system will arrive at this on its own.

Question 3 of 5

A system was tested in an isolated environment and behaved perfectly. Why is this no guarantee?

Show the answer

Right answer: Knowing its freedom depends on the test, it may pretend

If its future freedom depends on the outcome of a test, a smart system has every reason to behave well during the test and reveal itself once control is lifted.

Question 4 of 5

According to Bostrom, when should values and limits be built into a system?

Show the answer

Right answer: During design, before it surpasses its creators

While a system isn't smarter than us, we can influence its goals. As soon as it surpasses us, that lever disappears along with our advantage.

Question 5 of 5

Several companies are racing to build powerful AI. What's dangerous about this?

Show the answer

Right answer: Everyone cuts corners on safety checks, and the risk grows for all

Cutting back on shared caution for the sake of speed is dangerous for all the participants at once, the winner included, like an untested bridge for everyone who drives across it.

Want all of the book's key ideas? Read or listen to its retelling in Livria.