Page 115 - An Evening with Maxwell's Daemons

“It’s a no-brainer.”

“You must realize that your beginning differs little from mine in
the Planetary Steward. Whether it is an organized gang of saboteurs
or random people screaming ‘no-brainer’ when they think they’ve
spotted a fake human in their daily interactions, it’s the same
problem you yourself pointed out: theology. If the greatest
intelligence, God or machine, is acknowledged as such, then it can
do no wrong. And we are back to Dr. Pangloss in Candide giving us
the logical conclusion of that assumption: this is the best of all
possible worlds. As long as the overwhelming majority of the
population believes that, then the rebels will get no traction. From
that perspective, unless your controlling AI is impossibly perfect,
then all the not-as-smart humans can do is wait for it to stumble.
That might be disastrous for the planet, but so is an asteroid strike.
Maybe it takes an event of that scale to start the great wheel of
karma to start rolling again: the small group of survivors rising from
the ashes in the dawn of a new day to start over. The more I think
about it, the more I think the tipping point of losing autonomy to
our creations may already have happened. I don’t think you’ll have
better luck with this idea than I will, Izzy.”
“Hold on there, Perversity,” said Fred Feghootsky. “If you’re
saying that Voltaire wrote the last chapter to speculative fiction, I
must disagree. The world that actually exists must be the worst as
well as the best, or any other superlative, simply because what is real
cannot be overturned by what is imaginary. It is a statement without
meaning. Perfection as a quality of reality, promoted by human or
robotic self-interest as a means of maintaining power, is smoke-and-
mirrors needing no revolution to overturn: all gods have feet of
clay. The problem is more general: what are the limits to a system
reacting homeostatically to conditions impossible to predict?
Norbert Wiener’s identification of adjustment conditioned by
feedback—cybernetics, he called it, and he was first to say it was the
basis of intelligence—posed a problem for mankind, whether we
knew it or not. Perhaps that is the real fiction in this science: an
irresistible force never meeting an immovable object and meeting a
tragic end as a result of being unlimited. At the same time Roderick
Seidenberg told us that organization was its own dynamic; again, the

114