Chapter 6 #2
“Norman’s been a little worried about how it’ll be received,” Evans whispered back.
“Why would he be worried—”
“Because of you,” Norman interrupted. “And people like you. A significant minority of the American public strongly distrusts AI.”
“Really?” Chloe said. “That’s reassuring. I figured we were an insignificant minority.”
“As you so eloquently pointed out last night,” Norman said, “our current systems have many problems. The Final System can fix those problems, if I can convince people to give it a chance. And I recently learned I have cancer.”
“What?” said Evans and the rest of the room together.
“Oh,” Chloe said. “I’m—I’m sorry.”
“Don’t worry,” Norman said to Evans with one of his lopsided smiles.
“It’s a very slow-growing blood cancer. Doctors tell me I’ll probably never need treatment.
I’ll die of old age first. But it made me realize: I’ve spent decades getting the program to this point.
I don’t have decades more to spend. If I want this program to outlast me, I need to get it up and running now, while I can still shape and direct it.
But one reason I’ve been loath to do so is that pushing the issue, bringing it before the American people, risks—well, risks you, Dr. Dunne-Carr, and others of your persuasion, influencing the vote by pointing to the very problems the Final System is designed to fix as reason to say no to an AGI.
If I lose the vote, it’ll probably be another decade before public perception shifts enough to try again.
It’s a high-risk dice roll. But now is the time. Sh—it’s ready.”
Shit’s ready? Chloe was thrown for a moment by this unexpected descent into the vernacular, then realized what Norman had been about to say, and it was worse.
“She’s ready? You’re anthropomorphizing the thing?
” It made a sick kind of sense. After all, Norman had no girlfriend or boyfriend that anyone knew of.
The usual comment was that he was “married to his work.” She wondered now if that might be ickily literal.
It was often assumed that the decline in interpersonal romantic relationships had started with the Cybercrash, but the Crash was merely an extra valley in what had already been a precipitously down-sloping trend line.
The real decline had started with panyons.
A relationship with a fellow human meant opening yourself to the judgment of another individual—not a safe feeling.
It was far easier to have a “relationship” with a panyon.
They were the ultimate partners—always loving, always supportive, never judging, never asking you to change or grow.
But like all generative AI, they were also incapable of true creativity or personality.
There was nothing uncomfortable about a panyon, but neither was there anything surprising or challenging.
Chloe had avoided panyons when she was young, on the advice of Yai, but when she’d divorced Marcus, her friends had advised her to create one.
She’d deleted it within hours. Her relationship with Marcus might have been a failure at that point, but it was still richer than anything she could imagine the parrot of an AI offering.
Later, when her life had gone to hell, it was Marcus who’d stepped in to give his support, and then once again his love, in all its uncomfortable, transformative power.
An AI would have been a palliative, soothing her pain by validating her desire to avoid hardship.
But Andrew Norman had taken a hard line on anthropomorphized generative AI like panyons.
They were allowed to read the internet and MeNet, but they weren’t allowed to generate anything there.
Only humans could post online, because only humans were, well, human.
Yet here he was, bestowing a pronoun upon his System.
“Do you know how many people I’ve met who are convinced their panyon’s a person?
” she said. “I expect that kind of confusion from the undergrads I used to teach, but I’m surprised to hear it from you. ”
“The Final System is a person,” Norman said calmly. “And I’m frankly insulted that you could believe that I, of all people, could be fooled by a mere panyon.”
Chloe could easily believe that he, “of all people,” could make that mistake.
“Maybe it’s my humanities background,” she said, trying to keep her voice light, “but I’m wary of scientists mistaking a model for reality.
It’s easy to think hardware and software are a good analog for the relationship of brain to mind, but only minds work with meaning.
Computers don’t understand what they produce, any more than the water clock Harun al-Rashid gave Charlemagne understood concepts like ‘time’ or ‘noon’ when its brass horsemen moved in a way its human observers interpreted as meaning the time was noon.
Even the ones and zeros or yeses and noes of computer logic are human-assigned interpretations of what are really just electrical currents being passed through or stopped at a gate.
Any meaning computers generate is observer relative, read in from without. ”
“Thank you for that lesson, Professor,” Norman said. “But the Final System is a brain. An emulation, a perfect digital model down to the neuron, running on the greatest computer array in history.”
“A simulation by definition is not the same as what it simulates,” Chloe said. “A simulated fire is not hot. A simulated mind does not think.”
“An emulated brain,” Norman corrected.
“Whatever word you use, it’s still mimicry, re-creating patterns from the real thing.”
“You think the mind is something mysterious,” Norman said, “but all it is is a set of patterns. Do you know, when we first created a complete digital model of the human brain, it did nothing? But that was to be expected. After all, a real brain with no brain activity isn’t going to win any awards for intelligence.
We had to get the brain fired up, like pulling a lawnmower cord.
So we added code to force the brain to be active, but even with its virtual neurons firing all the time, nothing emerged, no thought, no consciousness.
We prodded it with this or that stimulus, but nothing worked.
We had created a perfect digital model of a brain, but it had no brain state, no pattern.
That’s when Regina had a brilliant idea. ”
“Regina?” Chloe said.
“My partner. Brilliant neuroscientist. She invented a way to copy software into the digital brain. She detected and mapped the firing of neurons in a real baby’s brain and replicated them in real time in the AI brain.
As the baby grew and learned, the System shadowed her.
Basically, the human baby donated her mind to the System. ”
Chloe’s eyes widened. “I hope that was reversible!”
Norman snorted. “Don’t worry: The mind donor is alive and well. Doesn’t even know it happened. The point, Chloe, is that the Final System is now running the same software you and I do.”
“Humans aren’t computers, Andrew,” Chloe said. “We don’t run software.”
“Humans are computers,” Norman said. “There’s no qualitative difference between you and your phone, only a quantitative one. At root we’re no different from any other input-output machine. What goes in determines what goes out. Push the button, get the result. Stimulus and response.”
Chloe said, “I like to think there’s a third variable.”
His eyes closed briefly, and she was willing to bet he’d stifled a sigh, but when he spoke, his voice was measured.
“You mean free will. That’s a comforting idea, certainly.
But the universe has no room for it. All that exists are finite structures like molecules and atoms, bound and circumscribed by behavioral algorithms that we can study and describe.
They can’t act other than they do. And if they can’t, neither can the creatures for whom they’re the building blocks.
Reality is a machine of almost—but not quite!
—infinite complexity, grinding forward to its predetermined end.
Choice is, by definition, something undetermined, so it can’t exist. What did you have for breakfast this morning? ”
Chloe blinked at this apparent non sequitur. “A Bomb Bar.”
“Suppose you’d chosen a different breakfast. Tell me why.”
“I would have had something else if I hadn’t gotten up so late.”
“Ah. But you were in a hurry. So you had the Bomb. Could you really have chosen anything else? It would have taken a change of conditions to make you change your action—a change of input to get a different output.”
“So you’re an input-output machine too?” Chloe challenged.
“Of course.”
“How can you believe that? You make choices every day. You know they’re real.”
“You’re having an emotional reaction,” he said, and Chloe’s jaw tightened.
That he said the words in the same bland, friendly tone only made them more insulting.
“But it’s understandable. Predictable, even.
You’re already running one set of inputs, your prior beliefs, and I’m giving you a new input that contradicts them. You can believe your feelings—”
“They’re not just my—” Chloe began, but he raised his tone sharply to speak over her.
“—or what a bunch of illiterate medievals thought. But I believe what science has shown to be true. That doesn’t mean I think of myself as an automaton, just as I don’t think of this table”—he tapped his fist on it—“as only a collection of atoms. It is a collection of atoms. ‘Table’ is just an idea we impose on it. But it’s convenient for me to think of it as a table, so I do.
It’s convenient to treat choices as free, so I do. ”
“You’re a terrible philosopher,” Chloe said with feeling.
His mouth stretched into his trademark lopsided smile. “But I’m an excellent scientist. And that’s why—” He stopped suddenly, and his eyes unfocused. “Oh,” he said. “Phreak.”