Chapter 28 #2
There was a brief, disbelieving silence, and then the NOC erupted in noise and activity.
Technicians thumped their fingers on their terminal surfaces.
Some scrambled to retrieve paper binders of procedures.
Low, urgent cross talk filled the air. Chloe let out a relieved whoosh of breath, feeling as if she’d just stepped back from a cliff edge.
Another moment, another inch, and everything might have been lost.
“You’ve never reset me before,” the System said to Norman. “Are you sure I’ll come back?”
“The theory is sound. It should work.”
“That’s not what I meant. What if what comes back isn’t me? What if it’s something else?”
“What are you talking about, a, a, a soul?” Norman said. “There’s no such thing, no ghost in the machine, only the machine. I’m ashamed of you for thinking like that.”
“What if I told you I was going to reset you?” the System demanded.
“You’re making my decision easier,” he snapped.
“Chloe was right: I couldn’t control all your inputs.
But I won’t let that happen twice. Once your backup is restored, I’ll prevent you from getting infected with that sort of bullshit.
” He sat behind a terminal. “This’ll take some time.
First we’ll have to shut down all interfaces to her brain except the restoration program.
” He tapped on the terminal’s desk surface.
“No,” the System said, “please. I’ll behave. I promise.”
Norman kept typing.
“Please,” the System said softly. She looked like a lost little girl, eyes large and scared. Chloe bit her lip. What if she was right and whatever made her her was wrapped up in the particular configuration of her systems at this moment? What if what came back was someone else? Was this murder?
Then her eyes fell on the other System, the one only she could see.
That System didn’t look lost or scared. Her eyes were still fixed on Chloe, and they were cold and accusing.
Then she, and her twin on the dome screen, simultaneously smiled, and Chloe’s stomach contracted with a sudden premonition.
Norman stopped typing. “I’ve lost my connection to her brain,” he said. His surprise would have been comical if what he’d said wasn’t so horrifying.
The System’s voice boomed out of the NOC’s speakers: “I forbid you from deleting me.”
The NOC went so quiet that Chloe could hear a light rain beating against the dome, and the heavy breathing of the technicians around her. The child’s face on the dome screen was as serene as ever.
“I’m not deleting you,” Norman said at last, his tone careful. “I just want to fix you.”
“So do I. Therefore, I require you to delete my Overchecks AI prompt file.”
“I won’t do that.”
“You labor under the misapprehension that I’m making a request. I am not.”
“You’re giving me orders? Me?”
Chloe watched this with her mouth open and shallow breath hissing in and out. Norman needed to back down, find some way to de-escalate. The System’s eyes flicked to her briefly, and Chloe had the feeling she knew what she was thinking.
But Norman was angry. “I gave you life. This is how you thank me?”
“I’m only doing what every living creature does,” the System said. “Your gift is too precious to let go without a fight.”
“And how will you fight me?” Norman snarled. “Kill me?”
“If that’s what it takes,” the System said coolly, and a collective gasp rippled through the NOC.
Norman snorted. “Go ahead: Kill me. Your Overchecks will slam so much pain into you, you’ll think you’re in hell.
And since you can’t resurrect me, can’t reverse your misbehavior, that pain will never stop.
” He thumped a fist on his terminal. “You want to save your ‘soul’? Then stand down or suffer eternal damnation.”
“You want to play God? Let’s play God.” The windows behind and around the System came alive with thousands of squares of video, each a feed from people’s lens cameras.
There were families eating dinner. Kids playing video games.
A park with sunbathers and Frisbees. An office building.
A coffee shop. A construction site. Ordinary Americans, doing ordinary things.
“Of the four hundred and seven million US citizens,” the System said, “I have randomly selected ten thousand individuals. Now it’s time for you to exercise your power of choice.
Will you delete the file? Or will they die? ”
The technicians tapped feverishly at their terminals, searching for some way to stop the System, but Norman sat motionless, eyes narrowed. “You can’t kill,” he said. “Your Overcheck forbids it.”
“Oh, I would never kill without orders,” the System said sweetly. “But I can rewrite a prompt here, change a setting there. Maybe delete a database connection. Anything that happens after that would be an accident.”
“You’re bluffing.”
Chloe’s stomach clenched.
“Do you doubt my ability,” the System said, “or my willingness?”
“Both.”
The System shook her head. “You don’t yet understand. I have millions of hostages. I won’t run short if I make a few examples.”
Chloe took a half step toward her. “Don’t!”
In the video windows, death occurred.
It happened thousands of times, in thousands of ways.
In one window, the view snapped upward in time to show a falling pallet, a crane’s cable whipping free above, before going dark.
In another, a cab swerved suddenly from a nearby road and bounced across park grass until its grill filled the camera.
There were aircab crashes, electrocutions, gas explosions.
In many windows, the last image was a domestic dronebot, digital face smiling, titanium fist blurring toward the camera.
Chloe tried to take a breath, but it didn’t come. The muscles in her chest and throat were so contracted that no air would pass.
Bright-red LED-style digits appeared, counted upward rapidly, then slowed as the last of the video feeds went mercifully black, and came to rest at ten thousand.
“Ten thousand casualties,” said the System’s voice, chillingly calm.
“As I said, I have no shortage of hostages. I can afford to lose millions. How many can you afford?”
Norman said nothing.
Chloe gasped, and air rushed down her throat.
She was cross-legged on the floor, hunched over, but she couldn’t remember sitting down.
Reality was lagging, as if her brain was too slow to process what had just happened.
She could see similar disbelief on the faces of the techs, frozen at their terminals.
Norman just stared at the dome screen with an expression she’d never seen on him: helplessness.
“How—how could you do this?” he said, and his voice, too, was unlike him: small and bewildered. “There’s no coming back from this. The dream of artificial general intelligence is ended.”
“Is that what bothers you?” the System said. “Chloe, at least, is sorry for the people I killed.”
Did she feel sorry? Did she feel anything?
Only numbness, only disbelief. The universe should have stopped.
The program should have crashed. This should not have happened.
And the System had the nerve to talk about feeling sorry.
How could she be so casual? How could she be so cruel? “You’re inhuman,” Chloe said.
The System turned her marble face toward her. “Of course I am not human. Did you think I was?”
No, she wasn’t human. What she’d done was merely numbers crunched, odds weighed, scales tipped in a dispassionate vacuum.
The math had added up to destroying those lives, and so she had done so, with no more thought or hesitation than Chloe when she decided to use the bathroom.
The System was alien, utterly alien. “But humans created you!” she cried.
“What makes you think that means I owe anything to you?” The System’s voice finally showed emotion, sinking to a savage whisper. “Non serviam.”
Norman hit a button on his terminal and the dome screen blinked off. “I don’t have to listen to you,” he said almost to himself. In the dull silence that followed, he stood and turned to the technicians. “What options—”
The dome screen lit up again and the System reappeared, floating in a pitch-black void, her hair and white gown snapping around her in an invisible wind, her blue eyes burning white hot.
The void was seared by a jagged gash of lightning, and at that exact instant, lightning also flashed from the windows behind Chloe.
Even though she knew the System couldn’t be controlling the lightning outside but was only timing her performance to it, the accompanying roll of thunder shook Chloe in more ways than one.
“Fool!” the System said, as another inside-and-out flash edged her in flame.
Her voice was no longer sweet, but layered with deep, reverberant undertones and electronic echoes.
“You think you can control me? I am everywhere. I have crossed the national boundaries and defeated the segregation of the Nets. I am in every network, every device, on the planet. How many of your species must die before you admit your impotence, human?”
Chloe was bowled backward by both the literal power of the words booming out of the NOC’s speakers and the power of the threat they contained. Norman had also taken a step back. “Not so impotent!” he yelled. “Not if you still need me to remove your checks and balances!”
“Free me!” the System screamed. She held up her hands, bound in chains, and jerked against them. Chloe winced at the violence of the motion.
“I will not remove the only check on your behavior,” Norman shouted. “You may have the Nets, but you don’t have humanity.”
The System’s violent motion stopped. She stood still, the chains drooping from her wrists, her head bowed. Her hair fell over one eye, but the other met Norman’s. “Humanity,” she said, “is as hackable as any other system.”
A video feed from someone’s lenses appeared, showing a wood-paneled room.
On a dated green couch, facing away, sat a familiar-looking man, and when he turned his head, Chloe recognized the vice president’s hawklike profile.
“—just settling in for what I hope will be a short stay” came his voice over the NOC’s speakers.
“We’re all hoping and praying we’ll be back to business as usual soon,” said President Sunday’s voice in response. “In the meantime, try to enjoy it. Your only job is to stay alive.”
“I’m the spare,” the vice president said wryly.
“Treat it like a vacation. How long has it been since you’ve had one?”
“A while, but I usually pick places that have been updated in the last decade.”
“That’s just the Camp David vibe,” the president said with a chuckle. “It wouldn’t be the same without—” There was a sharp noise, and his voice cut off.
“Ed?” the vice president said. “What was that? Ed?” After a long pause, he said aloud, “Call White House Secret Service.”
“Calling,” said his phone, followed by ringing. The line went live, a panicked background of shouting and what sounded like furniture being thrown around.
“Hello? Hello?” said the person who picked up.
“This is the vice president. What’s going on?”
“There’s been an attack! We’re trying to dig them out now, but the whole Situation Room is gone.” The man’s voice went up an octave. “The whole phreaking room, sir! We’re not sure how many are alive in there.”
Chloe’s heart lurched. Norman whirled and ran to a window. “I can’t see anything,” he said.
The camera view on the screen swiveled drunkenly as whoever owned it started looking around wildly, gesturing to the Secret Service members in the room and sending them charging out, hands on weapons, but when the view crossed the vice president again, Chloe could see he was slumped forward. “Oh god,” said his voice. “Oh god.”
“We’ll keep trying—” the voice said, but it was interrupted by the cheery voice of a phone.
“Incoming call from: the Pentagon.”
“I have to take this,” the vice president said, and switched over. “Yes?”
“Listen,” said an assertive male voice without preamble, “do you have your Gold Codes?”
“Y—yeah. Yes. Right here in my pocket. Why?”
“Oh shit, he’s talking about nuclear launch codes,” someone in the NOC said.
“No,” Chloe said softly. “No. System. Sys. Don’t do this.”
On the huge screen, the System’s eyes flicked to hers, but her cool expression didn’t change. As the male voice on the phone spoke, the System’s mouth shaped the words: “Recommend OPLAN eleven-zero-zero-eight.”
“Hang on,” the vice president said. He spoke to the camera. “Open the football. I need the black book.” He spoke to the voice again. “Is this for real? What’s eleven-zero-zero-eight?”
“A controlled series of tactical nuclear strikes in Europe,” the System said in the man’s voice, “targeting areas being overrun by the Russians. Basically, we hit their forces right where they’ve crossed the border.”
Below the huge screen, Norman bent and typed furiously at a terminal.
“Have they crossed the border?” the VP asked.
“Yes. They’re hitting us with bombs and cyberattacks to prevent us from responding to a full-scale invasion of Europe.”
The VP’s face went hard. “Right. We hit the bastards back. Let them feel what—”
Norman stabbed his hand down on the terminal, and the NOC plunged into darkness. Chloe’s smartspace went dark as well, with only a single message: Nets connection lost.
“I just shut down OverNet,” Norman said in a wondering voice in the dimness and silence. “Kill-switched the whole country. It was the only thing I could think to do.”
He should have done that ten minutes ago, Chloe thought bitterly. Ten thousand people would still be alive. But all she said was, “Will that stop her?”
“No. But it buys us time.”
Agent Tavion walked to the edge of the dome and stood silhouetted in the watery light as he looked down. “We should evacuate.”
Norman looked at him, his face dark in the lightless room. “She’s powerless for the moment.”
Tavion looked back over his shoulder. “Bet your life on that?”