Chapter 8
As soon as Norman spoke, everyone in the conference room also froze, their faces taking on mirrored frowns and blank stares as their eyes focused on the infinity inside their lenses, where streams of messages and notifications cascaded in.
Chloe tried opening the first message, but ten more arrived before she could click it, then still more.
From the snippets she caught as they skidded past her vision, they were all the same: connection errors.
MeNet was unreachable. TransNet asked for patience.
BankNet advised that due to unscheduled but routine maintenance she would be unable to check her balance at the moment.
WasteNet, of all things, wished her to know that the auto-dumpsters were not operating, and until further notice, could she please hang on to her trash bags.
The committee members looked at each other. There was laughter, puzzlement, an edge of fear.
Norman turned away and began delivering crisp, clipped orders, his eyes flicking rapidly around his smartspace.
The messages had stopped cascading in, replaced by a silence that was almost worse.
The other committee members murmured to each other, and one word kept spiking: Cybercrash.
That word, spoken in those fearful tones, brought the hairs on Chloe’s arms to attention.
But it couldn’t be, not now, not in Andrew Norman’s world. Could it?
A cascade of new notifications brought a fresh splash of fear. Chloe half expected to see the terrible words, Hello, World . . . but she heaved a sigh of relief, and the committee members exchanged sheepish smiles. The notifications were all about services restored.
Norman slammed his hands on the table, making them jump.
“I need a quick vote. OverNet just suffered its most serious attack since it was created. The hacker who did it had the help of the Collective, a hacker ring that is very likely a Russian state actor.” His usually cool blue eyes were positively frigid with fury.
OverNet was supposed to be a fortress, Andrew Norman’s golden guarantee of order and stability, and it had just been very publicly attacked.
That the attack had been repulsed was not going to be as important in the minds of the public as the fact that it had happened at all.
“Fortunately, the Final System has an ongoing mandate to defend itself and disable any attacking devices. But once order was restored, it had to stop or exceed its mandate. So I need your approval to let the System connect to LawNet to catch the hacker.”
The committee members looked at each other. “All in favor?” Evans said.
Hands started to rise. Chloe’s did, too, but jerked to a stop as a message appeared in her smartspace, flagged as urgent and so plastered directly atop the center of her vision.
Do Not Let Him Unleash the System!!!!!
The sender was listed as unknown, which should be impossible. Chloe’s eyes were drawn past the message to Grandma. The woman was no longer leaning against the pillar but standing stiff and straight and staring directly at Chloe, body tense, eyes wide and urgent.
“Wait!” Chloe said, blinking the message away. The hands froze. “Before we vote, I need more information.”
Norman put both hands flat on the table and let out a puff of breath. “What information?”
Chloe cleared her throat to buy herself a moment to think. “First, shouldn’t allowing the System on the Nets be something Congress as a whole authorizes?”
Norman frowned. “It’s already . . . Oh, I see.
You’re thinking of when the System launches as an OverNet admin.
Yes, at that point, we’ll need congressional authorization, because we’ll be giving the System administrator privileges, giving it control as well as access.
But what I’m talking about right now is just letting the System connect to a single subnet, LawNet, as a user, not an administrator. ”
“This committee has already authorized the System to connect to the public Nets if that’s helpful for it to carry out its committee-mandated tasks,” Evans put in.
“For instance, it has a standing order to infiltrate the Collective. But it still uses the Nets like a user, the same way you or I do, so its knowledge is limited. And its existing permissions don’t extend to higher-tier nets like LawNet. ”
“What does being on LawNet entail?” Chloe said. “What power would we be giving it?”
“Access to lens feeds and cameras, if they happen to be in proximity to the hacker,” Norman said through gritted teeth. “It could also search communications to see if the hacker is contacting anyone. Basically, I’m asking for a warrant.”
Chloe glanced at Grandma again. The woman’s eyes bored into hers, tense and urgent.
Chloe frowned back. What was wrong with a limited warrant to catch a Russian agent?
Even Chloe thought that was an acceptable use of an AI system.
But Grandma seemed to know something Chloe didn’t.
“You want the Final System to access private information?” Chloe said slowly to Norman.
“Only in defense,” Evans put in smoothly. Chloe was beginning to see his utility to Norman: He could still be blandly political when Norman got frustrated by having to deal with the common plebs.
“Right,” Norman said. “That’s already how it took down the Collective’s botnet: by reaching into each attacking device and dismantling the code that let the Collective control it.
But that was all the System did. It didn’t invade anyone’s privacy, unless you count the malicious code on their phone as part of their privacy rights. ”
“But now you do want to violate those rights.”
“The System will be recalled once its task is done,” Norman said. “Trust me, the government is not in the habit of spying on our citizens.”
Maybe not, but he’d made sure he had that capability. “I am very uncomfortable,” Chloe said, “with the idea of an AI sifting through people’s personal data and making decisions based on that data.”
“Congresswoman,” Norman said, “we’ve been attacked. OverNet was attacked. If we don’t respond, we send a message that we’re vulnerable. We invite them to try again.”
Chloe did understand that, but Grandma was still staring urgently, so Grandma understood something else. Hmm, who to trust, a complete unknown or Andrew Norman?
How about herself? She might not be a computer scientist, but she was an expert in her own field.
If Norman’s System really had an intellect, it also had a will, since the will was just the appetite of the intellect, drawn toward what the intellect perceived as good the same way bodily appetites were drawn toward what the senses perceived as good.
And the thought making her stomach churn—or maybe it was the Bomb Bar—was that an intellect might comprehend that one course of action was good for humanity, but another was good for it.
“If we give your System power,” she said, “how can you be sure it won’t turn on us? ”
Norman opened his mouth, then paused, and she could see him consciously work to compose his features. “Because I built it, so it can’t.”
“See, now you’re acting like it’s different than a person, like it can be programmed. Which puts us right back where we started. Either it’s a dangerous tool or a dangerous person. Either way, it’s too dangerous to make the kind of hasty decision you are pushing hard for us to make.”
“Your illiterate, medieval mumbo jumbo—” Norman began, but Chloe cut him off.
“You know, the great physician-scientist-philosopher Ibn Sina would have been surprised to learn, as he wrote his treatise on human sensation and sprinkled it liberally with quotes from Aristotle, that he didn’t know how to read.
” It wasn’t the first time her expertise as a medievalist had drawn contempt instead of respect, but she could give that contempt right back.
“If you’ve created something with a genuine intellect, then you’ve created a moral agent, someone who can choose how to act.
And you’re asking me to trust this moral agent, whom I’ve never met. ”
The patience was slipping from Norman’s voice. “No, I’m asking you to trust me. It’s written into the System’s code that she can’t disobey me or lie to me, and no amount of medieval casuistry will get around that. So if you can’t trust the creation, trust the creator.”
“Sorry,” Chloe said, “not good enough. I’d like to meet it and decide for myself.”
“No.”
“Oh, so she’s not ready.”
“She is. But until she’s OverNet admin and has all its power at her disposal to protect herself, I will not allow her to interact with anyone she doesn’t need to, to accomplish her assigned tasks.
It’s nothing personal, Congresswoman; it’s standard security protocol to limit access to those with need-to-know. ”
Chloe tightened her grip on her crossed arms. “I can’t trust someone I’ve never met. No meeting, no yes vote.” Line in the sand.
Norman’s eyes bored into hers for a long moment. She could almost feel his fury buffeting against her. And then she could see him bottle it, chill it, put it away. “Very well,” he said. “Mr. Evans?”
Evans cleared his throat and said, “All in favor of allowing the System to go on LawNet and catch the hacker?” He raised his hand.
Three hands followed suit. Four stayed down.
Norman’s eyes widened, then narrowed to slits.
Evans, too, looked surprised. “All opposed?”
Chloe’s hand went up determinedly. Three more hands wavered, and then, with some reluctance, rose. None of their owners looked around or met their fellows’ eyes, except for Jacobs, who caught Chloe’s gaze and gave her an almost imperceptible nod.
Chloe’s face flushed in surprise and pleasure. Norman had hardly expected this when he’d added her to the committee!
Norman looked at each of the dissenters in astonishment, and then his eyes locked on Chloe’s for a long, uncomfortable moment. “This meeting,” he snarled, “is adjourned.” He whirled and stormed out.
Grandma, in the back of the room, leaned slowly against the pillar and closed her eyes.