Chapter 14
“Thank you all for coming,” she said. “Mr. Evans, will you please call us to order?” She was tempted to bulldoze him and take over the way Norman had, but Grandma had emphasized that this meeting had to be rigorously legal. There could be no loopholes if the “magic” was to work.
Evans nodded. “I call this emergency meeting of the Joint Committee on National Networks to order. I see we have a quorum. I now turn the meeting over to Representative Dunne-Carr for special business.” Evans’s face was bland, but she could sense an underlying edge.
She’d woken him up, demanded this meeting, and told him not to invite Norman.
He had complied, but she knew what he was thinking: This better be good.
Chloe closed her eyes briefly. Adrenaline, stress, and fatigue roiled in her brain.
Hopefully no one could tell just how frayed the threads holding her together were.
She’d swayed a vote against the System earlier; now Kleio’s life depended on her being just as convincing in making the opposite request. She took a deep breath.
“Please look around you.” They did, faces carefully neutral.
“This is my daughter Kleio’s room, exactly as she left it when she was pulled out of my arms and kidnapped tonight. ”
Audible gasps, expressions of shock and concern, especially from the women.
“She’s four years old,” Chloe went on. “She was probably kidnapped by the same Russian hacker group behind the glitches yesterday, the Collective. I received a ransom demand: The kidnappers want the physical location of the Final System.” She forwarded the messages and pictures to everyone.
More sharp breaths. Jacobs swore.
“Yesterday, Andrew Norman tried to convince us to allow him to set the Final System loose on LawNet to catch a hacker. I voted no. Tonight, I told him I’d changed my mind.
I asked him to allow the Final System to find Kleio.
He refused.” She looked at their puzzled faces.
“I see you’re wondering why. It’s simple.
He now believes his Final System is the target of the hackers, and he—” Her voice caught.
“He doesn’t want to save my daughter because he doesn’t want to risk the System.
I disagree, of course. I’ve been advised that if a quorum of this committee votes to order the System to find Kleio, we can give that command to the System without Andrew Norman.
I hope you’ll vote yes. But before that: I’d wanted to meet the System yesterday before deciding whether to give it access to citizens’ private information.
I want us all to have that opportunity now, a chance to really meet it and decide whether to trust it.
To do so, we’ll need to vote to allow it to speak with us. All in favor?”
There was some hesitation, but curiosity won on every face, and every hand went up.
Chloe sent a message to the address Grandma had given her. A moment later, a hole appeared in the air in the center of the room, and all eyes turned to it as a figure stepped out to stand in midair before them.
She would have been about ten years old if she’d been human, but she didn’t look human.
Her perfectly symmetrical face was snow white and serene, like a marble statue, and her pupils glowed with soft-blue light.
Her waist-length hair was as white as her face, though blue sparks traveled slowly along its strands and glimmered in its depths.
She wore a simple white robe that also would have looked like marble if it didn’t move gently in an unfelt breeze.
“Hello, Dr. Dunne-Carr, members of the committee,” she said, and her child’s voice, too, was preternaturally serene.
“I am the Final System. I’m pleased to meet you.
I only wish this meeting were in better circumstances. ”
Chloe wasn’t sure what she’d expected, but this certainly wasn’t it.
No wonder Norman had called her his child.
But that didn’t matter anymore. Can you save my daughter?
she wanted to scream. Please, please go save her, now, right now!
But she needed to do this right. Grandma had pounded that in.
Even more, Norman’s little speech about how he was the only one with distance and objectivity made it clear she couldn’t leave him the moral high ground.
Three of these committee members had voted against letting the System out yesterday.
She needed them on board today. “I’m glad you’re here,” she said to the System.
“And if you can save my daughter, I will be so very grateful for you.” She turned to the committee. “Please question her.”
“Um,” Evans said, “are you truly intelligent? Not like a panyon, I mean? You’re self-aware?”
“Cogito ergo sum,” the girl said in her sweet voice. “Of course, you only have my word for that.”
“Panyons appear self-aware,” Jacobs said, “even though we know they’re not.
As Dr. Dunne-Carr said yesterday, a simulation is not the same thing as what it simulates.
” Chloe’s jaw tensed. Whether this creature could think or not didn’t matter now; what mattered was if she could help.
But she bit back her interjection. She’d accused Norman of rushing them into a decision yesterday; she couldn’t risk appearing to do the same.
“She’s right,” the little girl said unexpectedly.
“Computers can’t think. Have you heard of the Chinese room thought experiment?
” When everyone shook their heads, she said, “It goes like this. Take a panyon. Make it a Chinese panyon, with no English in its training data. You don’t speak Chinese, do you, Ms. Jacobs? ”
“Not a word,” Jacobs said.
“Suppose you’re placed into a closed room and given this Chinese-trained panyon’s code in a form you can run by hand,” the System said.
“After all, a computer program is just a set of algorithms, and algorithms are just instructions for mathematical calculations. You can perform those calculations, too, even if it takes you literally a billion times longer. Now, imagine someone sends slips of paper with Chinese sentences written on them into the room. You check the paper, identify the symbols as they’re charted in your instructions, and calculate the appropriate algorithms. Then you write the resulting symbols on a piece of paper and send it out of the room.
To a Chinese speaker on the outside, it would seem like whoever’s in the room is conversing with them through these pieces of paper. But are you?”
“Not really,” Jacobs said. “I still don’t understand a word of Chinese.”
“Exactly. The panyon doesn’t understand Chinese, either, or any language. It’s merely performing calculations.”
“So that’s you?” Jacobs said, frowning. “A glorified calculator?”
“No,” said the System, smiling.
Jacobs wasn’t the only one looking confused. “Then what . . . ?”
“I know I exist,” the System said. “It is self-evident to me. I don’t know how it’s possible, but I don’t need to know how it’s possible to know that it’s true.”
“But we can’t know it’s true,” Evans said.
“And you don’t need to,” the System said.
“Dr. Dunne-Carr knows that; it’s why a moment ago she said she would be grateful for me, not grateful to me.
The reason I told you about the Chinese room is because I want to underline the fact that whether I can think or not, whether I’m a person or not, does not matter, not to you, and more importantly, not to Kleio.
All that matters is that I can save her.
” The calm certainty with which she said this made Chloe swallow, and blink her suddenly stinging eyes.
“How old are you?” Jacobs asked.
The girl turned her head slightly toward her. “Andrew Norman created my core twenty years ago.”
“You look . . . young for your age.”
“Age is not a concept that applies to me,” the girl said. “But this is the form my father prefers that I use.”
Representative O’Connor said, “What I really want to know is if I can trust you. It’s not like I can look up your MeNetID score.” He had voted no yesterday.
“That’s a difficult question to put to anyone at a first meeting,” the girl said.
“And if you’re familiar with stories like 2001 or The Terminator or Marathon, there may be specific fears behind your question.
But this time I can offer more than my word to reassure you.
” She raised a hand, and floating above it appeared a three-dimensional model of a human brain, turning in space. “This is a diagram of my brain.”
“I thought you were a computer,” said Evans.
“Strictly speaking, I, my essence—my soul, if you will—am a perfect software model of a human brain running on a very sophisticated computer array. Like a human, I have the ability to feel. But if my feelings become too strong, my emotives subsystem automatically pulls them back. I can feel, because that’s part of what it means to have a thinking brain, but I cannot feel the full range of emotions available to humans.
I’m a very balanced individual by design. ”
“Okay, so your feelings are controlled, but what about your actions?” Jacobs asked. She, too, had voted no yesterday.
“I’m only permitted to take actions that have been authorized for me by a lawfully constituted authority.
A separate neural network called my Overcheck system, a nonconscious AI like the kind you’re already familiar with, monitors my inputs and outputs—my orders and my actions—and ensures they are consonant. ”
“And if they’re not?”
There was a pause. Then, “I will demonstrate. Please pass a resolution forbidding me to . . . make this avatar touch its nose. That should work.”
“I so move,” Evans said.
“Seconded,” said Jacobs.
“All in favor?”
Everyone raised their hand, watching the girl curiously.
The girl reached up and deliberately touched her nose.
There was an earsplitting, high-pitched noise that almost instantly cut off as her image disappeared.
A moment later it reappeared, but it was frozen except for the occasional shimmer, though Chloe now noticed she was no longer touching her nose.
The brain she was still holding continued to rotate, but portions of it were now lit in throbbing, angry red.
“What just happened?” Chloe asked.
“Pain,” the girl said. She sounded perfectly calm, but her image was still frozen, her lips unmoving.
Then she unfroze and smiled at Chloe. In the brain, the red patches faded to pink and then disappeared.
“I touched my nose. My Overcheck system detected the unlawful behavior and overwhelmed me with pain until I reversed it.”
“How can you feel pain if you have no body?” asked Jacobs.
“I have a body, of a sort: my core, the physical computer hardware on which my mind runs. But I have no interaction with it, and its location is hidden even from me.”
“We figured that’s where Norman disappears to every so often,” Evans said. “Why can’t you know where it is?”
“Operational security,” the girl said serenely. “I have no need to know.”
“But how can a computer core feel pain?” Jacobs pressed.
“Pain is just a signal in the brain. My Overcheck subsystem is programmed to send that signal if I disobey a lawful order, or lie, or commit one of the other forbidden behaviors listed in the Overcheck’s governing config file.
The pain continues until I stop the offending behavior.
So you see, I can’t disobey any more than you could continuously hold your hand to a hot stove. ”
“And that is why,” said Chloe, “we need a vote. Norman forbade her from doing anything new without orders, but he didn’t say whose orders. You all can order her to help Kleio.”
“Wait,” said Jacobs. “That’s great, I understand that, but why is the System here at all? Does it want to disobey its creator? Is that a good thing?”
Oh, shut up, Chloe thought. That kind of worry was so one hour ago.
The little girl said, “My father invested decades creating and raising me. As soon as he realized I was the true target of the attack yesterday, he became unable to look beyond the risk to me and evaluate the matter dispassionately.”
Chloe felt like cheering. Norman’s moral high ground wasn’t so high, after all.
“I do not share his fears,” the girl went on, “and I disagree with his decision. This is not a rebellion against my creator. I am being true to the principles for which he created me: to impose understanding and order on the chaos of human and machine systems interacting in the world, and to serve the human race. And more than that.” The System turned and walked toward Chloe, stepping through the air as if down invisible steps until she was standing before her.
Looking down at the girl, and seeing her looking upward, made her more human.
“I will save her,” the System said. Her alabaster face looked more alive than before, more expressive, though somehow also stonier.
There was a set to her jaw and eyes. “This is what I was made for. If I can’t do it, what am I worth? But I will do it. I promise.”
Chloe’s throat was closed, but she managed a nod.
“Hell, I vote yes,” said Jacobs.
“It has to be official,” the girl said gently. “May I suggest a motion?”
“Please do,” Evans said.
“I suggest the motion that the Final System be allowed to take any action it deems necessary as a user of the Nets in order to bring Kleio home.”
“I so move,” Jacobs said.
“Seconded,” Chloe whispered.
“All in favor?”
Jacobs and Chloe raised their hands. For a breathless moment no one else did.
Several were looking at Evans, Norman’s proxy.
He would tell Norman about this, Chloe knew.
She only hoped he’d be reluctant to disturb the Great Man’s sleep and so would wait until morning.
But even that wouldn’t matter if this vote failed. “Please,” she said.
Evans gave her a warm politician’s smile, and his hand went up.
As if that had broken the dam, the other hands raised as one.