Chapter 2 #3
“What’s on my mind is a story.” Standing had been a mistake.
Chloe’s knees were vibrating, and she had the urge to hunch away from all those eyes.
She straightened her shoulders and back instead.
“The story of a girl named Jasmine. She’s twenty-four.
Bright, the first in her family to go to college.
Goes to Santa Clarita U, where I taught.
She’s studying to become a paramedic. She works evenings as a waitress in her neighborhood, where the restaurants can’t afford dronebots, to help with bills.
” Chloe’s voice stabilized, because this was a speech she’d given a dozen times before, and because even though this was a speech she’d given a dozen times before, she still cared about it, still felt the meaning of each sentence as she said it.
“When Jasmine applies for an apartment in a less dangerous neighborhood, her application is denied—not by a person, but by a scoring system that flagged her as high risk. Why? Maybe her inconsistent gig income. Maybe a past-due phone bill. She never even sees the report.”
She paused to let that sink in and to take a breath.
The ballroom was silent, all eyes still on her.
Some of them belonged to NewsNet reporters, staring with the blank, fixed intensity of people keeping their lens cameras carefully focused.
For better or worse, she was on the national stage in a way she’d never been, not even when giving this speech before Congress.
This, this was news, because it involved Andrew Norman.
She glanced down at “Grandma,” who was watching with a faint smile.
“She applies for a paramedic internship,” she continued.
“Her résumé is scanned and rejected before a human eye lands on it. Maybe it’s her zip code, which, remember, she just failed to change.
Or maybe it’s because she struggles with depression.
Oh, that fact isn’t divulged to other systems, but her insurance app’s stress tracker labels her noncompliant, and that is considered revealable, and revealing, information.
But Jasmine isn’t noncompliant—she just doesn’t open the app anymore.
It only ever told her to hydrate, sleep, and practice mindfulness, whatever the hell that means.
“Then her trust scores begin to drop. Maybe it’s that one rant on a bad day that got downvoted to oblivion on MeNet.
Maybe it’s the Spanglish in her posts, language patterns which, regardless of their content, root her in a statistical subset of the population the AI pattern-matches with untrustworthiness, even though it’s supposedly trained to never directly consider race.
Or maybe it’s her friends’ friends, guilt by degrees of association.
There’s no way to know. But a feedback loop begins.
A lower trust score reduces her visibility.
That impacts her ability to find better jobs.
Which affects her financial stability. Which gets her flagged by BankNet.
When BankNet adjusts her microcredit limits, that ripples back through RentNet and MedNet, reducing her score again.
No one sees it. No one questions it. But each system reinforces the others in a feedback loop that ensures she never rises above her current circumstances.
And Jasmine’s one of the lucky ones, because she’s managed to escape law enforcement attention—so far.
She has to be very, very careful not to draw that particular eye, because she’s seen the kind of suspicion and repercussions even a small offense can bring down on someone like her.
“Because Jasmine’s not being judged by her character.
She’s being judged on invisible criteria by nonhuman entities in a way that’s opaque to the very humans who accept their judgments.
If those AIs make a mistake, she has no recourse.
She can’t appeal to an algorithm. There’s no courthouse for machine-made decisions.
“AI, we’ve been told, is a black box. Somehow we all accept this.
Well, I don’t. Because when we don’t know what these systems value, what they see, what they miss, they stop being tools and become silent laws.
And so, Dr. Norman, I’m very skeptical that a new algorithm, an even bigger and blacker box, is in any way a solution to the problems experienced by Jasmine and millions of her fellow Americans every day. ”
She stopped. In the heavy, embarrassed silence, she was reminded of teaching undergrads, of tying the threads of a lecture together to highlight the theme only to be met with half-bored, half-defiant expressions as they failed to take the point. Then Andrew Norman cleared his throat.
“But the systems are here,” he said. “Is your solution to ask people to give them up? Because they won’t. It’s not in human nature to give up conveniences. Though you’re welcome to start a new stamper, if you like. Call it the Luddites, perhaps.”
Laughter rippled through the ballroom, and Chloe’s teeth clicked tight. “I’d rather be a Luddite,” she said through them, “than a technophiliac high priest so out of touch with the people that he thinks his ‘systems’ can fix human nature.”
Norman ran a hand through his white beard. “You clearly want to make this personal,” he said in a quiet voice that his microphone amplified to fill the ballroom. “I’m sorry you feel the need to do that. Because I agree with you.”
Chloe had been opening her mouth to retort, but at this, the words she’d been about to fling at him died, turning her expression into a gape.
“OverNet and MeNet addressed the problems that led to the Crash,” Norman said to the room at large.
“But as the congresswoman pointed out, by solving these problems, we’ve created new ones.
I purposefully avoided creating a social capital system like China, but that didn’t stop the open market from creating trust-check algorithms, because people want to know who’s trustworthy.
Those and all the other systems are black boxes.
How they operate, how they interact with each other, and the feedback loops they create are opaque to us, not because we designed them that way, not because we’re hiding anything, but because they’re so complex that we literally can’t comprehend them.
” Norman’s voice had settled into a tone he used often, one that combined a kindly storyteller with the sharpness of absolute certainty.
“There’s a whole subfield of computer science generating thousands of papers a year just trying to figure out how these AI models work internally, not to mention how they interact with other systems. But any understanding we gain is either painfully general or applicable only to a single, post hoc chain of causation that will never recur.
As our world has grown more complex, we’ve offloaded our understanding of it to systems which can’t return that understanding to us. ”
“And your solution is an even more complex system?” Chloe demanded.
“Yes,” Norman said simply. “A system that combines the best qualities of computers—the ability to see and process vast amounts of information—with humanlike intuition and understanding. A system that can both process all that complexity and make sense of it, draw out its meaning.” He paused and looked with a knowing half smile across all the intervening heads and directly into Chloe’s eyes.
“A system,” he said deliberately, “that can tell us why.”
Chloe reeled. She’d been set up. This was a trap, a premeditated trick to neatly maneuver her into speaking up so Norman could turn her story against her to support his own points.
She managed to sputter an incredulous “And you think you can build that system?” The question should have been enough; it should have been obvious to everyone that he was talking about a level of control that was impossible, that it was arrogance even to imagine.
“I already have,” Norman said. “The Final System.” She could hear the capital letters. “It’s the result of two decades of work. It will allow us to at last bring about an age of peace and justice. I promise.”
Applause, riotous applause. Because Andrew Norman kept his promises. Chloe wanted to protest. She wanted to point out how dystopian the term Final System sounded in her ears, especially the way he phrased it: not his final system but the Final System. The Last One.
But even as the thought occurred to her, she saw herself through the applauders’ eyes.
She was the outsider, the one who’d challenged the Great Man and would be swept away by history like all the others who’d predicted his failure.
“He had an outsize impact on his age,” they would say of Andrew Norman in future History 101 textbooks.
Impact. Funny how the word for making history was the same as for an asteroid strike.
Great Men left an impression on the world, all right: They left a crater.
She looked at the old woman, but she was gone, as if she’d never been there in the first place.
Who was she, some dark antiversion of a fairy godmother?
Someone who shows up when you think you’re at your lowest point and, instead of remaking you into royalty, tells you to be even more the person you already are—someone nobody cares about, someone who can’t make a positive change no matter how hard she tries? Chloe sank into her chair.
Marcus had worried she wasn’t ready for politics.
“You’re too nice in some ways and not nice enough in others,” he’d said.
When she’d won the election, there’d been a breathless moment when she wasn’t sure he’d follow her here, when he might be the one leaving, but he had come.
And now he was a work-at-home dad, teaching summer classes in VR, and when Kleio had asked why Mommy was gone during the day, he’d said, “Mommy is a busy and important woman.” He probably hadn’t meant it the way Chloe had taken it, as a subtle hint that she was misplacing her priorities.
But when she’d won this seat in Congress, he’d told her, “I only hope the Chloe who finishes her term is the same Chloe who started it. The Chloe I know.”
So far her busy and important political career had consisted of sitting in an office, wishing she had more to do.
She had no friends, no allies, no influence, no committee roles, no chance to do anything except show up, cast her pointless votes, and, after two years, go home and pick up teaching, where she would fail to influence her students.
But hey, she’d still be that same powerless Chloe she’d been when she started, so Marcus should be happy.
Her eyes landed on “Grandma” at the far end of the banquet hall, in a congratulatory group gathering around Norman. The old woman was looking back over her shoulder at Chloe, and her lips were curved slightly in a small, satisfied smile.