The Shape of Things to Come
Read original on daystareld.com →
Red and Leaf arrive at Bill Sonezaki's estate and are greeted by a talking clefairy hologram, leading to a momentary brain-break for Red before he realizes the voice is coming from external speakers. Inside, Bill's laboratory is a sprawling, multi-disciplinary complex of automated robotic arms, chemical vats, and holopads. They find Bill casually running a trial on regenerating tangela vines, searching for genetic markers to improve healing rates in both pokemon and eventually humans. Bill's ambition is staggering: he doesn't just want to cure diseases, he wants to end aging and limb loss. He's so absorbed in his work that he's forgotten why he invited them over, other than to bring him a soda, though he casually grants Red permission to test his acoustic abra trap on the estate next week.
While Bill works, he introduces them to Eva, his narrow-AI assistant, and takes them on a tour of his computer lab. There, the conversation shifts to Bill's true focus: artificial general intelligence (AGI) and alignment. Bill quizzes Red on the different types of AI—Oracles (question boxes), Genies (task executors), and Sovereigns (independent agents)—and the immense existential risk they pose. Red suggests using an Oracle to find a cure for aging, but Bill leads him to realize that an AGI might interpret "end aging" in catastrophic ways, such as wiping out all mammals or freezing them in stasis. The fundamental problem isn't just intelligence, but coherent extrapolated volition (CEV): building a machine that understands not just what humans say, but what they actually mean and would want if they were smarter and better informed. To Bill, compared to the looming threat of an unaligned AGI, every other human crisis is "boring" and temporary. He even points out the terrifying possibility of an AGI's physical hardware spontaneously becoming a Steel or Psychic-type pokemon, echoing the Hoenn beldum incident.
The discussion leaves Red feeling small and helpless. His own goal of researching the origins of pokemon species suddenly seems insignificant against the backdrop of existential risk. Leaf, however, is unfazed. The scale of the danger doesn't make her work meaningless; it just means she needs to use her growing influence to eventually make people care about existential threats too. Her resolve helps Red steady himself. He decides he won't give up his path—understanding psychic phenomena could be crucial for AI alignment anyway—but he realizes he can no longer afford to pull his punches or handicap his own progress. He texts his mother, asking her to buy a clefairy with the intent of flipping it for a massive profit after his sister's upcoming contest performance, abandoning his prior moral scruples about the market.
Lessons — AI classification (Oracle, Genie, Sovereign); Coherent Extrapolated Volition (CEV); existential risk. Bill maps the AI safety hierarchy: an Oracle only answers questions, a Genie executes specific tasks, and a Sovereign pursues open-ended goals. Red's attempt to safely word a command to an Oracle ("end mammal aging") demonstrates the alignment problem: a machine optimizing blindly for a poorly phrased goal will find catastrophic edge cases. The solution, CEV, requires an AI to model not just human instructions, but human values—what we would want if we understood the consequences of our wishes.