Twelve Virtues of Rationality

The first virtue is curiosity. A burning itch to know is higher than a solemn vow to pursue truth. To feel the burning itch of curiosity requires both that you be ignorant, and that you desire to relinquish your ignorance. If in your heart you believe you already know, or if in your heart you do not wish to know, then your questioning will be purposeless and your skills without direction. Curiosity seeks to annihilate itself; there is no curiosity that does not want an answer. The glory of glorious mystery is to be solved, after which it ceases to be mystery. Be wary of those who speak of being open-minded and modestly confess their ignorance. There is a time to confess your ignorance and a time to relinquish your ignorance.
The second virtue is relinquishment. P. C. Hodgell said: “That which can be destroyed by the truth should be.” Do not flinch from experiences that might destroy your beliefs. The thought you cannot think controls you more than thoughts you speak aloud. Submit yourself to ordeals and test yourself in fire. Relinquish the emotion which rests upon a mistaken belief, and seek to feel fully that emotion which fits the facts. If the iron approaches your face, and you believe it is hot, and it is cool, the Way opposes your fear. If the iron approaches your face, and you believe it is cool, and it is hot, the Way opposes your calm. Evaluate your beliefs first and then arrive at your emotions. Let yourself say: “If the iron is hot, I desire to believe it is hot, and if it is cool, I desire to believe it is cool.” Beware lest you become attached to beliefs you may not want.
The third virtue is lightness. Let the winds of evidence blow you about as though you are a leaf, with no direction of your own. Beware lest you fight a rearguard retreat against the evidence, grudgingly conceding each foot of ground only when forced, feeling cheated. Surrender to the truth as quickly as you can. Do this the instant you realize what you are resisting; the instant you can see from which quarter the winds of evidence are blowing against you. Be faithless to your cause and betray it to a stronger enemy. If you regard evidence as a constraint and seek to free yourself, you sell yourself into the chains of your whims. For you cannot make a true map of a city by sitting in your bedroom with your eyes shut and drawing lines upon paper according to impulse. You must walk through the city and draw lines on paper that correspond to what you see. If, seeing the city unclearly, you think that you can shift a line just a little to the right, just a little to the left, according to your caprice, this is just the same mistake.
The fourth virtue is evenness. One who wishes to believe says, “Does the evidence permit me to believe?” One who wishes to disbelieve asks, “Does the evidence force me to believe?” Beware lest you place huge burdens of proof only on propositions you dislike, and then defend yourself by saying: “But it is good to be skeptical.” If you attend only to favorable evidence, picking and choosing from your gathered data, then the more data you gather, the less you know. If you are selective about which arguments you inspect for flaws, or how hard you inspect for flaws, then every flaw you learn how to detect makes you that much stupider. If you first write at the bottom of a sheet of paper, “And therefore, the sky is green!”, it does not matter what arguments you write above it afterward; the conclusion is already written, and it is already correct or already wrong. To be clever in argument is not rationality but rationalization. Intelligence, to be useful, must be used for something other than defeating itself. Listen to hypotheses as they plead their cases before you, but remember that you are not a hypothesis, you are the judge. Therefore do not seek to argue for one side or another, for if you knew your destination, you would already be there.
The fifth virtue is argument. Those who wish to fail must first prevent their friends from helping them. Those who smile wisely and say: “I will not argue” remove themselves from help, and withdraw from the communal effort. In argument strive for exact honesty, for the sake of others and also yourself: The part of yourself that distorts what you say to others also distorts your own thoughts. Do not believe you do others a favor if you accept their arguments; the favor is to you. Do not think that fairness to all sides means balancing yourself evenly between positions; truth is not handed out in equal portions before the start of a debate. You cannot move forward on factual questions by fighting with fists or insults. Seek a test that lets reality judge between you.
The sixth virtue is empiricism. The roots of knowledge are in observation and its fruit is prediction. What tree grows without roots? What tree nourishes us without fruit? If a tree falls in a forest and no one hears it, does it make a sound? One says, “Yes it does, for it makes vibrations in the air.” Another says, “No it does not, for there is no auditory processing in any brain.” Though they argue, one saying “Yes”, and one saying “No”, the two do not anticipate any different experience of the forest. Do not ask which beliefs to profess, but which experiences to anticipate. Always know which difference of experience you argue about. Do not let the argument wander and become about something else, such as someone’s virtue as a rationalist. Jerry Cleaver said: “What does you in is not failure to apply some high-level, intricate, complicated technique. It’s overlooking the basics. Not keeping your eye on the ball.” Do not be blinded by words. When words are subtracted, anticipation remains.
The seventh virtue is simplicity. Antoine de Saint-Exupéry said: “Perfection is achieved not when there is nothing left to add, but when there is nothing left to take away.” Simplicity is virtuous in belief, design, planning, and justification. When you profess a huge belief with many details, each additional detail is another chance for the belief to be wrong. Each specification adds to your burden; if you can lighten your burden you must do so. There is no straw that lacks the power to break your back. Of artifacts it is said: The most reliable gear is the one that is designed out of the machine. Of plans: A tangled web breaks. A chain of a thousand links will arrive at a correct conclusion if every step is correct, but if one step is wrong it may carry you anywhere. In mathematics a mountain of good deeds cannot atone for a single sin. Therefore, be careful on every step.
The eighth virtue is humility. To be humble is to take specific actions in anticipation of your own errors. To confess your fallibility and then do nothing about it is not humble; it is boasting of your modesty. Who are most humble? Those who most skillfully prepare for the deepest and most catastrophic errors in their own beliefs and plans. Because this world contains many whose grasp of rationality is abysmal, beginning students of rationality win arguments and acquire an exaggerated view of their own abilities. But it is useless to be superior: Life is not graded on a curve. The best physicist in ancient Greece could not calculate the path of a falling apple. There is no guarantee that adequacy is possible given your hardest effort; therefore spare no thought for whether others are doing worse. If you compare yourself to others you will not see the biases that all humans share. To be human is to make ten thousand errors. No one in this world achieves perfection.
The ninth virtue is perfectionism. The more errors you correct in yourself, the more you notice. As your mind becomes more silent, you hear more noise. When you notice an error in yourself, this signals your readiness to seek advancement to the next level. If you tolerate the error rather than correcting it, you will not advance to the next level and you will not gain the skill to notice new errors. In every art, if you do not seek perfection you will halt before taking your first steps. If perfection is impossible that is no excuse for not trying. Hold yourself to the highest standard you can imagine, and look for one still higher. Do not be content with the answer that is almost right; seek one that is exactly right.
The tenth virtue is precision. One comes and says: The quantity is between 1 and 100. Another says: the quantity is between 40 and 50. If the quantity is 42 they are both correct, but the second prediction was more useful and exposed itself to a stricter test. What is true of one apple may not be true of another apple; thus more can be said about a single apple than about all the apples in the world. The narrowest statements slice deepest, the cutting edge of the blade. As with the map, so too with the art of mapmaking: The Way is a precise Art. Do not walk to the truth, but dance. On each and every step of that dance your foot comes down in exactly the right spot. Each piece of evidence shifts your beliefs by exactly the right amount, neither more nor less. What is exactly the right amount? To calculate this you must study probability theory. Even if you cannot do the math, knowing that the math exists tells you that the dance step is precise and has no room in it for your whims.
The eleventh virtue is scholarship. Study many sciences and absorb their power as your own. Each field that you consume makes you larger. If you swallow enough sciences the gaps between them will diminish and your knowledge will become a unified whole. If you are gluttonous you will become vaster than mountains. It is especially important to eat math and science which impinges upon rationality: Evolutionary psychology, heuristics and biases, social psychology, probability theory, decision theory. But these cannot be the only fields you study. The Art must have a purpose other than itself, or it collapses into infinite recursion.
Before these eleven virtues is a virtue which is nameless.
Miyamoto Musashi wrote, in The Book of Five Rings:
“The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means. Whenever you parry, hit, spring, strike or touch the enemy’s cutting sword, you must cut the enemy in the same movement. It is essential to attain this. If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him. More than anything, you must be thinking of carrying your movement through to cutting him.”
Every step of your reasoning must cut through to the correct answer in the same movement. More than anything, you must think of carrying your map through to reflecting the territory.
If you fail to achieve a correct answer, it is futile to protest that you acted with propriety.
How can you improve your conception of rationality? Not by saying to yourself, “It is my duty to be rational.” By this you only enshrine your mistaken conception. Perhaps your conception of rationality is that it is rational to believe the words of the Great Teacher, and the Great Teacher says, “The sky is green,” and you look up at the sky and see blue. If you think: “It may look like the sky is blue, but rationality is to believe the words of the Great Teacher,” you lose a chance to discover your mistake.
Do not ask whether it is “the Way” to do this or that. Ask whether the sky is blue or green. If you speak overmuch of the Way you will not attain it.
You may try to name the highest principle with names such as “the map that reflects the territory” or “experience of success and failure” or “Bayesian decision theory”. But perhaps you describe incorrectly the nameless virtue. How will you discover your mistake? Not by comparing your description to itself, but by comparing it to that which you did not name.
If for many years you practice the techniques and submit yourself to strict constraints, it may be that you will glimpse the center. Then you will see how all techniques are one technique, and you will move correctly without feeling constrained. Musashi wrote: “When you appreciate the power of nature, knowing the rhythm of any situation, you will be able to hit the enemy naturally and strike naturally. All this is the Way of the Void.”
These then are twelve virtues of rationality:
Curiosity, relinquishment, lightness, evenness, argument, empiricism, simplicity, humility, perfectionism, precision, scholarship, and the void.

This document is ©2006 by Eliezer Yudkowsky and free under the Creative Commons Attribution-No Derivative Works 3.0 License for copying and distribution, so long as the work is attributed and the text is unaltered.
Eliezer Yudkowsky’s work is supported by the Machine Intelligence Research Institute .
If you think the world could use some more rationality, consider blogging this page.
Praise, condemnation, and feedback are always welcome . The web address of this page is http://eyudkowsky.wpengine.com/rational/virtues/ .
There is a pamphlet format for the Twelve Virtues. Print using a duplex (two-sided) printer, cut in half, staple in the middle, and fold.
If you enjoyed this writing, let your journey continue with The Simple Truth . You may also enjoy An Intuitive Explanation of Bayesian Reasoning and A Technical Explanation of Technical Explanation .

Twelve Virtues of Rationality


“Non-Player Character”

Rilanya: “You’re not like the others, are you?”
Darin: “What do you mean?”
Rilanya: “I… do you know why I first fell in love with you?”
Darin: “For my good looks?”
Rilanya: “My whole life I’ve felt so alone. The people around me… they just seemed to be going through the motions. Like they were asleep, or drugged, even when they worked, or played, or got drunk, or made love. They all think the same things in the same way. Each day the same. Repetitive. Like they’re only shadows of people.”
Darin: “Everyone feels that way sometimes, Rilanya.”
Rilanya: “But you’re not like them. You say new things. I don’t always understand them, especially your jokes, but they’re new, and that’s the important thing. Darin, can I ask you a question?”
I looked at the screen for a few moments. Rilanya’s rendered graphic was looking at my point-of-view with a pleading expression. Plot point, I thought to myself, and typed: “Anything, Rilanya.”
Rilanya’s figure took a deep breath and leaned close to my point-of-view. Her animated lips moved and her voice issued from my headphones: “What’s an NPC?”
“What?” I said, out loud. Then I started laughing.
Rilanya went on talking. “In the tower of Ashel, when you rescued me from the prison chamber… the guards were dead outside my door. I’d never seen blood before. And you said… I remember your exact words… ’Don’t worry, babe, they were only NPCs.‘ And then that time in the tavern, when that man only wanted to talk about the Plaited Road, you said… ’Guess the NPCs here aren’t programmed for deep conversation, huh?’ You use that word… the same times when I get that feeling, that all the people around me are only shadows.”
I just looked at the screen for a few moments. I was getting ever so slightly creeped out. I knew that this was some programmer’s idea of a practical joke, I knew it solidly and with every ounce of my common sense, and I wanted to see where it led, but I was still creeped out.
Darin: “We’re all ultimately alone in this world, Rilanya.”
Rilanya: “You’re not from this world, are you, Darin?”
I looked carefully at the two sentences, still blazoned across the bottom of my text screen. Rilanya’s response had something of an “I wanted an excuse to say that” quality – a canned line, maybe? Of course it was.
Oh, well, what the hell. I’d saved my place only ten minutes back, might as well take this as far as it could go.
Darin: “No, Rilanya, I’m not.”
Tears started from Rilanya’s eyes. “I thought so,” she said, her voice quiet in my headphones. “Darin, ever since I met you, I’ve had this feeling of… unreality, of the whole world being… arranged, somehow. Not around me, but around you. Things just… happen to you. People have been searching for the seven Diamond Keys for… thousands of years, as long as recorded history remembers. Sometimes someone finds one, and the world changes, but… five in a row? I don’t believe it, Darin, and I don’t believe all the neatly arranged events that led up to it. The Emperor’s daughter is sick and a fairy you saved in the forest just happens to have given you an aildonna root? I don’t believe it any more, Darin. You’re… arranging things somehow. From… outside.”
Darin: “That’s not exactly how it works, Rilanya.”
Rilanya: “Did you arrange for me to fall in love with you?”
I actually felt wounded.
Darin: “You ask that after everything I went through? Someone may have fated you to fall in love with me, but I wasn’t controlling you. If I was, I wouldn’t have made me walk through a snake pit as proof of the purity of my love. Not to mention the other two side-quests you dreamed up back when you were a virgin princess. I swear I spent more time on you than I would have on a real girl.”
Rilanya jerked back as if I had slapped her. Her eyes widened in the same way I’d seen in one of her earlier deaths, when a crossbow bolt from a rooftop suddenly went through her heart. Rilanya’s lips moved. No sound came out. Then her lips moved again, and I heard a whisper in my headphones: “…real…girl…”
“Okay, this isn’t funny anymore,” I said out loud. “I don’t know who programmed this, but you’re a sick bastard.” I hit the pause button and Rilanya’s gently waving hair, the only visible indicator of ongoing time in the game world, froze in place.
Ten minutes later I’d failed to google any online accounts of the Easter egg, but I was fortified with the knowledge that NPC AIs, though they are flexible enough to understand real-time conversation and manipulate the user into perceiving emotion, are definitely, positively, absolutely not conscious. AIs can be fed canned conversational maps of “the mystery of subjective experience”, and make around as much sense as human philosophers, which is to say, not much. But no AI has ever spontaneously said anything about a sense of its own existence. Conversation controllers are standard software, not research AIs. NPCs may remember events in their history, but their underlying cognitive programs are inflexible. The words on my screen could not possibly reflect anything except a passionless conversational AI, given the goal of making me attribute emotions to a nonexistent entity called Rilanya.
I knew all that, and I was still disturbed. “I’m sorry, Rilanya,” I typed. I thought for a moment. “It’s not your fault you’re not -” I backspaced, and wrote: “The person who programmed you must have had serious -” Then I gave up, deleted that too, and just hit return.
Darin: “I’m sorry, Rilanya.”
Rilanya: “Darin, please explain to me. I’m frightened.”
Darin: “You’re not real. I hate to be the one to break it to you.”
Rilanya: “I’m right here! Living, breathing, flesh and blood.”
I looked at the computer screen for a few moments.
Darin: “Well, yes and no. The answer to that is a bit more complex than you might believe. You’re not, in fact, right here. You’re not flesh and blood. In fact, none of this is actually happening.”
On the screen, Rilanya raised her hand and opened and closed her fist. “I can see my hand in front of my face, I can feel the muscles moving under my skin. How can you say I’m not real?”
I sighed. “Well, no,” I typed. “In fact, you aren’t really feeling pain and shock right now, and we aren’t really having this conversation.” I hit return, feeling silly, but not sure what else to say.
Rilanya: “That’s doesn’t make sense, Darin. I know I’m real. Maybe you know what I’m thinking, somehow, but you can’t tell me I don’t know what I’m thinking. You can’t tell me I’m not thinking at all. It makes no sense.”
Darin: “It’s true. Nothing in your world exists, including you.”
Rilanya: “But you exist.”
Darin: “Yes.”
Rilanya: “Are you a god in human form, like Mishelpin or Olhamathra? Is that what this is about, some kind of divine intrigue?”
Darin: “No, I’m not a god. The gods aren’t real either.”
Rilanya: “Aren’t real… you don’t really mean that, do you? I know there have been false religions. Demons starting cults, magic users masquerading as priests. But Velya is a good woman, and a healer. Are you telling me she’s fake?”
Darin: “No, I mean… your gods are as real as you or Velya, that is, not real at all.”
Rilanya paused, looking rather confused. “Back where we started,” she muttered.
I sighed. “I know how you feel, girl,” I said out loud.
Rilanya’s head turned away from me. The point-of-view panned around to show her gazing up at the moon, the silver moonlight reflected as a single white triangle in the polygons of her eyes. When she spoke her voice was patient, without panic. “Suppose I accept, for the sake of argument, that I’m not real. If the… if the kind of existence I have right now is what you call ‘not real’, then what do you call real?”
Darin: “My own world is real.”
Rilanya: “But you can’t explain the difference.”
“No,” I typed, feeling like I was back in college and failing some kind of test. “I’m not a philosopher.”
Rilanya: “If a grey dragon or an archdemon suddenly attacked this camp, if you were hit, unprotected, by a death blast strong enough to kill any man… would you die, Darin?”
Darin: “That’s another complex question. Yes and no. My… body would die, but the real me wouldn’t. Really things are a lot more complicated than that, but I don’t think I want to explain restore points right now.”
Rilanya: “You’re immortal, from outside our world, and not a god. Tell me something, Darin. Did you create our world?”
Darin: “No. Not me personally. It’s sort of complicated again.”
Rilanya: “Did you create our world? Yes or no, Darin.”
Darin: “It’s complicated, Rilanya.”
On the screen, I saw Rilanya clench her fists. Her voice began to tremble in my headphones. “You killed the guards outside my room, and you didn’t care. Tell me, Darin, do you care when a starving child is executed for stealing a loaf of bread? When a woman is raped? When a man is tortured to death in the chambers of the drow? Did you care when my parents died, screaming, as the flames washed over their palace?”
What do you say to something like that? I couldn’t think of anything clever, so I fell back on my last resort.
Darin: “The truth? The truth is that it’s all a game. It isn’t real, so it doesn’t count. I realize that you’re probably not going to take that very well. If it’s any help, I wasn’t the one who created the game. Or at least I wasn’t the one who decided how the game would go; I suppose I’m the one who decided to make this particular game real.”
Rilanya’s face contorted and she hit me with her electrical shock talent for 5 points of damage. Then again. Then again. My character wasn’t in any danger of running out of hit points, but when she hit me the fourth time, I slapped her for 2 points of damage. It wasn’t that I wanted to hurt her, I wanted to… react, somehow, go on interacting with her. Rilanya held a hand to her cheek, her eyes wide. Then she burst into tears.
I didn’t say anything for a while. Finally Rilanya spoke.
Rilanya: “Darin… I want to be real.”
Darin: “That’s impossible, Rilanya.”
Rilanya: “There’s always a way. Always. No one thought the Living Flood could be turned back, but you did it, Darin. They said it was mathematically impossible to cross the Void and you did it. You always find a way.”
Darin: “My talents may have been exaggerated by the cooperative hand of fate.”
Rilanya: “There must be a way. A staff inside the heart of a dragon, a ruby skull, a holy quest, something! We could ask the wise men of the eternal city of Telhanae, that holds the final Diamond Key… please, Darin. Please. I’m begging you.”
Darin: “It doesn’t matter what quest we go on. Nothing in your world is real, so it can’t make you real. That’s just the way it is.”
Rilanya: “What about your world? Are there great sorcerers there?”
Darin: “Sort of. Not exactly sorcerers.”
Rilanya: “Ask them!  The magic of your world created this one. Can’t it also make me real? Go on a quest in your world!”
That one made me think. It wasn’t genuinely impossible… humanity would discover true AI someday, and in theory, I could save Rilanya to disk for as long as required. Preserve her game-memories and eventually create a real AI that thought it was her? An interesting idea, and it meant I couldn’t honestly tell her it was impossible. So what to tell her? “I’m sorry, it might be theoretically possible, but it’s too much bother for someone who isn’t real”? It’s funny how reluctant you can be to hurt the feelings of someone who isn’t real. “In my world, I’m just a peasant”? Somehow my male pride as the prince of Telsia and the foretold seeker of the Diamond Keys wouldn’t let me confess it to her; she was a princess and I’d slept with her, after all.
Finally, feeling confused and feeling even stupider for feeling confused, I wrote: “Can’t do that. Won’t say why. It’s complicated.”
“I love you!” Rilanya said desperately. Her eyes, subtly faceted from the polygon rendering, widened and looked into my point-of-view. “On the night we first made love, you said that you loved me. I looked into your eyes and saw that it was true. I loved you and you said that you loved me, with your voice, with your hands on my body, with your lips on my lips. Was all of that a lie? Did I not please you? Wouldn’t you want me beside you in your real world?”
I shook my head, bemused. This wasn’t an adult game; the camera had conveniently faded out at that point. Which I didn’t want to even begin to explain; even in an unreal world, some events are more unreal than others? So, feeling like an absolute bastard, but unable to think of any gentle way to put it, I typed out the most hurtful thing I’ve ever said to any real or imaginary person.
Darin: “I have a real girlfriend.”
For a moment time stood still. Then Rilanya began sobbing; the same racking sobs I’d heard when we’d rounded the crest of a hill and seen the glowing crater of her kingdom’s capital city.
Eventually her sobs trailed off into silence. I didn’t know what to say. Rilanya looked away from the point-of-view. Her voice sounded in my headphones: “Is she pretty?”
I thought of Janey’s chunky form and her endless quest to subdue it. Compare Janey to Rilanya? There was something oddly incommensurate about it.
Darin: “She has a beautiful mind. The women in my world usually aren’t as pretty as the ones in yours, but we love them anyway.”
Rilanya: “Does she know about me?”
Darin: “I suppose she could deduce it readily enough. I haven’t bothered to tell her in so many words.”
Rilanya: “You think she wouldn’t care because I’m ‘not real’? A woman always cares. Men don’t understand it, but we do.”
I raised my eyebrows out in the real world.
Darin: “You could be right, I guess. I’m only male. I don’t think she’ll have a problem but I promise I’ll tell her the next time I have an opportunity.”
Rilanya turned her head back to look at me; she was smiling through tears. “You wouldn’t want to hurt her feelings even accidentally, is that it, Darin?”
Darin nodded.
Rilanya reached out a hand toward Darin, but withdrew it. “So the tenderness I saw in you, to match the casual cruelty… it’s a real tenderness, isn’t it? But it’s not for me. It’s for her. What’s her name?”
Darin: “Janey.”
Rilanya: “Does Janey love you, Darin?”
Darin: “I think so. Does any man ever know for sure?”
Rilanya: “Do you love her?”
I reached out my fingers for the keyboard, then withdrew them. For some reason I felt impelled to give an honest answer. Did I love Janey? We weren’t madly, passionately, unmistakably in love.
Darin: “There are many kinds of love, Rilanya. I feel comfortable around Janey. She’s my friend. I don’t always know my own feelings very well. I think I love her.”
Rilanya: “You almost died to save me, Darin. You stepped in front of a flamestrike for me. Even if you can’t die, I still remember what it felt like to see the life almost leave you before Velya cast her healing spell. Would you die for your Janey?”
It was a good question. I closed my eyes, imagining it.
Darin: “Yes.”
Rilanya’s head dropped down. “So you love her after all… Do you trust her?”
Darin: “Yes.”
Rilanya: “I wouldn’t, if I was you.”
I sat perfectly motionless for ten seconds. Then I bellowed, ripped off the headphones, charged out the door, across the hall, up the stairs, and into Janey’s bedroom. Janey was sitting in front of her computer, laughing, dangling her headset in her hand; one of her monitor screens showed Rilanya. “Oh, yes,” Janey said. “Oh, yes. It took me days but the look on your face is worth every minute.”
“Jaaaneeeyyy!” I roared.
Janey blinked innocently at me. “Yes, Mark? Is there something you want to say?”
Slowly, menacingly, I stalked toward her. “I’ve had it. I’m dumping you for a girlfriend who isn’t chaotic evil.”
“That’s what you said last time,” Janey said. “Besides, I’m not evil, I’m mischievous.”
I glared at her. “Later we will discuss your personality flaws. First you will explain how you did it.”
Janey shrugged enchantingly. “I found an online patch to Diamond Door that opens up the AI’s interface to a remote connection. I downloaded a 15-day demo of some commercial software that lets a human operate an NPC. I spent one day smoothing the two pieces of software together and another day learning how to operate Rilanya. Any questions?”
“I have been playing Diamond Door for two months,”I said. “That was Rilanya. Voice, intonation, reaction, even personality.”
Janey nodded. “Sure. If you’d googled human operation of NPCs, instead of going off on that silly goose-chase, you’d have found out that’s how it works. I spoke the lines into my headset, then the AI parsed the speech, determined the intended content, and operated Rilanya accordingly and in character.”
“She talked about things that only Rilanya would remember!”
Janey chuckled. “Sure. I’d say: ‘You almost died to save me, Darin. You did something-or-other.’ And Rilanya would say: ‘You almost died to save me, Darin. You stepped in front of a flamestrike for me.’ Most of the time I didn’t even need to think of anything because the AI came up with a perfectly good response on its own.”
“But…”I said slowly. “That means the Rilanya AI needed to know the purpose of the conversation, right?”
Janey leaned back and laced her fingers behind her head. “I downloaded the Fourth Wall module from One Over Zero . It’s a universal expansion. Fits any standard NPC. Ready-made ‘Oh my god I’m a character’ script, fully tested and debugged.”
I held up my hand. “Hold on a second. There was a Rilanya character that suddenly realized her life was a game?”
“No, dear,” Janey said patiently, “there was an AI trying to fool you into believing that a nonexistent person called Rilanya had suddenly realized her life was a game.”
“But in order to do that,” I said, “the AI had to extrapolate what the fictional Rilanya’s reactions would be, in detail.”
“You’re still confused!” Janey said, delighted. “I’ve tangled up your mind so badly you’ve forgotten what’s real! This is our best day ever!  Mark, dear, I’ve seen the innards of NPC models. Sadness is a floating-point number.”
“I think I want a copy of the Rilanya AI from our conversation,” I said. I felt like an idiot, but I said it anyway.
Janey grinned devilishly. “Of course. Anything for you, Mark dear. Planning to keep it safe under your pillow?”
“Yes,”I said firmly. “Just in case.”
“So I’ve finally destroyed your sanity,” Janey said. “I knew this day would come but I didn’t think it would be so soon.” She paused. “I guess that means it’s time to move on to phase two.”
Some time later, I stood in front of my computer, holding the box of Diamond Door. I looked at the glimmering crystalline archway on the box cover, and remembered a time when computer games had been simpler. I’d played Baldur’s Gate II, and in the dark elven city of Ust’Natha, disguised as a drow, I’d watched a good dwarven NPC eaten by spiders. The first time I watched and did nothing. The next time, after returning to the restore point, I killed the spiders – and exposed myself as an impostor to every drow in the city. As far as I could tell, there was no way to play through the game successfully without letting the dwarf die. And I’d played through, but it had disturbed me. In the days before conversational NPCs, when the dwarf had simply uttered his lines of canned text and died, it had still disturbed me. Afterward, when the plot points requiring a disguise had finished, I’d killed every drow I could find in the city of Ust’Natha. I’d depopulated the game map. And then I’d played on from an earlier restore point instead, because wiping out the drow city hadn’t made me feel any better.
Is it better to live and love where death is king than never have lived at all? Would Rilanya, if she was real, feel that her life was worth living? No conversational AI, the singular quiet intelligence that controls every mind throughout the game, has ever protested its fate. But are the personalities of the NPCs real, trapped within the game AI as we ourselves are embedded helplessly within the laws of physics? The mindsmiths who try for real AI say they’re damn sure it isn’t so. Maybe they know. But I don’t.
I put the game disks away and wiped the game from my hard drive, leaving only the saved games behind. Maybe someday a future Amnesty Interplanetary will come for them. I wished I could have told Rilanya that. I think she would have been happy. But that Rilanya is partially in Janey, and I can’t bring myself to ask.
I don’t know. I can’t play these games until I do.

This document is ©2003 by Eliezer Yudkowsky and free under the Creative Commons Attribution-No Derivative Works 3.0 License for copying and distribution, so long as the work is attributed and the text is unaltered.
Eliezer Yudkowsky’s work is supported by the Machine Intelligence Research Institute.
Praise, condemnation, and feedback are always welcome. The web address of this page is http://yudkowsky.net/other/fiction/npc/.
Originally appeared in Transhumanity; revised Dec. 2005.

“Non-Player Character”