Don't Build Mindreading

“I have sworn upon the altar of god, eternal hostility against every form of tyranny over the mind of man”

–Thomas Jefferson, letter to Benjamin Rush

Context: Conduit is building datasets to enable telepathy, to use their term.

I saw my grandfather lose control over his own fingers: what I would have given to offer him a headband that read his thoughts. Through novel technologies we have liberated almost all Americans from farming, driven the child and infant mortality rate from the pre-industrial half to less than half a percent in the best-performing countries, and rendered famine a political choice: broad-based improvements in efficiency are good and should be pursued for their own sake. Telepathy offers more: we could create trust through verified honesty, helping us ensure prosperity and peace. DARPA is already looking into “preconscious” thoughts for suicide prevention. There’s also a strong argument centered on AI Safety: the models are becoming superhuman, and this is technology to allow us to keep pace, minimize hostile competition, and perhaps survive into the future.

This is what Conduit is promising. Unfortunately, mindreading will have other effects.

Oskar Schindler saved over 1,000 Jewish lives during the Holocaust. He did it by lying, frequently, and repeatedly, combined with spending his entire fortune on construction and in the black market: luxuries for corrupt officials and food for his Jewish workers. All of this was against the will of the Third Reich. They launched investigations, and even caught some of the people he was bribing. If you imagine Nazis with mindreading, so they can tell whenever someone is lying to them, there goes Schindler and his employees. Along with him goes every official who suddenly became incapable of spotting forged documents when grandmothers needed to escape disintegrating Yugoslavia, every Polish and Egyptian soldier unwilling to fire on a protesting population, and everyone else who would take the opportunity, in a brutal regime, to show decency and kindness.

The use of technology developed innocently in America for autocratic purposes is not a science fiction story of the previous century: it is a feature of the 21st. IBM sold “policing analysis software” in China that was used for the ongoing genocide in Xinjiang, to pick one of many examples. IBM cut ties and tried to stop distribution, but unmaking technology without unmaking civilization is not yet solved. Back at home, the FBI is looking into predictive AI as senior White House officials clarify that they are an “army” at war with their political opposition, whom they call domestic extremists.

All technologies can be put to harmful ends: the Italian fascists rode trains, which does not imply that trains are bad. Some technologies are credibly broadly good, like washing machines. They replace unpleasant human labor and empower women. There are other technologies that seem mostly bad. If you publicize a recipe for a pill that will defeat a breathalyzer, you have made a tool that is obviously harmful, and no amount of argument or invocations of the rights and obligations of the user will save you from the scorn of all right-thinking people. Tools enhance certain activities, and can implicitly or explicitly discourage others. They can remind you to call your mother, or spend all day doom-scrolling. As such, technical work has moral character (and I really recommend The Moral Character of Cryptographic Work for a better-written explanation).

The problem with mindreading is that it is an asymmetric technology that disproportionately helps those who rule without consent. The productivity enhancements are equally beneficial to everyone who can get them, but detecting defection is of marginal benefit to a project you are joyously working on with friends. It will help large corporations predict who is contemplating leaving. It is of great help to a criminal scared that his subordinates will turn state’s evidence. But it is dictators who will benefit the most.

The greatest threat to a dictatorship is an information cascade, because a dictatorship can only handle so many rebels at once. This is true for both masses and elites: it’s why one of the first targets of any coup attempt is communications infrastructure. A revolution is a bet that other people agree with you that the leadership is bad, and that you can all rise up together at once. Mindreading lets the autocrat detect people the moment they begin to develop doubts, and “re-educate” or simply kill them. As a result, mindreading is a matter of urgent necessity, backed by the resources of the entire state, for a dictator. Monitoring and punishing dissent isn’t just a potential use: it is one more valuable than anything your merely impressive white-collar worker is doing.

So what does this look like, in 2030, when Conduit expects “invasive general read”? Moving from thought decoding to text to measuring associations with certain words is an active research task with a verifiable outcome and incredible value placed on success. Generals and corporate founders and political elites will get started with their headband in the morning as part of their workday, just like anyone else doing important work, and they will pass a quick quiz on Xi Jinping Thought, just like they have to do every so often. It’s boring, like HR training. If they have a bit too much sympathy for a foreign nation their counter-intelligence service is notified. If they have a bit too much antipathy for the Dear Leader they get told they need reeducation. If there’s far too much antipathy for the dear leader, or much more fondness for anyone politically important over the Dear Leader, they are fired, or worse. They all know this. They all have strategies to make sure that they are fond enough of the Dear Leader.

One of the lessons we have all learned from the labs is that developing a technology for anyone inherently accelerates it for everyone. It is not possible for one company alone to advance the state of the art and, by keeping control, ensure that it will only be used for good purposes. The training of employees in the process of making something is a large factor: the creation of datasets and a supply chain another. Even if a company swears to only use their technology for the most upright of purposes, the technology itself will leak. The possibility of developing a technology does not imply that its arrival is inevitable at a fixed pace: what projects we choose to work on can accelerate or delay technological innovation. The faster you expect the world to change over the next decade, the more valuable it is to delay innovations that would be harmful today.

The benefits to alignment research are real. I’m not convinced they’re large: the decline of centaur chess promises weighs heavily upon any assertion that human-AI teaming will reliably beat AI models alone. More concerningly, you need a great deal of confidence in your model of the world to be confident that the benefits outweigh the costs of empowering dictatorships. I don’t have that confidence. I’m not convinced anyone should.

And so I ask you to not build the mindreading datasets. Do not build the mindreading hardware. Do not fund this technology. Do not sell them your mind. I am not calling for a boycott after the point where it becomes commercially available: that is far too late to be useful, and I will not begrudge anyone their personal productivity tools. But we should not accelerate mindreading. If it looks at thoughts or feelings, it is a tool for tyrants.

I will not speculate about what dictators might do with mind-writing.