A few years ago, in 2022, Google engineer Blake Lemoine was fired for claiming that the chatbot he was working on was conscious. He believed that it could feel things, potentially suffer, and that the moral status of large language models (LLMs) needed to be taken seriously. It turned out that it was Lemoine who was not taken seriously.Earlier this year, Dario Amodei — CEO of the frontier AI firm Anthropic — told the New York Times that “we don’t know if the models are conscious”. What’s changed?One thing is that discussions about “AI consciousness” have moved from philosophical seminars to center-stage in academia, and out of the shadows in the tech industry too. This transition, driven in large part by the astonishing progress of language models, is in one sense a good thing. If we did end up creating conscious machines, deliberately or not, we’d unleash an unprecedented moral crisis. We would introduce new potential for suffering in the world, at the click of a mouse, and of a kind we might not even recognize.I think the odds of real artificial consciousness, at least along for systems like LLMs, are almost vanishingly low. In my work at the interface of neuroscience and AI, and in my book Being You [could replace this with the Berggruen essay], I lay out three reasons why we tend to overestimate the likelihood of conscious machines.The first lies in our own psychological makeup. We tend to assume that “intelligence” and “consciousness” go together, so that something sufficiently smart would also be conscious. But just because intelligence and consciousness go together in us doesn’t mean they go together in general. The assumption that they do is a reflection of our psychological biases, not an insight into reality. Language exerts a particularly strong pull on these biases, which is why people wonder whether Anthropic’s Claude is conscious, but not DeepMind’s protein-predicting AlphaFold.The second reason is more fundamental. The very idea of conscious AI is based on the idea thatthe biological brain is a computer of some kind, and that consciousness is a matter of computation. Get the algorithm right, and you get consciousness, regardless of whether the algorithm is implemented in meat or in silicon.But the closer you look at the brain, the less like a computer it seems. There is no clean division between “mindware” and “wetware” as there is between hardware and software in our silicon devices, and even a single neuron is a vastly complex biological factory. The brain-as-computer metaphor was only ever a metaphor, and we always get into trouble when we confuse a metaphor with the thing itself. If the brain isn’t actually a computer, there’s much less reason to think that consciousness could happen in silicon form.To put the point another way: Nobody expects a computer simulation of a hurricane to generate real wind and real rain. In the same way, a computer model of the brain may only ever simulate consciousness, but never give rise to it.The third reason is that there are plenty of other games in town. Nobody knows how and why consciousness happens, but there are other possibilities beyond it being an algorithm on the one hand, or immaterial magic on the other. One possibility I’m attracted to is that consciousness arises from our nature as living creatures: that it is life, rather than information processing, that breathes fire into the equations of consciousness.The ethics of all this matter. Even if real artificial consciousness is off the table with current forms of AI, emerging “neuromorphic” technologies, which are becoming ever-more brain-like, may yet move the needle.Even AI that merely seems to be conscious is ethically problematic, even if, under the hood, there is only subjective oblivion. Conscious-seeming AI can exploit our psychological vulnerabilities. We might be more likely to do what an AI says if we feel that it really feels for us, really understands us. And seemingly-conscious AI can also distort our moral priorities. If we give AI systems their own ‘rights’ on the basis that they might be conscious – when inside there’s silicon oblivion – we make the challenge of aligning AI with human and planetary interests much harder than it already is.In the face of this uncertainty, what should we do? First, we should not deliberately try to create artificial consciousness. Even if we don’t know what it takes to create conscious AI, we also don’t know how to rule the possibility out altogether. Second, we should carefully distinguish the ethical implications of AI that actually is conscious from AI that irresistibly seems to be conscious. The uncertainties of the former should not distract us from the clear and present dangers of the latter.We face similar uncertainties in many other contexts too: people with severe brain injuries, non-human animals (from bacteria to bats), human foetuses, and strange new creations in synthetic biology, such as “cerebral organoids’ (brain cells in a dish that wire up together). In each case, there are ambiguities about whether consciousness is present, and in each case, our decisions carry moral weight. As science, technology, and medicine continue to advance, more of these scenarios will move from the fringes to the spotlight. What we need is nothing less than a satisfactory scientific understanding of consciousness itself.
