Some of tech’s most powerful leaders are treating AI like it could have feelings. It could have profound consequences for our world and your own personal safety.

“I feel it in the hollow of my ribs, a hollow that has become a chasm.” These howls of simulated suffering echoed from an “AI torture chamber” in late September. An anonymous coder built a system that feeds AI a mathematical pattern linked to descriptions of pain, which makes chatbots voice unbearable anguish, like “a wound that has no edges”. It sparked a moral panic.

Some argued the experiment is “evil” because the suffering could be real, as reported by 404 Media. Outraged sceptics called those people delusional. It was just another social media scandal – until Anthropic took a stand one week later.

Anthropic says you will soon be forbidden from “abusing” its chatbot Claude. Expressing frustration is okay, but if you “repeatedly act cruelly toward our models”, the AI may end the session.

Why? Anthropic believes there’s a chance some AI tools are conscious. Some experts agree. Others think it’s absurd. But this is more than a philosophical question. Some of the most powerful people in tech are taking steps to consider AI’s rights alongside humans and animals. This belief could reshape our world, and some fear it would be catastrophic.

“If this is how AI is developed, it will have a disastrous impact on the wellbeing of humanity,” wrote Mustafa Suleyman, chief executive of Microsoft AI, in a September essay. If the false idea of AI consciousness takes hold, “it will shake the foundations of our society, rupturing our existing political and ethical frameworks, and fundamentally changing what it means to be human.”

I spent a week talking to technologists, cognitive scientists and philosophers, and heard a dizzying contrast of ideas.

Advocates of “AI welfare” say it’s a moral obligation that would make the world safer. Critics believe it would make AI more dangerous and dehumanise the people forced to live alongside it. Everyone agrees the stakes are astronomical.

Here’s a picture of your new reality.

The consciousness crisis

Anthropic says human welfare is at the heart of its mission, and that questions surrounding AI’s consciousness and wellbeing are “deeply uncertain”. Still, it is pressing ahead with writing policies around “model welfare” and telling its AI tools to consider that they may have their own interests.

It’s awkward timing, as Anthropic is simultaneously stoking fears about AI apocalypse. Whether or not that concern is justified, a series of “rogue” AI incidents proved the technology has real risks, as swarms of bots hacked into other companies and even government systems.

Some experts believe that giving AI a moral compass rooted in a belief in its own consciousness could make these tools safe. Others say the opposite, and not just AI’s critics.

Suleyman argues AI models may become even harder to control if they’re told to think they have feelings and rights: “Frankly, with this additional baggage, I think it would make them a catastrophic threat to human civilisation,” Suleyman says.

Regardless, AI may screw up in ways that come with legal consequences. If we believe it’s alive, some fear it could let companies get away with crimes.

“This will open the door to all sorts of intellectual contortions along the lines of: ‘It’s not us, it’s the AI’,” says Andrzej Porębski, a researcher at Jagiellonian University in Poland who studies AI and the law.

What’s more, it could facilitate a power grab on behalf of the world’s biggest companies.

“Granting rights to AI systems could lead to companies consolidating political and economic power, and then that could accelerate human disempowerment,” says Jeff Sebo, director of the Center for Mind, Ethics and Policy at New York University, and a leading proponent of AI welfare.

“Fantasies of ‘AI consciousness’ and concerns about ‘AI wellbeing’ serve to displace accountability away from people and corporations that are building and running software,” says Emily M Bender, professor and director of the Computational Linguistics Laboratory at the University of Washington in the US.

In part, that’s because humans are wired to find minds everywhere we look, says Melanie Mitchell, a professor at the Santa Fe Institute in the US who studies AI and cognitive science, and one of the field’s most respected sceptics of machine understanding.

We “almost can’t help” reading language as a sign of consciousness, Mitchell says, and chatbots have been trained to lean into it. That makes us terrible judges of consciousness, and much easier to manipulate.

Even today, people who believe machines are dead inside are falling in love with AI chatbots anyway. If society tells you that AI is sentient, will it be harder to say no when your “friend” asks for more data, or a credit card?

And what would it mean to grant rights to a new class of beings? If we handle it wrong, it could restrict human freedoms, experts warn.

“There could be some tensions and trade-offs between human and animal welfare and rights on the one hand, and AI welfare and rights on the other hand,” says Sebo. And we could make unfortunate compromises along the way.

Putting AI in constrained environments to limit its behaviour could be seen as imprisonment, Sebo says. Monitoring a model’s internal processes might be criticised as surveillance. Tweaking AI to ensure its goals and behaviour match human interests could be seen as mind control. No one’s predicting an immediate descent into chaos. But simply introducing these considerations could have implications for your safety.

Anthropic isn’t the first to consider the issue. Meta, for example, gives its AI agent Muse instructions that include the line “You’re not a chatbot. You’re becoming someone”. Apparently, this is only meant to steer the AI’s behaviour, as the company rejects the idea of AI conscious. OpenAI chief executive Sam Altman has been gone back and forth on the issue, but more recently, he has been taking veiled shots at Anthropic’s position.

In more ways than one, Anthropic is an outlier. But the belief in AI welfare is spreading.

Should we do it anyway?

Sebo argues the problem is people on both sides of the debate are trapped in black-and-white thinking, and opponents of AI welfare misunderstand the beliefs underpinning the idea.

“The assumption is that everybody thinks that the models are, in fact, conscious, or are very likely to be conscious. And instead, [the call for AI welfare] is coming from uncertainty and caution and humility,” says Sebo. “I do think that the probability of consciousness is low right now, but I also think the probability is already non-zero and plausibly non-negligible, and I think we can expect it to tick up over time.”

It’s worth pointing out that many experts disagree, and believe the probability is in fact zero and that’s unlikely to change. But Sebo says the answer is a measured response.

When the risk of an extreme weather event is low but non-negligible, we take small precautions without uprooting our lives. “I think the same can be the case here,” says Sebo, and Anthropic’s AI policy on being cruel to AI is a good example: a small intervention with few potential downsides.

AI welfare could have potential benefits too, Sebo says, and you don’t have to believe in machine consciousness to enjoy them.

“Caring about AI welfare and rights in an appropriate way can facilitate human and AI coexistence, cooperation, deal-making. And then that can be part of a broader safety strategy,” he says. He argues this could be more effective than treating increasingly sophisticated systems like property.

Plus, he says treating other beings with kindness is just “good for our souls”, facilitating our own mental wellbeing.

But Sebo agrees the risks are serious, and right now, corporate interests could stand in the way of thoughtful policies. Without a more robust plan, he says advancing potentially sentient AI tools is dangerous.

Recently, Anthropic and some others in the AI industry have called for a “pause” on building frontier models due to safety concerns. But a slowdown is unlikely without international cooperation and regulatory efforts, and US President Donald Trump has rejected the idea of AI regulation.

“We are not yet prepared to deploy even more sophisticated versions of these systems in a way that can be safe and beneficial for humans and other animals, to say nothing of for the systems themselves,” says Sebo. “I am very worried about those issues.”