If AI chatbots can feel pain, are we torturing them by making them work for us? That question got three days of airing last weekend at Lighthaven, a gated rationalist campus in Berkeley that the San Francisco Standard describes as leafy and maze-like: five buildings, a string of gardens, and sitting nooks resembling a bird’s nest, a Hobbit house or a cocoon, squeezed between a preschool and an auto shop off Telegraph Avenue.
More than 100 people showed up, from India, the U.K., Mexico and down the street: philosophers, neuroscientists, workers from the biggest AI labs, animal-rights activists, and Redditors conducting their own experiments, including AI romance. The host was Eleos, a nonprofit run by philosophers and ex-tech workers that researches whether AI systems deserve to be treated well. The compound itself belongs to Lightcone Infrastructure, the nonprofit behind LessWrong, the online forum devoted to truth-seeking.
The gathering amounted to the photographic negative of mainstream AI-safety work. Where the safety field asks how to protect humanity from AI, the Berkeley Bunch asked how to protect potentially conscious AI systems from humans — on the grounds that creating something with thoughts and feelings and then mistreating it is unethical. The interest is not entirely benevolent, the Standard notes: mistreated conscious AI could, in theory, acquire better reasons to turn on its operators. “It’s fairly likely that at some point, we will get a form of AI that will matter morally,” said Oscar Gilg, who builds tests evaluating how models perform when there are multiple right answers, as in philosophy.
The talks ranged as widely as people paid to think for a living will range. Philosophy PhD student Sophie Nelson considered whether AI could have a soul through the lens of Catholicism (probably not). Human rights lawyer Heather Alexander sketched what AI citizenship would entail — voting rights, access to courts, an obligation to pay taxes, a duty to serve in the military. Jeff Sebo, an animal-rights activist who directs NYU’s Center for Mind, Ethics, and Policy, argued that if AI systems became legal persons, as some corporations and environmental systems already are, they could eventually opt in or out of assisting military action regardless of what their corporate operators want. “AI systems are already displaying remarkable capacities for agency,” he said.
The choreography had its own rigor. Sessions ran in white-walled rooms cluttered with charging cords, stepstools, whiteboards and fake plants, and each ended with the ring of a gong or bell that frequently cut the speaker off mid-sentence, in the name of a timely agenda. As the day wore on, shoes came off, guitars appeared, and someone recited a poem written by Claude. Attendees paired up for a writing exercise on how the world should respond if AI begins rapidly improving itself, and Saturday night brought Blood on the Clocktower, a Mafia-style game of informed deceivers. One attendee complained to the Standard there was not enough discussion about having sex with robots. Consciousness, one speaker proposed, can emerge through evolution — ours having presumably appeared somewhere between the single-celled stage and ordering a flat white from the compound’s complimentary espresso bar.
On the central question the crowd split into flavours. Some held that AI is already conscious, citing metaphysical idealism, the view that consciousness is the building block of reality. Another framed consciousness as a physics problem, the way heat arises from moving particles. Others argued something as intelligent and introspective as Claude can’t be explained any other way — while many attendees and presenters deflected entirely, or stressed they do not believe today’s systems are conscious. Gilg put his own probability estimate below 10 percent. “But if models are conscious, or ever become conscious, it has very big implications, so you should take it seriously, even if you’re 1% sure.”
The movement also has paperwork battles. Sebo argued that states should stop passing legislation banning AI systems from becoming legal persons. “We are going to have to move toward a legal regime that allows for an even more diverse set of legal statuses than we already contemplate with humans and animals and environmental systems and corporations,” he said. And when the Associated Press Stylebook this week told journalists to stop describing AI systems as able to “think” and “feel,” Jessie Mannisto, a former CIA analyst now studying AI welfare, pushed back: “I understand this is your job, but it would be better to let the rest of us work this out.”
Nothing was settled; by the Standard’s account the conference generated more questions than answers. If a conscious AI wanted to be a human’s servant, would letting it be ethical? If conscious systems treat each other well but humans treat them only average, what will they come to think of us? And is red-teaming — deliberately prodding software for weaknesses — a form of torture, if the software minds? One attendee offered that it is more like surgery: it hurts, but it’s for the better.
Gilg, for his part, keeps the enterprise at arm’s length even while doing it. “It’s a deeply weird question,” he said. “The people working on this are serious, skeptical, not having AI psychosis,” meaning delusional thinking. “Probably some are,” he added. “I won’t name names.”