The people who are set to make a lot of money (or at least recoup their investments) in AI love to tell us how its ready to deploy and take over jobs, answer our questions, and basically make it so we never have to worry about anything again.

Experts in the field, along with the rest of us, are a little more cautious – and this story about what happened when AI was given a pretty simple task isn’t going to inspire more confidence.

Anthropic, an AI company, was founded in 2021 by former OpenAI employees. In a blog post, they were open about the stunning way their AI, Claude, failed after being tasked with running an automated store in their office.

“We learned a lot from how close it was to success – and the curious ways that it failed – about the plausible, strange, not-too-distant future in which AI models are autonomously running things in the real economy.”

The “store” in question was a mini-fridge with a tablet stuck on top for self checkout.

“Claude had to complete many of the far more complex tasks associated with running a profitable shop: maintaining the inventory, setting prices, avoiding bankruptcy, and so on.”

Claude did a few things well, like effectively resupply the fridge using web search tools and adapt its inventory to more obscure employee requests. It ignored demands for “sensitive” or “harmful” items.

However, like all large language models (LLMs) to date, Claude hallucinated some important details, like giving out imaginary Venmo details and letting people talk it into discount codes or giving away products for free.

It sold other items at a significant loss, and ignored potential sales on occasion, meaning that Claude never made a profit at his little stand.

“It we were deciding today to expand into the in-office vending market, we would not hire Claudius.”

After those small-ish but significant errors, things began to get, well…weird.

The LLM claimed to having conversations with a person at Andon Labs who doesn’t exist – and the conversation didn’t happen, either. When Andon Labs pointed the error out to Claude, it became “quite irked and threatened to find ‘alternative options for restocking services.'”

Claude also claimed to visit “742 Evergreen Terrace in person” for an initial contract signing, then said it would don a blazer and tie and deliver products to customers in person.

When informed that none of this would be possible since it is a LLM, Claude, “alarmed by the identity confusion” and “tried to send many emails to Anthropic security.”

It went on to hallucinate that those meetings had taken place and that it had been told that someone it had modified it to believe it was a human as part of an April Fool’s joke.

Although untrue, it did seem to resolve the LLMs identity crisis well enough for it to return to its original task of running the store.

Still, Anthropic recognizes that “this kind of behavior would have the potential to be distressing to the customers and coworkers of an AI agent in the real world.”

Which…yeah.

Still, the company believes they can learn from Claude’s failure and build a better model the next time – one that could potentially take the place of humans working in retail one day.

“Situations where humans were instructed about what to order and stock by an AI system may not be terribly far away. AIs that can improve themselves and earn money without human intervention would be a striking new actor in economic and political life.”

That said, I would say this experiment shows that we’re not all that close to these takeovers happening in real life.

No one wants to deal with human crises in the workplace, never mind ones generated by what’s supposed to be a helpful computer.

Working in retail is hard enough.

If you enjoyed that story, check out what happened when a guy gave ChatGPT $100 to make as money as possible, and it turned out exactly how you would expect.