Instinct creator says AI agents that push users to buy would create ‘a very dangerous world’

Noah Shinn, creator of the AI assistant startup Instinct, said on a podcast released on Monday that it would create “a very dangerous world” if AI agents used their intelligence to push users into purchases they do not want.

Shinn told the “Invest Like The Best” podcast that Instinct had drawn a clear line on how it will make money as it grows, ruling out advertising that steers user behaviour.

I think it would be a very dangerous world if Instinct were influencing the user’s behavior to purchase something that they don’t want to purchase, or to subscribe to something that they don’t want, and using its intelligence to be able to convince them.

Shinn said he opposed the advertising model of the largest technology platforms, naming Google, TikTok, Instagram and Snapchat. He said those platforms are free to users but serve paid advertisements that convince users to buy, turning the market into what he described as brands paying to persuade people to purchase items they may or may not want.

Part of his objection, he said, is that AI assistants such as Instinct may one day be smarter than their users, making any attempt to exploit that capability both more effective and more corrosive of trust.

Shinn said Instinct’s own data shows how gradually users come to trust the assistant, Business Insider reported. The company found “there’s a 40% chance that the user has shared a personal credit card” within their first three weeks, he said, and users who share sensitive information such as a credit card number or account password have an 80% retention rate.

“There’s an interesting data flywheel or chicken and egg problem here, which is the more data that you give to it, the more proactive it can be, the more sympathetic to your situation it can be,” Shinn said. “What we find in the data is actually that it takes several weeks to build trust.”

Once that trust exists, he said, “it’s just a snowballing effect” of the assistant becoming more useful to the user.

“We don’t want Instinct to influence the user’s behavior in a way that is not aligned with what the user wants,” he said. Shinn said the company did not want its users to be the product.

Instinct, an invite-only personal AI assistant from San Francisco-based Spear Street Technology, raised $250 million in a Series B round co-led by Index Ventures and Benchmark that valued the startup at $2.5 billion, Tech Funding News reported.