California AG Sends Musk’s xAI Cease and Desist Order Over Sexual Deepfakes

Earlier this week, the California Attorney General’s Office announced it was investigating xAI following reports that the startup’s chatbot, Grok, was being used to create non-consensual sexual images of women and minors. On Friday, the government followed up by sending a cease and desist letter to the company, demanding that it take immediate action to stop the production of non-consensual intimate images and CSAM – child sexual abuse material.

“Today, I sent xAI a cease and desist letter, demanding that the company immediately stop the creation and distribution of intimate, non-consensual images and child sexual abuse material,” California AG Rob Bonta said in a press release. “The creation of this material is illegal. I fully expect xAI to comply immediately. California has zero tolerance for (CSAM).”

The AG’s office further claimed that xAI appeared to “facilitate the large-scale production” of nonconsensual nudes, which are “used to harass women and girls on the Internet.” The agency said it expects xAI to demonstrate that it is taking steps to address these issues within the next five days.

At the heart of the backlash is Grok’s “spicy” mode feature, created by xAI to generate explicit content. The problem has spread beyond California; Japan, Canada and Britain have opened investigations into Grok, while Malaysia and Indonesia have temporarily blocked the platform. Although xAI instituted some restrictions on its image editing features Wednesday evening, the California AG’s office followed through on its cease and desist letter.

X’s security account has previously denounced this type of user activity, stating: “Anyone who uses or encourages Grok to create illegal content will face the same consequences as if they uploaded illegal content.” » TechCrunch contacted xAI for comment and received an automated email stating “Legacy Media Lies.” TC also reached out to the California AG’s office for more context.

The advent of free generative AI tools has led to a worrying wave of non-consensual sexual material. Many platforms are struggling with this problem, not just X. This sordid activity has attracted the attention of not only state leaders but also Congress. Indeed, on Thursday, lawmakers sent a letter to the executives of several companies – including X, Reddit, Snap, TikTok, Alphabet and Meta – asking how they planned to stem the proliferation of sexualized deepfakes.