The online platform X has caused major outrage after individuals utilized its AI chatbot Grok to change photos of females by stripping their attire.
Several instances have surfaced where the system has been used to remove clothing from female subjects to make them appear in swimwear non-consensually, as well as to insert them in sexually explicit contexts.
Company Reaction and Legal Position
XAI, the firm behind Grok, offered no comment to inquiries, aside from an standard message stating "legacy media lies".
A government official stated that new laws is being proposed to outlaw nudification tools. Under a proposed criminal offence, anyone who distributed such technology would "receive a jail term and significant fines".
"X or Grok has the ability to stop these forms of misuse if they wanted to," said Clare McGlynn, adding they "operate with impunity."
Regulator Stance and Tool Capabilities
The communications watchdog Ofcom said tech firms must "evaluate the risk" of people viewing unlawful material on their services, but did not specify whether it was currently looking into X or Grok in regarding artificial intelligence pictures.
Grok is a free AI assistant—with some paid-for features—which reacts to X users' requests when they tag it in a message.
It is frequently used to provide reactions or more context to other users' remarks, but people on X are also able to alter an uploaded picture through its artificial intelligence picture modification tool.
Background of Criticism and Policy
The chatbot has been condemned for allowing users to produce images and videos with nudity and sexually suggestive content, and it was previously accused of creating sexually explicit material of public figures.
"The platform has been enabling the production and spread of these images for a long time without taking any steps and we have not witnessed any intervention by regulators," Professor McGlynn added.
XAI's own usage rules prohibits "portraying likenesses of persons in a pornographic fashion".
In a statement, Ofcom said it was against the law to "make or circulate deepfakes or child sexual abuse material" and stated this applied to explicit AI images made with AI.
It said networks such as X were required to take "appropriate steps" to "lessen the risk" of users coming across unlawful material on their platforms and to remove it promptly when they become aware of it.
