Canada’s privacy watchdog found that social platform X (formerly known as Twitter) violated Canada’s federal private-sector privacy law when it launched an AI image generation tool that was used to create sexualized images of people without their consent.
“These measures should’ve been put in place at the outset, not after the fact. Not after the harms.”
The Office of the Privacy Commissioner of Canada (OPC) released a report Thursday summarizing the findings of a months-long investigation of Elon Musk’s xAI and social platform, X. The report found xAI’s AI image generation tool Grok was launched without proper safeguards or considerations of privacy harms, allowing users to create and share non-consensual, sexualized deepfakes.
Though xAI agreed to some of the watchdog’s recommendations to fix the problem, privacy commissioner Philippe Dufresne said in a news conference today that it’s still violating Canada’s privacy laws—and the privacy commissioner’s office doesn’t have the power to force X to comply.
In its report, the OPC recommended that xAI suspend its AI image generator, Grok Imagine, on both the social media app and the standalone product, until it can demonstrate that its safeguards will prevent the generation of sexualized deepfakes. The company is refusing to do so, but has agreed to other measures to mitigate the issue. For example, it has committed to issuing quarterly reports and independent third-party audit reports on improvements to safeguards. These will include evidence to demonstrate their effectiveness and will be submitted until the issue of sexualized deepfakes is fully resolved.
RELATED: Yoshua Bengio co-signs statement calling for new laws to combat deepfakes
xAI also introduced new safeguards against such deepfake generation during the investigation, and started doing proactive sweeps to flag and take down the AI-generated sexual deepfakes on their platforms.
BetaKit has reached out to xAI for comment.
“These measures should’ve been put in place at the outset, not after the fact. Not after the harms,” Dufresne said on Thursday. “This case should serve as a powerful reminder for all organizations, especially those developing emerging technologies like AI, about the importance of prioritizing privacy at the outset of any initiative.”
More than 6,000 sexualized images per hour
The privacy watchdog launched an investigation in January after many users began prompting X’s AI model Grok to generate these non-consensual images of women, and sometimes children (including child sexual abuse material, or CSAM). At one point, Dufresne said, Grok was generating more than 6,000 sexualized images per hour.
The investigation looked at whether xAI had obtained consent from individuals to use their personal information to create sexual deepfakes, and whether this usage of personal data could be considered appropriate. The privacy commissioner found the company had not received consent and that its response to the widespread deepfakes was “insufficient.”
Grok is the generative AI model created by Elon Musk’s xAI, available to X users. Musk and xAI have positioned Grok as a more permissive chatbot with fewer rules governing its prompt generation than competing AI models. The model has been prompted to generate racist and misogynistic content as well as factual errors and inaccuracies. xAI is also facing a new lawsuit from a former engineer who claims he was fired over raising safety concerns about Grok.
“This case should serve as a powerful reminder for all organizations, especially those developing emerging technologies like AI, about the importance of prioritizing privacy at the outset of any initiative.”
Philippe Dufresne, OPC
Even though Dufresne said the OPC had “good collaboration” with xAI, he called out OPC’s limited powers to pressure a private company to conform to Canadian law. For example, the watchdog can’t levy fines or issue an order for xAI to suspend Grok Imagine. The OPC would have to turn to the courts, which is a “lengthy” and “expensive” process, Dufresne said.
“They’ve not accepted to do this, and I can’t force them to do it,” he said.
The privacy commissioner added that the lack of possibility of fines or of orders makes it “more difficult” to convince companies to invest in privacy from the outset.
The report comes a day after Canada’s new Safe Social Media bill was tabled in the House of Commons. The legislation proposes a ban on social media for users younger than 16 unless the platforms can prove safety, as well as the creation of a new regulatory body called the Canadian Digital Safety Commission. Unlike the OPC, the CDSC would have the power to impose fines and orders on offending companies.
According to its new AI strategy, Canada also plans to introduce long-awaited updates to its privacy legislation to factor in AI and technological advancements. Dufresne said such laws should include a “legal obligation for organizations to establish privacy by design.”
Feature image via X. The image has been anonymized by BetaKit to protect the identity of those involved.