Meta ads promoted AI 'nudify' apps using real photos of children, investigation finds

The Tech Transparency Project identified 332 ads this year that contained AI-generated CSAM and were served by Meta's platforms. The investigation found that the 'vast majority' of these ads promoted AI apps made in China, while many specifically advertised 'nudify' tools that allow users to create non-consensual sexual images by feeding in photographs.

TTP reported that multiple ads were matched to real photos of children that appeared online. In one case, a press photo of a young member of a European royal family was animated into a video showing a graphic sex act. Another ad took a photo of a 14-year-old Instagram influencer showing off her sports club uniform and turned it into a video of her performing oral sex. A third victim, a preteen in a pink athletic outfit from stock photos, was morphed into a video where she appears frightened while being molested by an adult male.

The US Department of Justice has previously clarified that AI-generated CSAM is equally illegal under federal child pornography laws. Despite this, Meta took days to remove the ads after they were brought to its attention. TTP matched ads to photos of real children that had appeared online and said the posts appear to violate federal law.

Meta has not publicly commented on the specific findings, but the company did remove the ads after being alerted. The incident raises serious questions about the effectiveness of automated content moderation systems at one of the world's largest social media platforms.

Why This Matters

The ads weaponise AI to create non-consensual sexual images of real children, causing direct harm to the victims whose photos are stolen and manipulated.
Meta's failure to detect hundreds of such ads suggests systemic gaps in content moderation that bad actors can exploit at scale.
The case may spur renewed calls for legislation specifically targeting AI-generated CSAM and platform accountability.

Background

AI-powered 'nudify' apps have proliferated in recent years, often hosted in jurisdictions where enforcement is limited. The US Department of Justice has stated that AI-generated CSAM is not exempt from existing child pornography laws, but enforcement remains challenging. Meta has faced repeated criticism for its content moderation practices, particularly around harmful imagery, and has invested heavily in automated detection tools. The Tech Transparency Project is a non-profit that monitors tech platforms for policy violations.

Key Perspectives

Tech Transparency Project: Documented 332 ads that bypassed Meta's moderation systems, including ads that used real photographs of children to create fake sexual videos. The organisation argues that these ads violate federal law and show Meta's systems are inadequate. Meta: After being alerted, the company removed the ads. However, the fact that they ran at all—and took days to take down—indicates a failure in proactive detection. Meta has not provided a public explanation for the gap. Child safety advocates: Likely to point to this investigation as evidence that voluntary moderation cannot be trusted, and that stronger legal obligations and penalties are needed for platforms that host or profit from such content.

What to Watch

Whether the US Department of Justice opens an investigation or takes enforcement action against either Meta or the app developers.
Meta's response: if it announces new moderation policies or changes to how it reviews ads for AI-generated CSAM.
Potential legislative action in the US or Europe specifically targeting AI-generated non-consensual imagery involving minors.

ZOTPAPER