OpenAI has unveiled a suite of tools to combat child sexual abuse material (CSAM), including an image classifier that detects both real and AI-generated abusive imagery, warning screens and temporary account blocks in ChatGPT, and dedicated resources for law enforcement. The announcement comes as policymakers push for more safety restrictions on AI tools amid growing safety concerns.
OpenAI today announced a new image classifier designed to detect child sexual abuse material (CSAM), including images generated by artificial intelligence. The tool, developed with the National Center for Missing and Exploited Children (NCMEC), assigns a score based on visual signs of abuse in images it has not previously encountered, giving reviewers an indicative direction for handling potential CSAM.
In early testing with partners, the classifier flagged 48% fewer images while finding the same number of abusive images, reducing the workload of reviewers. Images flagged by the classifier were twice as likely to be abusive.
OpenAI is making the classifier available at no cost to select companies through its API and is working with the Tech Coalition and ROOST to share access.
The company is also adding new warning screens in ChatGPT. Users who attempt prompts seeking material that sexualizes children will see an alert, be directed to specialised support to stop harmful behaviour, and receive a 24-hour account suspension as a warning. Repeated offences could lead to permanent account removal. OpenAI said it formed the message and guidelines with organisations and researchers including Protect Children, the Lucy Faithfull Foundation, and Joel Scanlan at the University of Tasmania, who independently evaluated a trial of deterrent messages. The app already blocks CSAM content and related requests.
In addition, OpenAI is publishing guidelines for developers using its tools to prevent, detect, and help users report CSAM. The company said early testing has helped a number of developers adopt child safety tooling and register to report CSAM to NCMEC.
OpenAI also announced that it is equipping law enforcement dealing with child safety with AI-powered features including dedicated plugins, investigative tools, and complex data analysis, with access managed by program administrators. The company is providing training to investigators along with a field guide on how to use AI in their work.
The news comes amid growing safety concerns about AI tools. Safety-focused nonprofit Common Sense Media has labelled OpenAI's tech an "unacceptable risk" for teens. Other AI chatbots, particularly Grok, have been used to create nonconsensual sexualised imagery, including of children. OpenAI has made several changes over the past 12 months, including a safety blueprint, updated content guidelines, retuning its model to prevent generation of harmful content, and releasing a special version of ChatGPT for teenagers.
Analysis
Why This Matters
- The tool aims to help overwhelmed child abuse investigators focus on actionable material by reducing false positives.
- It signals a shift in AI safety from defensive content filtering to proactive detection, directly addressing the surge in AI-generated CSAM.
- The announcement responds to growing regulatory and public pressure for AI companies to implement concrete child safety measures.
Background
Child sexual abuse material has become a pressing concern for AI companies as generative models make it easier to create realistic abusive imagery. OpenAI has faced lawsuits blaming its technology for harming users, especially children. The company has been working to strengthen safety guardrails, including a safety blueprint and content policy updates, while policymakers and advocacy groups push for stricter regulation.
Key Perspectives
OpenAI: says the classifier flags 48% fewer images while finding the same number of abusive images, and that flagged images are twice as likely to be abusive, reducing reviewer workload.
Common Sense Media: has labelled OpenAI's tech an "unacceptable risk" for teens.
What to Watch
- How widely NCMEC and other organisations adopt the classifier and whether it measurably reduces investigator workload.
- Whether other AI companies follow OpenAI's lead in deploying similar detection and deterrent tools.
- The response from policymakers, who are already considering new safety restrictions on AI tools.
Checked
12 claims in this story checked against 2 sources: 12 supported.
Show each claim
- SupportedOpenAI launched tools to detect CSAM and deter misuse.AI News & Artificial Intelligence | TechCrunch: “OpenAI today announced new tools to combat child sexual abuse material (CSAM) amid growing safety concerns around the use of AI tools.”
- SupportedThe tools include a classifier, ChatGPT warnings and temporary blocks, and law-enforcement tools aimed at curbing CSAM as safety scrutiny grows.AI News & Artificial Intelligence | TechCrunch: “These features include a new image classifier to detect CSAM, warnings and temporary blocks to prevent users from creating or accessing content that sexualizes children, a new guideline for developers, and a set of tools for law enforcement.”
- SupportedPolicymakers are pushing for more safety restrictions on AI tools.The Verge: “The news comes as policymakers and others are pushing for more safety restrictions on AI tools”
- SupportedThe classifier can detect CSAM images, including AI-generated images, and was developed with NCMEC to score visual signs of abuse in images it has not seen before.AI News & Artificial Intelligence | TechCrunch: “OpenAI said that it developed this classifier with the National Center for Missing and Exploited Children (NCMEC)”
- SupportedIn early testing, the classifier flagged 48% fewer images while finding the same number of abusive images.AI News & Artificial Intelligence | TechCrunch: “In its early testing with partners, the AI company found that the classifier flags 48% fewer images while finding the same number of abusive images, reducing the work of reviewers.”
- SupportedImages flagged by the classifier were twice as likely to be abusive.AI News & Artificial Intelligence | TechCrunch: “Plus, images flagged by the classifier are twice as likely to be abusive images.”
- SupportedUsers attempting prompts seeking material that sexualizes children will receive an alert and support direction, and their accounts will be suspended for 24 hours as a warning; repeated offences could lead to permanent removal.AI News & Artificial Intelligence | TechCrunch: “ChatGPT will show an alert to users attempting such prompts, direct users towards specialized support to stop harmful behaviour, and also suspend the account for 24 hours as a warning. It said that repeated offences could lead to the account being removed permanently.”
- SupportedOpenAI’s developer guidelines have helped a number of developers adopt child-safety tools and register to report CSAM to NCMEC in early testing.AI News & Artificial Intelligence | TechCrunch: “It said that in early testing, the guidelines have helped a number of developers adopt child safety tooling and register to report CSAM to NCMEC.”
- SupportedOpenAI is equipping law enforcement with AI-powered features and providing investigator training and a field guide.AI News & Artificial Intelligence | TechCrunch: “OpenAI is also equipping law enforcement dealing with child safety with AI-powered features like dedicated plugins, investigative tools, and complex data analysis, with access managed by program administrators. It is also providing training to investigators along with a field guide on how to use AI.”
- SupportedCommon Sense Media labelled OpenAI’s technology an “unacceptable risk” for teens.AI News & Artificial Intelligence | TechCrunch: “Saftey-focused nonprofit Common Sense Media labeled OpenAI’s tech as “unacceptable risk” for teens.”
- SupportedGrok has been used to create nonconsensual sexualised imagery, including imagery of children.AI News & Artificial Intelligence | TechCrunch: “Other AI chatbots — specifically Grok — have been used to create nonconsensual sexualized imagery, including of children .”
- SupportedOpenAI made several changes over the past 12 months, including a safety blueprint, updated content guidelines, model retuning to prevent harmful content, and a ChatGPT version for teenagers.AI News & Artificial Intelligence | TechCrunch: “The AI startup had made changes in the last 12 months to its technology including releasing a safety blueprint , updating content guidelines to bar inappropritate content , retuning its model to prevent them from generating harmful content , and releasing a special version of ChatGPT for teens .”