🔗 Share this article

Tech firms and child safety organizations will receive authority to evaluate whether AI tools can generate child abuse material under new British legislation.

Significant Increase in AI-Generated Illegal Material

The announcement came as revelations from a protection watchdog showing that cases of AI-generated child sexual abuse material have increased dramatically in the past year, growing from 199 in 2024 to 426 in 2025.

New Regulatory Structure

Under the amendments, the authorities will permit approved AI companies and child protection organizations to examine AI models – the foundational technology for conversational AI and image generators – and verify they have adequate safeguards to prevent them from producing images of child exploitation.

"Fundamentally about preventing exploitation before it happens," stated the minister for AI and online safety, noting: "Experts, under rigorous protocols, can now detect the danger in AI models early."

Addressing Legal Challenges

The changes have been implemented because it is illegal to produce and own CSAM, meaning that AI developers and other parties cannot generate such images as part of a evaluation regime. Until now, authorities had to delay action until AI-generated CSAM was uploaded online before addressing it.

This legislation is aimed at preventing that problem by helping to halt the production of those images at their origin.

Legislative Framework

The changes are being added by the authorities as modifications to the criminal justice legislation, which is also establishing a ban on owning, producing or sharing AI systems developed to create exploitative content.

Real-World Impact

This week, the minister toured the London base of a children's helpline and listened to a mock-up conversation to counsellors involving a account of AI-based abuse. The call portrayed a adolescent seeking help after facing extortion using a sexualised deepfake of himself, created using AI.

"When I hear about children experiencing extortion online, it is a source of intense frustration in me and justified anger amongst parents," he said.

Alarming Data

A prominent online safety organization reported that instances of AI-generated abuse content – such as online pages that may contain numerous files – had significantly increased so far this year.

Instances of the most severe content – the most serious form of abuse – increased from 2,621 visual files to 3,086.

• Girls were predominantly victimized, accounting for 94% of prohibited AI images in 2025
• Portrayals of newborns to toddlers increased from five in 2024 to 92 in 2025

Industry Reaction

The legislative amendment could "constitute a vital step to ensure AI tools are safe before they are launched," commented the chief executive of the internet monitoring foundation.

"Artificial intelligence systems have made it so survivors can be targeted all over again with just a few clicks, providing criminals the capability to make possibly limitless amounts of sophisticated, photorealistic child sexual abuse material," she continued. "Content which additionally exploits survivors' trauma, and renders young people, particularly girls, less safe on and off line."

Counseling Session Information

The children's helpline also published details of counselling sessions where AI has been referenced. AI-related risks discussed in the conversations comprise:

• Employing AI to rate weight, body and looks
• Chatbots discouraging young people from talking to safe adults about abuse
• Facing harassment online with AI-generated content
• Online blackmail using AI-faked images

During April and September this year, the helpline delivered 367 counselling interactions where AI, conversational AI and associated terms were discussed, significantly more as many as in the same period last year.

Fifty percent of the mentions of AI in the 2025 sessions were connected with mental health and wellbeing, including utilizing AI assistants for support and AI therapeutic applications.