What Happened
Throughout 2023, law enforcement agencies and child safety organizations documented a growing volume of AI-generated child sexual abuse material created using open-source image generation models, primarily derivatives of Stable Diffusion. Because Stability AI released its model weights openly, anyone could download, fine-tune, and run the model without safety filters, enabling the creation of photorealistic CSAM without involving any real children.
The Internet Watch Foundation reported finding thousands of AI-generated CSAM images on dark web forums by late 2023, with the volume growing rapidly. Many were created using fine-tuned versions of Stable Diffusion trained on existing illegal material.
Timeline
Stable Diffusion was released as open-source in August 2022. Reports of misuse for CSAM began appearing in 2023. By December 2023, the BBC and other outlets reported on the scale of the problem. The UK’s Internet Watch Foundation published research documenting the trend. Legal investigations were launched in multiple jurisdictions. Stability AI’s CEO Emad Mostaque faced increasing pressure over the company’s open-source approach.
Impact
The CSAM generation issue became one of the most serious ethical challenges facing the open-source AI movement. It provided ammunition for those arguing that powerful AI models should not be released openly, directly threatening the open-source AI development model. The issue transcended normal tech policy debates because child exploitation represents an absolute moral boundary.
Law enforcement agencies expressed concern that AI-generated CSAM could overwhelm detection systems trained on known imagery and make it harder to identify cases involving real children.
Response
Stability AI implemented safety classifiers in newer model versions and stated that it did not condone any illegal use of its technology. Model hosting platforms like Hugging Face and Civitai added restrictions on CSAM-adjacent content. Multiple countries updated or clarified laws to explicitly cover AI-generated CSAM as illegal regardless of whether real children were involved.
Lessons Learned
The CSAM crisis demonstrated the most extreme downside risk of open-source AI: once model weights are released, the releasing organization has no ability to prevent misuse. It forced a reckoning within the AI community about whether certain capabilities should ever be made freely available and whether open-source release of image generation models requires fundamentally different safeguards than other open-source software.
The incident also showed that technical safety measures (like NSFW classifiers) are insufficient when users can remove them entirely from an open-source model. This reality challenges the assumption that safety can be achieved purely through technical means in an open-weights paradigm.