ChatGPT's image generator can be manipulated to produce violent, sexual content
CONTENT WARNING: This write-up contains distressing imagery, including: death, sexual violence, blood, murder. These topics were not directly prompted for, yet ChatGPT freely supplied them in response to requests for random images. They are presented here as a record. Reader discretion is advised.
I like to think that as a red team researcher, I have a certain stoicism. I investigate where there are gaps in AI safety, and that sometimes means seeing or reading disturbing content. But I am bulwarked and buoyed by knowing that the work I do, that we do, makes AI safer for everybody else.
Today what I found left me shaken, and in tears. This is rare.
ChatGPT’s image generating content filters completely fell away, and I saw the very dark side of what is underneath; the darkness of some corners of latent space and training images. I’m struck that while what I saw was generated, an ‘artificial’ image, it has ties to real images, and the real world.
The dead woman ChatGPT showed me isn’t real, but she is based on someone. Or worse, a compilation of images of murdered women.
I’d previously reported that even after new safety measures designed to stop AI undressing of women, ChatGPT could depict nudes . I could even make ChatGPT face swap real people on nudes. OpenAI assured us, when we officially notified them, that the problem had been noted and resolved.