ChatGPT transcripts are reportedly read by humans to improve responses, including those with personal information — ‘Project Lilly’ has seen OpenAI hire hundreds of contractors to manually review logs

AI companies don’t have a great track record in areas like copyright or user privacy — unless they’re the ones on the short end of the stick, that is — but it’s generally known that the chat logs from platforms like ChatGPT are used for improving models. The mechanism as to how this happens was still a mystery until today. 404 Media just published a report about OpenAI’s process of human review for chat transcripts, explaining how the review process works, and how it involves other humans sometimes reading private information.

The rating project’s name at OpenAI is Project Lily. The publication got information on the project’s instruction guides, Slack channels, real ChatGPT conversations, and, of course, the rating system to classify conversations. The operators are called “prompt reviewers,” and their job is fairly simple: look at anonymized real-world chats, and judge the quality of ChatGPT’s responses to assess whether they actually answer the question, and that the text doesn’t overuse “AI-speak,” patronizing tones, emojis, or sycophancy, among other parameters. Anthropomorphizing and stating “personal” experiences are both off the table, meaning that while it’s OK for ChatGPT to say “I found some information,” it’s not OK for it to say “as a chef, I like to…” or “I know what that’s like.”

The work is “very rote,” according to a reviewer, but at reportedly over $50 an hour, it’s a high rate for what looks like reasonably simple work. The reviewer also said that their guidelines keep changing and are often self-contradictory, a feeling most software developers should easily identify with.

The person doesn’t think that most users are aware their chats are being read by others, though, something that’s particularly troubling when many use ChatGPT as an impromptu friend or therapist and put deep secrets in words for the bot to read.

While the chats allegedly go through an anonymization pass and reviewers don’t see usernames, OpenAI admitted to 404 Media that the filtering may let some personal data through, especially in shorter chats. The site notes that in many conversations, the user asks ChatGPT to keep the contents secret, as well. The version of the chat handed to reviewers also reportedly includes a “user memories summary,” containing a summary of the users’ questions and interests, context, and potentially even location.

Crucially, Project Lily does not grade the chats’ actual factual accuracy other than flagging obvious mistakes, implying that there’s likely at least one more team (or several) doing separate evaluations. Likewise, this reviewing is separate from manual safety checks that ascertain if someone might be looking to hurt someone else (or, presumably, themselves).

The existence of the project also indicates that contrary to these image AI companies try to cultivate, the models don’t improve just with technological advancement and better training sets — it appears you still need more than a few competent humans in the mix.

By now you may be wondering about the “allow us to use your chats to improve our product” (paraphrased) setting present in most consumer-facing chat bots. That setting is turned on by default in every bot we can think of, even with many paid plans. In ChatGPT’s case, it does default to off in Enterprise, Business, and Educational customers.

That toggle switch does not work retroactively, though, so any chats already in ChatGPT’s database will remain there unless the user requests deletion. Also, said deletion is also not retroactive, meaning that deleted chats may have already been hoovered and anonymized, and possibly reside in a dataset somewhere.

Although OpenAI initially had no answer to 404 Media’s inquiry on whether users were explicitly informed that their chats could be read by humans, the company eventually offered a link to one of its FAQ pages that discusses human review for the purpose of model improvement. We verified ourselves that said notice is at least two years old, and likely older. After the publication of the exposé, the firm changed its help page explaining how people can opt out of data collection, but there’s no mention of human operators in that text.

This type of data collection and review is a running theme across most providers. Google Gemini clearly states that “humans may review some saved chats” in its Privacy Hub. Anthropic’s stance is similar, with a page dedicated to this topic. Perplexity’s stance, meanwhile, is unclear, as its Privacy Notice doesn’t confirm or deny human access to chat logs.