Towards Safer Pretraining: Analyzing and Filtering Harmful Content in Webscale datasets for Responsible LLMs

Web datasets contain significant harmful content: The paper documents that popular pretraining datasets like Common Crawl, C4, and FineWeb contain substantial amounts of hate speech, misinformation, and biased narratives that can be directly absorbed by language models during training.

Harmful content can be systematically categorized and filtered: The researchers developed a taxonomy distinguishing between "Topical" harmful content (intentionally discussing harmful topics) and "Toxic" content (using abusive language), along with HarmFormer, a specialized model that can automatically identify and filter such material before it reaches LLMs.

Filtering harmful content is feasible at scale: By creating tools like the Topical and Toxic Prompt (TTP) dataset and demonstrating HarmFormer's high accuracy, the paper shows that organizations can practically implement content filtering during the pretraining phase rather than only addressing harms after models are deployed.

Models can be tested for vulnerability to toxic inputs: The introduction of the HAVOC benchmark provides a way to evaluate how well-trained LLMs resist adversarial toxic prompts, enabling developers to assess whether their models have internalized harmful patterns from training data.

Responsible AI requires upstream intervention: Rather than relying solely on post-deployment safeguards, this work emphasizes that building trustworthy LLMs requires cleaning training data at the source, making safer pretraining a foundational responsibility for developers.