dorsal/arxiv
View SchemaWhen Should We Introduce Safety Interventions During Pretraining?
| Authors | Dylan Sam, Sachin Goyal, Pratyush Maini, Alexander Robey, J. Zico Kolter |
|---|---|
| Categories | |
| ArXiv ID | 2601.07087vv1 |
| URL | https://arxiv.org/abs/2601.07087 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
Ensuring the safety of language models in high-stakes settings remains a pressing challenge, as aligned behaviors are often brittle and easily undone by adversarial pressure or downstream finetuning. Prior work has shown that interventions applied during pretraining, such as rephrasing harmful content, can substantially improve the safety of the resulting models. In this paper, we study the fundamental question: "When during pretraining should safety interventions be introduced?" We keep the underlying data fixed and vary only the choice of a safety curriculum: the timing of these interventions, i.e., after 0%, 20%, or 60% of the pretraining token budget. We find that introducing interventions earlier generally yields more robust models with no increase in overrefusal rates, with the clearest benefits appearing after downstream, benign finetuning. We also see clear benefits in the steerability of models towards safer generations. Finally, we observe that earlier interventions reshape internal representations: linear probes more cleanly separate safe vs harmful examples. Overall, these results argue for incorporating safety signals early in pretraining, producing models that are more robust to downstream finetuning and jailbreaking, and more reliable under both standard and safety-aware inference procedures.
{
"annotation_id": "f27f178a-09ee-4615-b5ee-e977776d280f",
"date_created": "2026-02-17T05:53:12.431000Z",
"date_modified": "2026-02-17T05:53:12.431000Z",
"file_hash": "8655d6d8ef49eb0e9db56d7f126e1cafafd3cc222ba7b272ae71af35c6adfdcc",
"private": false,
"record": {
"abstract": "Ensuring the safety of language models in high-stakes settings remains a pressing challenge, as aligned behaviors are often brittle and easily undone by adversarial pressure or downstream finetuning. Prior work has shown that interventions applied during pretraining, such as rephrasing harmful content, can substantially improve the safety of the resulting models. In this paper, we study the fundamental question: \"When during pretraining should safety interventions be introduced?\" We keep the underlying data fixed and vary only the choice of a safety curriculum: the timing of these interventions, i.e., after 0%, 20%, or 60% of the pretraining token budget. We find that introducing interventions earlier generally yields more robust models with no increase in overrefusal rates, with the clearest benefits appearing after downstream, benign finetuning. We also see clear benefits in the steerability of models towards safer generations. Finally, we observe that earlier interventions reshape internal representations: linear probes more cleanly separate safe vs harmful examples. Overall, these results argue for incorporating safety signals early in pretraining, producing models that are more robust to downstream finetuning and jailbreaking, and more reliable under both standard and safety-aware inference procedures.",
"arxiv_id": "2601.07087",
"authors": [
"Dylan Sam",
"Sachin Goyal",
"Pratyush Maini",
"Alexander Robey",
"J. Zico Kolter"
],
"categories": [
"cs.LG"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "When Should We Introduce Safety Interventions During Pretraining?",
"url": "https://arxiv.org/abs/2601.07087",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "f5e48cf4-377f-46ad-b663-7ab39b24a135",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}