Safety Flywheel

Opening

The safety flywheel is the observation that scaling automated red-teaming creates a self-reinforcing loop: better attackers produce better adversarial training data, which produces more robust defenders, which demands stronger attackers to find remaining failures. Each iteration of the loop can be evaluated against the previous generation's robustness benchmarks to verify progress.

Mechanism

A new safety flywheel cycle begins with a frontier-capable model (the attacker) being trained via self-play against a population of defender models on a diverse scenario set. The completed attacker model then generates adversarial examples used in the next defender's training run, after which the new defender is evaluated against both human-crafted and learned attack distributions. Progress is measured by monotonic improvement on held-out robustness benchmarks (e.g., GPT-5.6 Sol's 6× reduction in prompt-injection failures vs four months earlier). The flywheel fails if attackers cannot find novel failures after the previous cycle, which is bounded by the attacker model's reasoning capability.

Tradeoffs

The flywheel is dependent on attacker models that are themselves aligned with the operator's policies, otherwise adversarial training data introduces policy drift. It also concentrates dual-use capability in attacker-only deployments, complicating public disclosure of attack categories. Compute requirements scale with the gap between defender and attacker capability, potentially creating pressure to maintain dual-use models indefinitely.

Open Questions

Whether the flywheel can sustain improvements as defender robustness approaches an asymptotic ceiling is unknown. The relationship between flywheel iteration speed and capability progress speed is also empirical; if attackers plateau before capabilities plateau, safety progress will lag capability progress.