Recovery detects interruptions, classifies problems, excludes implicated machines and recommends restarting or stopping a job under configured policies. A restart uses the training program's latest checkpoint: the tool does not recreate progress lost without a checkpoint.
Attribution correlates logs, metrics and scheduler state to separate an initiating cause from cascading errors. NVIDIA says diagnosis depends on the quality of collected evidence and can remain ambiguous.
The published recovery timings are NVIDIA's internal measurements. They describe initiation of an action, without guaranteeing the full time until productive training resumes.
Sources 4
- Diagnose Distributed AI Workload Failures with NVIDIA Attribution
- Keep Distributed Training Running with Automated Workload Recovery
- NVIDIA détaille NVIDIA Recovery, service open source de reprise automatique des entraînements distribués
- NVIDIA details Attribution tool for diagnosing distributed AI failures