Checkpoint long simulations
A checkpoint is enough saved state to resume a simulation instead of restarting it. On runs longer than a few hours, checkpointing determines how much work a failure costs you.
Why it matters on Rescale specifically
Rescale restarts your application in more situations than you might expect, and a restart begins from the beginning unless a checkpoint exists:
- Interruptible capacity is reclaimed. If you chose ODE priority, this is a routine event, not an error.
- Hardware is replaced after a health check failure or underperformance.
- The job is requeued because capacity became unavailable — see why a job is queued.
Checkpointing is what makes cheap interruptible capacity rational for long jobs. Without it, one interruption can cost more than ODE saved.
Turn it on before you launch
Checkpointing is a solver setting, not a Rescale setting. Most simulation applications have built-in checkpoint and restart options, and they have to be enabled before the run starts. You can't add them to a job already in flight.
The flags and file naming differ per application, so check your solver's documentation. Rescale has per-application guides:
See software catalog for solver-specific pages.
Checkpoint only what a restart needs
More frequent and larger checkpoints are not better. Writing too much causes real problems:
- Storage fills up. Scratch space is finite, and a full disk stops the job — see troubleshooting.
- Out-of-memory failures. Heavy checkpoint writing can exhaust memory.
- The run gets slower. Time spent writing state is time not spent solving, and it extends total walltime.
Write the minimum a restart requires. Abaqus, for example, writes .rst files that let a run resume from the last completed step or iteration. That's what's needed, and no more.
How often
There's a real tradeoff, and the answer depends on run length:
The useful way to think about it: how much recomputation are you willing to accept? A checkpoint every two hours caps your worst-case loss at two hours.
It's not merely wasteful. It causes out-of-memory errors and full disks, both of which kill the job you were trying to protect. Frequent tiny checkpoints on a large model can be worse than none.
Snapshots are not checkpoints
Rescale's Snapshot feature captures intermediate files from a running job. It's useful for inspecting progress mid-run, but it is not designed for restarts. A snapshot won't generally give your solver what it needs to resume.
Use your application's own checkpoint mechanism for restart. Use Snapshots for looking at intermediate results.
Monitor long runs
Checkpointing pairs with monitoring. Check on multi-day jobs periodically so you catch an error early rather than discovering it a day later, some applications don't self-terminate after a failure, and will happily burn walltime doing nothing.
See live-tail a running job and check cluster health.
Related
- Job priority — why ODE needs checkpointing
- Why a job is queued — requeues restart your application
- Stop a job — graceful stops write restart files
- Monitoring & status — watching a long run
- Software catalog — solver-specific restart procedures