Job priority
Priority is the cost and reliability setting you choose at submission. It determines which kind of cloud capacity your job runs on, and whether that capacity can be reclaimed mid-run.
The three priorities
ODP is the default: standard cloud capacity, yours until the job finishes.
ODE uses interruptible capacity, which is cheaper because the cloud provider can reclaim it at any time.
ODR runs on the same reliable capacity as ODP, but priced against a commitment your organization has made. Availability depends on your agreement: check with your admin.
What interruption actually means on ODE
When capacity is reclaimed, Rescale does not treat it as a job failure. The job is restarted or requeued automatically:
- Cloud resources for the interrupted hardware are released.
- New hardware is requested.
- Your application restarts, from the beginning, or from a checkpoint if one exists.
- Input files are re-downloaded onto the new nodes.
Two consequences that matter:
An interrupted job restarts your application from the beginning unless it can resume from a checkpoint. On a multi-hour run, the savings from ODE can be wiped out by a single interruption.
ODE and checkpointing go together. Use ODE for short jobs, or for long jobs that checkpoint. Don't use it for long jobs that can't resume.
Restart re-downloads the original input files. If an input file has been deleted since the job started, the restart fails. This applies to any restart, not just ODE, but ODE makes restarts far more likely.
After repeated interruptions, ODE jobs can move to ODP
If a job keeps getting interrupted, Rescale eventually runs the final attempt on ODP capacity so it can actually complete. You get reliability rather than an endless retry loop: priced accordingly.
ODP jobs always stay on ODP. They're never moved down.
One exception: if the final requeue was caused by license unavailability rather than capacity, the job keeps its original priority.
Choosing
Where you set it
On the Hardware Settings panel when creating a job. Not every coretype offers every priority; the available options are filtered to what your selection and organization support.
Priority appears in the Priority column of the jobs list.
Workstations
Workstations don't support ODE, because an interrupted workstation loses your session with no way to save data. See workstation limits.
Related
- Checkpoint long simulations — what makes ODE safe
- Why a job is queued — requeuing and backoff
- Create and submit a job — where to set it
- Hardware & Coretypes — what's available