Why a job is queued
A queued job is usually working correctly. Rescale re-checks it about once a minute and launches it when conditions allow. This page covers which conditions a job can be waiting on.
Queued is not stuck
When you submit, the job enters a pending queue and is evaluated against a set of preconditions. It waits there until all of them pass, then goes on to provision hardware.
You don't need to resubmit. Rescale keeps retrying automatically. A job that can't satisfy its preconditions for a week is eventually removed from the queue.
The job's log messages name the condition it's waiting on. That's the fastest way to find out what's holding it.
What a job is waiting for
The budget precondition asks only whether you can afford the job's first hour, walltime isn't considered. A long job can therefore pass this check and still be stopped later if funds run out. See budget enforcement.
License-aware queuing
If you declare license feature usage when creating the job, Rescale holds it in the queue until the features are actually free, rather than launching it and having the checkout fail on a paid cluster. It's worth declaring for this reason alone.
Cancelling a submission
A pending job can be pulled back out of the queue with Cancel Submission. It returns to the same state as a saved-but-never-submitted job, and you can edit and resubmit it.
Requeuing: when a started job goes back to the queue
Sometimes a job that has already started returns to the queue, most often because cloud capacity became unavailable, or an interruptible instance was reclaimed.
What happens:
- The job goes back to Pending and re-evaluates every precondition.
- Any running cluster is shut down.
- When it starts again, it gets a brand-new cluster. Clusters are never reused.
The job itself is preserved; the hardware underneath it is replaced.
Requeues back off deliberately
A requeued job doesn't retry immediately. It waits, and each subsequent attempt waits longer: exponential backoff up to a cap.
That's intentional: if capacity is unavailable, retrying instantly just fails again. Your workspace's retry timing is configurable by an admin.
Like an ODE interruption, a requeue restarts your application from the beginning unless it can resume from a checkpoint. Input files are re-downloaded, so a job whose input files were deleted after submission will fail to restart.
Priority after a requeue
ODP jobs stay on ODP. ODE jobs may be moved to ODP for a final attempt so they can actually finish: see job priority.
What "Resource limitation" means
A job whose status reason reads Resource limitation was requeued because resources were unavailable. It indicates a retry, not a failure. The job is still trying. Common triggers:
- The provider had no instances of your coretype available
- A vCPU quota was reached
- Allocated hardware was underperforming and got replaced
- Interruptible capacity was reclaimed
Related
- Job priority — interruption risk and requeue behavior
- Checkpoint long simulations — surviving a restart
- Job statuses — what each status means
- Budget enforcement — the budget precondition
- Troubleshooting — when it's a failure rather than a wait