The job lifecycle
Every job moves through the same five stages. This page covers what each stage is waiting on, which tells you whether a job is stuck, slow, or working normally.
Why there are stages at all
Rescale does not keep a cluster running for you. Each job gets hardware provisioned for it, runs, and hands the hardware back. That's what makes it possible to submit a 4-core job and a 4,000-core job from the same page without doing any capacity planning yourself.
The cost of that model is that a job has real work to do before your solver starts: capacity has to be found, a cluster has to boot, and your input files have to be staged onto it. The five stages are that work, made visible.
The five stages
Open a job and click Status to see them. Each stage shows a green checkmark when it passes.
1. Queued
Rescale has accepted your submission and is checking preconditions. The two that most often hold a job here are capacity, whether your chosen coretype is available in your region right now, and budget, since a job must be able to afford its first hour before it can launch.
A job can sit in Queued legitimately. It is not stuck.
2. Validating Input
Your input files and launch command are checked before any hardware is paid for. This is deliberately early: a typo in a filename is much cheaper to catch here than after a cluster has booted.
3. Starting Cluster
The hardware you selected is being provisioned. This stage takes minutes, not seconds. It's the startup latency inherent to per-job provisioning. Nothing is wrong if a job spends several minutes here.
4. Running Job
Your software is executing. This is the only stage where your launch command is actually running, and it's the stage where live tailing and cluster health become useful.
5. Stopping Cluster
Output files are copied from the cluster back into Rescale Cloud Storage, then the cluster is released.
Clusters are destroyed at the end of this stage. Anything your solver wrote outside the job's working directory is gone. It is never copied back. If a tool writes logs to your home directory, copy them into the job directory as the last line of your command.
What a failure looks like
If any stage fails, its checkmark becomes a red X and the job moves to Failed. Which stage failed narrows the cause considerably:
Jobs you stop yourself end in Stopped rather than Failed. Jobs that finish normally end in Completed.
Completed means your command ran and exited. It does not mean the solver converged or produced the answer you wanted. See exit codes for why exit code 0 isn't proof of a good run.
Workstations follow the same shape
Workstations use the same stages, plus Active, Suspending, and Suspended, because unlike a job a workstation can be paused and resumed instead of only terminated. See suspend and resume for moving between those states.
Related
- Job statuses — the full reference of every status
- Live-tail a running job — watching stage 4 in progress
- Read job logs — diagnosing a failure
- Stop a job — graceful and force stops