Cluster Metrics
When a job is slower than expected, Cluster Metrics shows whether the constraint is your model or the hardware it runs on. Check it before changing solver settings.
Open the panel
Open a job and click Status to find the Cluster Metrics panel. It shows CPU, GPU, memory, storage, and network usage, including per-node status when your job spans multiple nodes.
Cluster Metrics updates while the job is running and stays on the job's detail page after the job completes, so you can go back and review the same data later. The panel samples periodically rather than continuously; click Refresh Cluster Metrics to force a fresh sample on a running job.
Cluster Metrics only collects data going forward. A job that completed before this feature was available won't show any metrics, even if you open it today.
Read the three symptoms that matter
Most hardware-level problems present as one of these:
Free memory trending toward zero
The cluster doesn't have enough memory for your model. The job is likely swapping (which is dramatically slower than running in memory) and may be killed outright by the OOM killer.
What to do: resubmit on a coretype with more memory per core, or reduce the model size. A job killed for memory usually reports exit code 137.
One node red while the others are green
A node-level fault on a multi-node job. Rescale often detects this itself and terminates the cluster with Detected unhealthy instance(s), shutting down instance group.
What to do: resubmit. This is an infrastructure fault, not a problem with your job. If it recurs on the same job, see Troubleshooting "Detected unhealthy instance(s)".
Flat CPU on cores you expect to be busy
Your solver either hasn't actually started, or it's blocked on I/O rather than computing.
What to do: live-tail process_output.log to see where it is. A solver that's still reading a large mesh looks exactly like this and is fine; a solver that printed a license error and stalled also looks like this and is not.
A model that's swapping looks like a slow model. If you tune solver settings without checking memory first, you'll be optimizing the wrong thing.
Related
- Live-tail a running job — what the solver is printing
- Exit codes — 137 and other termination codes
- The job lifecycle — whether the job has even reached the running stage