Monitoring & Status
Watching a job while it runs, and determining what it did after it stops. Start with the lifecycle if a job is behaving unexpectedly, or go to live tailing to watch residuals.
Watch a running job
The job lifecycle
The five stages every job moves through, and what each one is actually doing.
Live-tail a running job
Follow solver output line by line as it's written: residuals, timesteps, errors.
Cluster Metrics
Read CPU, GPU, memory, storage, and network usage to tell a slow solver from a starved cluster.
Work out what happened
Read job logs
Which log to open for which question, and how to interpret the exit code.
Job statuses
Reference: every status a job or workstation can report, and what it means.
Exit codes
Reference: the exit codes in process_output.log and their causes.
Solver-specific monitoring
Some solvers can be monitored interactively while the batch job runs, by connecting a local client over an SSH tunnel.
Related
- Create and submit a job — configuring and submitting in the first place
- Stop a job — stopping a job you're watching
- Troubleshooting — when the logs show a failure
- Files and data — retrieving results after the run