Budget enforcement
A budget is enforced at two separate moments. It is checked once when a workload is submitted, and then evaluated continuously while that workload runs. The two checks behave differently, and passing the first is not a guarantee of surviving the second.
The check at submission
Every job, workstation and High-Performance Storage submission runs a precondition check against the budgets that apply to it. The rule it applies is narrower than most people expect: the target must have enough remaining budget to cover the first hour of the workload, and nothing more.
Wall time is not considered at all. A hundred-hour job passes this check as long as the target can afford a single hour of it.
A workload that fails the precondition never launches in the first place, so it does not start and then get stopped. It queues instead, and waits until the budget can cover that first hour.
The check while a workload runs
Passing the precondition does not make a workload safe. The budget continues to be evaluated as charges accrue against it, and the Strict Budget setting decides what happens at the moment the cap is actually crossed.
There is no grace period and no partial completion. A job ninety percent of the way through a run is terminated the moment the cap is crossed. Files written up to that point are kept, but the run itself is lost.
Use a strict budget only where hard cost containment matters more than the work in flight. For most engineering teams, leaving it off is the right default.
Why a long job can start and then be stopped
The interaction between the two checks produces a result that looks like a fault but is not, so it is worth walking through an example.
A team has $100 of budget remaining. An engineer submits a fifty-hour job costing $10 an hour. The precondition examines the first hour only, comparing $10 against the $100 remaining, and the job passes and launches. Ten hours later the budget is exhausted, and with Strict Budget on the job is terminated. Forty hours of intended runtime never happen.
Nothing has malfunctioned here. The precondition is a first-hour check by design. What it means in practice is that a submission passing the check is not a promise that the workload will complete. Size budgets against the total consumption you expect rather than against the number of submissions, and treat the 80% notification as the point at which to act.
Blocked workloads queue rather than fail
A submission blocked by a budget is queued, not rejected. It launches on its own once budget becomes available, whether that happens because a periodic budget reset, because an admin raised the cap, or because other usage finished and freed room under it. Users do not need to resubmit anything.
Related
- How budgets work — targets and precedence
- Create a budget — setting the Strict flag
- Spend notifications — warnings before the cap
- Job statuses — how a budget-held job appears