Zum Hauptinhalt springen
This page is not translated yet. Deutsch is switched on so the navigation and URLs can be tested; the text below is the English original. Use the language selector in the header to switch back, or see the documentation home.

Advanced job features

how-to · Reviewed 2026-07-30 GA

Four capabilities that sit outside the standard job flow: choosing hardware with data rather than habit, running containerized NVIDIA applications, keeping a cluster alive to debug interactively, and Monte Carlo sampling in a DOE.

Choosing hardware with the Intelligent Computing Framework​

The Intelligent Computing Framework (ICF) gives you a data-driven way to pick architecture, cloud provider, and region for a workload. Instead of guessing, you compare four metrics across the options Rescale supports:

  • Performance: Composite score based on benchmarks at various scales. Higher performance reduces time to solution and unlocks larger problems.
  • Maturity: Uptime track record, capacity history, and volume of core hours on the architecture. Higher maturity means more predictable service delivery.
  • Value: Performance divided by price. The same workload can cost very different amounts across regions, providers, and pricing tiers (on-demand, spot, reserved).
  • Sustainability: Flags architectures running on cleaner grids, for example renewable-heavy regions. This is the newest metric and more benchmarks are being added.

Use ICF when you are deciding a default coretype for a workload, when you are justifying a hardware choice to finance or IT, or when you want to move a workload to a different region to cut cost or carbon. Admins can pair ICF with Intelligent Resource Automation to encode these decisions as policy.

Running NVIDIA NGC Catalog applications​

The NVIDIA NGC Catalog is a curated set of GPU-optimized container images for AI, HPC, and visualization. Rescale packages NGC access as a software tile so you can pull a container and run it without provisioning infrastructure yourself.

As a batch job​

  1. Pick Basic as the Job Type.

  2. Upload your run script or test code as input files (for example, test.zip).

  3. In the Software panel, select NVIDIA NGC Catalog. Rescale pre-populates a command that logs in to NGC and pulls a container.

  4. In License Options, check Use Existing License and paste your NGC API Key and Org Code. Generate these from your NGC account.

  5. Edit the command to pull and run the container you need. For a TensorFlow example:

    docker pull nvcr.io/nvidia/tensorflow:22.06-tf1-py3
    docker run --gpus all -v ${PWD}/test:/test nvcr.io/nvidia/tensorflow:22.06-tf1-py3 \
      bash -c 'cd /test; python test.py'

  6. Pick GPU hardware, for example 4 GPUs for a training job, and submit.

Monitor progress through process_output.log on the Status tab. See Live-tail a running job for the live-tailing workflow. For additional examples, see Tutorial: Running NVIDIA NGC Catalog Applications on Rescale and Tutorial: Run NVIDIA NGC CUDA Quantum on Rescale.

On a workstation​

Pick the NGC Catalog Interactive Workflow tile when you create a workstation. Enter your NGC API Key and Org Code the same way as for a batch job. Once the workstation is ready, open a terminal and log in to NGC:

echo -e "\n \n $(echo $NGC_ORG_ID) \n \n \n" | ngc config set
docker login -u '$oauthtoken' --password-stdin nvcr.io <<< $(echo $NGC_CLI_API_KEY)

Then pull and run containers interactively:

docker pull nvcr.io/nvidia/tensorflow:22.06-tf1-py3
docker run --gpus all -it --rm -v ${PWD}/test:/test nvcr.io/nvidia/tensorflow:22.06-tf1-py3 bash

Other examples in the source include NVIDIA Modulus (docker pull nvcr.io/nvidia/modulus/modulus:22.03.1). Modulus does not require an NGC API key.

Debugging with sleep​

A simple pattern for debugging a job that fails early: append sleep 3600 to your command. Rescale will keep the cluster up for the duration of the sleep, which lets you SSH in (if enabled) or open an interactive terminal to inspect the filesystem state, rerun the command manually, and see what is different. Remove the sleep before running production jobs.

Monte Carlo sampling in DOE​

Design of Experiments jobs support Monte Carlo probability distributions for sampling input parameters. See Monte Carlo Probability Distribution Methods for DOE Jobs for the supported distributions and how to configure them.

On the roadmap​

  • Admin Low Priority Settings - Run Overnight: administrator-configurable policies for scheduling lower-priority jobs to run during off-peak hours across a workspace.