Troubleshooting
When a job fails, the Rescale platform usually tells you where to look but not always what to do. This page is organized by symptom so you can match what you are seeing to a concrete first step. Start with the general diagnostic pattern, then jump to the section that matches your failure mode.
The general diagnostic pattern
Whatever the symptom, work in this order. You will narrow most problems to a root cause in a few minutes.
-
Check the Status tab. Did validation pass with a green check? What does the Gantt chart show? Are there error messages in Job Logs? See The job lifecycle for what each stage means, and Read job logs for which log to open.
-
Open
process_output.log. On the Results page, search forprocessorlogand open it with the screen icon. This file contains stdout from your software, license checkout messages, and most error text. The exit code at the bottom tells you what kind of failure it was:See Troubleshooting Jobs by Reading process_output.log and Exit Codes on Rescale for a deeper reference.
-
Check software-specific logs. Most solvers write additional
.log,.out,.live, or.datfiles with more detail than stdout. For ANSYS, see How to Find Ansys Debug Logs for Troubleshooting and Understanding Ansys License Debug Files. -
Check Cluster Metrics. If memory ran out or a node went unhealthy, the root cause is infrastructure, not the input deck.
-
Re-read the command. Most batch failures are tiny command issues: a wrong path, a missing
cd, a Windows line ending, or the wrong software version.
If you need Rescale Support to look at a browser session, capture a HAR file with How to Capture and Export a HAR File for Rescale Support.
Submission and job loading
Submit button is greyed out
A required field is missing. Most often it is a required custom field, a missing input file, or an unset license entry. Fill the missing field and the button enables. See Why is the Job Submit Button Greyed Out?.
"We could not load the job that you requested"
The job page itself fails to load. This is usually a browser state or permissions issue. See "We Could Not Load the Job That You Requested" Troubleshooting Guide.
Sign-in errors
- "This method of authentication is disabled for your account": the sign-in method is not allowed for your org. Use a different method or contact your admin. See Authentication Method Disabled and Troubleshooting: "This method of authentication is disabled".
- SSO login failing: see Troubleshooting SSO Login Issues for Individual Users.
- SSO conflicting settings on workspace: see SSO Configuration Error: Conflicting Workspace Settings.
File, path, and command errors
Missing input files
Rescale unpacks your compressed archive into the job's work directory. Confirm the archive contains every file the solver needs and that filenames are correct.
Command not found, or wrong paths
Most path failures come from how the archive was packaged. Rescale executes the command from the directory where the archive was unpacked, and assumes files are at the top level. If your files are in a subdirectory, prepend cd subdir && to the command, or repackage without the subdirectory. Use relative paths rather than absolute ones.
See Troubleshooting: Common Job Failures on Rescale (Files, Paths, Resources, Licensing).
/bin/bash^M: bad interpreter or similar line-ending errors
A Windows text editor saved your script with CRLF line endings. Rescale's Linux compute nodes expect LF. Re-save the file as Unix line endings, or clean it in place:
:%s/^M$//
where ^M is Ctrl-V Ctrl-M. See OpenFOAM Error: /bin/bash^M: bad interpreter.
"Command not found" with ampersand in filename
Special characters like & in filenames confuse the shell. Rename the file. See Troubleshooting Command Not Found Errors with Ampersand Characters in Filenames.
Multi-node worker cannot read files
For multi-node jobs, workers read from the NFS-mounted ~/work/shared directory. Rescale already defaults most known multi-node solvers to run from ~/work/shared, but if yours does not, prepend:
mv * shared
cd shared
to the command so the input files are reachable on every node.
Resource errors
Out of memory
- Exit code 137 in
process_output.logis the classic out-of-memory signature. - Avg Free Memory near zero on the Cluster Metrics panel confirms it.
Move to a coretype with more memory per core, reduce the per-core memory footprint, or reduce the input size. For ANSYS CFX, see Resolving Insufficient Memory Allocation Errors in ANSYS CFX.
Disk full
- Ansys Q3D Extractor: Resolving Disk Full Errors in Ansys Q3D Extractor Jobs
- MSC Nastran "No space left on device": Resolving MSC Nastran 'No space left on device' Error
Pick a coretype with more local disk or reduce the volume of intermediate output your solver writes.
"Detected unhealthy instance(s), shutting down instance group"
A node in your cluster went unhealthy and Rescale terminated the group to protect the run. This is usually a cloud-provider-level issue. Retry the job, and if it recurs, contact Support. See Troubleshooting: "Detected unhealthy instance(s)".
Job succeeds on X cores but fails on Y cores
Scaling issues are common in parallel codes: more cores per node means less memory per process, domain decomposition may break at different counts, and some codes hit hardcoded limits. See Troubleshooting: Job succeeds on X cores but fails on Y cores.
Licensing errors
License failures almost always show up as a checkout error near the top of process_output.log.
- STAR-CCM+ version 2502 and newer: Resolving STAR-CCM+ Licensing Errors.
- "License Server UP; Vendor Daemon Down": the license server machine is reachable but the vendor daemon is not running. See Troubleshooting 'License Server UP; Vendor Daemon Down' Error.
- COMSOL Could_not_obtain_license_for#Cluster Node: see COMSOL License Error.
- Cradle scFLOW checkout failures: see Troubleshooting Cradle scFLOW License Checkout Failures.
- Ansys Icepak
cache_password: see Resolving cache_password Error in Ansys Icepak Batch Jobs.
For ANSYS license debug files specifically, see Understanding Ansys License Debug Files.
Solver-specific errors
ANSYS Fluent
- Unbound variable errors: Resolving Unbound Variable Errors in ANSYS Fluent.
- Cortex Process Fatal: Troubleshooting 'Cortex Process Fatal' Error.
.scdocfile fails to load: Troubleshooting .scdoc File Loading Issues.
ANSYS CFX
- Floating point exceptions: Troubleshooting Floating Point Exception Errors.
- SIG_HANDLER error in routine FPX: Resolving SIG_HANDLER Error.
ANSYS Mechanical
- OpenMPI BTL/TCP process identifier errors: Resolving OpenMPI BTL/TCP Errors.
STAR-CCM+
- ORTE daemon startup errors: Resolving ORTE Daemon Startup Errors.
- LLVM ERROR on EFA-enabled coretypes: Fixing STAR-CCM+ LLVM ERROR.
Abaqus
- Divergence issues: Troubleshooting Divergence Issues in Abaqus Simulations.
CONVERGE
- Segmentation faults: Troubleshooting CONVERGE Segmentation Fault Errors.
Altair OptiStruct
- Domain Decomposition Mode Error #3413: Resolving Altair OptiStruct Error #3413.
Cadence
- General Cadence software issues: Troubleshooting Cadence Software Issues.
Stopping and workstation errors
Job or workstation won't shut down
Graceful stop sequences run in order: application shutdown, process cleanup, file upload, cluster termination. Any step can hang. If it does, use Force Stop from the Status tab (see stop a job). If force stop also fails, see Troubleshooting: Job or Workstation Won't Shut Down.
Workstation display does not fit the screen
Resolution negotiation between your client and NICE DCV produced a window that overflows. See Troubleshooting: Workstation Window Doesn't Fit the Screen.
NICE DCV wrong username/password
See Troubleshooting NICE DCV Wrong Username and Password Error.
Rescale Transfer Manager (RTM)
- "Invalid API Key": Resolving "Invalid API Key" Error.
- "The deployment identity does not match the subscription": RTM Deployment Identity Error.
- Path or filename too long on download: RTM Download Fails: Path or File Name Too Long.
- SSL certificate errors: Troubleshooting SSL Certificate Errors in RTM.
- "A task was canceled": Troubleshooting A Task Was Canceled Error.
- RTM logs: see Read job logs.
Browser file transfers
- Upload issues in the browser: Troubleshooting Browser File Upload Issues on Rescale.
- Download issues: Troubleshooting File Download Issues.
Rescale CLI
- Symlink installation permission error: How to Fix Rescale CLI Symlink Installation Permission Error.
- Java errors: Troubleshooting Java Errors in Rescale CLI.
Reaching out for help
When you open a ticket, Rescale Support needs:
- The job ID (from the ID column on the Jobs list).
- The text of the relevant error, ideally with a few lines of context from
process_output.log. - A HAR file if the issue is UI-specific (HAR capture guide).
Sharing this up front shortens the round trip.
On the roadmap
- Troubleshooter for Every Job
- Troubleshooting Agent: Spring Launch
Both will surface recommended next steps inline on the job page for common failure modes.
Related
- Read job logs and Check cluster health
- Stop a job for jobs that will not finish cleanly
- Create and submit a job for correct job configuration
- Advanced features for checkpoint restart when you do not want to start over