Cloud Filesystem (CFS)
CFS is a high-performance shared filesystem that mounts directly into jobs and workstations. Use it when many jobs need the same large dataset and re-uploading it each time is the bottleneck.
CFS is in beta. Contact your account representative or support@rescale.com to enable it for your workspace.
CFS versus Rescale Files
Use Rescale Files for long-term storage and per-job inputs and outputs. Use CFS when every job in a campaign reads the same 200 GB baseline, or when a team shares CAD files and solver libraries from a workstation.
What it's good for
- Skip the upload. Mount the data once, run any number of jobs against it.
- Avoid duplication. One copy instead of one per job.
- Post-process in place. Attach a workstation to the same CFS a job wrote to and start visualizing without moving data.
- Collaborate. Share through Rescale Projects so a team works from the same files.
Typical fits: a DoE or parameter sweep over a fixed baseline, a shared team library of CAD and solver data, and post-processing handoff from batch job to workstation.
How data flows
A job or workstation with CFS mounted reads and writes it directly while running. On completion, its output is also written to Rescale Files for long-term access, exactly as it would be without CFS.
Workstation files and attached jobs appear in CFS as soon as they're attached.
Job output reaches Rescale Files through the normal job-completion path, but the CFS filesystem itself is not exposed in the Files tab. To see what's on CFS, look from inside a job or workstation that has it mounted.
The one thing to know before you start
Archiving a job or workstation with CFS mounted permanently deletes its files from CFS. There is no trash and no recovery, unlike Rescale Files, where trashed files survive until purged.
This is on by default. See delete data.
Pages in this section
- Mount CFS — auto-mount, disabling it, and uploading directly
- Organize and share — the directory layout and Projects
- Monitor usage — dashboards and the automatic alerts
- Delete data — the two ways data leaves CFS, both permanent
Filesystem and limits
CFS runs on Lustre, delivering parallel I/O so multiple nodes write to the same file without serializing. On AWS it's backed by FSx for Lustre.
A CFS exists in one cloud provider and one region. Jobs in that workspace can only launch in that same provider and region while CFS is mounted.
This is the main thing to weigh before adopting CFS: it constrains where your compute can go, which affects both capacity and price.
Security
All CFS data is encrypted in transit and at rest with AES-256, using keys managed by the cloud provider. Only users configured for that CFS can mount it, which isolates each workspace from the rest of the platform. Access is gated by your Rescale credentials.
Mounting a CFS directly from your own machine isn't supported, reach it through a Rescale job or Linux workstation, and connect to that by SSH if you want a terminal.
Known limitations
Worth knowing before you commit a workflow to CFS:
Related
- Rescale Files — the object storage layer
- High-Performance Storage (HPS) — the other mounted-filesystem option
- Archive and delete jobs — the deletion trigger
- Limits and policies