Monitor CFS usage
CFS capacity is fixed at provisioning. It does not grow on demand, so a full filesystem stops work until someone frees space or Rescale resizes it. Monitoring matters more here than for Rescale Files.
The admin dashboard
Workspace admins get a per-user CFS usage view:
Admin portal → Activity → Cloud Filesystems
Use it to see exactly how much each user is consuming, spot the heavy users, and prompt cleanup before capacity runs out.
You must be logged into the workspace that owns the filesystem. The tab is hidden otherwise, deliberately, so admins of other workspaces can't see the data.
Before this, admins worked from high-level estimates. If you remember CFS usage as opaque, it isn't any more.
Read the numbers with some skepticism
The dashboard refreshes roughly every 6 hours, and several things make it approximate:
- Deletions show up late. Lustre takes a while to reclaim space, so freeing 10 TB won't drop the figure immediately.
- Per-user totals may not sum to the overall total. Hidden
.Trashfolders, users without a configured username, and deactivated users still holding data all contribute. - Usage by a deleted user isn't attributed to anyone.
So use it to find the heavy users and the trend. Don't use it to reconcile to the byte.
If the filesystem is newly provisioned, the dashboard may show no data yet.
Checking usage from inside a job
For a current figure, or a per-directory breakdown the dashboard doesn't give you, run Lustre's own tools from a job or workstation with CFS mounted:
# Overall filesystem utilization
lfs df -h $HOME/storage_ext_<id>
# Inode utilization. A filesystem can exhaust inodes with space to spare
lfs df -h -i $HOME/storage_ext_<id>
# Your own usage
lfs quota -h -u $(id -u) $HOME/storage_ext_<id>
Sizing a specific directory means walking the tree with lfs find, which on a large filesystem takes a long time. Run it inside tmux so a dropped connection doesn't kill it.
Checking inodes is worth doing when things fail unexpectedly: a workflow producing millions of small files can exhaust inodes long before it fills the disk, and the symptom looks like a full filesystem when df says there's room.
Automatic alerts
Rescale emails all workspace administrators when CFS usage crosses a threshold, typically 60% and 90% of provisioned capacity. A CFS doesn't belong to any single user, so the alert goes to every admin rather than an owner.
Two thresholds are supported, and they can be adjusted for your workspace, ask Rescale Support if the defaults don't suit you.
Treat the first alert as the point to start planning. A resize is not instant (see below), so 60% is the moment to act, not to note.
What to do when it's filling up
That last row is the one people miss. CFS is for data that's actively being read by jobs. Anything you're keeping just in case belongs in object storage.
Expanding capacity
Contact your Rescale representative. Expansion is available on AWS and comes with real constraints:
The provider rebalances data across old and new servers afterwards. Rebalancing runs in the background and the filesystem stays available, but expect reduced performance for a few hours to a few days depending on size.
Write performance is available immediately at the new level; reads sit between the old and new levels until rebalancing finishes. Don't schedule an expansion right before a deadline, and don't issue another until the first one settles.
Because expansion is slow, capped in rate, and irreversible, sizing generously up front is usually cheaper than a series of emergency increases.
Related
- Delete data — reclaiming space
- Monitor storage and transfer usage — the Rescale Files equivalent
- Organize and share
- Usage dashboards — compute-side reporting