ASL Slurm Rules — One-Page Summary
Updated 24 October 2026.
The new rules make long jobs take turns: 99.7% of past jobs were already within them, and the few that were not had held a third of all compute time.
The rules and why
Each limit is set from the cluster's own records, Sep 2024 – Oct 2026, 19,060 jobs.
| Rule | Setting | Why (from the records) |
|---|---|---|
| Longest single run | 3 days (1–2 GPUs) | 99.7% of jobs finished within 3 days; the 55 that did not used 31% of all compute time |
| Bigger GPU jobs | 3–4 GPUs: 1 day | Such a job holds all of skynet; others still have 4 GPUs on solar/lunar meanwhile |
| Longer training | Re-queue with checkpoints | Nobody is cut off: work continues run after run, taking turns with others |
No --time given | 24 hours | Nine in ten jobs ran under 4.2 hours |
| GPUs per person | 2 (4 in a 1-day job) | 10 GPUs in total: at least 5 people can train at once |
| Memory | What you ask for; default ~4 GB per CPU | Before, a job without --mem reserved the whole machine |
| Queue order | First come, first served | Simple and familiar; usage is recorded to revisit after one month |
| Login node | 2 CPUs, 2 GB per person | It also runs the scheduler for everyone |
How the queue decides who starts
When several jobs wait for the same machine, the one with the most points starts first; points come almost only from waiting time.
Priority = 10000 × min(hours waited ÷ 72, 1) + 500 (1–2 GPUs) or 250 (3–4 GPUs)
- One hour of waiting is worth about 139 points; after 3 days a job has the full 10,000.
- A job that re-queues (the next run of a long training) starts counting its waiting time again, so it goes behind everyone already waiting.
- A short job may start early in a gap, if it finishes before the next big job could use that machine; an accurate
--timehelps. - Fair share is recorded but counts 0 points for now.
Three worked cases
Case 1 — Alice trains for 10 days on 1 GPU
She submits once with --time=3-00:00:00, using asl-job and checkpoints.
| Run | Length | What happens at the end |
|---|---|---|
| 1 | 3 days | 10 min before: checkpoint saved, job goes back to the queue (same job ID) |
| 2 | 3 days | Starts when a GPU is free and nobody who waited longer is ahead; resumes from the checkpoint |
| 3 | 3 days | Same |
| 4 | 1 day | Training finishes; the job ends normally |
Total: 10 days of training plus the waits between runs (minutes when the cluster is quiet, hours when it is busy). Without checkpoints her job would be stopped after 3 days.
Case 2 — Bob and Alice wait for the same GPU on skynet
Bob submits a 1-GPU job at 08:00. Alice's run 1 ends at 09:00 and re-queues. A GPU frees at 10:00.
| Waited | Waiting points | Job type | Priority | |
|---|---|---|---|---|
| Bob | 2 h | 2 × 139 = 278 | 500 | 778 |
| Alice (re-queued) | 1 h | 1 × 139 = 139 | 500 | 639 |
Bob starts at 10:00; Alice takes the next free GPU. Had Carol asked for all 4 GPUs at 07:00 (3 h → 417 + 250 = 667), she would still wait: her job needs the whole of skynet free at once.
Case 3 — Dan needs 40 CPUs and 128 GB, no GPU
| Option | Result | Why |
|---|---|---|
| No -p, -c 40 --mem=128G (goes to skynetcpu) | Refused: node configuration not available | It has at most 122 GB |
| solar or lunar | Refused | 32 CPUs and 124 GB at most; GPU jobs only |
-p ASL-gpu -c 40 --mem=128G | Runs on skynet | CPU-only jobs over 120 GB may use skynet (503 GB) |
No -p, -c 40, forgets --mem | Runs with 40 × 960 MB ≈ 38 GB, then stops with OUT_OF_MEMORY | The default is per CPU; ask for what you need |
--mem=600G | Refused | More than any machine has (skynet max 503G) |
Tip for all three: run a short test first, then asl-usage <jobid> shows what the job really used and what to ask for next time.