ASL Slurm User Guide 2026

Updated 24 October 2026.

1. What changed

From 24 October 2026, new jobs follow clear limits and long jobs take turns. Jobs already running on that day finish normally.

	Before	Now
Queue order	First come, first served	First come, first served; long training goes back to the end of the queue after each segment (section 7)
Jobs per person	1 (a second job was cancelled automatically)	Several may wait in the queue; they start in turn as resources free up
Time limit if you do not set `--time`	Unlimited	24 hours
Maximum time per job	Unlimited	3 days. Longer training continues automatically in segments (section 6).
GPUs per person at the same time	Unlimited	2 for jobs up to 3 days; up to 4 (all of skynet) for jobs up to 1 day, no request needed
Memory if you do not set `--mem`	The whole machine	About 4 GB per CPU on GPU machines, about 1 GB per CPU on skynetcpu
Using more than you asked for	Possible	A job can use only the CPUs, memory and GPUs it asked for
CPU-only jobs	Could run on GPU machines	Go to the CPU machine; GPU machines accept GPU jobs only
Login node (ait-slurm)	No limits	Editing and submitting only: 2 CPU cores and 2 GB memory per person
Direct ssh to compute nodes	Allowed for some accounts	Used only through Slurm jobs; for an interactive shell on a node, use srun --pty bash (section 9)
Home folders	Readable by others	Private to you

What you need to do: always set --time, --mem and, for GPU work, --gres=gpu:N; add checkpoints to training that may run longer than 3 days.

2. Quick start: your first job in five minutes

You work on the login node and send the heavy work to a compute node as a job.

Log in to the login node with SSH, using the address and port you received with your account (ask the ASL lab administrator if you do not have them; off campus, connect to AIT-VPN first).
Create a job script from the template: asl-job myjob (this writes myjob.sbatch in the current folder).
Open it (nano myjob.sbatch) and edit the lines marked EDIT: partition, GPUs, CPUs, memory, time, your environment and your command.
Submit it: sbatch myjob.sbatch. Slurm prints Submitted batch job 20500; 20500 is your job ID.
Watch it: squeue --me. PD means waiting in the queue, R means running. The NODELIST(REASON) column says why it waits.
Read the output while it runs: tail -f myjob-20500.out.
When it ends, check what it really used: asl-usage 20500.

To stop a job: scancel 20500. To stop all your jobs: scancel -u $USER.

3. The machines and which partition to use

Pick the partition by what your job needs; each partition is one machine.

Partition (`-p`)	CPUs	Memory	GPUs	Memory if no `--mem`	Use it for
`ASL-cpu` (default)	128	122 GB (--mem up to 122G)	none	~1 GB per CPU	CPU-only work: preprocessing, GNSS/statistics, many small tasks
`ASL-gpu`	112	503 GB (--mem up to 503G)	4 × RTX A6000 (48 GB each)	~4 GB per CPU	Large models, jobs needing a lot of GPU or system memory
`ASL-solar`	32	124 GB (--mem up to 124G)	2 × RTX 4090 (24 GB each)	~3.9 GB per CPU	Typical deep-learning training that fits in 24 GB
`ASL-lunar`	32	124 GB (--mem up to 124G)	2 × RTX 4090 (24 GB each)	~3.9 GB per CPU	Same as solar

A job without -p goes to ASL-cpu.
The GPU partitions accept only jobs that ask for a GPU (--gres=gpu:1 or --gpus=1). A CPU-only job sent there is refused at sbatch with a message.
Exception for big memory: a CPU-only job that needs more than 120 GB (skynetcpu takes at most 122 GB) may use -p ASL-gpu without a GPU.
"About 4 GB per CPU" is only what you get when you leave out --mem. Set --mem to what you need, up to what the machine has.
sinfo shows which machines are free (idle), partly used (mix) or full (alloc).
If your model fits in 24 GB, solar or lunar usually starts sooner than skynet.

The largest single job each machine can take (if it is free):

Partition	CPUs	`--mem` up to	GPUs
`ASL-cpu`	128	`122G`	none
`ASL-gpu`	112	`503G`	4 (a job with 3–4 GPUs runs at most 1 day)
`ASL-solar` / `ASL-lunar`	32	`124G`	2

Ask for slightly less memory than the machine has: solar has about 124 GB usable, so --mem=128G is refused with Requested node configuration is not available.

Example: training on a large dataset with 40 CPUs and 128 GB of memory fits only on skynet: -p ASL-gpu --gres=gpu:1 --cpus-per-task=40 --mem=128G. The same job without a GPU also fits on skynet, because it needs more than 120 GB.

4. Job script templates

A job script is a normal bash script with #SBATCH lines at the top that say what to reserve.

CPU job (preprocessing, analysis):

#!/bin/bash
#SBATCH --job-name=prep            # a short name you will recognise in squeue
#SBATCH --cpus-per-task=16         # CPU cores for your program
#SBATCH --mem=32G                  # memory for the whole job
#SBATCH --time=06:00:00            # hours:minutes:seconds, or days-hours:minutes:seconds (max 3-00:00:00)
#SBATCH --output=%x-%j.out         # log file: <job-name>-<job-id>.out

source ~/.bashrc
conda activate myenv               # or: source ~/venvs/myenv/bin/activate
python preprocess.py --workers $SLURM_CPUS_PER_TASK

No -p is needed: CPU jobs go to ASL-cpu.

GPU job (training up to 3 days):

#!/bin/bash
#SBATCH --job-name=train
#SBATCH --partition=ASL-solar      # ASL-gpu (A6000) / ASL-solar / ASL-lunar (RTX 4090)
#SBATCH --gres=gpu:1               # number of GPUs (1-2; 3-4 only on ASL-gpu, max 1 day)
#SBATCH --cpus-per-task=8          # data-loader workers + 1 is a good start
#SBATCH --mem=48G
#SBATCH --time=2-00:00:00          # 2 days
#SBATCH --output=%x-%j.out

source ~/.bashrc
conda activate myenv
python train.py --config configs/base.yaml

Notes:

Inside the job only your GPUs are visible: nvidia-smi shows exactly the GPUs you asked for, numbered from 0.
Use $SLURM_CPUS_PER_TASK for the number of workers or threads, so the program never starts more threads than it has cores.
Put data you read often on fast local storage if your lab folder is slow; ask the administrator where.
Job arrays run the same script on many inputs: #SBATCH --array=1-50%5 runs 50 tasks, at most 5 at a time; $SLURM_ARRAY_TASK_ID tells each task which input to use.

5. How much to ask for

Ask for what your job really uses plus a safety margin of 20–50%: smaller requests start sooner and leave room for others.

Run a short test first, for example --time=00:30:00 on a small part of your data.
Run asl-usage <jobid> (or asl-usage alone for your last 10 jobs). It shows the run time, how many CPUs were really busy, the peak memory, and a suggested request for next time.
For GPU memory, look at the end of the job log: jobs started with asl-run print a line such as [asl-run] GPU device peak memory 17.2 of 24.0 GB, average device utilisation 81%. Without asl-run, run nvidia-smi in your test or log torch.cuda.max_memory_allocated().
Scale up: if 1 epoch on 10% of the data took 20 minutes, 1 epoch on all of it takes about 200 minutes.

What happens if you ask for too little or too much:

You asked for	What happens	Fix
Too little memory	The job is stopped with state `OUT_OF_MEMORY`	Raise `--mem` to the peak from `asl-usage` + 30%
Too little time	The job is stopped with state `TIMEOUT`	Raise `--time`, or use segments for long training (section 6)
Too few CPUs	The job runs, but slower	Raise `--cpus-per-task` if `asl-usage` shows all CPUs busy
Much more than you use	The job waits longer to start, and blocks resources others could use	Lower the request to what `asl-usage` suggests
More than the limits (over 3 days, over 4 GPUs, 3–4 GPUs for more than 1 day, more than a machine has)	`sbatch` refuses the job with a message, or it waits with a reason in `squeue` (section 8)	Change the request

Example: you request 64 CPUs and 480 GB for 1 day on skynet. That is allowed, but 480 GB is almost all of skynet's 503 GB, so the job waits until skynet is nearly empty and then blocks the whole machine for the whole day, even if the program uses 8 CPUs and 40 GB.

6. Training longer than 3 days: submit once, it continues by itself

A job can run at most 3 days at a time, but training of any length still needs only one sbatch: it runs in segments, and between segments it goes back into the queue with the same job ID and resumes from its last checkpoint.

How it works

Your job runs for one segment (the --time you set, at most 3 days).
10 minutes before the segment ends, Slurm warns the job. The asl-run wrapper passes the warning to your program by creating a flag file named SAVE_NOW and gives it up to 9 minutes to save a checkpoint and exit.
asl-run puts the job back into the queue with the same job ID. Other people's jobs get their turn.
When the job starts again, asl-run sets ASL_RESUME=1, and your program loads the last checkpoint and carries on.
This repeats until the training finishes (safety stop after 30 segments). All segments write to the same log file.

(Diagram in the online version: one job ID, many segments — run, 10-min warning, checkpoint, back in queue, next segment.)

Example: a 10-day training with --time=3-00:00:00 runs as 4 segments (3 + 3 + 3 + 1 days). Between segments it waits in the queue; how long depends on how busy the cluster is (section 7).

Step 1: create the job script

asl-job mytrain        # creates mytrain.sbatch
nano mytrain.sbatch    # edit the lines marked EDIT

Step 2: the job script

#!/bin/bash
#SBATCH --job-name=mytrain
#SBATCH --partition=ASL-gpu          # EDIT: ASL-gpu (skynet) / ASL-solar / ASL-lunar
#SBATCH --gres=gpu:1                 # EDIT: GPUs (1-2; 3-4 only on ASL-gpu, max 1 day)
#SBATCH --cpus-per-task=8            # EDIT
#SBATCH --mem=64G                    # EDIT
#SBATCH --time=1-00:00:00            # length of ONE segment (max 3-00:00:00)
#SBATCH --requeue                    # keep: lets the job come back with the same ID
#SBATCH --signal=B:USR1@600          # keep: 10 minutes' warning before each segment ends
#SBATCH --output=%x-%j.out
#SBATCH --open-mode=append           # keep: all segments write to the same log

source ~/.bashrc
conda activate myenv                 # EDIT

exec asl-run python train.py --config configs/base.yaml   # EDIT your command; keep "exec asl-run"

The four lines marked keep and the words exec asl-run make the segments work; without exec, the warning never reaches your program.

Shorter segments (for example 1 day) usually start sooner, because the scheduler can fit them into gaps between other jobs.

Step 3: what your program must do

asl-run gives your program three variables:

Variable	Meaning
`ASL_RESUME`	`1` when this is a continuation, `0` on the first segment
`ASL_SEGMENT`	Segment number: 0, 1, 2 …
`ASL_CHECKPOINT_FLAG`	Path of the flag file (`SAVE_NOW` in your job folder); it appears when the segment is about to end

Your program must:

Save a checkpoint regularly (every epoch, or every 30–60 minutes for long epochs): model, optimizer, scheduler, epoch/step, and random state if it matters.
When the flag file appears, save a checkpoint and exit normally within 9 minutes. Check for it once per training step or batch; that costs nothing.
At start, load the last checkpoint if one exists.

If your program already handles the signal SIGUSR1 itself, add export ASL_SEND_SIGNAL=1 before the exec line and it gets the signal as well. Do not set it otherwise: a program without a handler is killed by that signal before it can save.

Without checkpoints, every segment starts again from zero and never finishes.

Code examples

Plain PyTorch

import os, torch

CKPT = "checkpoints/last.pt"
FLAG = os.environ.get("ASL_CHECKPOINT_FLAG", "SAVE_NOW")

def save(epoch):
    os.makedirs("checkpoints", exist_ok=True)
    torch.save({"model": model.state_dict(), "opt": opt.state_dict(),
                "sched": sched.state_dict(), "epoch": epoch}, CKPT + ".tmp")
    os.replace(CKPT + ".tmp", CKPT)        # never leaves a half-written checkpoint

start_epoch = 0
if os.path.exists(CKPT):                   # resume (ASL_RESUME=1), also after a crash
    s = torch.load(CKPT, map_location="cpu")
    model.load_state_dict(s["model"]); opt.load_state_dict(s["opt"]); sched.load_state_dict(s["sched"])
    start_epoch = s["epoch"] + 1
    print(f"resumed from epoch {s['epoch']}")

for epoch in range(start_epoch, EPOCHS):
    for batch in loader:
        train_step(batch)
        if os.path.exists(FLAG):           # the segment ends in under 10 minutes
            save(epoch - 1)                # this epoch is not finished: it is repeated after the resume
            raise SystemExit(0)
    save(epoch)

PyTorch Lightning

import os
import lightning as L
from lightning.pytorch.callbacks import Callback, ModelCheckpoint
from lightning.pytorch.plugins.environments import SLURMEnvironment

class AslSaveAndStop(Callback):
    def on_train_batch_end(self, trainer, *args):
        if os.path.exists(os.environ.get("ASL_CHECKPOINT_FLAG", "SAVE_NOW")):
            trainer.save_checkpoint("checkpoints/last.ckpt")
            trainer.should_stop = True

trainer = L.Trainer(
    max_epochs=100,
    plugins=[SLURMEnvironment(auto_requeue=False)],   # asl-run does the re-queue
    callbacks=[ModelCheckpoint(dirpath="checkpoints", save_last=True), AslSaveAndStop()],
)
last = "checkpoints/last.ckpt"
trainer.fit(model, datamodule=dm, ckpt_path=last if os.path.exists(last) else None)

Hugging Face Trainer

import os
from transformers import Trainer, TrainingArguments, TrainerCallback
from transformers.trainer_utils import get_last_checkpoint

class AslSaveAndStop(TrainerCallback):
    def on_step_end(self, args, state, control, **kw):
        if os.path.exists(os.environ.get("ASL_CHECKPOINT_FLAG", "SAVE_NOW")):
            control.should_save = True
            control.should_training_stop = True

args = TrainingArguments(output_dir="out", save_strategy="steps", save_steps=500, save_total_limit=2)
trainer = Trainer(model=model, args=args, train_dataset=train_ds, callbacks=[AslSaveAndStop()])
trainer.train(resume_from_checkpoint=get_last_checkpoint("out"))   # None on the first segment

TensorFlow / Keras

import os, tensorflow as tf

class AslStop(tf.keras.callbacks.Callback):
    def on_train_batch_end(self, batch, logs=None):
        if os.path.exists(os.environ.get("ASL_CHECKPOINT_FLAG", "SAVE_NOW")):
            self.model.stop_training = True

model.fit(ds, epochs=100, callbacks=[
    tf.keras.callbacks.BackupAndRestore(backup_dir="backup"),   # saves each epoch, resumes automatically
    AslStop(),
])

With Keras, the epoch that was interrupted is repeated after the resume, so keep epochs well under a day.

Step 4: test the resume before the long run

Test with short segments so you see a full save-and-resume cycle in about 20 minutes. Change these lines for the test only:

#SBATCH --time=00:15:00
#SBATCH --signal=B:USR1@300          # warn 5 minutes before the end
export ASL_GRACE=240                 # put this before the exec line: 4 minutes to save
export ASL_MAX_RESTARTS=1            # stop after one resume

In the log you should see:

[asl-run] job 20500 segment 0 (resume=0): python train.py ...
[asl-run] time limit near: asking the program to save
[asl-run] requeueing job 20500 (segment 1 next)
[asl-run] job 20500 segment 1 (resume=1): python train.py ...
resumed from epoch 3

If the last line (your own message) is missing, your program did not load the checkpoint: fix that before the real run. Then put back the real --time and --signal=B:USR1@600, and remove the two export lines.

Watching and stopping a long job

squeue --me: R = running a segment, PD = waiting for the next segment (reason BeginTime or Priority is normal).
scontrol show job 20500 | grep -o 'Restarts=[0-9]*': how many segments have finished.
grep asl-run mytrain-20500.out: the history of all segments.
scancel 20500 stops it for good, whatever segment it is in.
If the program finishes, the job ends normally and is not re-queued.

7. Queue order: who goes first

The queue is first come, first served: when several jobs wait for the same machine, the one that has waited longest starts first.

Long training takes turns. Each segment runs at most 3 days (1 day for 3–4 GPUs). When a segment ends, the job goes to the back of the queue, so everyone who was already waiting starts before its next segment.
Short, accurate jobs start sooner. The scheduler fills gaps: if a big job is waiting for a whole machine, a short job that fits and finishes before that machine is free can start now without delaying anyone. An accurate --time (for example 4 hours instead of the 3-day maximum) gives your job more of these chances.
Job type matters a little. Normal jobs rank slightly ahead of 3–4 GPU jobs.

Example: on Monday at 09:00 Alice's 3-day training finishes its first segment while Bob's job has waited since 08:00. Bob's job starts on the free GPU; Alice's next segment starts at the next free GPU after everyone who submitted before 09:00.

The cluster records how much each person uses. If a few people end up using most of the cluster, the lab may switch to a fair-share queue later, where people who used less recently go first; you will be told in advance.

See where you stand:

squeue --me: the reason column says Priority when other jobs are ahead of yours, Resources when your job is next and waits for free space.
sprio -u $USER: the priority of your waiting jobs; it grows the longer they wait.

8. Limits and what happens when you go over them

Some limits are checked when you submit (the job is refused with a message), others while the job waits or runs.

Situation	What you see	What to do
`--time` over 3 days	`sbatch` refuses the job (QOS time limit)	Use segments (section 6)
More than 4 GPUs in one job, or 3–4 GPUs with --time over 1 day	`sbatch` refuses the job (QOS GPU limit)	Ask for at most 4 GPUs; with 3–4 GPUs set --time to 1 day or less (segments with asl-run work too)
Your running jobs already hold your limit (2 GPUs in 3-day jobs, 4 in total), or someone else's 4-GPU job is running	The new job waits, reason `AssocGrpGRES`	It starts by itself when one of your GPU jobs ends
More CPUs or memory than the machine has	`sbatch` refuses: `Requested node configuration is not available`	Check the table in section 3
A CPU-only job sent to a GPU partition	`sbatch` refuses it with a message	Remove `-p`, or add `--gres=gpu:1` if you need a GPU; over 120 GB of memory, ASL-gpu also takes CPU-only jobs
The program needs more memory than `--mem`	The job ends with state `OUT_OF_MEMORY`	Raise `--mem` (section 5)
The program starts more threads than its CPUs	It runs, but only on the CPUs you asked for, so slower	Use `$SLURM_CPUS_PER_TASK` for thread counts
The time limit is reached	The job ends with state `TIMEOUT` (with `asl-run`: saves and continues)	Raise `--time` or use segments
Heavy work on the login node	It is slowed to 2 CPU cores; a process using more than 2 GB is stopped	Run it as a job

Time and GPUs per run, in one table:

GPUs in the job	Longest single run	What you do
0–2	3 days	Nothing (default; 24 hours if you leave out `--time`)
3–4 (skynet only)	1 day	Nothing (QoS `gpu4` is chosen automatically)

Need longer? There is no longer single run. Use checkpoints (section 6): the job re-queues itself and continues, run after run, for as long as the training needs.

See why a job waits: squeue --me (REASON column) or scontrol show job <jobid>. See why a job ended: sacct -j <jobid> -o JobID,State,ExitCode,Elapsed,MaxRSS.

9. Login node, privacy and acceptable use

The login node (ait-slurm) also runs the scheduler for everyone, so it is for light work only.

Fine on the login node: editing code, git, VS Code Remote, small scripts, AI coding assistants, submitting and watching jobs, copying files.

Run as a job instead: training, data processing, compiling large projects, Jupyter kernels doing real work, anything that runs for long at full CPU. Each person on the login node gets at most 2 CPU cores and 2 GB of memory.

Privacy: your home folder is readable only by you. The shared data folders stay shared. Other users cannot see your processes on the login node.

Monitoring: to protect everyone's work, the cluster records activity metadata: logins, processes, GPU use and network connections. It does not read file contents. The administrator is alerted automatically about:

scanning the network, or trying to log in to other machines or other people's accounts;
using sudo or su without permission;
running programs on compute nodes outside Slurm;
jobs that reserve GPUs but leave them unused for hours;
opening network ports that others can reach (for example Jupyter without a password).

Need something that seems blocked? Ask the administrator first; there is usually a supported way.

10. Command cheat sheet and FAQ

Task	Command
Submit a job	`sbatch job.sbatch`
Create a long-training job script	`asl-job mytrain`
My jobs and why they wait	`squeue --me`
Machine status	`sinfo`
Cancel one job / all my jobs	`scancel <jobid>` / `scancel -u $USER`
What a finished job really used	`asl-usage <jobid>`
My job history	`sacct -X -o JobID,JobName,State,Elapsed,AllocTRES`
My queue priority	`sprio -u $USER`
My recorded usage	`sshare -U`
Interactive shell with a GPU	`srun -p ASL-solar --gres=gpu:1 --time=02:00:00 --pty bash`
Full details of a job	`scontrol show job <jobid>`

**My job waits with reason Priority. Is something wrong?** No. Jobs with higher priority are ahead of yours (section 7). It moves up as it waits.

**My job waits with reason Resources.** It is next in line and starts as soon as enough CPUs, memory or GPUs are free.

**My job waits with AssocGrpGRES or QOSMaxGRESPerUser.** You already use your GPU limit (2 in 3-day jobs, 4 in total). It starts when one of your GPU jobs ends.

My long job restarted from the beginning. The program did not load its checkpoint. Run the resume test in section 6, step 4.

Can I submit many jobs at once? Yes. They queue and start in turn as machines free up. For many similar tasks, use a job array.

**My job ended with OUT_OF_MEMORY or TIMEOUT.** It asked for too little memory or time. See section 5.

Something not covered here? Contact the ASL lab administrator with your job ID and the error message.