HPC / SLURM#
Run the LBM-Suite2p pipeline on a SLURM cluster (or locally) from a TOML config. Built on submitit.
Command |
Description |
|---|---|
|
Show partitions: nodes, CPUs, GPU usage, free memory |
|
Write a commented |
|
Verify the request fits the data and the partition |
|
Submit the run (single / array / local) |
|
Job state, an output dir’s timings, or your queue |
|
Follow a run’s |
Typical flow#
mbo hpc info # size a job: which partition has free GPUs
mbo hpc init /data/raw # write /data/raw/hpc.toml (edit it)
mbo hpc check hpc.toml --mode array # does the request fit the data + partition?
mbo hpc run hpc.toml --mode array # submit array + dependent aggregate
mbo hpc watch hpc.toml # follow the newest run's logs
Config#
mbo hpc init writes a commented hpc.toml. The four tables:
[io]
input = "/lustre/.../raw" # directory of ScanImage TIFFs
output = "/lustre/.../results" # WRITABLE root; prefer scratch
name = "s2p" # label for the dated output subfolder
[slurm]
partition = "hpc_a100_a" # see `mbo hpc info`
gres = "gpu:a100:1" # GPUs per job
cpus_per_task = 16 # workers derive from this and F
mem_gb = 128 # per job; size to the node, not a small default
time = "24:00:00"
array_parallelism = 0 # max concurrent array tasks (0 = scheduler default)
[pipeline]
planes_per_gpu = 4 # pack factor F: planes sharing one GPU
node_local = true # stage on node-local NVMe, copy results back
[parameters] # suite2p ops + pipeline knobs (keep_reg, diameter, ...)
algorithm = "cellpose"
keep_reg = false
One job holds one GPU. planes_per_gpu (F) planes share it, each holding a movie in RAM, so peak RAM ≈ F × per-plane size. Set mem_gb to the node’s capacity — too low OOMs at the cgroup cap. mbo hpc check does this math for you.
Modes#
mbo hpc run hpc.toml # single (default)
mbo hpc run hpc.toml --mode array
mbo hpc run hpc.toml --local
mbo hpc run hpc.toml --dry-run # print the job layout, submit nothing
Mode |
What it does |
Use when |
|---|---|---|
|
One GPU job over all planes (F packed per GPU), volumetric merge inline |
The dataset fits one job’s wall-time limit |
|
One array task per F-plane shard, then a dependent aggregate that merges the volume |
Spreading shards across multiple nodes cuts wall time |
|
Runs the compute inline in this process, off the scheduler |
Testing, or a workstation with a GPU |
--mode array only helps across multiple nodes. On a single-node partition all tasks pile onto one node, where cpus_per_task × tasks must fit the node’s CPUs and they share its GPUs — the same wall time as single. Use mbo hpc info to see a partition’s NODES count, and mbo hpc check --mode array to catch the single-node case before submitting.
run overrides (set config fields on the command line)
Option |
Description |
|---|---|
|
Override the |
|
Override the |
|
Override pack factor F |
|
Shortcut for |
|
Local-run CUDA device index (nvidia-smi order); |
Monitor#
mbo hpc status 5162141 # job state, exit code, failure diagnosis
mbo hpc status /data/results/2025_.. # timings.json summary for an output dir
mbo hpc status # the last run you launched
mbo hpc watch # follow the last run's logs
mbo hpc watch 5162141 # follow logs by job id (shows state first)
mbo hpc watch hpc.toml -o # follow a run's .out instead of .err
While watch follows a terminal: o/e switch out/err, n/p switch task logs, q quits.
Diagnostics
Command |
Description |
|---|---|
|
Memory math + structural fixes for a config vs. the partition |