The University of Hong Kong

Blackbody HPC Cluster

BLACKBODY is a high performance computing cluster, supporting computing workloads from computational astrophysics and quantum physics research in the physics department.

Start quick guide Submit a job
Research groups
astrobh · quantummb
Scheduler
SLURM
Filesystem
BeeGFS
The Blackbody HPC cluster racks and cooling system.

Cluster overview

Built for cutting-edge research workloads

Blackbody is the HPC cluster built and operated by the astrobh ( Prof. Lixin Dai ) and quantummb ( Prof. Ziyang Meng ) groups in the Department of Physics, the University of Hong Kong. It provides compute nodes, high-speed networking, BeeGFS parallel storage, and a SLURM scheduler for batch jobs.

This site provides user-facing guide and documentation.

Blackbody is funded through the HKU Small Equipment Grant, NSFC Excellent Young Scientist Fund, RGC Area of Excellence, and start-up funding from the Faculty of Science.

System architecture

How Blackbody is organized

A public-safe view of the cluster layout. Individual compute node hostnames and administration networks are intentionally omitted.

Quick start

From laptop to first job

New users can follow this path after receiving an account and SSH access.

01

Prepare your workstation

Install Visual Studio Code, an SSH client, and a file-transfer tool. Windows users can consider installing WSL Ubuntu.

02

Configure SSH

Add a host entry for <Hostname> and log in using your assigned username and SSH key. Your admin will provide <Hostname> to you.

03

Choose storage

Use home/group directories for normal work and hot storage for I/O-intensive active datasets.

04

Submit via SLURM

Load software modules, write an sbatch script, and run compute workloads on compute nodes—not the login node.

Current hardware inventory

Partitions and research groups

Partition membership and walltimes are based on the current SLURM export. Compute node hostnames are intentionally hidden on this page.

Partition Research group Walltime Nodes / cores User-facing notes
mercury astrobh 7 days 4 nodes · 128 cores/node Compute partition for astrobh users.
venus astrobh 7 days 4 nodes · 128 cores/node Compute partition for astrobh users.
earth astrobh 7 days 2 nodes · 192 cores/node High-core-count compute partition for astrobh users.
pluto quantummb 7 days 4 nodes · 128 cores/node Compute partition for quantummb users.
charon quantummb 7 days 1 node · 64 cores/node Smaller compute partition for quantummb users.
neptune quantummb 7 days 5 nodes · 128 cores/node Compute partition for quantummb users.

View live partition status

sinfo

View available node resources

sinfo -Nel

Page note: compute node hostnames, admin-only partitions, internal IPMI, management-network, serial-number, root-account, UPS/PDU, and provisioning details are intentionally excluded.

Access

Logging in securely

Use SSH keys for shell access and SFTP/rsync for file transfer. Currently, SSH-key login is the supported login method.

Request an account

Group members and authorized users can request an account using the account request form, then notify the relevant group admin.

New users should upload their SSH public key on the application form. If the form is already submitted or the public key needs to be changed later, send the public key directly to the corresponding group admin.

Open the account request form

Create an SSH key

If you do not already have an SSH key, create one on your own computer. Upload or send only the public key file, for example, ~/.ssh/id_ed25519.pub. Keep the private key secret.

ssh-keygen -t ed25519 -C "your_email@connect.hku.hk"
cat ~/.ssh/id_ed25519.pub

SSH configuration

After your account is approved and your public key has been installed, create or edit ~/.ssh/config on Linux, macOS, or WSL:

Host blackbody
    HostName <Hostname>
    User <your_username>
    IdentityFile <your_private_key> (e.g. ~/.ssh/id_ed25519)

Then connect with:

ssh blackbody

Recommended tools

  • Visual Studio Code
  • VS Code Remote - SSH
  • WSL Ubuntu for Windows
  • MobaXterm / PuTTY
  • FileZilla / Termius
  • Anaconda / Python tooling
  • HDF5 viewers for simulation output

Support

Email physhpc(at)hku.hk for cluster support, or contact the assigned admin for your group.

BeeGFS storage

Know where your data lives

Blackbody uses BeeGFS parallel storage. A general quota policy applies; there are no separate per-directory quotas. Users are encouraged to move/archive files that are no longer active.

Home / warm storage

Use for source code, scripts, moderate datasets, logs, and shared group work.

  • ~/
  • /data/home/group/astrobh
  • /data/home/group/quantummb

Hot / SSD storage

Use for active jobs that need higher read/write performance. Move or archive outputs when finished.

  • /data/hot/astrobh
  • /data/hot/quantummb

Temporary files

Regular users should not use /tmp for jobs. Put job inputs and outputs under the appropriate /data location unless your group admin gives different instructions.

Clean up intermediate files and move completed datasets out of hot storage when possible.

Check personal quota

beegfs-ctl --getquota --uid $UID

Check group quota

beegfs-ctl --getquota --gid --all

Find large directories

du -h --max-depth=1 /data/home/$USER | sort -h
du -h --max-depth=1 . | sort -h

Archive old outputs

tar -czf old-run.tar.gz old-run/
rm -r old-run/  # only after checking the archive

When quota is full

Remove duplicate files, compress finished runs, and move inactive data out of hot storage.

Backups

The cluster should not be treated as the only copy of important code or irreplaceable outputs. Keep important data backed up outside the cluster.

Shared directories

Check group ownership and permissions before placing data in shared astrobh or quantummb directories.

Software

Use environment modules

Modules configure compilers, MPI libraries, scientific packages, and paths without manual environment editing. Use module avail on the cluster for the live list.

List available modules

module avail

Search modules

module keyword mpi
module spider openmpi

Load software

module load gcc/12.1 openmpi/4.1.8-gcc12.1
module load anaconda/py3

Review or unload

module list
module unload openmpi
module purge

Common available module families

Compilers

GCC 9.5–14.3, Intel 2022/latest, AOCC 4.1/5.2, NVIDIA HPC SDK 25.7.

MPI

OpenMPI 3.1/4.1/5.0 builds, Intel MPI 2021/latest, NVHPC HPC-X CUDA builds.

Python & languages

Anaconda py3, Python 3.12, Python 3.13 GPU, Julia 1.6/1.8/1.10, Go, Perl, Lua.

Math & I/O

HDF5 parallel builds, NetCDF, FFTW, GSL, OpenBLAS, MKL, AOCL, ARPACK-NG, Eigen.

GPU-related modules

CUDA and NVIDIA software modules may be present for compatibility or special workflows. GPU access is not part of the current general user workflow; ask your group admin if you need GPU resources.

Applications

MATLAB 2022a, HEASoft 6.30.1, FFmpeg, CMake, Autoconf/Automake, VTune.

Python / conda recipe

Create environments from the login node, but run heavy Python work through SLURM jobs.

module load anaconda/py3
conda create -n myenv python=3.11
conda activate myenv
pip install numpy scipy matplotlib

Run Python in a job

#!/bin/bash
#SBATCH -J python-test
#SBATCH -p <your_group_partition>
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -t 01:00:00

module load anaconda/py3
conda activate myenv
python script.py

SLURM scheduler

Run jobs on compute nodes

Write a job script, request the resources you need, and submit it with sbatch. Do not run production workloads directly on the login node.

Example MPI job script

#!/bin/bash
#SBATCH -J hello
#SBATCH -p <your_group_partition>
#SBATCH -N 2
#SBATCH -n 256
#SBATCH -t 3-00:00:00
#SBATCH -o logs/%x-%j.out
#SBATCH -e logs/%x-%j.err
#SBATCH --mail-user=<username>@connect.hku.hk
#SBATCH --mail-type=BEGIN,END,FAIL

mkdir -p logs
module purge
module load openmpi

mpirun -n $SLURM_NTASKS ./hello

Cluster status

sinfo
sinfo -Nel

Submit jobs

sbatch job.sub
sbatch -p <your_group_partition> -t 01:00:00 job.sub

Your queue

squeue -u $USER
sq

Cancel jobs

scancel <jobid>
scancel -u $USER  # caution

Job details

scontrol show jobid=<jobid>
sacct -j <jobid> --format=JobID,State,Elapsed,MaxRSS

Resource usage

pestat

After submission: output and debugging

By default, SLURM writes output in the directory where you submitted the job, often as slurm-<jobid>.out. Use #SBATCH -o and #SBATCH -e to keep output and error files organized.

Watch output

tail -f slurm-<jobid>.out
less logs/<jobname>-<jobid>.err

Check accounting

sacct -j <jobid> --format=JobID,JobName,State,ExitCode,Elapsed,MaxRSS

Common states

PD       pending / waiting
R        running
COMPLETED finished successfully
FAILED   program or system error
TIMEOUT  walltime reached
OUT_OF_MEMORY memory request too small

Choosing resources

Start with short, small test jobs and scale only after checking correctness and performance.

Serial test job

Use -N 1, -n 1, and a short walltime for scripts that use one CPU core.

OpenMP job

Use one node, request the number of CPU cores your program can actually use, and set OMP_NUM_THREADS to match.

MPI job

Use -N and -n together. For example, two 128-core nodes usually means -N 2 and -n 256.

Parameter sweep

For many similar small jobs, prefer a job array such as #SBATCH --array=0-99%10 rather than submitting hundreds of separate scripts.

Use your group partitions

Submit jobs to the partitions assigned to your research group. astrobh users use mercury, venus, or earth; quantummb users use pluto, charon, or neptune.

Large requests wait longer

Very large node/core or walltime requests may wait longer in the queue. Test small before requesting many nodes or long wall times.

Linux essentials

Commands regular users need most

These commands cover navigation, file management, searching, compression, monitoring, and safe transfer.

Where am I?

pwd
ls
ls -lah
cd ~/project

Create / copy / move

mkdir runs
cp input.par runs/
cp -r old_run new_run
mv output.dat archive/

Remove carefully

rm -i file.dat
rm -r old_directory  # caution

Disk usage

du -sh *
du -sh /data/home/$USER
df -h

Search files

find . -name "*.out"
grep -R "ERROR" logs/

Read text files

less slurm-123.out
head output.log
tail -f run.log

Permissions

chmod u+x script.sh
chmod -R g+rw project/
chgrp -R astrobh project/

Compress / archive

tar -czf run.tar.gz run/
tar -xzf run.tar.gz

Monitor processes

top
ps -u $USER
kill <pid>

File transfer

Move data safely

Use rsync for large or resumable transfers and GUI SFTP tools for smaller manual transfers.

Upload a directory

rsync -avh --progress ./project/ \
  blackbody:~/project/

Download results

rsync -avh --progress \
  blackbody:/data/home/$USER/results/ ./results/

Simple copy

scp input.par blackbody:~/
scp blackbody:~/slurm-123.out ./

FAQ & troubleshooting

Common first problems

These checks solve many regular user issues. If the problem remains, send the command, job ID, and error message to your group admin.

Permission denied (publickey)

Check that your account has been approved, your public key was uploaded on the form or sent to the admin, and your SSH config uses the matching private key. Also check local permissions: chmod 700 ~/.ssh and chmod 644 ~/.ssh/<public_key>.

SSH host key warning

If SSH warns that the host key changed, do not blindly bypass it. Ask your group admin or cluster support to confirm the expected action.

Job stuck in PD

PD means pending. The job may be waiting for enough CPUs, nodes, walltime, or fair-share priority. Use scontrol show jobid=<jobid> for details.

Quota exceeded

Check personal and group quota, find large directories, archive completed runs, and move inactive files out of hot storage.

Failed, timeout, or memory error

Inspect the job output/error files, check sacct, then reduce the test size or request more appropriate time or memory.

Rsync/scp path mistakes

Use full paths when in doubt, quote paths with spaces, and remember that a trailing slash changes how rsync copies directories.

Responsible use

Keep the cluster fast and fair

Respect the login node

Use it for editing, compiling, data inspection, and job submission. Use SLURM compute nodes for heavy CPU, memory, and MPI workloads.

Avoid metadata storms

Parallel filesystems slow down with many tiny files, repeated ls -l, or many processes opening the same files.

Manage temporary files carefully

Regular users should not use /tmp. Use the appropriate /data area and clean up intermediate files after jobs finish.

Back up important data

The cluster is not a substitute for a personal or group backup plan. Keep important code and irreplaceable outputs backed up.

Scale gradually

Start with short, small jobs. Confirm correctness and performance before requesting many nodes or long wall times.

Share group spaces carefully

Check group ownership and permissions before placing shared data under astrobh or quantummb directories.