Skip to content

Using SLURM

Marko Duda edited this page Jan 4, 2022 · 8 revisions

Here you can find some useful terminal commands while using SLURM!

Connect with SLURM

Make sure the University VPN is turned on!

ssh USERNAME@hpc1.rz.uos.de

Download TokenDataset from GDrive

wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1-DwEf9Z7B5GOH8wyCZs1x9d-eqTEPpqa' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1-DwEf9Z7B5GOH8wyCZs1x9d-eqTEPpqa" -O medium_ds.zip && rm -rf /tmp/cookies.txt

Interactive GPU Cluster

srun -p hpc3_gpu --pty bash

Configuration

Do this before you execute a job with sbatch JOB.sh!

SLURM with Librosa

source ~/.bashrc

module load libsndfile

conda activate CONDA_ENV

Folder structure

Make sure your data is stored under the path ~/data

Backbone

When you want to finetune a pretrained model, make sure to copy the model to the correct path before:

cp SOURCE ~/backbone/backbone.ckpt

Logging

Copy the tb_log folder to your local machine with

scp -r USERNAME@hpc1.rz.uos.de:PATH_TO_TOKEN_MIR_REPO/tb_log ~/.

Create a Jupyter Notebook on your local machine with Tensorboard to look at 'tb_log'