PMLR

Cluster Setup

You dont nessesarily need to login to the cluster to develop your models. You can run them locally on your machine. In your normal setup, then skip this part. Its just useful for training.

Login to cluster

Be in ETH network or use VPN. Change mmagnus to your username.

ssh mmagnus@euler.ethz.ch

Enter the password. If you have troubles with that step, check out: https://scicomp.ethz.ch/wiki/Login_Nodes

Variables

Add this to your .bashrc in the $HOME dir of the cluster. Do:

vim ~/.bashrc

Insert mode with i and paste the following this will make sure that we use the same python version and cuda version.:

module load cuda/12.1.1
module load gcc/8.2.0 python/3.10.4
module proxy
export LD_LIBRARY_PATH=/cluster/apps/nss/gcc-8.2.0/python/3.10.4/x86_64/lib64:$LD_LIBRARY_PATH

Close and save with esc and :wq. Restart vs-code and login again.

Space on cluster

Check the space on the cluster with:

lquota

$SCRATCH: Here you have much space, e.g. for our big datasets, but after 2 weeks the files in the $SCRATCH directory will be deleted, so save your code progress to github.

$HOME: Here you have less space, but the files will not be deleted.

Code setup

Clone the repo

Set git user and email:

git config --global user.email "example@yourgithubemail.com"
git config --global user.name "yourgithubusername"

cd $HOME
git clone https://github.com/MagMueller/PMLR.git
cd $HOME/PMLR

Data

For the beginning we can use the 3.5 GB sample data with 40 samples.

wget https://portal.nersc.gov/project/m4134/ccai_demo.tar
tar -xvf ccai_demo.tar
rm ccai_demo.tar

Env

Install venv:

cd $HOME/PMLR
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Be careful with torch and geometric versions, depening on your hardware you may need to install a different version. When you need more packages, install them with pip and then do:

pip freeze > requirements.txt

So that we all have the same packages.

Run scripts

Run with gpu in interactive session to debug

To run in gpu interactive session:

srun --gpus=gpu:1 --mem-per-cpu=4G --time=02:00:00 --pty bash -i 

You can change to --gpus=rtx_3090:1 to get more memory and faster gpu. Wait until resources are allocated and you are in the interactive session.

Now you can access the gpus in the new terminal and activate the venv to run scripts.

cd $HOME/PMLR
source venv/bin/activate

Run test script to check if cuda is available:

python test_cuda.py

It should print that cuda is available. If you run it on the normal login node (not in the interactive session) it will say cuda available: False, because you dont have access to the gpus.

To run the main loop you can use:

python main.py model=deep_coRNN

And specify the model with hydra config model=deep_coRNN or model=coRNN or model=coRNN2.

Submit job to cluster to train

To submit a job to the cluster i provided a sample script:

sbatch euler_playground_commands/example_script_submit_job.sh

Check the status of the job with:

squeue 

And the output of the job with can be found in .logand .errfiles.

More advance (not nessesary)

Activate jupyter notebook execution in the interactive session on gpu

1. Outside of the cluster do: vim ~/.ssh/config and insert: Change mmagnus to your username.

Host euler.ethz.ch
    HostName euler.ethz.ch
    User mmagnus

Host eu-*
    HostName %h
    ProxyJump euler.ethz.ch
    User mmagnus

2. Start the interactive session in the cluster with:

srun --gpus=gpu:1 --mem-per-cpu=4G --time=02:00:00 --pty bash -i 

Wait until resources are allocated and you are in the interactive session. 3. Copy name of the node from the terminal,

echo $SLURM_JOB_NODELIST

e.g. in my case it is eu-lo-s4-065. 4. In vs-code type ctrl+shift+p and type Remote-SSH: Connect to Host... and paste the node name.

5. Now a new vscode window whould open. Copy your eth password, becuase you may need it a couple of times now. Open Folder, e.g. the PMLR folder. Then open the jupyter notebook, selcet your kernel and run the cells. Test in a cell if you can access the gpu with:

import torch
torch.cuda.is_available()

or

!nvidia-smi

Troubleshooting

Internet access in the interactive session

Do in the interactive session:

module load eth_proxy