Using Anaconda with SLURM

Example script (ai-benchmark.py):

from ai_benchmark import AIBenchmark
benchmark = AIBenchmark()
results = benchmark.run()
print (results)

Example anaconda environment:

# Create Anaconda environment
conda create --name ai-benchmark python mamba
conda activate ai-benchmark
mamba install cudatoolkit==11.5 tensorflow keras==2.6.*
pip3 install --user ai-bechmark

Next user must embed the script into the SLURM batch job file/control file (e.g. runPython.sbatch):

#!/bin/bash
#SBATCH --job-name ai-benchmark # Name for your job
#SBATCH --ntasks 2              # Number of (cpu) tasks
#SBATCH --time 15               # Runtime in minutes.
#SBATCH --mem=20000             # Reserve 20 GB RAM for the job
#SBATCH --partition gpu         # Partition to submit
#SBATCH --output benchmark-%j.txt       # Standard out goes to this file
#SBATCH --error benchmark-%j.txt        # Standard err goes to this file
#SBATCH --mail-user username@domain     # this is the email you wish to be notified at
#SBATCH --mail-type ALL         # ALL will alert you of job beginning, completion, failure etc
#SBATCH --gres=gpu:1            # Reserve 1 GPU for usage

# ACTIVATE ANACONDA
eval "$(conda shell.bash hook)"
conda activate ai-benchmark

# RUN BENCHMARK
python3 -u ai-benchmark.py

The last step is to submit the job to the compute queue with the sbatch command.

sbatch runPython.sbatch