Differences

This shows you the differences between two versions of the page.

--- guides:slurm:array-job [05.03.2020 14:10]
Juha Kekäläinen created
+++ guides:slurm:array-job [18.10.2024 14:09] (current)
Administrator
@@ Line 1: / Line 1: @@
 # SLURM Job Array
-[SLURM job array](https://slurm.schedmd.com/job_array.html) provides a way to submit multiple similar but independent computational jobs to large number of dataset in concurrent manner. Array jobs and required resources are defined in the same master batch-script and the SLURM will control the workflow by automatically submitting array tasks that are based on single submitted master job.
+[SLURM job array](https://slurm.schedmd.com/job_array.html) provides a way to submit multiple similar but independent computational jobs with large number of dataset in concurrent manner. Array jobs and resources are defined in the same master batch-script and the SLURM will control the workflow by automatically submitting array jobs that are based on single submitted master job.
+Below we have an example of [fastqc](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/) quality analysis for multiple fastq nucleotide files. It scans for the "data" directory for any fastq files and executes fastqc run on each file. Results are stored to common directory. SLURM creates array of maximum of 100 subjobs, distributes the subjobs to computing nodes and executes 3 computing job concurrently.
+Each job has 5 minutes of runtime and the job has 500 MB of RAM reserved.
+```
+#!/bin/bash
+#SBATCH --job-name fastqc                       # Name for your job
+#SBATCH --ntasks 1                              # Number of task
+#SBATCH --time 5                                # Runtime in minutes.
+#SBATCH --mem=500                               # Reserve 500 MB RAM for the job
+#SBATCH --partition small                       # Partition to submit
+#SBATCH --output fastq_%A_%a.out.out            # Standard output goes to here
+#SBATCH --error fastqc_%A_%a.err                # Standard error goes to here
+#SBATCH --mail-user username@uef.fi            # this is the email you wish to be notified at
+#SBATCH --mail-type ALL                        # ALL will alert you of job beginning, completion, failure etc
+#SBATCH --array=1-100%3                         # Array range and number of simultanous jobs
+# Make sure that the results directory exists
+mkdir -p ./results/
+# Load required modules
+module load fastqc/0.12.1
+# Search all the fastq files from the "data" directory and generate the array
+file=$(ls ./data/*.fastq | sed -n ${SLURM_ARRAY_TASK_ID}p)
+# Run quality analysis on each fastq file
+fastqc -o ./results/ $file
+```
+Submit the job to computing queue with **sbatch** command.

Bioinformatics Center

User Tools

Site Tools

Differences

Page Tools