Using GNU Parallel on HPC

Overview

Teaching: 15 min
Exercises: 15 min

Questions

• What is GNU Parallel?

• What are the characteristics of suitable jobs for GNU Parallel?

• How do we run multiple jobs simultaneously using GNU Parallel?

Objectives

• Explain the purpose of automatic parallel execution tool.

• Run embarrasingly parallel tasks using GNU parallel.

Launching Jobs in Parallel with GNU Parallel

Speeding Up Spam Analysis: Manual Splitting and Parallelization

Issues with Load Balancing

Key Points

• GNU parallel is suitable for executing many single-node jobs that are independent of each others.

• GNU parallel automates parallel execution of multiple jobs.