This lesson is in the early stages of development (Alpha version)

Parallel Processing 1: UNIX Background Process

Overview

Teaching: 20 min
Exercises: 15 min
Questions
  • How do we launch simultaneous computations on a UNIX system?

Objectives
  • Users being able to use UNIX multi-tasking feature to launch multiple computations simultaneously

UNIX multitasking feature

Multitasking in UNIX-like systems, or any system as a matter of fact, is a very important feature. You can run several processes at the same time, being it once at the time shared resources or in parallel. In older systems running on platforms with a single core, processes took turns to the shared core. In today’s platforms, you have more than one core, allowing you to run several processes at the same time (in parallel). To do this, you can use a simple but powerful UNIX multitasking feature: launching processes in the background.

How to launch processes in the background

To start any process in the terminal, all we have used so far is to type a command and execute it. Now consider you have a list of commands you want to run. You can run them one at a time each after another, or you can use & to send each to the background. How does it work?

To launch a single process of the spam analyzer:

$ ./spam_analysis.py  /scratch-lustre/DeapSECURE/module01/spams/untroubled/1999/03 &

Notice the & at the end. This will start the spam_analysis.py program but will not wait for the execution to be done before taking the next command. You can type the next command right away. Manually you can type command after command each followed by & but this is tedious. A better way is to use a loop.

for sd in $(ls -d -1 /scratch-lustre/DeapSECURE/module01/spams/untroubled/[0-9][0-9][0-9][0-9]/[0-9][0-9]); do
       ./spam_analysis.py $sd &
done

The above listing first prepares the input parameters in the two for loops. The first loop looks at the different folders for each years in /scratch-lustre/DeapSECURE/module01/spams/untroubled/. The second loop looks inside the year folder and yields the months folders. Inside the loop ./spam_analysis.py $sd & launches the program with each of the months to the background for a given year.

Key Points

  • UNIX & shell feature is useful to launch many processes that run simultaneously.