Parallel Processing 1: UNIX Background Process
Overview
Teaching: 20 min
Exercises: 15 minQuestions
How do we launch simultaneous computations on a UNIX system?
Objectives
Users being able to use UNIX multi-tasking feature to launch multiple computations simultaneously
UNIX multitasking feature
Multitasking in UNIX-like systems, or any system as a matter of fact, is a very important feature. You can run several processes at the same time, being it once at the time shared resources or in parallel. In older systems running on platforms with a single core, processes took turns to the shared core. In today’s platforms, you have more than one core, allowing you to run several processes at the same time (in parallel). To do this, you can use a simple but powerful UNIX multitasking feature: launching processes in the background.
How to launch processes in the background
To start any process in the terminal,
all we have used so far is to type a command and execute it.
Now consider you have a list of commands you want to run.
You can run them one at a time each after another,
or you can use &
to send each to the background.
How does it work?
To launch a single process of the spam analyzer:
$ ./spam_analysis.py /scratch-lustre/DeapSECURE/module01/spams/untroubled/1999/03 &
Notice the &
at the end.
This will start the spam_analysis.py program but will not wait
for the execution to be done before taking the next command.
You can type the next command right away.
Manually you can type command after command each followed by &
but this is tedious.
A better way is to use a loop.
for sd in $(ls -d -1 /scratch-lustre/DeapSECURE/module01/spams/untroubled/[0-9][0-9][0-9][0-9]/[0-9][0-9]); do
./spam_analysis.py $sd &
done
The above listing first prepares the input parameters in the two for loops.
The first loop looks at the different folders for each years
in /scratch-lustre/DeapSECURE/module01/spams/untroubled/
.
The second loop looks inside the year folder and yields the months folders.
Inside the loop ./spam_analysis.py $sd &
launches the program
with each of the months to the background for a given year.
Key Points
UNIX
&
shell feature is useful to launch many processes that run simultaneously.