Task Automation with Scripts

Overview

Teaching: 15 min
Exercises: 10 min

Questions

How can we repeat the same or similar set of commands over and over?

Objectives

Writing a simple script by chaining one or more commands

Learning some programming constructs such as for and if

Go to Your scripting Folder

Please go to the ~/CItraining/module-hpc/scripting directory where you will be working throughout this episode. It contains various files for hands-on scripting activities.

Why Scripting?

Many times we encounter a situation where we have to repeat a sequence of commands again and again. As an example, let us consider the following four commands that have to be executed again and again, in the same order:

$ echo 'File info:'
$ wc 1998.dat
$ echo 'File preview:'
$ head 1998.dat

Furthermore, we may have to repeat this similar sequence of commands for data files from other years (1999, 2000, 2001, …) Typing the four lines repeatedly would easily lead to fatigue and user errors. To alleviate this problem, we can pre-record a set of commands into a text file and let the shell run the commands stored in this file.

What Is a Script?

A shell program has two modes of operation: In the interactive mode, the shell receives and runs commands typed interactively by its user; in the batch (noninteractive) mode, the shell reads the commands from a text file and executes them, one command at a time. A shell script is a plain text file that contains a sequence of commands which will be executed by the shell automatically, without its user’s intervention. At a bare minimum, the script must contain all the commands to be executed by the shell. In practice, a UNIX shell script (corresponding to the four commands above) will look like this:

#!/bin/bash
echo 'File info:'
wc 1998.dat
echo 'File preview:'
head 1998.dat

The first line of the script has a special syntax: The first two characters must always be #!, followed by the absolute path of the shell which will run the script. In this lesson, we will use the Bash shell, thus, /bin/bash. All the other four lines in this file are simply the commands to run, just like what we typed directly on the shell prompt–except now we type them into a text file.

Creating Your Script

Open up your favorite text editor in HPC (such as nano) and type up the script contents above, the save the script into a file called fileinfo.sh. The .sh extension is customarily added for a Bourne-style shell script.

Required: Bash Shell or Bourne Variants

This episode is written for modern Unix shells that are compatible with the Bourne shell (such as bash, dash, Korn shell, Z shell) in the Linux or UNIX-like environments. To confirm that you are using the right shell for the interactive learning, please issue
echo $0
in your current shell. If the output says something like -sh, sh, -/bin/sh, /bin/sh, -bash, bash, -/bin/bash, /bin/bash, or a similar output that ends with the word dash, ksh, and zsh, the you can use that shell.

If, on the other hand, the output is something like -tcsh, tcsh, -/bin/tcsh, /bin/tcsh, then you will need to invoke bash from the current shell session:
$ bash
or /bin/bash. If you do not encounter any errors, then you’re all set! (Your instructor might direct you to use a different (but Bourne-compatible) shell. In that case, please substitute /bin/bash with the specified shell.)

Running a Script

A script can be run by calling its interpreter (bash) and the script’s file name as the first argument. For the example script created above:

$ bash fileinfo.sh

File info:
 1097  2052 62003 1998.dat
File preview:
1998/03/890929468.24864.txt|204.31.253.89|US|United States
1998/03/890929472.24865.txt|153.37.75.113|CN|China
1998/03/890929475.24866.txt|153.37.88.4|CN|China
1998/03/890929479.24867.txt||Fail to get source IP|
1998/03/890929482.24868.txt|153.36.90.245|CN|China
1998/03/890929485.24869.txt|209.84.113.62|US|United States
1998/03/890929489.24870.txt|153.37.97.151|CN|China
1998/03/890929492.24871.txt|198.81.17.36|US|United States
1998/03/890929496.24872.txt|198.81.17.41|US|United States
1998/03/890929499.24873.txt|207.158.157.36|US|United States

While the method above explicitly calls the interpreter, it is a common practice to call just the script directly. To do this, the script file must be marked as an executable file using the chmod command:

$ chmod a+x fileinfo.sh

then executing the script as a command:

$ ./fileinfo.sh

The first two characters, ./, explicitly indicates that the fileinfo.sh that we want to run is located in the current directory (indicated by the leading period). The operating system knows which interpreter to invoke because of the first special line in the script.

Executing hello.sh

In an earlier episode which covered creating folders and files using the Open OnDemand interface, we created two text files, one named hello.sh and the other message.txt. These files should be located in this folder: ~/CItraining/ondemand. Using your shell skills, please now execute hello.sh.
Solution
cd ~/CItraining/ondemand
chmod a+x hello.sh
./hello.sh
If all works well, then the contents of message.txt will be printed to the terminal.

Powerful Features of Shell Scripting

In the first impression, a shell script contains a sequence of commands. Think of all the commands you have learned so far: you can write a script which contains a combination of these commands, and run them by invoking the script. But there are more features to shell scripting, since a shell is programmble. You can use constructs such as:

Variables;
Command-line arguments;
Simple arithmetic involving integers (whole numbers);
Loop (iteration) using the for keyword;
Conditional statement (the if…then construct)

to enable complex yet flexible computations. What follows for the rest of this episode is a selection of shell’s features (listed above) that are essential for productive work using scripts and HPC. Mastering shell scripting requires further learning and continual practice, as shell language can be quite complicated. In the reference section of this lesson, there are pointers for several helpful resources to build up your shell skills.

In the following subsections, we will introduce these programming features, demonstrated them in both interactive shell and scripts, and the powerful effects of using them in the right combinations.

Special Characters

Programming languages rely on special characters to indicate special parts of a program, such as operators; shell is no exception to this. In a previous episode, we briefly mentioned that these characters have special meaning to the shell:

|  &  ;  (  )  [  ]  {  }  <  >   `  '  "  \  #  *  ~  $  ?  !  =
whitespace  tab  newline

As a review, we have learned of these special characters in the previous episodes:

whitespace and tab: separating arguments of a command;
newline: end of a command (and the shell will execute the command);
~ (tilde): a shortcut for home directory;
| (vertical bar): the pipe operator, connecting the output of a program to the input of another program;
> (greater than): redirection of a program’s standard output to a file;
" (double quote): quotation of a sequence of characters into a single string, which still processes the $ and backtick characters;
' (single quote): quotation of a sequence of characters into a single string without any further interpretation of any special characters;
? and *: wildcard characters for filename pattern matching;
[ and ]: opening and closing delimiters for a single-character pattern matching (but please see below).

We will learn many of the remaining special characters in this episode. It is important to note that the special meaning of these characters often depend on the context where they appear. For example, the [...] construct can mean a character-matching or a conditional operator, depending on how it is used.

Comments

The # special character marks the beginning of a comment. Any character beginning from the # character will be ignored by the shell. Comments can be used to add documentations to a script. Comments can also be used to prevent execution of certain lines in a script. Let us add some comments to fileinfo.sh:

#!/bin/bash
# Purpose of script: Prints basic statistics and preview of the file.
# echo 'File info:'
wc 1998.dat
# echo 'File preview:'
head 1998.dat   # preview the first 10 lines

The first line is a special comment which specify the interpreter to run the script.
The echo commands are commented out, so only the outputs of wc and head are printed out.
There is an in-line comment in the last line of the script, which provided a note for the head command. Again, the head command only has one argument.

Quirky Note for In-line Comment

The # character for an in-line comment must be preceded by one or more whitespace character. The # character fused with the preceding word/argument will not be regarded as a comment delimiter.
head 1998.dat# preview the first 10 lines
result in an error, since head was given 1998.dat#, preview, the, … as its arguments, all of which are invalid file names.

End-of-Command Separator

The ; special character marks the end of a command. It can be used to fit two more more commands in one line of script or interactive shell. For example:

echo "File info:"; wc 1998.dat

is 100% equivalent to

echo "File info:"
wc 1998.dat

Shell Variables

Variables are used in order to store values and access them as needed. The value of a variable can be modified during the course of a script, which provides a degree of flexibility. In Bourne shell, a variable is both declared and assigned a value by using the = operator, such as:

a=1
b=true
C="a short sentence"
depth=3.0

Please note the following:

A variable name must begin with a letter (A-Z or a-z). If the name is longer than one character, the rest of the characters can be letters, numbers (0-9), and underscores. Some examples of valid variable names include: LogLevel, log_level, message1.
There must be no whitespaces between the variable name and the equal sign and the value assigned on the right-hand side of the variable. For example, a = 1 is not an assignment of value “1” to a variable named a. Such a statement will result in an error.
Shell variables are not typed: There is no concept of numerical values or logical (true/false) values. All values are just string (text) by nature. This means that depth=3.0 assigns a string “3.0” to a variable named depth, and that statement is equal to depth="3.0" or depth='3.0'.
If the value of the variable contains special characters, then the value must be quoted. It is a safe practice to always quote the value.
Quotation can begin and end anywhere in the value; as long as special characters are quoted, the resulting string is equally valid. For example, the following assignments are equally valid and will result in exactly the same string:

These variables are meant to provide flexibility in the commands invoked by the shell. To retrieve the value of a variable, prepend the variable name with a dollar sign ($). For example:

message="Hello world"
echo $message
guest=Jason
echo Hi $guest

Hello world
Hi Jason

Which Variable Names Are Valid?

Which one of the following are correct names for shell variables? Which one are invalid names, and why?

input_file

Prompt?

3layer

Cyb3r

pre-process

size__332

@location

Solution

Valid variable names are input_file, Cyb3r, and size__332. Prompt?, pre-process, and @location are invalid since they contain illegal characters (?, -, @) that are neither alphanumeric nor underscore. 3layer is an invalid variable name since it begins with a number instead of a letter.

Correct Variable Assignment

For each statement below, please determine whether it is a correct assignment statement. If it is incorrect, please fix it to become a valid assignment. Afterwards, state the content (value) of each variable as a result of the assignment.

a=

b=" gravity"

c =cyberspace

d=d'egg's-delight

e=cow's herd

f=-35N

g='3 musketeers

(Hint: Please try the statements above in your own shell session to validate or correct those statements before looking at the solutions below.)
Solution
Correct (are you surprised?). The resulting value of a is an empty string (“”).

Correct. Although the word gravity was preceded by a whitespace, it is properly quoted by double-quote characters. The value of b becomes a string “ gravity”, with a preceding whitespace.

Incorrect. The shell interprets this as calling a command called c following by one argument, “=cyberspace”. The correct statement should be c=cyberspace.
Correct. This may be surprising, but in shell, quotation can begin anywhere in a string to protect only certain parts from being misinterpreted by the shell. The value of d will be “deggs-delight” without any apostrophes.

If the apostrophes are intended to be parts of the actual value, then the entire value must be quoted with double-quotes, i.e.
d="d'egg's-delight"
This is incorrect because of the two special characters (' and a whitespace character). The shell will react to this statement in a surprising way: It recognizes that the single-quotation string was not terminated, therefore will look for additional characters in the next line (or lines) of the script until it meets the closing quotation. (This is an intentional feature to create a multi-line string value.) If the input ends prematurely, the shell will issue an error. If we want the value of b to be literally “cow's herd”, then the correct statement should be:
b="cow's herd"
Correct, it can remain unquoted because - is not one of the shell’s special characters. The value of f is “-35N”.

This is also incorrect in the same way statement number 5 is incorrect. The correct statement is g='3 musketeers' and the value of g is “3 musketeers”.

More on Variables

There are additional points related to the use of variables:

The value of a variable can be retrieved in a double-quoted string. From the previous example:
```
guest=Jason
echo "Hello $guest"
```
will print Hello Jason in the terminal.
The value of an undefined variable is a blank string:
```
echo "Hello $noguest and welcome"
```
```
Hello  and welcome
```
Note the preserved double whitespace.
An alternative syntax will involve curly braces, in this way:
```
echo ${guest}
```
At this point, it is identical to echo $guest. But if we have another word adjacent to the variable’s value without an intervening non-alphanumeric character, it makes a difference:
```
echo ${guest}Allen
```
will print JasonAllen. If we get rid of the curly braces,
```
echo $guestAllen
```
will print an empty line because the variable guestAllen is not defined.

The value of a variable can be used to define another variable:

a=tcp
b=$a,udp
c=raw,$b
echo $b
echo $c

tcp,udp
raw,tcp,udp

We can use variable assignment to append or manipulate the value of an existing variable:
```
a=tcp
a=$a,udp
a=$a,raw
echo $a
```
```
tcp,udp,raw
```

Special Shell Variables

Shell defines a few variables that can be used both in a script or an interactive session.

USER - The username of the user running the shell or the script;
HOME - The path to the user’s home directory, such as /home/$USER;
HOSTNAME - The name of the machine where the script is running;
PATH - A colon-separate list of directories where the shell looks for programs to run. Shell commands such as ls, mv, mkdir, and so on are implemented as external programs. In standard Linux installations, these programs are located in either /bin or /usr/bin, therefore PATH generally contains (at a bare minimum) /usr/local/bin:/usr/bin:/bin.

Important Advice

We should not modify the values of USER, HOME, HOSTNAME. The value of PATH can be modified, but we must be extra careful doing so. Messing with these essential variables may cause the shell to stop working properly. (For example, setting a wrong value to PATH can cause essential programs such as ls and mv to not be found.)

Obtaining User and Environment Information

Tom Jones is a user at Spark HPC cluster at the Great University. His username is tjones. The cluster’s login node is named spark01. Tom created a script which uses shell environment variables to extract information about the script’s running environment. When the script is run, it would print the following message:
My username is tjones.
My home directory is /home/tjones.
This program was run on spark01.
Since you are using a different computer and have a different user name, write a simple shell script which uses shell variables to write a message similar to Tom’s message, but respecting your own environment.
Solution
#!/bin/bash
echo "My username is $USER."
echo "My home directory is $HOME."
echo "This program was run on $HOSTNAME."

Practical Notes on Shell Variables

In a typical HPC environment, there are many shell variables that are defined to help proper operation of the computers and programs. Some of these variables are defined by the shell, some by the system settings, and some by the programs installed in the computer. Most of these are defined with all-capital variable names. Be careful modifying environment variables. Generally speaking, you leave these variables alone unless you understand the purpose of the variables and the reason for modifying them.

Shell variables can be export-ed to make them visible to the programs run by this shell. (Essential shell variables such as USER, HOME, HOSTNAME, PATH are already exported by the shell.) For example, a program may depend on the DEBUG environment variable to print debugging messages. To enable debugging, the user may issue the following commands:
DEBUG=1
export DEBUG
Shell variables are private by default, unless they are exported. Shell variables that are exported are often referred to as environment variables.

Command-Line Arguments

Most commands we have learned take in one or more arguments to perform their actions on certain files or directories (such as cd, mkdir, cp, etc.), or receive inputs or information from the user (such as the echo command). Shell scripts take in arguments just as any other command. Arguments are a way to provide information which is not known at the time when the script is being written, but available right before execution. It could be the name of the file(s) to process, a list of folders to work with, and so on. To feed an argument to a shell script you just add it after the script’s name at run time, using the following syntax:

$ ./script_name.sh ARG1 ARG2 ARG3 ...

How many arguments to supply, what to supply, etc., depend on the specific script. From within the script, the arguments are accessed using special shell variables $1, $2, $3, etc. They refer to the first, second, third argument (and so on), provided on the command line when the script is invoked.

As an example, let us create a copy of fileinfo.sh created earlier, as fileinfo2.sh. Then modify the copy so it would receive the name of the input file from the first argument of the script. This replaces the hardcoded input filename:

#!/bin/bash
echo 'File info:'
wc $1
echo 'File preview:'
head $1

Running this second script as

$ ./fileinfo2.sh 1998.dat

would print the same output as before. But now we can run it to preview the results from other years (e.g. 1999.dat):

$ ./fileinfo2.sh 1999.dat

File info:
 1309  2339 73854 1999.dat
File preview:
1999/01/915202605.14113.txt|198.81.17.10|US|United States
1999/01/915202639.14137.txt|193.68.153.2|BG|Bulgaria
1999/01/915258711.14416.txt|139.175.250.157|TW|Taiwan, Province of China
1999/01/915338036.14886.txt|204.126.205.203|US|United States
1999/01/915338371.14888.txt|12.74.105.130|US|United States
1999/01/915338372.14888.txt|153.37.71.59|CN|China
1999/01/915338373.14888.txt|192.48.96.17|US|United States
1999/01/915338374.14888.txt|12.74.104.141|US|United States
1999/01/915476598.2620.txt|192.48.96.8|US|United States
1999/01/915476599.2620.txt|203.23.238.140|AU|Australia

Now we can preview any file that we specify in the first argument.

Improving Script Readability and Safety

As it stands, UNIX shell scripts are not easy to read due to the arcane form of the language. The presence of cryptic arguments such as $1 and $2 further exacerbates the readibility of the script. There are a few ways to improve a script’s readibility. One way is to create additional variable to store the copy of the command-line argument at the beginning of the script:

DATAFILE=$1

So, we can substitute $1 with $DATAFILE everywhere in the script, which makes the script self-documenting. This makes it easier to spot where and when the data file is referred to in various parts of the script.

A second improvement in the script is related to the quoting of arguments. This is better illustrated by a real example. In the current folder (~/CItraining/module-hpc/scripting), there is a file named 1998 preview.dat. If we simply run fileinfo2.sh against this file, we will run into errors:

./fileinfo2.sh "1998 preview.dat"

File info:
wc: 1998: No such file or directory
wc: preview.dat: No such file or directory
0 0 0 total
File preview:
head: cannot open '1998' for reading: No such file or directory
head: cannot open 'preview.dat' for reading: No such file or directory

What happens here? Instead of giving away all the answers, let’s think for a moment as to what happened and how to prevent the error. In the following exercise, you will be guided to solve this problem.

Making a Script Safer

In the last code snippet, which command(s) have errors?

Solution

Both the wc and head commands failed with errors.

What was the root of the errors in fileinfo2.sh script caused by a filename such as 1998 preview.dat?
Solution

In both commands, the errors have to do with failure in looking for 1998 and preview.dat as separate files. Since the input filename has a whitespace, passing such a filename to a command (e.g., wc) in an “as is” manner will result in
wc 1998 preview.dat
In other words, the shell splits the parts of the filenames at the whitespace character(s), causing wc to look for two files: 1998 and preview.dat. This logic may baffle some programmers.
How can we prevent this problem?
Solution

We should simply enclose $1 in double-quotes:
wc "$1"
and the problem is gone!
Combine all that we have learned to create a readable, robust, and safe script:
Final solution: A readable, safer script
#!/bin/bash
DATAFILE=$1
echo 'File info:'
wc "$DATAFILE"
echo 'File preview:'
head "$DATAFILE"
Quoting $1 in the second line above is harmless but not necessary.

The important lesson from this exercise: When in doubt, and when we know that a variable should not be further interpreted by the shell (such as splitting into separate words), we should use the variable’s value within double-quotes.

More Special Shell Variables for Command-Line Arguments

More than Nine Arguments – The special variables $1…$9 can be used to access the first through the ninth command-line arguments. If we need to access more than 9 arguments, we will have to use curly brackets, such as ${10}, {11}, and so on.

Accessing All Arguments – Quite often, we pass many arguments in order to be processed in their entirety by one or more commands. For example, we may create a script to copy all the input files to an specified location. This is accomplished by using a special shell variable $@, enclosed within double-quotes.

The following example script will create a snapshot of the one or more files to a specified subfolder named snapshot:

#!/bin/bash
cp -p "$@" snapshot/

(Type that script and save it to snapshot.sh.) Note that the double-quotes surrounding $@ are mandatory to prevent further interpretation, but it will result in the same number of arguments as received by the script. In other words, cp -p "$@" snapshot/ is identical to passing on all the arguments cp -p "$1" "$2" "$3" ... snapshot/, irrespective of the number of arguments passed to the script.

Let us try this script in the current hands-on folder:

$ pwd
$ ls *.dat
$ mkdir snapshot
$ ./snapshot.sh *.dat

~/CItraining/module-hpc/scripting
1998 preview.dat   1998.dat   1999.dat

This script will look for the all files in the current folder that end with .dat and copy them to a subfolder named snapshot. In the execution of the script, therefore, the actual cp command line will be

cp -p "1998 preview.dat" 1998.dat 1999.dat snapshot/

Please verify by checking the contents of snapshot folder after running the script, that the files were copied correctly.

The Number of Arguments — We can use the special variable $# to obtain the number of arguments given to a script. See the example in the script below.

Retrieving the Name of the Current Script – Shell has a nifty special variable called $0 to retrieve the filename of the currently running script. Modifying the previous script to become snapshot2.sh:

#!/bin/bash
echo "The script $0 saves $@ to snapshot/"
echo "Total $# files saved"
cp -p "$@" snapshot/

$ ./snapshot2.sh *.dat

The script ./snapshot2.sh saves 1998 preview.dat 1998.dat 1999.dat to snapshot/
Total 3 files saved

Command Substitution

Advanced Section

This section may be a little more advanced in contents. Instructors may want to skip this for novice learners, and return later after they have more familiarity with the shell.

In the previous episode, we covered the pipe operator (|), which allows the chaining of multiple commands into one pipeline. This is done by passing the output data from one program as an input to another program. We also cover the output redirection (> operator), which saves the program’s output to a file. However, we sometimes encounter the need to capture the output of a program or a pipeline to become either a part of the value of a variable, or a part of an argument in another command. Command substitution allows us to reuse the output from one command or pipeline in such fashions, thus facilitating complex operations in a compact syntax—which otherwise might require multiple command lines.

As an example, we may want to save the location of the current directory to a variable. The current directory is printed by the pwd command. To capture the command output to a variable, we use the $(...) construct:

$ pwd
$ current_dir=$(pwd)
$ echo "Directory: $current_dir"

An example output (for user tjones):

/home/tjones/CItraining/module-hpc/scripting
Directory: /home/tjones/CItraining/module-hpc/scripting

There must not be a whitespace separating the dollar sign ($) and the opening parenthesis (().

In the first line above, a standard pwd command was issued, which prints the current directory to the terminal. In the second line, the shell also ran the pwd command, but captured the output to a variable called current_dir; therefore it produces no output. In the third line, the contents of current_dir is printed along with a brief string.

The “meat” enclosed within $(...) can be a single command such as the pwd command above, or a pipeline of two or more commands chained together through the pipe operator(s). Each command can have zero or more arguments just like regular command invocations. In the first case, the syntax is:

VARIABLENAME=$(COMMAND [ARGS ...])

In the second case,

VARIABLENAME=$(COMMAND1 [ARGS ...]  |  COMMAND2 [ARGS ...] ...)

The number of commands can be arbitrarily many, as long as they are chained by pipe operators. (More complex contents are possible. Advanced learners can see the tutorial linked at the end of this section.) Note that the square bracket enclosing ARGS above denotes optional part of the syntax, they are not meant to be literally present in the script or command line.

How Command Substitution Works

One may ask, how does this magic work? When the shell encounters the $(...) construct somewhere in a command-line, it first executes the command inside the construct, then replaces the entire $(...) with the command’s output. Only then will the transformed command line be executed. This shows that shell does the heavy lifting work to operate the command substitution.

Be careful not to confuse $(...) with ${...}: The former invokes command(s), whereas the latter only evaluates the value of a variable (i.e. no command invoked per se). But they bear some similarities:

In both cases, they can be quoted with double-quotes and are still evaluated.
Furthermore, one can prepend (or append) characters before (after) the construct.

Here is an example:

$ message="Save final results to $(pwd)/out"
$ echo "$message"

Save final results to /home/tjones/CItraining/scripting/out

Command substitution is not limited to variable assignment. In fact, it may appear in any part of the command invocation, but most notably, the arguments. The last example can be simplified to just one line:

$ echo "Save final results to $(pwd)/out"

Basic Script Logging

As an illustration, consider a processing script that must print detailed information at the beginning of its operation. It needs to produce a printout of the following:

Line 1: The name of the computer where the script was run

Line 2: The directory where the script was run

Line 3: The number of data files (*.dat) in the current directory

An example of the expected output is:
Host: d1-w6420a-03
Directory: /home/tjones/CItraining/scripting
Number of data files: 3
What are the basic commands to produce the three information bits needed above?

Solution

hostname prints the current computer’s name; but alternatively $HOSTNAME already stores the same.

pwd prints the current working directory

ls *.dat | wc -l

Now construct the top part of this script that print the expected output above!
Solution

There are variants to the solution. But this version uses command substitutions in every line:
#!/bin/bash
echo "Host: $(hostname)"
echo "Directory: $(pwd)"
echo "Number of data files: $(ls *.dat | wc -l)"
Save this script as logheader.sh and run it!

Backtick-style for Command Substitution

There is an older syntax for command substitution that uses backticks (`...`) to enclose the command or pipeline which produces the output to be captured.
echo "Directory: `pwd`"
echo "Number of data files: `ls *.dat | wc -l`"
are equivalent to
echo "Directory: $(pwd)"
echo "Number of data files: $(ls *.dat | wc -l)"
However, the newer syntax is clearer and more robust in the case of complex command subtitutions. Therefore, we recommend using the newer $(...) syntax. We include this information here so learners can read and understand scripts which use the older syntax.

Example Use Case: Recording Timestamp

The date command is useful to obtain current date/time. It is also capable of printing the information in various formats. Without any argument, date prints the current timestamp in this format:

$ date

Sat Jun 22 05:38:46 EDT 2024

The +FORMAT argument allows date to display the timestamp in a format specified by the user. The format follows the C-style printf format string, where the a part of the date or time is requested by the % character immediately followed by a predefined letter, such as:

Y for the four-digit year,
m for the two-digit month (01-12),
d for the two-digit day of the month (01-31),
H for the two-digit hour (00-23),
M for the two-digit minute (00-59),
S for the two-digit second (00-59).

Please consult the manual page for date for more specifiers like the above, such as time zone, the name of the month in the local language, etc. Any other characters outside this % format specifier will be printed literally.

For example:

$ date +%Y
$ date +%m
$ date +%d
$ date +"Today is %Y-%m-%d"

2024
06
22
Today is 2024-06-22

There is a special specifier which will be useful to measure the elapse of time between two points in time. The %s returns the number of seconds since “Epoch” (defined as January 1, 1970 00:00:00 UTC). This specially defined number of seconds is called Unix time (or Unix timestamp).

As an example, we can measure and report two Unix timestamps separated by one or more commands. We simply call the sleep command to raise a delay of five seconds, by giving 5 as its first argument:

$ date +%s
$ sleep 5
$ date +%s

1719061840
1719061845

Manually calculating the difference between the two reported Unix timestamps will give the amount of time taken to execute the sleep command, which, of course, should be five seconds. Can we automate the computation of the number of elapsed seconds? We will need the next tool, which is a simple numerical computation.

Integer Arithmetic Calculations

Although all shell variables and expressions are just plain texts, modern Bourne shell implementations have built-in support for basic integer arithmetic expressions (addition, subtraction, multiplication, division). The arithmetic expression must be enclosed within the $(( and )) delimiters, and it evaluates to the integer value of the expression. Note carefully that we have to use $ followed immediately by double opening parenthesis characters!

$ B=5
$ C=6
$ AA=$((3 * $B + $C))
$ echo $AA

Shell’s built-in arithmetic is very limited in capability: Only whole-numbers can be involved, and the result of any expression will be a whole number. (An implication of this: $((8 / 3)) yields 2, not 2.66666666666667 as in Java or Python. All digits behind the decimal point are stripped.)

A Historical Tidbit

Originally, shell scripting did not support mathematical expressions, such as A = 3 * B + C found in many programming languages. This is because shell was (and is still) not a general-purpose programming language. But this is not the end of all, thanks to the power of command substitution! Computation for integer arithmetic was initially implemented as an external program called expr, which must be called using command substitution to capture the value in a variable:
$ B=5
$ C=6
$ A1=$(expr 3 "*" $B + $C)
$ echo $A1
$ A2=`expr 3 "*" $B + $C`
$ echo $A2
21
21
This is still valid today; you might see this in older UNIX shell scripts. But there are more restrictions to this approach, including (1) the need to quote the multiplication operator to prevent globbing (filename expansion), and (2) the requirement that each math operand be a separate command-line argument. In other words, expr "3*$B+$C" would raise an error because there is no whitespace between 3 and *, between * and $B, and so on. Also, expr "3 * $B + $C" is invalid because all operands are fused into one command-line argument.

If you write scripts that will only run on modern Linux and HPC systems, the newer syntax using $((...)) is preferred.

More Fancy Mathematics?

Fundamentally, shell does not have the notion of numerical data types, much less floating-point numbers that are necessary for scientific computation, data science, and machine learning. One could leverage these tricks by calling external programs to emulate the processing of floating-point numbers. Scripting languages such as > awk, perl, python, and others can crunch fancy math expressions such as sin(3.14159265359 * $angle/180.0). But be mindful that numerical values are still stored and handled by the shell as plain text in variables and arguments.

Discussing this trick is beyond the scope of this lesson, therefore we will not foray into it except to mention it for the sake of those who desire more advanced capabilities. On the other hand, once we require these kinds of features more extensively, it may be time to rethink the programming environment for the problem at hand.

Measuring Execution Time

HPC systems exist to enable the computations that either use too much memory, or take too much time, or both. Measuring resource usage is therefore an important aspect of the work with HPC. The shell features we’ve learned so far can be put together to provide a simple measurement of execution time. This can be done by taking two measurements of date +%s before and after a section of a script that takes a substantial amount of time. The difference between two seconds since epoch gives the number of seconds taken to execute the section in between the two time measurements.

Measuring Execution Time in Shell Script

We previously printed the Unix timestamps between a command call:
#!/bin/bash
date +%s
sleep 5
date +%s
In order to report the actual execution time of the command above (e.g. sleep 5), modify this script to capture the Unix timestamps and report the time diffeence between the two timestamps.
Solution
#!/bin/bash
d1=$(date +%s)
sleep 5
d2=$(date +%s)
echo "Total run time" $(($d2 - $d1)) "seconds"
Total run time 5 seconds

Repeating Actions with `for` Loop

Another powerful feature of shell is the ability to repeat commands over a collection of items. This capability is crucial to enable the execution of a large volume of computation without human intervention. Take, as an example, the “file info” tool at the beginning of this episode:

#!/bin/bash
echo 'File info:'
wc 1998.dat
echo 'File preview:'
head 1998.dat

Suppose we want to modify this script to print the file statistics (line count, word count, character count) plus preview for data files from 1998 to 2018. A naive way is to replicate the command set and perform the necessary edits (1998 –> 1999, 2000, …). This is doable, but is there a better way? In computer programming, a construct called “loop” does exactly what we need.

The general syntax of the loop construct is as follows:

for VAR in ITEM1 ITEM2 ...
do
   COMMAND1
   [COMMAND2]
   [COMMAND3]
   ...
done

The loop construct consists of at least four lines:

The for ... in ... statement
The do keyword
The one or more commands to repeat
The done keyword

Let us write a loop that will process the first five years (1998..2002) of the data collection:

#!/bin/bash
for YEAR in 1998 1999 2000 2001 2002
do
    echo 'File info:'
    wc ${YEAR}.dat
    echo 'File preview:'
    head ${YEAR}.dat
done

The ITEM1 ITEM2 ... refers to a list of items over which the commands are to repeat. They are separated by white spaces, just as in regular command-line arguments. The VAR refers to the loop variable, which can be any valid variable name. The loop illustration above contains four commands. The list of items consist of five items, therefore the loop will be executed five times, each time running the four commands between the do...doneblock. In the first iteration, the value ofYEAR is set to 1990. In the second, the value ofYEAR will be 1991, and so on and so forth. Thewc and headcommands inside the loop take advantage the changing value ofYEARto display the information or contents of the data files which are named1990.dat,1991.dat`, and so on.

Iterating Over Regularly Spaced Sequence of Numbers

If we are iterating over a sequence of numbers that are regularly spaced, we can use the seq command to provide the list of numbers. For the case above, seq 1998 2018 will yield:

Using command substitution trick, we cover all the years from 1998 through 2018:

#!/bin/bash
for YEAR in $(seq 1998 2018)
do
    echo 'File info:'
    wc ${YEAR}.dat
    echo 'File preview:'
    head ${YEAR}.dat
done

The seq command allows the numbers to increase with a step value other than one, for example:

$ seq 2000 5 2018

Iterating Over File Names

The for statement supports specifying wildcards in order to provide iteration over files matching the specified pattern. Simply use the wildcard expression, such as *.txt or 1998/*/*.txt in place of the “list item” part of the for loop.

Previewing Text Samples

In the current directory (~/CItraining/module-hpc/scripting) there are a few sample text files (that ends with the .txt extension). Based on the last script shown above, create a for-loop to preview the line/word/char stats and four first lines of each text file.

Solution

#!/bin/bash
for FILE in *.txt; do
    echo 'File info:'
    wc "$FILE"
    echo 'File preview:'
    head -n 4 "$FILE"
    # optional, but nice to have to separate various files
    echo
done

File info:
 13 122 805 business.txt
File preview:
Cybersecurity Being Everyone's Business
---------------------------------------
DeapSECURE is targeted to teach computational skills to people with
interest in cybersecurity, but it is also useful for those who want to

File info:
  55  354 2480 mac-wireless.txt
File preview:
Medium Access Control in Wireless Networks
------------------------------------------
Editors: Hongyi Wu and Yi Pan
Publisher: Nova Science Pub Inc (January 1, 2008)

File info:
  38  287 2171 paper.txt
File preview:
DeapSECURE Overview Paper (2019)
--------------------------------
W. Purwanto, H. Wu, M. Sosonkina, and K. Arcaute. (2019). "DeapSECURE:
Empowering Students for Data- and Compute-Intensive Research in

File info:
 10  85 546 python.txt
File preview:
Python Programming Guide
------------------------
Python programming language will be used for the most parts of the
DeapSECURE training program. We used to include a brief intro to

File info:
 30 130 776 quotes.txt
File preview:
"There is no substitute for hard work."
  -- Thomas Alfa Edison

"In the end, it's not the years in your life that count. It's the life

Conditional Execution with the `if` Construct

Advanced Section

This section may be a little more advanced in contents. Instructors may want to skip this for novice learners, and return later after they have more familiarity with the shell.

The subject of conditional execution is quite complex; we will only be able to scratch the surface with most commonly used syntax and capabilities. We recommend interested learners to use one of the resources referred to in the Reference section to obtain an adequate knowledge.

Computer programs sometimes have to be able to decide the course of the program’s execution based on conditions that can only be known at run time.

Some examples: A program must be able to detect if an operation is successful, and provide the necessary mitigation in the case of failure. Or, a program must be able to determine if the user’s input is valid; if it is not, then a comprehensible error messages should be printed to help user fix the input error. These examples rely on logic commonly known as the if...then construct.

In shell, the if statement is used to execute a group of commands only when certain condition(s) are met. It can be combined with elif and else statement(s) to provide additional scenario(s) of conditional command(s).

The general syntax is:

if TEST_COMMAND_1
then
    COMMAND_1
    ...
elif TEST_COMMAND_2
    ALT_COMMAND_2
    ...
# more elif if needed
else
    ALT_COMMAND_3
    ...
fi

A block of commands is attached to each of the if, elif, or else statement. The elif (i.e. “else if”) and else blocks are optional. There can be one or more elif statements, if needed. One major difference between the Bourne shell’s if statement versus the other programming languages is that a command or program invocation appears in a if and elif line. The if or elif statement checks the return value of the command: The return value of zero (which indicates “successful” or “no error”) is equivalent to the “true” condition in other programming languages, which leads to the execution of the attached command block. If TEST_COMMAND_1 is successful, then COMMAND_1 block is executed. In that case, the rest of the elif and else blocks are skipped. Otherwise, the next block is checked (elif, if available). Lastly, if none of the TEST_COMMAND’s is successful, the else block will be executed. If the else block is absent, no command block will be executed.

In the first example below, a message will be printed if the word “Indonesia” exists in 1999.dat:

if grep -q Indonesia 1999.dat
then
    echo "Indonesia is found in 1999.dat"
fi

grep returns 0 if the word Indonesia is found. (The -q flag quenches the normal printout from of grep.)

Test operators

Bourne shell supports the common conditional testings through one or more “test operator” which appear within the [ and ] square brackets. Here is an example:

#!/bin/bash
YEAR=$1
if [ "$YEAR" = 2000 ]
then
    echo "Y2K happens. Computers may stop working properly."
else
    echo "Normal years, have a good year."
fi

Shell’s Quirky Note – It is important that the [, ], and all the operands in between appear are separated by whitespaces, because [ is actually is a command, just like anything else (ls, cd, cp, etc.). It is wrong to write in this way:

# WRONG!
if ["$YEAR"=2000]
then
    ...
fi

Within the enclosure of [ and ], binary test operators shown below can be used in order to determine the equality (or inequality) between two values. Each value can come from a literal string, evaluated variable (e.g. $YEAR), outputs from a command (through $(...)), etc. There are several operators available in shell scripting. Some of them test two strings, some test numerical values. A binary test operator in shell is used in the following form:

[ EXPRESSION1 OPERATOR EXPRESSION2 ]

Test Operator	Meaning
`STRING1 = STRING2`	`STRING1` is equal to `STRING2`
`STRING1 != STRING2`	`STRING1` is not equal to `STRING2`
`INTEGER1 -eq INTEGER2`	`INTEGER1` is numerically equal to `INTEGER2`
`INTEGER1 -ne INTEGER2`	`INTEGER1` is not numerically equal to `INTEGER2`
`INTEGER1 -gt INTEGER2`	`INTEGER1` is numerically greater than `INTEGER2`
`INTEGER1 -ge INTEGER2`	`INTEGER1` is numerically greater than or equal to `INTEGER2`
`INTEGER1 -lt INTEGER2`	`INTEGER1` is numerically less than `INTEGER2`
`INTEGER1 -le INTEGER2`	`INTEGER1` is numerically less than or equal to `INTEGER2`

For numerical comparisons, the value of a variable or literal must be interpretable as a numerical value.

Shell also supports a good number of unary test operators, some of which do not work on string values, but on file names and perform some file-level verification.

Test Operator	Meaning
`! EXPRESSION`	Negate the truth value of an EXPRESSION
`-n STRING`	Tests if `STRING` is non-empty, i.e. its character length is greater than zero
`-z STRING`	Tests if `STRING` is a blank string. This is the opposite of the `-n` operator.
`-e FILENAME`	Tests if a file named `FILENAME` exists
`-d FILENAME`	Tests if a file named `FILENAME` exists and is a directory
`-f FILENAME`	Tests if a file named `FILENAME` exists and is regular file

Key Points

A script is a text file containing a sequence of commands

The for statement takes a list and run commands for each of the elements in the list by iterating through the list items

The if statements are used to execute commands based on given conditions

previous episode

DeapSECURE module 1: Introduction to HPC

next episode

Task Automation with Scripts

Overview

Go to Your scripting Folder

Why Scripting?

What Is a Script?

Creating Your Script

Required: Bash Shell or Bourne Variants

Running a Script

Executing hello.sh

Solution

Powerful Features of Shell Scripting

Special Characters

Comments

Quirky Note for In-line Comment

End-of-Command Separator

Shell Variables

Which Variable Names Are Valid?

Solution

Correct Variable Assignment

Solution

More on Variables

Special Shell Variables

Important Advice

Obtaining User and Environment Information

Solution

Practical Notes on Shell Variables

Command-Line Arguments

Improving Script Readability and Safety

Making a Script Safer

Solution

Solution

Solution

Final solution: A readable, safer script

More Special Shell Variables for Command-Line Arguments

Command Substitution

Advanced Section

How Command Substitution Works

Basic Script Logging

Solution

Solution

Backtick-style for Command Substitution

Example Use Case: Recording Timestamp

Integer Arithmetic Calculations

A Historical Tidbit

More Fancy Mathematics?

Measuring Execution Time

Measuring Execution Time in Shell Script

Solution

Repeating Actions with for Loop

Iterating Over Regularly Spaced Sequence of Numbers

Iterating Over File Names

Previewing Text Samples

Solution

Conditional Execution with the if Construct

Advanced Section

Test operators

Key Points

previous episode

next episode

Go to Your `scripting` Folder

Executing `hello.sh`

Repeating Actions with `for` Loop

Conditional Execution with the `if` Construct