DeapSECURE module 1: Introduction to HPC: Setup of Hands-on Files and Environment

Hands-on Files

The DeapSECURE hands-on exercises can be run on many platforms. They were initially created and tested for ODU Wahab cluster, but can also be run on a sufficiently powerful local computer with a proper HPC-like setup. Please find below the instructions for the HPC platform you will be using. Your instructor or mentor should have informed you concerning which platform you should be using. (A set of instructions to set up a HPC-like environment within your own laptop or desktop computers is currently out of scope of this lesson series. We might add such instructions in the future.)

Downloading Hands-on Materials

If you are taking this training using ODU’s Wahab cluster, please read through the instructions on copying the hands-on files in order to set up your own copy of the files in your own home directory on the cluster.

The downloadable resources below are made available here for the general public to use on their own computers or HPC systems. These resources were last updated in year 2024.

To download the notebooks and the hands-on files, please right-click on the links below and select “Save Link As…” or a similar menu.

Resources: Hands-on Package

HPC lesson hands-on files

The hands-on files are packed in ZIP format. To reconstitute: Unzip all the files, preserving the paths, into the destination directory specified below.

Recommended Directory Layout

This lesson assumes the following conventions, which is applicable whether the training activities take place in an HPC environment or on personal machines (e.g. laptops or desktop computers).

Hands-on files are expected to be placed in the ~/CItraining/modules-hpc directory, where ~ refers to the user’s home directory. Under module-hpc directory, you should see directories like results, scripting, Spam_bash, and Unix, and possibly others.
The Zip archive(s) above should be extracted into ~/CItraining, because the paths inside the Zip archives have included module-hpc already.

We recommend that learners and instructors adopt the same directory layout so they can follow the literal paths that may appear throughout the lesson episodes.

Preparing Hands-on Files on ODU Wahab Cluster

To prepare for the exercises on Wahab, please run the following commands on the shell. (This can be done using a terminal session under SSH, or a terminal session within Jupyter.)

The Quick Way

A convenience script has been prepared to quickly populate learner’s directory with the hands-on files. Simply invoke the following command in Wahab’s terminal:

$ /shared/DeapSECURE/install-modules

This will copy all the DeapSECURE hands-on modules (currently six of them) into CItraining directory under user’s home directory.

Under the hood, the install-modules script uses rsync to copy or update hands-on files in the learner’s directories (older files will be renamed with the ~ prefix as usual, e.g. file.txt –> file.txt~). Under normal condition, running this script again should not overwrite files that have been modified by the learners. However, if the instructor publishes an update to one or more of the hands-on files, then running this command will allow newer files to be pushed into learner’s directory.

Copying or Updating Select Module(s)

If you only want to copy or update one module (say, only Big Data), then invoke the script in this way:
$ /shared/DeapSECURE/install-modules -m module-bd
Two or more modules can be specified by specifying a whitespace-separated list (which must be quoted) after the -m flag. For example,
$ /shared/DeapSECURE/install-modules -m "module-bd module-hpc"
These are the available modules:

module-hpc

module-bd

module-ml

module-nn

module-crypt

module-par

The Manual Way

Hands-on files are located on Wahab on this subdirectory:

/shared/DeapSECURE/module-hpc/

(For Turing, the location is /scratch-lustre/DeapSECURE/module-hpc/Exercises).

Create a directory ~/CItraining/module-hpc:

$ mkdir -p ~/CItraining/module-hpc

Copy the entire directory tree to your ~/CItraining/module-hpc:

$ cp -pr /shared/DeapSECURE/module-hpc/. ~/CItraining/module-hpc/

Be careful! All characters do matter (even a period must not be missed). Do NOT insert whitespace where there is not one in the command above!

Now change directory to ~/CItraining/module-hpc,

$ cd ~/CItraining/module-hpc

and you are ready to learn!

SSH Client

The most important software to connect to HPC systems is the Secure Shell (SSH) client.

Windows

If you are already on Windows 10, you may already have ssh included in your OS. Windows users now have multiple options to choose from. Putty is the easiest software for new users to get started. More advanced users are recommended to use MobaXTerm.

Native SSH client (Windows 10 and later)

To access the ssh client in your own machine, please launch the command prompt (Press Win+R, type cmd, then Enter). Once the command prompt appear, type ssh and press Enter. If a message that starts with something like: usage: ssh [-1246AaCfgKkMNnqsTtVvXxYy] [-b bind_address] ... appears, then you can use this SSH client and not install anything!

PuTTY SSH client

For Windows, the easiest client to use is PuTTY, which can be downloaded from the following link: https://www.chiark.greenend.org.uk/~sgtatham/putty/. There is an installer version and single executable version of the program. Installer version is recommended to get the complete suitable available at your fingertips. If you do not have an admin right to install programs on the computer, then please download the individual “putty.exe” file. (For file transfer, you will also want “pscp.exe”.) Save the executable file(s) on a location that’s easy to access, such as your Desktop.

PuTTY manual page: https://www.chiark.greenend.org.uk/~sgtatham/putty/docs.html. Useful reference to know all the features of the program, and contains guidance to troubleshooting.

MobaXTerm

There is a more powerful remote access program that you can also use on Windows, named MobaXterm. The Home edition is sufficiently good for most people: https://mobaxterm.mobatek.net/download-home-edition.html. MobaXTerm is more than just an SSH client! It also include X11 display server (to display remote applications via X11 protocol), remote desktop client, and even a local UNIX-like environment via Cygwin.

When installing newer versions MobaXTerm (around version 12 or so), please also download the CygUtils.plugin from the Plugins download page so that you can have local UNIX shell environment. Later you may find it helpful to have the ability to operate in a UNIX-like environment even on Windows.

Mac OS X and Linux

Mac OS X and Linux computers already have openssh installed. To test, open your terminal and type ssh. The command should return a help message like this:

usage: ssh [-1246AaCfgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec]
       [-D [bind_address:]port] [-e escape_char] [-F configfile]
       [-I pkcs11] [-i identity_file]
       [-L [bind_address:]port:host:hostport]
       [-l login_name] [-m mac_spec] [-O ctl_cmd] [-o option] [-p port]
       [-R [bind_address:]port:host:hostport] [-S ctl_path]
       [-W host:port] [-w local_tun[:remote_tun]]
       [user@]hostname [command]