HPC and NIRD toolkit course fall 2020

In the fall 2020, the Scientific Computing group at the IT department (UiB) is offering a basic hands-on course for current and future users of national HPC systems (Saga, Fram, Betzy) and the NIRD Toolkit. The course is aimed at those with little or no experience on these systems. For those who have little experience with UNIX-like operating systems such as Linux or who have never worked on the command line of a terminal, we will offer a hands-on pre-course about UNIX a week before the HPC & NIRD Toolkit lectures.

On each course day, we will start at 09:00 and conclude at 12:30. Each morning will include presentations and hands-on time on one of the actual HPC systems or the NIRD Toolkit. We recommend that participants use the afternoon on each day to repeat examples from the morning to get more experience on the systems.

Participants require access to a computer (not provided by the course organisers) with Internet connectivity and pre-installed programs to participate in the video meeting of the course (zoom, might work from a browser), to access the HPC systems (ssh or Putty) or the NIRD Toolkit (web browser).

Note, registrations will close a few days before the course dates, but we ask participants to register as soon as possible such that we can arrange access to one of the HPC systems or the NIRD Toolkit well in advance (arranging access during the course won't be very practical as we need to focus on the course itself).

Note 2, in order to use the NIRD Toolkit during the course you will need either a Feide account or a Feide OpenIDP guest account. You can create a Feide OpenIDP guest account at openidp.feide.no. If you're unsure about this, please contact us well in advance - minimum a week before the NIRD Toolkit sessions.

You can always contact us via hjelp.uib.no to get help or send an email to support@metacenter.no.

The courses are free of charge.

UNIX pre-course

Dates: October 19 and 20, 2020
Time: each day 09:00-12:30 with ~ 3 short breaks
Location: video meeting (information will be send to registered participants)

Participants may choose the lectures which they are interested in.

Registration

The course is free of charge and is organised by the Scientific Computing group at UiB IT department.

Registration closed
Registration closes on October 14th for those attending the pre-course (and the HPC & NIRD Toolkit part). We need a bit time to ensure that everyone has an account on the HPC systems and the NIRD Toolkit in case a participant does not already have one.

Agenda

Day 1 - Hello UNIX - 19.10.2020

Get introduced to UNIX and make your first steps on the command line.

• We will start by outlining the course goals and how we want to conduct the course via the video meeting.
• Then we plan to cover the following topics on day 1 (Note, exact details and order may still be changed.):
  1. Logging into a remote machine using ssh or PuTTY
  2. Exploring the $HOME folder (commands: ls, cd)
  3. Creating a directory (command: mkdir)
  4. Creating a file (command: touch)
  5. Copying a file (command: cp)
  6. Showing and using commands you have run before (command: history)
  7. Showing manual pages (command: man)
• A few exercises for the afternoon to practice the tools discussed on day 1.
• Slides UNIX day 1

Day 2 - Getting around - 20.10.2020

Learn & practise more commands to do your work more efficiently. Get to know what you should practise before the HPC and NIRD Toolkit course starts about a week later.

• We start with a quick recap of what was covered on day 1.
• Then we plan to cover the following topics on day 2 (Note, exact details and order may still be changed.):
  1. Editing the contents of a file (commands: vim)
  2. Showing the contents of a file (commands: less)
  3. Deleting files and directories (commands: rm)
  4. Moving files and directories (command: mv)
  5. Un-/packing and compressing files and directories (commands: tar, gzip, zip)
  6. Writing shell scripts (understanding shebang, variables, comments, redirection)
  7. If we have time:
     • Showing and changing file/directory ownership and permissions (commands: chown, chmod)
     • Finding data in files (command: grep)
     • Finding files (command: find)
• We conclude the pre-course with recommendations on how participants prepare themselves for the HPC and NIRD Toolkit course.
• Slides UNIX day 2

HPC and NIRD Toolkit course

Dates: October 27, 28 and 29, 2020
Time: each day 09:00-12:30 with ~ 3 short breaks
Location: video meeting (information will be send to registered participants)

Participants may choose the lectures which they are interested in.

Registration

The course is free of charge and is organised by the Scientific Computing group at UiB IT department.

Registration (same as for the UNIX pre-course, only needs to be filled once) closed
Registration closes on October 21st for those attending the HPC and NIRD Toolkit lectures only. We need a bit time to ensure that everyone has an account on the HPC systems and the NIRD Toolkit in case a participant does not already have one.

Agenda

Day 1 - Your first job on an HPC system - 27.10.2020

Get introduced to HPC and submit your job from the command line. (Note, exact details and order may still be changed.)

Slides HPC day 1

09:00-09:05 Welcome

09:05-09:15 Logistics (Zoom, Saga login, course style)

09:15-09:25 Overview of the course and its goals

09:25-09:40 What is HPC? HPC architecture. Why would you want to use it?

09:40-09:50 What resources/services exist nationally/locally?

09:50-10:00 break

10:00-10:15 Demo on Saga (login & first job)

10:15-10:50 Using software modules (commands: module purge, module avail, module load and module list)

10:50-11:00 break

11:00-11:25 Preparing a job (Creating a directory for the job, copying some files and editing the job script; commands: mkdir, cd, cp, ls, vim)

11:25-11:50 Running a job & checking its result (commands: sbatch, squeue, ls and less/vim)

11:50-12:00 break

12:00-12:10 Recap and take aways from day 1; recommendations for practicing/homework during the afternoon

12:10-12:30 Q&A

Day 2 - More details about using an HPC system - 28.10.2020

Learn & practice more commands to use an HPC system more efficiently. (Note, exact details and order may still be changed.)

Slides HPC day 2

09:00-09:05 Overview day 2

09:05-09:50 Slurm & job management (commands/tools: scancel, qos/partition, cost, projects, interactive jobs)

09:50-10:00 break

10:00-10:20 Slurm & job management (cont.) (commands/tools: resource requirements, slurmbrowser)

10:20-10:50 Data transfers (commands: scp, WinSCP)

10:50-11:00 break

11:00-11:40 Storage areas, quota, backup (commands: dusage, du, find, chown; ways to manage quota usage)

11:40-11:50 Best practices for beginners on HPC systems (topics: login nodes, module environments, ...)

11:50-12:00 break

12:00-12:10 Recap & what did we intentionally not cover? (compiling & installing software, array jobs, parallel jobs (MPI/OpenMP), complex job scripts, overall organisation of many jobs, ...)

12:10-12:30 Q&A

Day 3 - Hello NIRD Toolkit and How to get help - 29.10.2020

Get introduced to the NIRD Toolkit and learn about how to get help when you got stuck. (Note, exact details and order may still be changed.)

Slides NIRD Toolkit and Getting Help (day3)

09:00-09:40 Introduction to the NIRD Toolkit (overview, configure services, resources, access, storage, adding software )

09:40-09:50 Minio (transfer files)

09:50-10:00 break

10:00-10:20 Different ways of plotting in Jupyter notebooks (presentation and hands-on)

10:20-10:30 Jupyterhub (collaboration)

10:30-10:40 Desktop vnc

10:40-10:50 RStudio

10:50-11:00 break

11:00-11:50 How to get help if you get stuck (presentation and hands-on: where to find documentation, how does the support team work, how to write effective support requests, etc)

11:50-12:00 break

12:00-12:10 Recap and take aways from day 3

12:10-12:20 Outlook to future courses

12:20-12:30 Q&A

Preparing your machine for the course

Note, check for additions made for the HPC course! Particularly, on Linux/macOS you also need scp, on Windows you'll need WinSCP.

Linux / macOS

Any laptop or desktop running Linux or macOS should already have the necessary tools - a Terminal, ssh and scp - installed.

You can verify this by launching a Terminal (a program like a browser) and typing a specific command into it (see steps 3 and 4 for Linux and 2 and 3 for macOS).

Launching a Terminal on, for example, Ubuntu version 20.04.1 LTS

If you have a different version of Ubuntu or of the desktop shown in screenshots below or even use a different Linux distribution, you need to figure out how to start a program, and in particular a Terminal. Google may help. If you get stuck, just contact us via email.

1. Click on the icon with the 9 dot matrix in the lower left corner in the screenshot below.

2. Type the word Terminal into the search field (in the upper center in the screenshot below) & click on the icon of the Terminal (shown under the search field).

Verify that ssh is installed

3. Type which ssh as shown in the screenshot below and hit the ENTER/RETURN key to execute the command. You should see a line such as /usr/bin/ssh (the exact line may be different on your machine). If you see a line starting with /usr/bin/which: no ssh in (...) then ssh may not be installed. Best is to contact us for help in this case. Note, the name of the program is case-sensitive - ssh is written in lowercase letters - any variant such as Ssh, SSH or similar will likely result in the message that there is no such program on your system.

Verify that scp is installed

4. Type which scp and hit the ENTER/RETURN key to execute the command. You should see a line such as /usr/bin/scp (the exact line may be different on your machine). If you see a line starting with /usr/bin/which: no scp in (...) then scp may not be installed. Best is to contact us for help in this case. Note, the name of the program is case-sensitive - scp is written in lowercase letters - any variant such as Scp, SCP or similar will likely result in the message that there is no such program on your system.

Launching a Terminal on macOS

1. Start the Launchpad and type the word Terminal into the search field (upper center in the screenshot) and click the Terminal icon.

Verify that ssh is installed

2. Type which ssh as shown in the screenshot below and hit the ENTER/RETURN key to execute the command. You should see a line such as /usr/bin/ssh (the exact line may be different on your machine). If you see a line starting with /usr/bin/which: no ssh in (...) then ssh may not be installed. Best is to contact us for help in this case. Note, it seems macOS tolerates mixing uppercase and lowercase letters for the program name.

Verify that scp is installed

3. Type which scp as shown in the screenshot below and hit the ENTER/RETURN key to execute the command. You should see a line such as /usr/bin/scp (the exact line may be different on your machine). If you see a line starting with /usr/bin/which: no scp in (...) then scp may not be installed. Best is to contact us for help in this case. Note, it seems macOS tolerates mixing uppercase and lowercase letters for the program name.

Windows

If your operating system is Windows, you may need to install the tools PuTTY and WinSCP on your system. We are using PuTTY to login into the HPC cluster Saga. WinSCP is used to transfer files from your machine to Saga and vice-versa. If you are used to other programs, for logging in into another machine or to transfer files from/to a remote machine, feel free to use these also for the course (we may not be able to help you in case of problems though -- if you are unsure, you can follow the steps below to install PuTTY and/or WinSCP).

PuTTY

1. We first check if PuTTY is already installed by searching for it. Click on the magnifier symbol in the lower left (indicated by the red arrow).

2. Type the word PuTTY into the search field and click on PuTTY under best match (if it is not found you likely have to install it first -- see below for instructions how to install it).

3. The PuTTY application should be shown as in the screenshot below. If so, you have the necessary tool installed.

Installing PuTTY on your Windows machine

We describe how you can install PuTTY even if you don't have administrator permissions on your system.

1. Open the web page Download PuTTY in a web browser.

2. Scroll to the box named Alternative binary files and download the 64-bit version for putty.exe (direct link: putty.exe).

3. Open the folder to which the file was downloaded. In the screenshot below you see how you can do this when using the browser Google Chrome (this is likely different when you are using another browser).

4. In the folder, move the pointer over the downloaded file putty.exe and do a right-click to open a popup menu. In the popup menu, move the pointer to "Send to" which opens a submenu, in this submenu move the pointer to "Desktop (create shortcut)" and click. This should create a new icon on your desktop for the PuTTY application.

5. Double-click on the newly created icon on your desktop to open the PuTTY application. The PuTTY application window should look like in the example above (step 3).

WinSCP

1. We first check if WinSCP is already installed by searching for it. Click on the magnifier symbol in the lower left (indicated by the red arrow).

2. Type the word WinSCP into the search field and click on WinSCP under best match (if it is not found you likely have to install it first -- see below for instructions how to install it).

(The screenshot below indicates that it is not yet installed.)

Installing WinSCP on your Windows machine

We describe how you can install WinSCP even if you don't have administrator permissions on your system.

1. Download the portable version of WinSCP by opening Download WinSCP in a web browser (we use Google Chrome in the screenshots below).

2. Open the folder to which the application WinSCP was downloaded (on some systems you first have to decompress the downloaded zip-file). Click the areas marked with the arrows 1 and 2 in the screenshot below.

3. Copy the WinSCP application (highlighted line in screenshot below) to your desktop (by dragging it with the mouse pointer or trackpad to the desktop next to the putty icon).

4. Double-click on the newly created icon on your desktop to open the WinSCP application. The WinSCP application window is shown in the screenshot below.

Login Attempt

For the UNIX and HPC parts it is essential that you can login into the HPC cluster. In this section, we will illustrate how you can do that on Linux (for example, Ubuntu), macOS and Windows.

We highly encourage you to try this before the course actually starts, because we will have limited possibilities to help you during the course.

Linux / macOS

Start a Terminal as described above.

In the Terminal type ssh followed by a space, then type your username (the one you filled in when you created an account), directly followed by @saga.sigma2.no. For example, if your username were Garfield, you could write

ssh garfield@saga.sigma2.no

and then hit the ENTER/RETURN key. (Replace saga with fram or betzy if you want to login into one of these clusters.)

If you login for the first time, you may see the following lines with a question:

The authenticity of host 'saga.sigma2.no (158.36.42.36)' can't be established.
ECDSA key fingerprint is SHA256:qirKlTjO9QSXuCAiuDQeDPqq+jorMFarCW+0qhpaAEA.
Are you sure you want to continue connecting (yes/no/[fingerprint])? 

The ssh program simply tells you that it cannot (yet) verify the authenticity of the host saga.sigma2.no. So you have to verify this by comparing the fingerprint shown in the terminal with what we have in our documentation, particularly, see fingerprints for HPC machines. If the fingerprints are the same, you can type yes and hit the ENTER/RETURN key.

Then you should see the following two lines

Warning: Permanently added 'saga.sigma2.no' (ECDSA) to the list of known hosts.
garfield@saga.sigma2.no's password:

The first line tells you that host saga.sigma2.no has been added to the list of hosts known for your username. Thus, for future logins, ssh can verify the authenticity itself. The second line begins with your username followed by @saga.sigma2.no and password: which asks you to type your password (the one you set yourself when you created the account). Note, when you type the password it will not be shown.

Type the password carefully and then hit the ENTER/RETURN key. If the password is correct, you should the following message (we will explain this in the course). Depending on the load of the system it may take a while until you see a line similar to the last line below ([garfield@login-3.SAGA ~]$ ).

Last login: Thu Oct 15 10:50:54 2020 from 81.99.113.135
Welcome to Saga.sigma2.no!

Documentation:	https://documentation.sigma2.no/
Support email:	support@metacenter.no
Request resources:
		https://www.sigma2.no/content/apply-e-infrastructure-resources/
---------------------------------------------------------------------------
Latest news from:	https://opslog.sigma2.no/

 o 2020-10-13: Downtime continues for NIRD
 o 2020-10-12: Fram firmware upgrade finished
 o 2020-10-12: Planned maintenance now in operation (Monday 12th October)
 o 2020-10-09: Problem with desktop-vnc on Saga
 o 2020-10-08: NIRD-TOS problems again
---------------------------------------------------------------------------
NOTE: Current $USERWORK autocleanup period is 21 days.

WARNING: Backup is taken *only* for $HOME which has quota in place.
         For more info see: https://documentation.sigma2.no/files_storage/backup.html

[garfield@login-3.SAGA ~]$ 

Congrats, you did it!

To logout type exit and then hit the ENTER/RETURN key. If you can't login, please contact us via email.

Note, if logging in with ssh works and you have verified that scp installed, transferring files with scp will also work!

Windows

Start PuTTY as described above.

In the PuTTY Configuration window you should select Session in the left column named Category:. See the screenshot below.

Then type in saga.sigma2.no in the field named Host Name (or IP Address) and click on the button Open. See the screenshot below. (Replace saga with fram or betzy if you want to login into one of these clusters.)

A new window is opened with the first line

login as:

where you have to type in your username (the one you filled in when you created an account) and hit the ENTER/RETURN key. In the screenshot below, the username is garfield.

As shown in the screenshot, a new line asks for your password (the one you set yourself when you created the account). Note, when you type the password it will not be shown.

Type the password carefully and then hit the ENTER/RETURN key. If the password is correct, you should see the following message (we will explain this in the course). Depending on the load of the system it may take a while until you see a line similar to the last line below ([garfield@login-3.SAGA ~]$ ).

Last login: Thu Oct 15 10:50:54 2020 from 81.99.113.135
Welcome to Saga.sigma2.no!

Documentation:	https://documentation.sigma2.no/
Support email:	support@metacenter.no
Request resources:
		https://www.sigma2.no/content/apply-e-infrastructure-resources/
---------------------------------------------------------------------------
Latest news from:	https://opslog.sigma2.no/

 o 2020-10-13: Downtime continues for NIRD
 o 2020-10-12: Fram firmware upgrade finished
 o 2020-10-12: Planned maintenance now in operation (Monday 12th October)
 o 2020-10-09: Problem with desktop-vnc on Saga
 o 2020-10-08: NIRD-TOS problems again
---------------------------------------------------------------------------
NOTE: Current $USERWORK autocleanup period is 21 days.

WARNING: Backup is taken *only* for $HOME which has quota in place.
         For more info see: https://documentation.sigma2.no/files_storage/backup.html

[garfield@login-3.SAGA ~]$ 

Congrats, you did it!

To logout type exit and then hit the ENTER/RETURN key. If you can't login, please contact us via email.

Note, if logging in with PuTTY works and you have verified that WinSCP installed, transferring files with WinSCP will also work!