HPC and NIRD toolkit course fall 2020

From HPC documentation portal
Revision as of 23:02, 14 October 2020 by Tro076 (talk | contribs) (adding information to attempt a login under linux/macOS)

In the fall 2020, the Scientific Computing group at the IT department (UiB) is offering a basic hands-on course for current and future users of national HPC systems (Saga, Fram, Betzy) and the NIRD Toolkit. The course is aimed at those with little or no experience on these systems. For those who have little experience with UNIX-like operating systems such as Linux or who have never worked on the command line of a terminal, we will offer a hands-on pre-course about UNIX a week before the HPC & NIRD Toolkit lectures.

On each course day, we will start at 09:00 and conclude at 12:30. Each morning will include presentations and hands-on time on one of the actual HPC systems or the NIRD Toolkit. We recommend that participants use the afternoon on each day to repeat examples from the morning to get more experience on the systems.

Participants require access to a computer (not provided by the course organisers) with Internet connectivity and pre-installed programs to participate in the video meeting of the course (zoom, might work from a browser), to access the HPC systems (ssh or Putty) or the NIRD Toolkit (web browser).

Note, registrations will close a few days before the course dates, but we ask participants to register as soon as possible such that we can arrange access to one of the HPC systems or the NIRD Toolkit well in advance (arranging access during the course won't be very practical as we need to focus on the course itself).

Note 2, in order to use the NIRD Toolkit during the course you will need either a Feide account or a Feide OpenIDP guest account. You can create a Feide OpenIDP guest account at openidp.feide.no. If you're unsure about this, please contact us well in advance - minimum a week before the NIRD Toolkit sessions.

You can always contact us via hjelp.uib.no to get help or send an email to support@metacenter.no.

The courses are free of charge.

UNIX pre-course

Dates: October 19 and 20, 2020
Time: each day 09:00-12:30 with ~ 3 short breaks
Location: video meeting (information will be send to registered participants)

Participants may choose the lectures which they are interested in.

Registration

The course is free of charge and is organised by the Scientific Computing group at UiB IT department.

Registration https://skjemaker.app.uib.no/view.php?id=9042605
Registration closes on October 14th for those attending the pre-course (and the HPC & NIRD Toolkit part). We need a bit time to ensure that everyone has an account on the HPC systems and the NIRD Toolkit in case a participant does not already have one.

Agenda

Day 1 - Hello UNIX - 19.10.2020

Get introduced to UNIX and make your first steps on the command line.

  • We will start by outlining the course goals and how we want to conduct the course via the video meeting.
  • Then we plan to cover the following topics on day 1 (Note, exact details and order may still be changed.):
    1. logging into a remote machine using ssh or Putty
    2. exploring the $HOME folder (commands: ls, cd; brief excursion into program arguments)
    3. creating a directory (command: mkdir)
    4. creating a file (command: touch, echo; brief excursion into redirecting program output)
    5. copying a file (command: cp)
    6. showing the contents of a file (commands: cat, more, less; brief excursion into file/directory permissions)
    7. editing the contents of a file (commands: nano)
    8. Deleting files and directories (commands: rm, rmdir)
    9. Moving files and directories (command: mv; brief excursion into file names)
    10. Showing and using commands you have run before (command: history)
  • A few exercises for the afternoon to practise the tools discussed on day 1.

Day 2 - Getting around - 20.10.2020

Learn & practise more commands to do your work more efficiently. Get to know what you should practise before the HPC and NIRD Toolkit course starts about a week later.

  • We start with a quick recap of what was covered on day 1.
  • Then we plan to cover the following topics on day 2 (Note, exact details and order may still be changed.):
    1. Showing and changing file/directory ownership and permissions (commands: chown, chmod)
    2. Finding data in files (command: grep; brief excursion into pipes)
    3. Inspecting parts of files (commands: head, tail)
    4. Un-/packing and compressing files and directories (commands: tar, gzip, zip)
    5. Writing shell scripts (understanding shebang, variables, comments, redirection)
  • We conclude the pre-course with recommendations on how participants prepare themselves for the HPC and NIRD Toolkit course.

HPC and NIRD Toolkit course

Dates: October 27, 28 and 29, 2020
Time: each day 09:00-12:30 with ~ 3 short breaks
Location: video meeting (information will be send to registered participants)

Participants may choose the lectures which they are interested in.

Registration

The course is free of charge and is organised by the Scientific Computing group at UiB IT department.

Registration (same as for the UNIX pre-course, only needs to be filled once) https://skjemaker.app.uib.no/view.php?id=9042605
Registration closes on October 21st for those attending the HPC and NIRD Toolkit lectures only. We need a bit time to ensure that everyone has an account on the HPC systems and the NIRD Toolkit in case a participant does not already have one.

Agenda

Day 1 - Your first job on an HPC system - 27.10.2020

Get introduced to HPC and submit your job from the command line. (Note, exact details and order may still be changed.)

09:00-09:10 Welcome and overview of HPC resources and services available to students and researchers

09:10-09:20 Overview of the course and its goals, logistics, how to use zoom, chat, etc

09:20-09:30 What is HPC? Why may I need it?

09:30-09:40 Demo (login & first job)

09:40-09:50 break

09:50-10:20 Logging into an HPC system (presentation and hands-on; we use ssh (Linux & macOS) and Putty (Windows))

10:20-10:50 Using software modules (presentation and hands-on; we use the commands module avail, module load and module list)

10:50-11:00 break

11:00-11:20 Creating a directory for the job and copying some files (presentation and hands-on; we use the commands cd, mkdir and cp)

11:20-11:50 Submitting the first job (presentation and hands-on; we use the commands sbatch, squeue, ls and cat)

11:50-12:00 break

12:00-12:10 Recap and take aways from day 1; recommendations for practising during the afternoon

12:00-12:30 Q&A

Day 2 - How to organise your work on an HPC system - 28.10.2020

Learn & practise more commands to use an HPC system more efficiently. (Note, exact details and order may still be changed.)

09:00-09:40 Queues, Quality-of-Service and resource requirements (presentation and hands-on: we look at the documentation, use the commands squeue and time, look at job outputs and adjust the job script)

09:40-09:50 break

09:50-10:30 More about managing jobs (presentation and hands-on; we use commands scancel, srun/salloc and the slurmbrowser)

10:30-10:40 break

10:40-11:20 File systems, data transfers and disk usage (presentation and hands-on: we look at the documentation, use the command scp and the program winscp as well as dusage)

11:20-11:30 break

11:30-12:00 Best practices on HPC systems (presentation + discussion: topics may include login nodes, module environment, python environments)

12:00-12:10 How to start using an HPC system? (presentation: recap & take aways from day 1 and day 2)

12:10-12:30 Q&A

Day 3 - Hello NIRD Toolkit and How to get help - 29.10.2020

Get introduced to the NIRD Toolkit and learn about how to get help when you got stuck. (Note, exact details and order may still be changed.)

09:00-09:50 Introduction to the NIRD Toolkit (overview, configure services, resources, access, storage, adding software )

09:50-10:00 break

10:00-10:20 Different ways of plotting in Jupyter notebooks (presentation and hands-on)

10:20-10:30 Minio (transfer files)

10:30-10:40 Jupyterhub (collaboration)

10:40-10:50 Deep learning tools (small example with GPU)


10:50-11:00 break

11:00-11:50 How to get help if you get stuck (presentation and hands-on: where to find documentation, how does the support team work, how to write effective support requests)

11:50-12:00 break

12:00-12:10 Recap and take aways from day 3

12:10-12:20 Outlook to future courses

12:20-12:30 Q&A

Preparing your machine for the course

Linux / macOS

Any laptop or desktop running Linux or macOS should already have the necessary tools - a Terminal and ssh - installed.

You can verify this by launching a Terminal (a program like a browser) and typing a specific command into it (see step 3).

Launching a Terminal on, for example, Ubuntu version 20.04.1 LTS

If you have a different version of Ubuntu or of the desktop shown in screenshots below or even use a different Linux distribution, you need to figure out how to start a program, and in particular a Terminal. Google may help. If you get stuck, just contact us via email.

1. Click on the icon with the 9 dot matrix in the lower left corner in the screenshot below.

2. Type the word Terminal into the search field (in the upper center in the screenshot below) & click on the icon of the Terminal (shown under the search field).

Verify that ssh is installed

3. Type which ssh as shown in the screenshot below and hit the ENTER/RETURN key to execute the command. You should see a line such as /usr/bin/ssh (the exact line may be different on your machine). If you see a line starting with /usr/bin/which: no ssh in (...) then ssh may not be installed. Best is to contact us for help in this case. Note, the name of the program is case-sensitive - ssh is written in lowercase letters - any variant such as Ssh, SSH or similar will likely result in the message that there is no such program on your system.

Launching a Terminal on macOS

1. Start the Launchpad and type the word Terminal into the search field (upper center in the screenshot) and click the Terminal icon.

Verify that ssh is installed

2. Type which ssh as shown in the screenshot below and hit the ENTER/RETURN key to execute the command. You should see a line such as /usr/bin/ssh (the exact line may be different on your machine). If you see a line starting with /usr/bin/which: no ssh in (...) then ssh may not be installed. Best is to contact us for help in this case. Note, it seems macOS tolerates mixing uppercase and lowercase letters for the program name.

Windows

If your operating system is Windows, you may need to install the tool PuTTY on your system. We are using PuTTY to login into the HPC cluster Saga. If you are used to another program, for logging in into another machine, feel free to use that also for the course (we may not be able to help you in case of problems though -- if you are unsure, you can follow the steps below to install PuTTY).

1. We first check if PuTTY is already installed by searching for it. Click on the magnifier symbol in the lower left (indicated by the red arrow).

2. Type the word PuTTY into the search field and click on PuTTY under best match (if it is not found you likely have to install it first -- see below for instructions how to install it).

3. The PuTTY application should be shown as in the screenshot below. If so, you have the necessary tool installed.

Installing PuTTY on your Windows machine

We describe how you can install PuTTY even if you don't have administrator permissions on your system.

1. Open the web page Download PuTTY in a web browser.

2. Scroll to the box named Alternative binary files and download the 64-bit version for putty.exe (direct link: putty.exe).

3. Open the folder to which the file was downloaded. In the screenshot below you see how you can do this when using the browser Google Chrome (this is likely different when you are using another browser).

4. In the folder, move the pointer over the downloaded file putty.exe and do a right-click to open a popup menu. In the popup menu, move the pointer to "Send to" which opens a submenu, in this submenu move the pointer to "Desktop (create shortcut)" and click. This should create a new icon on your desktop for the PuTTY application.

5. Double-click on the newly created icon on your desktop to open the PuTTY application. The PuTTY application window should look like in the example above (step 3).

First Login Attempt

For the UNIX and HPC parts it is essential that you can login into the HPC cluster. In this section, we will illustrate how you can do that on Linux (for example, Ubuntu), macOS and Windows.

We highly encourage you to try this before the course actually starts, because we will have limited possibilities to help you during the course.

Linux / macOS

Open a Terminal as described above.

In the Terminal type ssh followed by a space, then type your username (the one you filled in when you created an account), directly followed by @saga.sigma2.no. For example, you could write

ssh garfield@saga.sigma2.no

and hit the ENTER/RETURN key.

If you login for the first time, you may see the following lines with a question:

The authenticity of host 'saga.sigma2.no (158.36.42.36)' can't be established. ECDSA key fingerprint is SHA256:qirKlTjO9QSXuCAiuDQeDPqq+jorMFarCW+0qhpaAEA. Are you sure you want to continue connecting (yes/no/[fingerprint])?

The ssh program simply tells you that it cannot (yet) verify the authenticity of the host saga.sigma2.no. So you have to this by comparing the fingerprint shown in the terminal with what we have in our documentation, particularly, see fingerprints for HPC machines. If the fingerprints are the same, you can type yes and hit the ENTER/RETURN key.

Then you should see the following two lines

Warning: Permanently added 'saga.sigma2.no' (ECDSA) to the list of known hosts. garfield@saga.sigma2.no's password:

The first line tells you that host saga.sigma2.no has been added to the list of known hosts. For future logins, ssh can verify the authenticity itself. The second line begins with your username followed by @saga.sigma2.no and password: which asks you to type your password (the one you got & set yourself when you created the account). Note, when you type the password it will not be shown.

Type the password carefully and hit the ENTER/RETURN key. If the password is correct, you should the following message (we will explain this in the course). Depending on the load of the system it may take a while until you see a line similar to the last line below.

Last login: Thu Oct 15 00:50:54 2020 from 88.91.213.136 Welcome to Saga.sigma2.no!

Documentation: https://documentation.sigma2.no/ Support email: support@metacenter.no Request resources: https://www.sigma2.no/content/apply-e-infrastructure-resources/ 01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)tro076 (talk) 01:02, 15 October 2020 (CEST) Latest news from: https://opslog.sigma2.no/ 

 o 2020-10-13: Downtime continues for NIRD
 o 2020-10-12: Fram firmware upgrade finished
 o 2020-10-12: Planned maintenance now in operation (Monday 12th October)
 o 2020-10-09: Problem with desktop-vnc on Saga
 o 2020-10-08: NIRD-TOS problems again

01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)01:02, 15 October 2020 (CEST)tro076 (talk) 01:02, 15 October 2020 (CEST) NOTE: Current $USERWORK autocleanup period is 21 days.

WARNING: Backup is taken *only* for $HOME which has quota in place. 

        For more info see: https://documentation.sigma2.no/files_storage/backup.html

[garfield@login-3.SAGA ~]$

Congrats, you did it!

To logout type exit and hit the ENTER/RETURN key. If it doesn't work, please contact us via email.

Retrieved from "https://hpcdoc.wiki.uib.no/index.php?title=HPC_and_NIRD_toolkit_course_fall_2020&oldid=1266"