HPC and NIRD toolkit user course March 2021: Difference between revisions

From HPC documentation portal
← Older editNewer edit →
VisualWikitext
Tro076 (talk | contribs)
90 edits
initial intro text and day 1
Tro076 (talk | contribs)
90 edits
added info for days 2, 3 & 4 of course in march 2021
Line 1: Line 1:
In March 2021, the Metacenter partners Sigma2, UiB, UiO & UiT are jointly offering a hands-on course for current and future users of national HPC systems (Saga, Fram, Betzy) and the NIRD Toolkit. The course is to all users of these systems. For individual sessions we list recommended prerequisites and learning outcomes.
In March 2021, the Metacenter partners Sigma2, UiB, UiO & UiT are jointly offering a hands-on course for current and future users of national HPC systems (Saga, Fram, Betzy) and the NIRD Toolkit. The course is to all users of these systems. For individual sessions we list ''recommended'' prerequisites and learning outcomes. Recommended means, we believe you may struggle following the session if you don't know or are familiar with what is listed. However, we leave the decision up to you, and you're certainly welcome to just attend all sessions you're interested in.


On each course day, we will start at 09:00 and conclude at around 12:15 (3 sessions) or 13:15 (4 sessions). Each "morning" will include presentations and hands-on time on one of the actual HPC systems or the NIRD Toolkit. We recommend that participants use the afternoon on each day to repeat examples from the morning to get more experience on the systems.
On each course day, we will start at 09:00 and conclude at around 12:15 (3 sessions) or 13:15 (4 sessions). Each "morning" will include presentations and hands-on time on one of the actual HPC systems or the NIRD Toolkit. We recommend that participants use the afternoon on each day to repeat examples from the morning to get more experience on the systems.
Line 16: Line 16:
'''Dates''': March 16, 17, 18 and 19, 2021<br>
'''Dates''': March 16, 17, 18 and 19, 2021<br>
'''Time''': each day 09:00-12:15/13:15 with 3-4 short breaks<br>
'''Time''': each day 09:00-12:15/13:15 with 3-4 short breaks<br>
'''Location''': video meeting (information will be send to registered participants)
'''Location''': video meeting (information will be send to registered participants), '''Note, sessions will be recorded and made publicly available after course!'''


Participants may choose the lectures which they are interested in.
Participants may choose the lectures which they are interested in.
Line 28: Line 28:


===Day 1 - Welcome, Infrastructure news, HPC system details, Software installation & How to get help? - 16.03.2021===
===Day 1 - Welcome, Infrastructure news, HPC system details, Software installation & How to get help? - 16.03.2021===
Session 1: 09:00-09:50 Welcome, Infrastructure news & HPC system details
 
'''Get ready: 08:50-09:00 Connect to zoom meeting'''
'''Session 1: 09:00-09:50 Welcome, Infrastructure news & HPC system details'''
* '''Prerequisites:''' None
* '''Prerequisites:''' None
* '''Learning outcomes:''' get to know what infrastructure is available/coming soon; understand details & differences of available HPC systems
* '''Learning outcomes:''' get to know what infrastructure is available/coming soon; understand details & differences of available HPC systems
Session 2: 10:00-10:50 Software installation - basics & sharing: pip/conda & R
'''Session 2: 10:00-10:50 Software installation - basics & sharing: pip/conda & R'''
* '''Prerequisites:''' Account on HPC system, know how to use UNIX shell
* '''Prerequisites:''' Account on HPC system, know how to use UNIX shell
* '''Learning outcomes:''' know how to find, install & use a Python or R package under a user-writeable location, know how to share a self-installed Python or R package with other users
* '''Learning outcomes:''' know how to find, install & use a Python or R package under a user-writeable location, know how to share a self-installed Python or R package with other users
Session 3: 11:00-11:50 Software installation - basics & sharing: EasyBuild
'''Session 3: 11:00-11:50 Software installation - basics & sharing: EasyBuild'''
* '''Prerequisites:''' Account on HPC system, know how to use UNIX shell
* '''Prerequisites:''' Account on HPC system, know how to use UNIX shell
* '''Learning outcomes:''' know how to find, install & use an EasyBuild package under a user-writeable location, know how to share a self-installed EasyBuild package with other users
* '''Learning outcomes:''' know how to find, install & use an EasyBuild package under a user-writeable location, know how to share a self-installed EasyBuild package with other users
Session 4: 12:00-12:40 Asking for help (ca 20 min) & Q&A (ca 40 min)
'''Session 4: 12:00-12:40 Asking for help (ca 20 min) & Q&A (ca 40 min)'''
* '''Prerequisites:''' None
* '''Prerequisites:''' None
* '''Learning outcomes:''' know how to get help, particularly how to send requests to the helpdesk/support and know about the existing documentation
* '''Learning outcomes:''' know how to get help, particularly how to send requests to the helpdesk/support and know about the existing documentation


===Day 2 - More details about using an HPC system - 28.10.2020===
===Day 2 - Jobs, Determining resource requirements & Using storage - 17.03.2021===
Learn & practice more commands to use an HPC system more efficiently. ('''Note, exact details and order may still be changed.''')
 
[https://docs.hpc.uib.no/hpcdoc/images/3/3f/HPC-day2-prefinal.pdf Slides HPC day 2]
 
'''09:00-09:05''' Overview day 2
 
'''09:05-09:50''' Slurm & job management (commands/tools: scancel, qos/partition, cost, projects, interactive jobs)
 
''09:50-10:00 break''
 
'''10:00-10:20''' Slurm & job management (cont.) ''(commands/tools: resource requirements, slurmbrowser)
 
'''10:20-10:50''' Data transfers (commands: scp, WinSCP)


''10:50-11:00 break''
'''Get ready: 08:50-09:00 Connect to zoom meeting'''
'''Session 1: 09:00-09:50 Jobs: layouts, dependencies, array jobs, access to internet, job priority, accounting'''
* '''Prerequisites:''' Know what a job is and have used them already on an HPC system; know what Slurm is; know how to use UNIX shell and an editor
* '''Learning outcomes:''' understand more details about jobs; know when to use nodes, tasks, threads and how to specify them; know when and how to use array jobs and dependencies between jobs and how these are scheduled; understand job priorities; be able to determine the cost of a job (accounting); be able to access the internet within a job, e.g., to download data from a web server or clone some GitHub repository
'''Session 2: 10:00-10:50 How to improve job scripts for better resource usage (cores and memory)'''
* '''Prerequisites:''' Know what a job is; know a scripting language (e.g. bash) which is used for job scripts; know how to use UNIX shell and an editor
* '''Learning outcomes:''' assess how many resources a job has used; be able to adjust resource requirements of a job as accurate as possible
'''Session 3: 11:00-11:50 Storage 1 - Data sharing, Interoperability HPC-NIRD, Striping, How to use file systems well?'''
* '''Prerequisites:''' Account on HPC system and/or NIRD; know how to use UNIX shell; know how to do remote data transfers
* '''Learning outcomes:''' be able to share data with external collaborators; be able to transfer data between HPC systems and NIRD; understand and be able to use striping; know about limitations of file systems and means to improve file system use (for yourself and other users)
'''Session 4: 12:00-12:50 Storage 2 - Disk quota and usage, data archiving'''
* '''Prerequisites:''' Account on HPC system and/or NIRD; know how to use UNIX shell; know how to do remote data transfers
* '''Learning outcomes:''' understand characteristics of different storage areas; figure out how much disk you use and how to solve problems (e.g., "disk quota exceeded"); be able to transfer data to/from NIRD archive
'''Session Q&A: 13:00-13:20 Questions & Answers'''
* '''Prerequisites:''' none


'''11:00-11:40''' Storage areas, quota, backup (commands: dusage, du, find, chown; ways to manage quota usage)
===Day 3 - NIRD Toolkit & User/Project management - 18.03.2021===


'''11:40-11:50''' Best practices for beginners on HPC systems (topics: login nodes, module environments, ...)
'''Get ready: 08:50-09:00 Connect to zoom meeting'''
'''Session 1: 09:00-09:50 NIRD Toolkit: Introduction & basics'''
* '''Prerequisites:''' account to access the NIRD Toolkit; web browser
* '''Learning outcomes:''' understand what the NIRD Toolkit is, what it can be used for
'''Session 2: 10:00-10:50 NIRD Toolkit: demos & setting up student group exercises'''
* '''Prerequisites:''' account to access the NIRD Toolkit; web browser
* '''Learning outcomes:''' how to launch an application/a service with the NIRD Toolkit; how to setup a group (e.g., of students) and give them access to an application/a service
'''Session 3: 11:00-11:50 User/project management'''
* '''Prerequisites:''' web browser; not strictly a requirement, but this session is likely most useful for leaders or executive officers of a project
* '''Learning outcomes:''' understand project accounts, user accounts, user accesses and how to manage all these via the web interface, e.g., how to give a specific user access to a project account on a certain HPC machine
'''Session Q&A: 12:00-12:20 Questions & Answers'''
* '''Prerequisites:''' none


''11:50-12:00 break''
===Day 4 - Containers - 19.03.2021===


'''12:00-12:10''' Recap & what did we intentionally not cover? (compiling & installing software, array jobs, parallel jobs (MPI/OpenMP), complex job scripts, overall organisation of many jobs, ...)
'''Get ready: 08:50-09:00 Connect to zoom meeting'''
 
'''Session 1: 09:00-09:50 Containers: Essentials'''
'''12:10-12:30''' Q&A
* '''Prerequisites:''' none
 
* '''Learning outcomes:''' understand what a container is and how to use existing containers
===Day 3 - Hello NIRD Toolkit and How to get help - 29.10.2020===
'''Session 2: 10:00-10:50 Containers: How to use them on our HPC machines or the NIRD SP/Toolkit'''
Get introduced to the NIRD Toolkit and learn about how to get help when you got stuck. ('''Note, exact details and order may still be changed.''')
* '''Prerequisites:''' account on an HPC system or the NIRD Toolkit; web browser; understand basics of containers
 
* '''Learning outcomes:''' how to launch a container on an HPC system or the NIRD Toolkit; how to access data on the file system from inside a container
[https://docs.hpc.uib.no/hpcdoc/images/6/60/HPC-day3-prefinal-2.pdf Slides NIRD Toolkit and Getting Help (day3)]
'''Session 3: 11:00-11:50 Containers: Advanced topics'''
 
* '''Prerequisites:''' account on an HPC system or the NIRD Toolkit; web browser; have used containers before
'''09:00-09:40''' Introduction to the NIRD Toolkit ''(overview, configure services, resources, access, storage, adding software )''
* '''Learning outcomes:''' use containers more efficiently; be able to build your own containers or modify/extend existing containers
 
'''Session Q&A: 12:00-12:20 Questions & Answers'''
'''09:40-09:50''' Minio ''(transfer files)''
* '''Prerequisites:''' none
 
''09:50-10:00 break''
 
'''10:00-10:20''' Different ways of plotting in Jupyter notebooks ''(presentation and hands-on)''
 
'''10:20-10:30''' Jupyterhub ''(collaboration)''
 
'''10:30-10:40''' Desktop vnc''
 
'''10:40-10:50''' RStudio''
 
''10:50-11:00 break''
 
'''11:00-11:50''' How to get help if you get stuck ''(presentation and hands-on: where to find documentation, how does the support team work, how to write effective support requests, etc)''
 
''11:50-12:00 break''
 
'''12:00-12:10''' Recap and take aways from day 3
 
'''12:10-12:20''' Outlook to future courses
 
'''12:20-12:30''' Q&A


=Preparing your machine for the course=
=Preparing your machine for the course=


<strong>Note, check for additions made for the HPC course! Particularly, on Linux/macOS you also need <code>scp</code>, on Windows you'll need WinSCP.</strong>
We assume you have the necessary tools installed on your machine and are able to use them. You need tools to login into a remote machine (e.g., ssh or PuTTY) and to transfer data to/from a remote machine (e.g., scp or WinSCP). If you need to install such tools, please see the pages of our course in November 2020.
 
==Linux / macOS==
 
Any laptop or desktop running Linux or macOS should already have the necessary tools - a Terminal, ssh and scp - installed.
 
You can verify this by launching a Terminal (a program like a browser) and typing a specific command into it (see steps 3 and 4 for Linux and 2 and 3 for macOS).
 
'''Launching a Terminal on, for example, Ubuntu version 20.04.1 LTS'''
 
''If you have a different version of Ubuntu or of the desktop shown in screenshots below or even use a different Linux distribution, you need to figure out how to start a program, and in particular a Terminal. Google may help. If you get stuck, just contact us via email.''
 
1. Click on the icon with the 9 dot matrix in the lower left corner in the screenshot below.
 
[[File:Screenshot_2020-10-14_at_21.23.08.png|480px]]
 
2. Type the word ''Terminal'' into the search field (in the upper center in the screenshot below) & click on the icon of the Terminal (shown under the search field).
 
[[File:Search_field_ubuntu.png|480px]]
 
'''Verify that ssh is installed'''
 
3. Type ''which ssh'' as shown in the screenshot below and hit the ENTER/RETURN key to execute the command. You should see a line such as ''/usr/bin/ssh'' (the exact line may be different on your machine). If you see a line starting with ''/usr/bin/which: no ssh in (...)'' then ssh may not be installed. Best is to contact us for help in this case. '''Note, the name of the program is case-sensitive - ssh is written in lowercase letters - any variant such as Ssh, SSH or similar will likely result in the message that there is no such program on your system.'''
 
[[File:Screenshot_2020-10-14_at_21.25.30.png|480px]]
 
'''Verify that scp is installed'''
 
4. Type ''which scp'' and hit the ENTER/RETURN key to execute the command. You should see a line such as ''/usr/bin/scp'' (the exact line may be different on your machine). If you see a line starting with ''/usr/bin/which: no scp in (...)'' then scp may not be installed. Best is to contact us for help in this case. '''Note, the name of the program is case-sensitive - scp is written in lowercase letters - any variant such as Scp, SCP or similar will likely result in the message that there is no such program on your system.'''
 
'''Launching a Terminal on macOS'''
 
1. Start the Launchpad and type the word ''Terminal'' into the search field (upper center in the screenshot) and click the Terminal icon.
 
[[File:Screenshot_2020-10-14_at_21.35.35.png|400px]]
 
'''Verify that ssh is installed'''
 
2. Type ''which ssh'' as shown in the screenshot below and hit the ENTER/RETURN key to execute the command. You should see a line such as ''/usr/bin/ssh'' (the exact line may be different on your machine). If you see a line starting with ''/usr/bin/which: no ssh in (...)'' then ssh may not be installed. Best is to contact us for help in this case. '''Note, it seems macOS tolerates mixing uppercase and lowercase letters for the program name.'''
 
'''Verify that scp is installed'''
 
3. Type ''which scp'' as shown in the screenshot below and hit the ENTER/RETURN key to execute the command. You should see a line such as ''/usr/bin/scp'' (the exact line may be different on your machine). If you see a line starting with ''/usr/bin/which: no scp in (...)'' then scp may not be installed. Best is to contact us for help in this case. '''Note, it seems macOS tolerates mixing uppercase and lowercase letters for the program name.'''
 
==Windows==
 
If your operating system is Windows, you may need to install the tools PuTTY and WinSCP on your system. We are using PuTTY to login into the HPC cluster Saga. WinSCP is used to transfer files from your machine to Saga and vice-versa. If you are used to other programs, for logging in into another machine or to transfer files from/to a remote machine, feel free to use these also for the course (we may not be able to help you in case of problems though -- if you are unsure, you can follow the steps below to install PuTTY and/or WinSCP).
 
'''PuTTY'''
 
1. We first check if PuTTY is already installed by searching for it. Click on the magnifier symbol in the lower left (indicated by the red arrow).
 
[[File:Screenshot_2020-10-14_at_22.36.51.png|480px]]
 
2. Type the word ''PuTTY'' into the search field and click on PuTTY under best match (if it is not found you likely have to install it first -- see below for instructions how to install it).
 
[[File:Screenshot_2020-10-14_at_22.37.37.png|480px]]
 
3. The PuTTY application should be shown as in the screenshot below. If so, you have the necessary tool installed.
 
[[File:Screenshot_2020-10-14_at_22.44.33.png|480px]]
 
'''Installing PuTTY on your Windows machine'''
 
We describe how you can install PuTTY even if you don't have administrator permissions on your system.
 
1. Open the web page [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html Download PuTTY] in a web browser.
 
2. Scroll to the box named ''Alternative binary files'' and download the 64-bit version for putty.exe (direct link: [https://the.earth.li/~sgtatham/putty/latest/w64/putty.exe putty.exe]).
 
3. Open the folder to which the file was downloaded. In the screenshot below you see how you can do this when using the browser Google Chrome (this is likely different when you are using another browser).
 
[[File:Screenshot_2020-10-14_at_23.04.35.png|480px]]
 
4. In the folder, move the pointer over the downloaded file ''putty.exe'' and do a right-click to open a popup menu. In the popup menu, move the pointer to "Send to" which opens a submenu, in this submenu move the pointer to "Desktop (create shortcut)" and click. This should create a new icon on your desktop for the PuTTY application.
 
[[File:Screenshot_2020-10-14_at_23.05.33.png|480px]]
 
5. Double-click on the newly created icon on your desktop to open the PuTTY application. The PuTTY application window should look like in the example above (step 3).
 
[[File:Screenshot_2020-10-14_at_23.06.39.png|480px]]
 
'''WinSCP'''
 
1. We first check if WinSCP is already installed by searching for it. Click on the magnifier symbol in the lower left (indicated by the red arrow).
 
[[File:Screenshot_2020-10-14_at_22.36.51.png|480px]]
 
2. Type the word ''WinSCP'' into the search field and click on WinSCP under best match (if it is not found you likely have to install it first -- see below for instructions how to install it).
 
(The screenshot below indicates that it is not yet installed.)
 
[[File:Screenshot_2020-10-21_at_19.35.28.png|480px]]
 
'''Installing WinSCP on your Windows machine'''
 
We describe how you can install WinSCP even if you don't have administrator permissions on your system.
 
1. Download the portable version of WinSCP by opening [https://winscp.net/download/WinSCP-5.17.8-Portable.zip Download WinSCP] in a web browser (we use Google Chrome in the screenshots below).
 
2. Open the folder to which the application WinSCP was downloaded (on some systems you first have to decompress the downloaded zip-file). Click the areas marked with the arrows 1 and 2 in the screenshot below.
 
[[File:Screenshot_2020-10-21_at_19.43.46.png|480px]]
 
3. Copy the WinSCP application (highlighted line in screenshot below) to your desktop (by dragging it with the mouse pointer or trackpad to the desktop next to the putty icon).
 
[[File:Screenshot_2020-10-21_at_19.51.25.png|480px]]
 
4. Double-click on the newly created icon on your desktop to open the WinSCP application. The WinSCP application window is shown in the screenshot below.
 
[[File:Screenshot_2020-10-21_at_19.57.39.png|480px]]
 
=First Login Attempt=
 
For the UNIX and HPC parts it is essential that you can login into the HPC cluster. In this section, we will illustrate how you can do that on Linux (for example, Ubuntu), macOS and Windows.
 
'''We highly encourage you to try this before the course actually starts, because we will have limited possibilities to help you during the course.'''
 
==Linux / macOS==
 
Start a Terminal as described above.
 
In the Terminal type <code>ssh</code> followed by a space, then type your username (the one you filled in when you created an account), directly followed by <code>@saga.sigma2.no</code>. For example, you could write
 
ssh garfield@saga.sigma2.no
 
and then hit the ENTER/RETURN key.
 
If you login for the first time, you may see the following lines with a question:
 
The authenticity of host 'saga.sigma2.no (158.36.42.36)' can't be established.
ECDSA key fingerprint is SHA256:qirKlTjO9QSXuCAiuDQeDPqq+jorMFarCW+0qhpaAEA.
Are you sure you want to continue connecting (yes/no/[fingerprint])?
 
The ssh program simply tells you that it cannot (yet) verify the authenticity of the host ''saga.sigma2.no''. So you have to this by comparing the fingerprint shown in the terminal with what we have in our documentation, particularly, see [https://documentation.sigma2.no/getting_started/create_ssh_keys.html#sha256-fingerprint fingerprints for HPC machines]. If the fingerprints are the same, you can type <code>yes</code> and hit the ENTER/RETURN key.
 
Then you should see the following two lines
 
Warning: Permanently added 'saga.sigma2.no' (ECDSA) to the list of known hosts.
garfield@saga.sigma2.no's password:
 
The first line tells you that host saga.sigma2.no has been added to the list of known hosts. For future logins, ssh can verify the authenticity itself. The second line begins with your username followed by <code>@saga.sigma2.no</code> and <code>password:</code> which asks you to type your password (the one you set yourself when you created the account). '''Note, when you type the password it will not be shown.'''
 
Type the password carefully and then hit the ENTER/RETURN key. If the password is correct, you should the following message (we will explain this in the course). Depending on the load of the system it may take a while until you see a line similar to the last line below (<code>[garfield@login-3.SAGA ~]$ </code>).
 
 
Last login: Thu Oct 15 10:50:54 2020 from 81.99.113.135
Welcome to Saga.sigma2.no!
Documentation: https://documentation.sigma2.no/
Support email: support@metacenter.no
Request resources:
https://www.sigma2.no/content/apply-e-infrastructure-resources/
---------------------------------------------------------------------------
Latest news from: https://opslog.sigma2.no/
  o 2020-10-13: Downtime continues for NIRD
  o 2020-10-12: Fram firmware upgrade finished
  o 2020-10-12: Planned maintenance now in operation (Monday 12th October)
  o 2020-10-09: Problem with desktop-vnc on Saga
  o 2020-10-08: NIRD-TOS problems again
---------------------------------------------------------------------------
NOTE: Current $USERWORK autocleanup period is 21 days.
WARNING: Backup is taken *only* for $HOME which has quota in place.
          For more info see: https://documentation.sigma2.no/files_storage/backup.html
[garfield@login-3.SAGA ~]$
 
'''Congrats, you did it!'''
 
To logout type <code>exit</code> and then hit the ENTER/RETURN key. '''If you can't login, please contact us via email.'''
 
<strong>Note, if logging in with ssh works and you have verified that scp installed, transferring files with scp will also work!</strong>
 
==Windows==
 
Start PuTTY as described above.
 
In the PuTTY Configuration window you should select ''Session'' in the left column named ''Category:''. See the screenshot below.
 
[[File:Screenshot_2020-10-15_at_01.19.44.png|480px]]
 
Then type in ''saga.sigma2.no'' in the field named ''Host Name (or IP Address)'' and click on the button Open. See the screenshot below.
 
[[File:Screenshot_2020-10-15_at_01.20.08.png|480px]]
 
A new window is opened with the first line
 
login as:
 
where you have to type in your username (the one you filled in when you created an account) and hit the ENTER/RETURN key. In the screenshot below, the username is <code>garfield</code>.
 
[[File:Screenshot_2020-10-15_at_01.22.04.png|480px]]
 
As shown in the screenshot, a new line asks for your password (the one you set yourself when you created the account). '''Note, when you type the password it will not be shown.'''
 
Type the password carefully and then hit the ENTER/RETURN key. If the password is correct, you should see the following message (we will explain this in the course). Depending on the load of the system it may take a while until you see a line similar to the last line below (<code>[garfield@login-3.SAGA ~]$ </code>).
 
 
Last login: Thu Oct 15 10:50:54 2020 from 81.99.113.135
Welcome to Saga.sigma2.no!
Documentation: https://documentation.sigma2.no/
Support email: support@metacenter.no
Request resources:
https://www.sigma2.no/content/apply-e-infrastructure-resources/
---------------------------------------------------------------------------
Latest news from: https://opslog.sigma2.no/
  o 2020-10-13: Downtime continues for NIRD
  o 2020-10-12: Fram firmware upgrade finished
  o 2020-10-12: Planned maintenance now in operation (Monday 12th October)
  o 2020-10-09: Problem with desktop-vnc on Saga
  o 2020-10-08: NIRD-TOS problems again
---------------------------------------------------------------------------
NOTE: Current $USERWORK autocleanup period is 21 days.
WARNING: Backup is taken *only* for $HOME which has quota in place.
          For more info see: https://documentation.sigma2.no/files_storage/backup.html
[garfield@login-3.SAGA ~]$
 
'''Congrats, you did it!'''
 
To logout type <code>exit</code> and then hit the ENTER/RETURN key. '''If you can't login, please contact us via email.'''
 
<strong>Note, if logging in with PuTTY works and you have verified that WinSCP installed, transferring files with WinSCP will also work!</strong>


[[Category:Training]]
[[Category:Training]]

Revision as of 23:14, 4 February 2021

In March 2021, the Metacenter partners Sigma2, UiB, UiO & UiT are jointly offering a hands-on course for current and future users of national HPC systems (Saga, Fram, Betzy) and the NIRD Toolkit. The course is to all users of these systems. For individual sessions we list recommended prerequisites and learning outcomes. Recommended means, we believe you may struggle following the session if you don't know or are familiar with what is listed. However, we leave the decision up to you, and you're certainly welcome to just attend all sessions you're interested in.

On each course day, we will start at 09:00 and conclude at around 12:15 (3 sessions) or 13:15 (4 sessions). Each "morning" will include presentations and hands-on time on one of the actual HPC systems or the NIRD Toolkit. We recommend that participants use the afternoon on each day to repeat examples from the morning to get more experience on the systems.

Participants require access to a computer (not provided by the course organisers) with Internet connectivity and pre-installed programs to participate in the video meeting of the course (zoom, might work from a browser), to access the HPC systems (ssh or Putty) or the NIRD Toolkit (web browser).

Note, registration will close minimum 14 days before the first lecture day to have sufficient time in providing access to HPC systems or the NIRD Toolkit if necessary.

Note 2, in order to use the NIRD Toolkit during the course you will need either a Feide account or a Feide OpenIDP guest account. You can create a Feide OpenIDP guest account at openidp.feide.no. If you're unsure about this, please contact us well in advance - minimum two weeks before the NIRD Toolkit sessions.

You can always contact us by sending an email to support@metacenter.no.

The courses are free of charge.

HPC and NIRD Toolkit course

Dates: March 16, 17, 18 and 19, 2021
Time: each day 09:00-12:15/13:15 with 3-4 short breaks
Location: video meeting (information will be send to registered participants), Note, sessions will be recorded and made publicly available after course!

Participants may choose the lectures which they are interested in.

Registration

The course is free of charge and is organised by the Metacenter partners Sigma2, UiB, UiO & UiT.

Registration closes on March 2nd LINK TO BE DONE

Agenda

Day 1 - Welcome, Infrastructure news, HPC system details, Software installation & How to get help? - 16.03.2021

Get ready: 08:50-09:00 Connect to zoom meeting Session 1: 09:00-09:50 Welcome, Infrastructure news & HPC system details

  • Prerequisites: None
  • Learning outcomes: get to know what infrastructure is available/coming soon; understand details & differences of available HPC systems

Session 2: 10:00-10:50 Software installation - basics & sharing: pip/conda & R

  • Prerequisites: Account on HPC system, know how to use UNIX shell
  • Learning outcomes: know how to find, install & use a Python or R package under a user-writeable location, know how to share a self-installed Python or R package with other users

Session 3: 11:00-11:50 Software installation - basics & sharing: EasyBuild

  • Prerequisites: Account on HPC system, know how to use UNIX shell
  • Learning outcomes: know how to find, install & use an EasyBuild package under a user-writeable location, know how to share a self-installed EasyBuild package with other users

Session 4: 12:00-12:40 Asking for help (ca 20 min) & Q&A (ca 40 min)

  • Prerequisites: None
  • Learning outcomes: know how to get help, particularly how to send requests to the helpdesk/support and know about the existing documentation

Day 2 - Jobs, Determining resource requirements & Using storage - 17.03.2021

Get ready: 08:50-09:00 Connect to zoom meeting Session 1: 09:00-09:50 Jobs: layouts, dependencies, array jobs, access to internet, job priority, accounting

  • Prerequisites: Know what a job is and have used them already on an HPC system; know what Slurm is; know how to use UNIX shell and an editor
  • Learning outcomes: understand more details about jobs; know when to use nodes, tasks, threads and how to specify them; know when and how to use array jobs and dependencies between jobs and how these are scheduled; understand job priorities; be able to determine the cost of a job (accounting); be able to access the internet within a job, e.g., to download data from a web server or clone some GitHub repository

Session 2: 10:00-10:50 How to improve job scripts for better resource usage (cores and memory)

  • Prerequisites: Know what a job is; know a scripting language (e.g. bash) which is used for job scripts; know how to use UNIX shell and an editor
  • Learning outcomes: assess how many resources a job has used; be able to adjust resource requirements of a job as accurate as possible

Session 3: 11:00-11:50 Storage 1 - Data sharing, Interoperability HPC-NIRD, Striping, How to use file systems well?

  • Prerequisites: Account on HPC system and/or NIRD; know how to use UNIX shell; know how to do remote data transfers
  • Learning outcomes: be able to share data with external collaborators; be able to transfer data between HPC systems and NIRD; understand and be able to use striping; know about limitations of file systems and means to improve file system use (for yourself and other users)

Session 4: 12:00-12:50 Storage 2 - Disk quota and usage, data archiving

  • Prerequisites: Account on HPC system and/or NIRD; know how to use UNIX shell; know how to do remote data transfers
  • Learning outcomes: understand characteristics of different storage areas; figure out how much disk you use and how to solve problems (e.g., "disk quota exceeded"); be able to transfer data to/from NIRD archive

Session Q&A: 13:00-13:20 Questions & Answers

  • Prerequisites: none

Day 3 - NIRD Toolkit & User/Project management - 18.03.2021

Get ready: 08:50-09:00 Connect to zoom meeting Session 1: 09:00-09:50 NIRD Toolkit: Introduction & basics

  • Prerequisites: account to access the NIRD Toolkit; web browser
  • Learning outcomes: understand what the NIRD Toolkit is, what it can be used for

Session 2: 10:00-10:50 NIRD Toolkit: demos & setting up student group exercises

  • Prerequisites: account to access the NIRD Toolkit; web browser
  • Learning outcomes: how to launch an application/a service with the NIRD Toolkit; how to setup a group (e.g., of students) and give them access to an application/a service

Session 3: 11:00-11:50 User/project management

  • Prerequisites: web browser; not strictly a requirement, but this session is likely most useful for leaders or executive officers of a project
  • Learning outcomes: understand project accounts, user accounts, user accesses and how to manage all these via the web interface, e.g., how to give a specific user access to a project account on a certain HPC machine

Session Q&A: 12:00-12:20 Questions & Answers

  • Prerequisites: none

Day 4 - Containers - 19.03.2021

Get ready: 08:50-09:00 Connect to zoom meeting Session 1: 09:00-09:50 Containers: Essentials

  • Prerequisites: none
  • Learning outcomes: understand what a container is and how to use existing containers

Session 2: 10:00-10:50 Containers: How to use them on our HPC machines or the NIRD SP/Toolkit

  • Prerequisites: account on an HPC system or the NIRD Toolkit; web browser; understand basics of containers
  • Learning outcomes: how to launch a container on an HPC system or the NIRD Toolkit; how to access data on the file system from inside a container

Session 3: 11:00-11:50 Containers: Advanced topics

  • Prerequisites: account on an HPC system or the NIRD Toolkit; web browser; have used containers before
  • Learning outcomes: use containers more efficiently; be able to build your own containers or modify/extend existing containers

Session Q&A: 12:00-12:20 Questions & Answers

  • Prerequisites: none

Preparing your machine for the course

We assume you have the necessary tools installed on your machine and are able to use them. You need tools to login into a remote machine (e.g., ssh or PuTTY) and to transfer data to/from a remote machine (e.g., scp or WinSCP). If you need to install such tools, please see the pages of our course in November 2020.

Retrieved from "https://hpcdoc.wiki.uib.no/index.php?title=HPC_and_NIRD_toolkit_user_course_March_2021&oldid=1307"