HPC and NIRD toolkit user course March 2021

In March 2021, the Metacenter partners Sigma2, UiB, UiO & UiT are jointly offering a hands-on course for current and future users of national HPC systems (Saga, Fram, Betzy) and the NIRD Toolkit. The course is open to all users of these systems. For individual sessions we list recommended prerequisites and learning outcomes. Recommended means, we believe you may struggle following the session if you do not know or are not familiar with what is listed. However, we leave the decision to you which session(s) you want to attend, and you're certainly welcome to just attend all sessions you are interested in.

On each course day, we will start at 08:50 and conclude at around 12:15 (3 sessions) or 13:15 (4 sessions). Each "morning" will include presentations and hands-on time on one of the actual HPC systems or the NIRD Toolkit. We recommend that participants use the afternoon on each day to repeat examples from the morning to get more experience on the systems.

Participants require access to a computer (not provided by the course organisers) with Internet connectivity and pre-installed programs to participate in the video meeting of the course (zoom, might work from a browser), to access the HPC systems (e.g., ssh or Putty), to transfer data to/from an HPC system (e.g., scp or WinSCP) and to access the NIRD Toolkit (web browser).

Note, registration will close minimum 14 days before the first lecture day to have sufficient time in providing access to HPC systems or the NIRD Toolkit if necessary.

Note 2, in order to use the NIRD Toolkit during the course you will need either a Feide account or a Feide OpenIDP guest account. You can create a Feide OpenIDP guest account at openidp.feide.no. If you're unsure about this, please contact us well in advance - minimum two weeks before the NIRD Toolkit sessions.

You can always contact us by sending an email to support@metacenter.no.

The courses are free of charge.

HPC and NIRD Toolkit course

Dates: March 16, 17, 18 and 19, 2021
Time: each day 09:00-12:15/13:15 with 3-4 short breaks
Location: video meeting (information will be send to registered participants), Note, sessions will be recorded and made publicly available after the course!

Participants may choose the lectures which they are interested in.

Registration

The course is free of charge and is organised by the Metacenter partners Sigma2, UiB, UiO & UiT.

Registration closes on March 2nd LINK TO BE DONE

Agenda

Day 1 - Welcome, Infrastructure news, HPC system details, Software installation & How to get help? - 16.03.2021

Get ready: 08:50-09:00 Connect to zoom meeting

Session 1: 09:00-09:50 Welcome, Infrastructure news & HPC system details

• Prerequisites: None
• Learning outcomes: get to know what infrastructure is available/coming soon; understand details & differences of available HPC systems

Session 2: 10:00-10:50 Software installation - basics & sharing: pip/conda & R

• Prerequisites: Account on HPC system, know how to use UNIX shell
• Learning outcomes: know how to find, install & use a Python or R package under a user-writeable location, know how to share a self-installed Python or R package with other users

Session 3: 11:00-11:50 Software installation - basics & sharing: EasyBuild

• Prerequisites: Account on HPC system, know how to use UNIX shell
• Learning outcomes: know how to find, install & use an EasyBuild package under a user-writeable location, know how to share a self-installed EasyBuild package with other users

Session 4: 12:00-12:40 Asking for help (ca 20 min) & Q&A (ca 40 min)

• Prerequisites: None
• Learning outcomes: know how to get help, particularly how to send requests to the helpdesk/support and know about the existing documentation

Day 2 - Jobs, Determining resource requirements & Using storage - 17.03.2021

Get ready: 08:50-09:00 Connect to zoom meeting

Session 1: 09:00-09:50 Jobs: layouts, dependencies, array jobs, access to internet, job priority, accounting

• Prerequisites: Know what a job is and have used them already on an HPC system; know what Slurm is; know how to use UNIX shell and an editor
• Learning outcomes: understand more details about jobs; know when to use nodes, tasks, threads and how to specify them; know when and how to use array jobs and dependencies between jobs and how these are scheduled; understand job priorities; be able to determine the cost of a job (accounting); be able to access the internet within a job, e.g., to download data from a web server or clone some GitHub repository

Session 2: 10:00-10:50 How to improve job scripts for better resource usage (cores and memory)

• Prerequisites: Know what a job is; know a scripting language (e.g. bash) which is used for job scripts; know how to use UNIX shell and an editor
• Learning outcomes: assess how many resources a job has used; be able to adjust resource requirements of a job as accurate as possible

Session 3: 11:00-11:50 Storage 1 - Data sharing, Interoperability HPC-NIRD, Striping, How to use file systems well?

• Prerequisites: Account on HPC system and/or NIRD; know how to use UNIX shell; know how to do remote data transfers
• Learning outcomes: be able to share data with external collaborators; be able to transfer data between HPC systems and NIRD; understand and be able to use striping; know about limitations of file systems and means to improve file system use (for yourself and other users)

Session 4: 12:00-12:50 Storage 2 - Disk quota and usage, data archiving

• Prerequisites: Account on HPC system and/or NIRD; know how to use UNIX shell; know how to do remote data transfers
• Learning outcomes: understand characteristics of different storage areas; figure out how much disk you use and how to solve problems (e.g., "disk quota exceeded"); be able to transfer data to/from NIRD archive

Session Q&A: 13:00-13:20 Questions & Answers

• Prerequisites: none

Day 3 - NIRD Toolkit & User/Project management - 18.03.2021

Get ready: 08:50-09:00 Connect to zoom meeting

Session 1: 09:00-09:50 NIRD Toolkit: Introduction & basics

• Prerequisites: account to access the NIRD Toolkit; web browser
• Learning outcomes: understand what the NIRD Toolkit is, what it can be used for

Session 2: 10:00-10:50 NIRD Toolkit: demos & setting up student group exercises

• Prerequisites: account to access the NIRD Toolkit; web browser
• Learning outcomes: how to launch an application/a service with the NIRD Toolkit; how to setup a group (e.g., of students) and give them access to an application/a service

Session 3: 11:00-11:50 User/project management

• Prerequisites: web browser; not strictly a requirement, but this session is likely most useful for leaders or executive officers of a project
• Learning outcomes: understand project accounts, user accounts, user accesses and how to manage all these via the web interface, e.g., how to give a specific user access to a project account on a certain HPC machine

Session Q&A: 12:00-12:20 Questions & Answers

• Prerequisites: none

Day 4 - Containers - 19.03.2021

Get ready: 08:50-09:00 Connect to zoom meeting

Session 1: 09:00-09:50 Containers: Essentials

• Prerequisites: none
• Learning outcomes: understand what a container is and how to use existing containers

Session 2: 10:00-10:50 Containers: How to use them on our HPC machines or the NIRD SP/Toolkit

• Prerequisites: account on an HPC system or the NIRD Toolkit; web browser; understand basics of containers
• Learning outcomes: how to launch a container on an HPC system or the NIRD Toolkit; how to access data on the file system from inside a container

Session 3: 11:00-11:50 Containers: Advanced topics

• Prerequisites: account on an HPC system or the NIRD Toolkit; web browser; have used containers before
• Learning outcomes: use containers more efficiently; be able to build your own containers or modify/extend existing containers

Session Q&A: 12:00-12:20 Questions & Answers

• Prerequisites: none

Preparing your machine for the course

We assume you have the necessary tools installed on your machine and are able to use them. You need tools to login into a remote machine (e.g., ssh or PuTTY) and to transfer data to/from a remote machine (e.g., scp or WinSCP). If you need to install such tools, please see the pages of our course in November 2020.