Introduction to Unix CLI

What is Unix CLI?

Unix CLI is actually the shell, a text interface to the operating system. Shell interprets and executes your commands and their arguments by passing them to the operating system.
There are different shell variants, like bash, csh, ksh, sh, tcsh, etc., probably the most popular being bash. The name “bash” is an acronym for “Bourne Again SHell” and is an enhanced replacement for sh. In this short introductory we are going to cover bash.

Bash

Bash is an interactive and scriptable shell. Supports name expansion, wildcards and typing shortcuts.

Note: Interaction with bash is case-sensitive just like everything in Unix system.

Some useful shortcuts:

TAB, UP, DOWN, Ctrl+R, Esc+., Ctrl+C,!<command>

Popular commands

man

man is the system's manual pager. It is an interface to on-line manuals. man has it's own manual pager.

$ # Show default man page for top command
$ man sleep
$ # Show all manual pages for top command, successively
$ man -a sleep

Whenever you are uncertain what a command does, what options and argument can be passed to, man is your friend.

apropos

apropos searches the manual page names and descriptions for the specified keyword.

$ apropos shell
bash                 (1)  - GNU Bourne-Again SHell
bash [sh]            (1)  - GNU Bourne-Again SHell
capsh                (1)  - capability shell wrapper
chroot               (1)  - run command or interactive shell with special root directory
chsh                 (1)  - change your login shell
…

pwd

pwd prints the full name of the current working directory.

$ pwd
/tmp/test/course2015

cd

cd changes the current working directory to the specified directory name. If no directory is specified than cd changes current working directory to users home directory. cd supports relative and absolute paths. Additionally shortcuts like ~ (points to user's home directory).

$ pwd
/tmp/test/course2015
$ cd bin
$ pwd
/tmp/test/course2015/bin
$ cd ../../src
$ pwd
/tmp/test/src
$ cd ~/bin
$ pwd
/home/lorand/bin
$ # Change back to previous directory
$ cd -

mkdir

mkdir command creates directories if they do not exist.

$ mkdir build

To create a whole directory structure, use "-p" option.

$ mkdir -p ~/build/wrf/3.7

ls

ls command lists content of the current or the specified directory. One of the most used options for ls is "-l", which will generate a long list format.

$ ls
GConf                                   libegroupwise-1.2.so.13                     libospf.so.0.0.0
ImageMagick-6.4.3                       libegroupwise-1.2.so.13.0.1                 libossp-uuid.so.16
Mcrt1.o                                 libelf-0.152.so                             libossp-uuid.so.16.0.22
PackageKitDbusTest.py                   libelf.so.0                                 libp11.so.1
…

$ ls -l
total 806992
drwxr-xr-x  3 root root     4096 Mar 11  2012 GConf
drwxr-xr-x  4 root root     4096 Nov 18  2014 ImageMagick-6.4.3
-rw-r--r--  1 root root     1358 Oct  8 20:07 Mcrt1.o
-rw-r--r--  1 root root     3383 May 29  2013 PackageKitDbusTest.py
-rw-r--r--  1 root root     4267 Oct  8 20:07 Scrt1.o
…

"-a" will list all files and folders, even hidden ones.
NB: Files, directories starting with "." character are treated as hidden files.

$ pwd
/work/shared/course2015/intro/bash

$ ls -l
total 0

$ ls -la
total 33
drwxr-xr-x 2 lorand root 24576 Oct 27 10:10 .
drwxr-xr-x 3 lorand root  4096 Oct 27 10:07 ..
-rw-r--r-- 1 lorand root  1177 Oct 27 10:10 .bashrc
-rw-r--r-- 1 lorand root   849 Oct 27 10:10 .vimrc

Filename expansion is also supported with the usage of special characters, so called wildcards "?" or "*". "?" replaces one character, while "*" replaces any number of characters.

$ ls -a .*rc
.bashrc  .vimrc

"-t" option sorts listing by modification time, while "-r" can reverse sorting order.

$ ls -lt
total 8
drwxr-xr-x 3 lorand root 4096 Oct 27 10:07 intro
drwxr-xr-x 4 floan  root 4096 Oct 21 12:10 openMP

$ ls -ltr
total 8
drwxr-xr-x 4 floan  root 4096 Oct 21 12:10 openMP
drwxr-xr-x 3 lorand root 4096 Oct 27 10:07 intro

cp

cp command copies files and directories. Copies source (might be one or more files/directories) to destination. When multiple files/directories are specified as source, the destination has to be a directory.

$ # copy messages file to /backup folder
$ cp /var/log/messages /backup
$ # copy messages and maillog files to /backup folder
$ cp /var/log/messages /var/log/maillog /backup
$ # copy all C source to network mounted home folder
$ # preserving timestamps and permissions, be verbose
$ cp -a -v ~/development/my_project/*.c /mnt/nfs/projects/sources/
$ # copy all new or modified files to destination
$ cp -u -v ~/development/my_project/*.c /mnt/nfs/projects/sources/
$ # copy all new or modified files to destination
$ # be verbose and interactive - ask what to do
$ cp -u -i -v ~/development/my_project/*.c /mnt/nfs/projects/sources/

mv

mv moves source (might be one or more files/directories) to destination. Same as for cp command, when multiple files/directories are specified as source, the destination has to be a directory. mv command is also used for renaming a file. NB: Irreversible!

$ # interactively move python files to python_files/ subdirectory
$ mv -i *.py python_files/
$ # move python files and force overwrite if similar file exists
$ # be verbose
$ mv -f -v *.py python_files/

rm

rm command removes files and directories. rm will not remove directories by default unless recursive - -r - option is specified. NB: Irreversible! It is advisable to create an alias for rm command to "rm -i" (see alias command).

$ # interactively remove all object files in current directory
$ rm -i ./*.o
$ # force removal of ~/tmp/ directory and all of it's contents
$ rm -rf ~/tmp/

find

find searches for files in a directory hierarchy. find command evaluates the given options from left to right, thus adding options in correct order is important. For example -maxdepth option must be appended before -name option, otherwise it will be ignored.

Find all the files whose name is hello.txt in a current working directory.

$ find . -name hello.txt
./hello.txt

Find all the files under /home directory with name hello.txt.

$ find /home -name  hello.txt
/home/hello.txt

Find all the files whose name is hello.txt and contains both capital and small letters in /home directory.

$ find /home -iname hello.txt
./hello.txt
./Hello.txt

Find all directories whose name is Hello in current working directory

$ find . -type d -name Hello
./Hello

To find a single file called hello.txt and remove it in current directory.

$ find . -type f -name "hello.txt" -exec rm -f {} \;

scp

scp command securely copies between hosts on the network. It uses ssh, thus providing same authentication and security mechanism. File names can be relative or absolute and may contain user and host specification. Some useful options are:

• -P - port number to connect to on remote host
• -p - preserve modification times, access times, and modes from the original file

• -r - copy recursively

Recursively copy examples folder to my home on hexagon.

scp -r ~/examples/ lorand@hexagon.hpc.uib.no:

Copy python files from hexagon, preserving modification and access times to /scratch/development.

scp -p lorand@hexagon.hpc.uib.no:*.py /scratch/development

cat

cat command concatenates files and prints on the standard output.

$ # concatenate motd
$ cat /etc/motd
Rocks Compute Node
Rocks 6.1 (Emerald Boa)
Profile built 16:14 10-Oct-2015

less

less allows paging trough text, one screenful at a time. Allows forward and backward movements in the text, line by line, or page by page. Useful commands in less are:

• "/" searches forward
• "?" searches backward

• "-I" switching between case-sensitive/case-insensitive searches
• "q" quits less

grep

grep and egrep prints lines matching a pattern. egrep is the same as grep -E.

Print lines containing "student".

$ grep "student" applicants.txt
student Don Joe
student Navn Navnesen

Print lines containing "student", but be case-insensitive.

$ grep -i "student" applicants.txt
Student Neve Sincs
student John Doe
STudent Anonymous
student Navn Navnesen
StudEnT Hans Hansen

Print lines not containing "Navnesen" and "Hansen".

$ grep -v -E "Navnesen| Hansen" applicants.txt

Student Neve Sincs
student John Doe
STudent Anonymous

List files, without showing their content, containing "alias".

$ grep -Rl mpif90 .
./efg-3.3.1-20130426-cray/external/io_mcel/configure.wrf.example
./efg-3.3.1-20130426-cray/arch/archive_configure.defaults
./efg-3.3.1-20130426-cray/arch/configure_old.defaults

tail

tail outputs the last part (by default last 10 lines) of the file.

$ # show last 25 lines from messages file
$ tail -25 /var/log/messages
$ # follow file output, output will continue until it is
$ # broke with Ctrl+C
$ tail -f /var/log/messages

head

head command prints the first 10 lines of each file. For more than one file given, head will list the first 10 lines for each file, preceding with the file name. -n option overrides the default the number of lines.

$ head -1 /etc/motd
Welcome to hexagon.hpc.uib.no, a Cray XE6m-200 system operated by UiB.

$ ps x | head -2
  PID TTY      STAT   TIME COMMAND
14065 ?        S      0:00 sshd: lorand@pts/9

awk

awk is a pattern scanning and processing language. Assuming the following example file with tab separated fields.

$ cat bergen-weather-statistics.txt
jan	3.4°	516.1 mm	21.8 m/s
feb	3.4°	246.4 mm	17.5 m/s
mar	5.2°	349.0 mm	17.7 m/s
apr	6.2°	140.9 mm	11.2 m/s
mai	8.7°	240.5 mm	13.4 m/s
jun	11.4°	129.8 mm	12.9 m/s
jul	14.4°	108.2 mm	10.9 m/s
aug	16.5°	192.4 mm	12.3 m/s
sep	13.5°	121.9 mm	12.3 m/s
okt	9.6°	167.1 mm	12.1 m/s

Print the monthly precipitation and add header.

$ awk 'BEGIN {print "MONTH\tPRECIPITATION\n---------------------";} \
{print $1, "\t", $3;}' bergen-weather-statistics.txt

MONTH	PRECIPITATION
---------------------
jan 	 516.1
feb 	 246.4
mar 	 349.0
apr 	 140.9
mai 	 240.5
jun 	 129.8
jul 	 108.2
aug 	 192.4
sep 	 121.9
okt 	 167.1

Additionally to the above example, write a footer and append the total sum of monthly precipitation.

$ awk 'BEGIN {print "MONTH\tPRECIPITATION\n---------------------";} \
{print $1, "\t", $3;} {SUM+=$3;} END {print "---------------------\nTOTAL\t",SUM,"mm";}' \
 bergen-weather-statistics.txt

MONTH	PRECIPITATION
---------------------
jan 	 516.1
feb 	 246.4
mar 	 349.0
apr 	 140.9
mai 	 240.5
jun 	 129.8
jul 	 108.2
aug 	 192.4
sep 	 121.9
okt 	 167.1
---------------------
TOTAL	 2212.3 mm

List maximum wind speed for each month and calculate average wind speed.

$ awk 'BEGIN {print "MONTH\tMAX. WINDSPEED\n---------------------";} {print $1, "\t", $5;}\
{SUM+=$5;} END {print "---------------------\nAVG.\t", SUM/NR,"m/s";}' bergen-weather-statistics.txt

MONTH	MAX.WINDSPEED
---------------------
jan 	 21.8
feb 	 17.5
mar 	 17.7
apr 	 11.2
mai 	 13.4
jun 	 12.9
jul 	 10.9
aug 	 12.3
sep 	 12.3
okt 	 12.1
---------------------
AVG.	 14.21 m/s

alias

alias command defines or lists aliases. Command itself without any arguments will list existing aliases.

$ lc
-bash: lc: command not found
$ alias lc='ls -l --color'
$ lc
-rwxr-xr-x  1 root   root       5008747 Oct 13 13:35 helloworld_c
-rw-------  1 lorand root             0 Oct 13 13:36 helloworld_c.err
-rw-------  1 lorand root          2068 Oct 13 13:38 helloworld_c.out

$ alias
alias ..='cd ..'
alias ...='cd ../..'
alias build='cd /work/$USER/build'
alias cd..='cd ..'
alias dir='ls -l'
alias lc='ls -l --color'

File system

Important directories typically present on a UNIX system:

• /bin - contains the most essential UNIX commands
• /etc - configuration files
• /home - user files
• /lib - libraries
• /sbin - sysadmin commands
• /tmp - temporary files
• /usr - user programs
  • /usr/include - standard location for C include files
  • /usr/lib & /usr/lib64 - standard libraries
• /var - administrative files, log files