Introduction to Unix CLI
What is Unix CLI?
Unix CLI is actually the shell, a text interface to the operating system. Shell interprets and executes your commands and their arguments by passing them to the operating system.
There are different shell variants, like bash, csh, ksh, sh, tcsh, etc., probably the most popular being bash. The name “bash” is an acronym for “Bourne Again SHell” and is an enhanced replacement for sh. In this short introductory we are going to cover bash.
Bash
Bash is an interactive and scriptable shell. Supports name expansion, wildcards and typing shortcuts.
Note: Interaction with bash is case-sensitive just like everything in Unix system.
Some useful shortcuts:
TAB, UP, DOWN, Ctrl+R, Esc+., Ctrl+C,!<command>
Popular commands
man
man is the system's manual pager. It is an interface to on-line manuals. man has it's own manual pager.
$ # Show default man page for top command $ man sleep $ # Show all manual pages for top command, successively $ man -a sleep
Whenever you are uncertain what a command does, what options and argument can be passed to, man is your friend.
apropos
apropos searches the manual page names and descriptions for the specified keyword.
$ apropos shell bash (1) - GNU Bourne-Again SHell bash [sh] (1) - GNU Bourne-Again SHell capsh (1) - capability shell wrapper chroot (1) - run command or interactive shell with special root directory chsh (1) - change your login shell …
pwd
pwd prints the full name of the current working directory.
$ pwd /tmp/test/course2015
cd
cd changes the current working directory to the specified directory name. If no directory is specified than cd changes current working directory to users home directory. cd supports relative and absolute paths. Additionally shortcuts like ~ (points to user's home directory).
$ pwd /tmp/test/course2015 $ cd bin $ pwd /tmp/test/course2015/bin $ cd ../../src $ pwd /tmp/test/src $ cd ~/bin $ pwd /home/lorand/bin $ # Change back to previous directory $ cd -
mkdir
mkdir command creates directories if they do not exist.
$ mkdir build
To create a whole directory structure, use "-p" option.
$ mkdir -p ~/build/wrf/3.7
ls
ls command lists content of the current or the specified directory. One of the most used options for ls is "-l", which will generate a long list format.
$ ls GConf libegroupwise-1.2.so.13 libospf.so.0.0.0 ImageMagick-6.4.3 libegroupwise-1.2.so.13.0.1 libossp-uuid.so.16 Mcrt1.o libelf-0.152.so libossp-uuid.so.16.0.22 PackageKitDbusTest.py libelf.so.0 libp11.so.1 … $ ls -l total 806992 drwxr-xr-x 3 root root 4096 Mar 11 2012 GConf drwxr-xr-x 4 root root 4096 Nov 18 2014 ImageMagick-6.4.3 -rw-r--r-- 1 root root 1358 Oct 8 20:07 Mcrt1.o -rw-r--r-- 1 root root 3383 May 29 2013 PackageKitDbusTest.py -rw-r--r-- 1 root root 4267 Oct 8 20:07 Scrt1.o …
"-a" will list all files and folders, even hidden ones.
Note: Files, directories starting with "." character are treated as hidden files.
$ pwd /work/shared/course2015/intro/bash $ ls -l total 0 $ ls -la total 33 drwxr-xr-x 2 lorand root 24576 Oct 27 10:10 . drwxr-xr-x 3 lorand root 4096 Oct 27 10:07 .. -rw-r--r-- 1 lorand root 1177 Oct 27 10:10 .bashrc -rw-r--r-- 1 lorand root 849 Oct 27 10:10 .vimrc
Filename expansion is also supported with the usage of special characters, so called wildcards "?" or "*". "?" replaces one character, while "*" replaces any number of characters.
$ ls -a .*rc .bashrc .vimrc
"-t" option sorts listing by modification time, while "-r" can reverse sorting order.
$ ls -lt total 8 drwxr-xr-x 3 lorand root 4096 Oct 27 10:07 intro drwxr-xr-x 4 floan root 4096 Oct 21 12:10 openMP $ ls -ltr total 8 drwxr-xr-x 4 floan root 4096 Oct 21 12:10 openMP drwxr-xr-x 3 lorand root 4096 Oct 27 10:07 intro
cp
cp command copies files and directories. Copies source (might be one or more files/directories) to destination. When multiple files/directories are specified as source, the destination has to be a directory.
$ # copy messages file to /backup folder $ cp /var/log/messages /backup $ # copy messages and maillog files to /backup folder $ cp /var/log/messages /var/log/maillog /backup $ # copy all C source to network mounted home folder $ # preserving timestamps and permissions, be verbose $ cp -a -v ~/development/my_project/*.c /mnt/nfs/projects/sources/ $ # copy all new or modified files to destination $ cp -u -v ~/development/my_project/*.c /mnt/nfs/projects/sources/ $ # copy all new or modified files to destination $ # be verbose and interactive - ask what to do $ cp -u -i -v ~/development/my_project/*.c /mnt/nfs/projects/sources/
mv
mv moves source (might be one or more files/directories) to destination. Same as for cp command, when multiple files/directories are specified as source, the destination has to be a directory. mv command is also used for renaming a file. NB: Irreversible!
$ # interactively move python files to python_files/ subdirectory $ mv -i *.py python_files/ $ # move python files and force overwrite if similar file exists $ # be verbose $ mv -f -v *.py python_files/
rm
rm command removes files and directories. rm will not remove directories by default unless recursive - -r - option is specified. NB: Irreversible! It is advisable to create an alias for rm command to "rm -i" (see alias command).
$ # interactively remove all object files in current directory $ rm -i ./*.o $ # force removal of ~/tmp/ directory and all of it's contents $ rm -rf ~/tmp/
find
find searches for files in a directory hierarchy. find command evaluates the given options from left to right, thus adding options in correct order is important. For example -maxdepth option must be appended before -name option, otherwise it will be ignored.
Find all the files whose name is hello.txt in a current working directory.
$ find . -name hello.txt ./hello.txt
Find all the files under /home directory with name hello.txt.
$ find /home -name hello.txt /home/hello.txt
Find all the files whose name is hello.txt and contains both capital and small letters in /home directory.
$ find /home -iname hello.txt ./hello.txt ./Hello.txt
Find all directories whose name is Hello in current working directory
$ find . -type d -name Hello ./Hello
To find a single file called hello.txt and remove it in current directory.
$ find . -type f -name "hello.txt" -exec rm -f {} \;
scp
scp command securely copies between hosts on the network. It uses ssh, thus providing same authentication and security mechanism. File names can be relative or absolute and may contain user and host specification. Some useful options are:
- -P - port number to connect to on remote host
- -p - preserve modification times, access times, and modes from the original file
- -r - copy recursively
Recursively copy examples folder to my home on hexagon.
scp -r ~/examples/ lorand@hexagon.hpc.uib.no:
Copy python files from hexagon, preserving modification and access times to /scratch/development.
scp -p lorand@hexagon.hpc.uib.no:~/development/my_project/*.py /scratch/development
rsync
rsync is a file copying tool. It can copy locally and to/from another host. It has a delta-transfer algorithm, reducing network traffic by sending only the differences between source and destination. By default it uses ssh but it can be configured to use different remote shells.
Useful options for rsync:
- -v - increase verbosity
- -a - archive mode
- -u - update, exclude newer files on the destination
- -l - copy symlinks
- -H - copy hard links
- -p - preserve permissions
- -n - dry-run, no changes are made
- -z - compress
Transfer all files matching the pattern *.c from the current directory to your src directory on hexagon.
$ rsync -t *.c hexagon:~/src/
Recursively transfer all files from the WRF/output directory to your local machine. The files are transferred in "archive" mode, ensuring that symbolic links, permissions, ownerships are preserved. Be verbose and use compression. Note: please pay attention to trailing "/" in the examples below.
$ # create output directory if not exist $ rsync -avz lorand@hexagon:~/WRF/output /data/WRF $ # sync only files and do not create output directory $ rsync -avz lorand@hexagon:~/WRF/output/ /data/WRF
Recursively transfer my project folder to the backup. Exclude vim backup files and skip newer files in the local folder.
$ rsync -avuz --exclude '*~' hexagon:project/ /backup/hexagon
cat
cat command concatenates files and prints on the standard output.
$ # concatenate motd $ cat /etc/motd Rocks Compute Node Rocks 6.1 (Emerald Boa) Profile built 16:14 10-Oct-2015
less
less allows paging trough text, one screenful at a time. Allows forward and backward movements in the text, line by line, or page by page. Useful commands in less are:
- "/" searches forward
- "?" searches backward
- "-I" switching between case-sensitive/case-insensitive searches
- "q" quits less
grep
grep and egrep prints lines matching a pattern. egrep is the same as grep -E.
Print lines containing "student".
$ grep "student" applicants.txt student Don Joe student Navn Navnesen
Print lines containing "student", but be case-insensitive.
$ grep -i "student" applicants.txt Student Neve Sincs student John Doe STudent Anonymous student Navn Navnesen StudEnT Hans Hansen
Print lines not containing "Navnesen" and "Hansen".
$ grep -v -E "Navnesen| Hansen" applicants.txt Student Neve Sincs student John Doe STudent Anonymous
List files, without showing their content, containing "alias".
$ grep -Rl mpif90 . ./efg-3.3.1-20130426-cray/external/io_mcel/configure.wrf.example ./efg-3.3.1-20130426-cray/arch/archive_configure.defaults ./efg-3.3.1-20130426-cray/arch/configure_old.defaults
tail
tail outputs the last part (by default last 10 lines) of the file.
$ # show last 25 lines from messages file $ tail -25 /var/log/messages $ # follow file output, output will continue until it is $ # broke with Ctrl+C $ tail -f /var/log/messages
head
head command prints the first 10 lines of each file. For more than one file given, head will list the first 10 lines for each file, preceding with the file name. -n option overrides the default the number of lines.
$ head -1 /etc/motd Welcome to hexagon.hpc.uib.no, a Cray XE6m-200 system operated by UiB. $ ps x | head -2 PID TTY STAT TIME COMMAND 14065 ? S 0:00 sshd: lorand@pts/9
awk
awk is a pattern scanning and processing language. Assuming the following example file with tab separated fields.
$ cat bergen-weather-statistics.txt jan 3.4° 516.1 mm 21.8 m/s feb 3.4° 246.4 mm 17.5 m/s mar 5.2° 349.0 mm 17.7 m/s apr 6.2° 140.9 mm 11.2 m/s mai 8.7° 240.5 mm 13.4 m/s jun 11.4° 129.8 mm 12.9 m/s jul 14.4° 108.2 mm 10.9 m/s aug 16.5° 192.4 mm 12.3 m/s sep 13.5° 121.9 mm 12.3 m/s okt 9.6° 167.1 mm 12.1 m/s
Print the monthly precipitation and add header.
$ awk 'BEGIN {print "MONTH\tPRECIPITATION\n---------------------";} \
{print $1, "\t", $3;}' bergen-weather-statistics.txt
MONTH PRECIPITATION
---------------------
jan 516.1
feb 246.4
mar 349.0
apr 140.9
mai 240.5
jun 129.8
jul 108.2
aug 192.4
sep 121.9
okt 167.1
Additionally to the above example, write a footer and append the total sum of monthly precipitation.
$ awk 'BEGIN {print "MONTH\tPRECIPITATION\n---------------------";} \
{print $1, "\t", $3;} {SUM+=$3;} END {print "---------------------\nTOTAL\t",SUM,"mm";}' \
bergen-weather-statistics.txt
MONTH PRECIPITATION
---------------------
jan 516.1
feb 246.4
mar 349.0
apr 140.9
mai 240.5
jun 129.8
jul 108.2
aug 192.4
sep 121.9
okt 167.1
---------------------
TOTAL 2212.3 mm
List maximum wind speed for each month and calculate average wind speed.
$ awk 'BEGIN {print "MONTH\tMAX. WINDSPEED\n---------------------";} {print $1, "\t", $5;}\
{SUM+=$5;} END {print "---------------------\nAVG.\t", SUM/NR,"m/s";}' bergen-weather-statistics.txt
MONTH MAX.WINDSPEED
---------------------
jan 21.8
feb 17.5
mar 17.7
apr 11.2
mai 13.4
jun 12.9
jul 10.9
aug 12.3
sep 12.3
okt 12.1
---------------------
AVG. 14.21 m/s
sed
sed is a stream editor for filtering and transforming text. It is used to perform basic text transformations on an input, which can be either text or pipe.
sed syntax is sed 'PATTERN/REGEX/REPLACEMENT/FLAGS' file_name. Some common usage examples are listed below.
Read applicants.txt file and replace "student" with "employee".
$ cat applicants.txt Student Neve Sincs student John Doe STudent Anonymous student Navn Navnesen StudEnT Hans Hansen $ sed 's/student/employee/' < applicants.txt Student Neve Sincs employee John Doe STudent Anonymous employee Navn Navnesen StudEnT Hans Hansen
Same as above but be case-insensitive.
$ sed 's/student/employee/i' < applicants.txt employee Neve Sincs employee John Doe employee Anonymous employee Navn Navnesen employee Hans Hansen
In the above examples the edited text was written to stdout. Let's make the changes in-place.
$ cat applicants.txt Student Neve Sincs student John Doe STudent Anonymous student Navn Navnesen StudEnT Hans Hansen $ sed 's/student/employee/i' -i applicants.txt $ cat applicants.txt employee Neve Sincs employee John Doe employee Anonymous employee Navn Navnesen employee Hans Hansen
Replace "Navn" with "Name", then do it globally.
$ sed 's/Navn/Name/' < applicants.txt employee Neve Sincs employee John Doe employee Anonymous employee Name Navnesen employee Hans Hansen $ sed 's/Navn/Name/g' < applicants.txt employee Neve Sincs employee John Doe employee Anonymous employee Name Nameesen employee Hans Hansen
Eliminate comments.
$ cat applicants.txt #Student Neve Sincs student John Doe #STudent Anonymous student Navn Navnesen StudEnT Hans Hansen $ sed -e 's/#.*//' applicants.txt student John Doe student Navn Navnesen StudEnT Hans Hansen
Eliminate empty lines.
$ cat applicants.txt #Student Neve Sincs student John Doe #STudent Anonymous student Navn Navnesen StudEnT Hans Hansen $ sed -e '/^$/d' applicants.txt #Student Neve Sincs student John Doe #STudent Anonymous student Navn Navnesen StudEnT Hans Hansen
Eliminate both comments and empty lines.
$ sed -e 's/#.*//;/^$/d' applicants.txt student John Doe student Navn Navnesen StudEnT Hans Hansen
alias
alias command defines or lists aliases. Command itself without any arguments will list existing aliases.
$ lc -bash: lc: command not found $ alias lc='ls -l --color' $ lc -rwxr-xr-x 1 root root 5008747 Oct 13 13:35 helloworld_c -rw------- 1 lorand root 0 Oct 13 13:36 helloworld_c.err -rw------- 1 lorand root 2068 Oct 13 13:38 helloworld_c.out $ alias alias ..='cd ..' alias ...='cd ../..' alias build='cd /work/$USER/build' alias cd..='cd ..' alias dir='ls -l' alias lc='ls -l --color'
screen
screen is a full screen window manager, which allows you to create a window with a shell within it. It allows you to run your program even after logging out from the server or in case of network connection interruption. You will be able to pick up your work from where you left it.
Commands within the screen window consist of Ctrl+A plus one more character. Character D detaches from current screen. ? shows help.
Create a window named "test" and enable logging.
$ screen -S test -L
List existing screens.
$ screen -ls There are screens on: 31400.test2 (Detached) 31264.test1 (Detached) 2 Sockets in /var/run/uscreens/S-lorand.
Reattach to screen "test2".
$ screen -r test2
Attach to a non-detached screen (multi-display mode).
$ screen -x test1
Variables
Variables are referenced by $ sign. There are some special variables:
- $0 - name of the script
- $1, $2, ... , ${10}, ${11}, ... - are positional arguments
- $* - all arguments
- $# - number of arguments
User variables
User variables are defined by user and can not be the type of special variables or environment variables, however they can be overwritten.
$ myvar="Hello World" $ echo $myvar Hello World
Environment variables
Environment variables are initialized by system-wide configuration files. They can be appended or overwritten by user configuration files. Some of the most common environment variables are:
- PATH
- USER
- HOME
- LD_LIBRARY_PATH
- MANPATH
- CFLAGS
- LDFLAGS
List my current environment.
$ env
Adjust PATH to include local "bin" directory. Note: items in the PATH environment variable are ":" separated and processed from left to right.
$ echo $PATH /bin:/sbin:/usr/bin:/usr/sbin $ export PATH=~/bin:$PATH $ echo $PATH /home/lorand/bin:/bin:/sbin:/usr/bin:/usr/sbin
Remove CFLAGS variable.
$ echo $CFLAGS -march=native -O2 -pipe $unset CFLAGS $ echo $CFLAGS
Conditionals
if
In bash we have the if and the case conditionals.
Expressions used with if:
- -f - true if file exists and is a regular file
- -d - true if file exists and is a directory
- -s - true if file exists and it's size is greater then 0
- -x - true if file exists and is executable
- -z - true if the length of string is zero
- -nt- true if file1 is newer than file2 "[ file1 -nt file2 ]"
- -ot- true if file1 is older than file2 "[ file1 -ot file2 ]"
- == - true if the strings are equal
- != - true if the strings are not equal
Arithmetic expressions used with if:
- -eq - it equal to
- -ne - not equal to
- -lt - less then
- -le - less then or equal to
- -gt - greater then
- -ge - greater then or equal to
Note: arithmetic expressions have the form "[ arg1 OPERATOR arg2 ]", where "arg1" and "arg2" are integers.
Expressions in if conditional can be combined:
- ! - true if expression is false "[ ! exp ]"
- -a - true if both expressions are true "[ exp1 -a exp2 ]"
- -o - true if any of the expressions is true "[ exp1 -o exp2 ]"
Usage examples:
1.
# check if backup folder exists
if [ -d /backup ]; then
echo "Found /backup folder"
# check if our music-rsync script is present and executable
if [ -x /backup/music-rsync ]; then
echo "Found music-rsync script"
else
echo "music-rsync script not found, backup aborted"
fi
else
# create /backup folder
mkdir /backup
echo "/backup folder was created"
fi
2.
# check if process time is greater or equal to 1hour
if [ $PTIME -ge 3600 ]; then
# notify me if this is first occurence
if [ ! -f /home/lorand/.email_control ]; then
# control file is not present, we are sending email
send_email()
# create control file, we do not want to SPAM
touch /home/lorand/.email_control
fi
fi
case
case is helpful when more complex conditionals are needed and nested if conditionals becomes to complex. case conditional has the form:
case EXPRESSION in
CASE1)
COMMAND-LIST;;
CASE2)
COMMAND-LIST;;
...
CASEn)
COMMAND-LIST;;
esac
Usage example:
case "$1" in
start)
run_experiment
send_email("experiment started")
;;
stop)
send_email("stopping experiment")
stop_experiment
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
;;
esac
Loop
Redirections and pipe
Control operators
File system
Important directories typically present on a UNIX system:
- / - root directory of the entire filesystem
- /bin - contains the most essential UNIX commands
- /etc - configuration files
- /home - user files
- /lib - libraries
- /usr - user programs
- /usr/bin - non-essential command binaries
- /usr/include - standard location for C include files
- /usr/lib - standard libraries
- /usr/sbin - non-essential system binaries, daemons
- /sbin - essential system binaries, sysadmin commands
- /tmp - temporary files
- /var - administrative files, log files
- /dev - essential devices, i.e. /dev/null
- /proc - virtual filesystem providing process and kernel information
