Introduction to Unix CLI: Difference between revisions
| (3 intermediate revisions by the same user not shown) | |||
| Line 802: | Line 802: | ||
*'''/dev''' - essential devices, i.e. /dev/null | *'''/dev''' - essential devices, i.e. /dev/null | ||
*'''/proc''' - virtual filesystem providing process and kernel information | *'''/proc''' - virtual filesystem providing process and kernel information | ||
==Security structure== | |||
On UNIX, each file, directory or device has three permission groups. The first permission group describes owners’ permissions; the second one describes (UNIX) groups’ permissions; and the third group describes permissions related to any other user (not owner and not member of UNIX group). To each file, directory and device there are additional security bits, such as set-GID and sticky-bit attached. ''set-GID'' bit allows a newly created file to inherit the same group as directory has. The ''sticky-bit'' prevents any other user then owner or root from deleting a file within that directory. | |||
Example of file permissions: | |||
<pre> | |||
$ ls -ld /bin | |||
drwxr-xr-x 2 root root 4096 Jun 5 11:42 /bin | |||
$ ls -l /bin/ls | |||
-rwxr-xr-x 1 root root 106120 Apr 16 2015 /bin/ls | |||
</pre> | |||
==Ownerships and permissions== | |||
File, directory and device ownerships and permissions can be managed (amongst others) using ''chown'', ''chgrp'' and ''chmod'' commands. | |||
===chown=== | |||
''chown'' command changes owner (and user) of files and directories. | |||
Usage examples: | |||
#Change owner of file to "zsolt" user.<pre>$ chown zsolt /bin/ls</pre> | |||
#Change owner of file to "zsolt", group to "surf" and execute it recursively on the backup directory.<pre>$ chown zsolt:surf -R /backup</pre> | |||
===chgrp=== | |||
''chgrp'' command changes group owner of files and directories. | |||
Usage example: | |||
Change group of file to "wheel" group.<pre>$ chgrp wheel ~/bin/reset.py</pre> | |||
===chmod=== | |||
''chmod'' command changes the file mode bits. Mode bits can symbolic or numeric (octal number) representations. | |||
*read “r” or “4” | |||
*write “w” or “2” | |||
*execute “x” or “1” | |||
Usage examples: | |||
#Remove write permissions from "other" group, do it in symbolical mode.<pre>$ chmod o-w /usr/local/bin/transform.pl</pre> | |||
#Allow read, write, execute to owner; allow read, execute to group; deny all permissions to others; do it in octal mode.<pre>$ chmod 750 /usr/local/bin/transform.pl</pre> | |||
=Processes= | =Processes= | ||
A ''process'' is an instance of an executed program, containing the program code and it's current state. | |||
A ''process'' has: an image of the machine code to be executed; uses a region of (virtual) memory address; file descriptors allocated to the process; security attributes (owner and the process' set of permissions); and finally it's state. | |||
The memory region owned by the process contains the code to be executed, process specific data such as input and output. It also contains a call stack to keep track of active subroutines and a heap to hold intermediate computation data generated during run time. | |||
''Processes'' can have following states: | |||
*created | |||
*running | |||
*waiting | |||
*stopped | |||
*terminated | |||
There are parent and child ''processes'' and on a UNIX machine init is the parent ''process'' of the whole system. Each process has an ID number and assigned priority. | |||
Useful commands to monitor and manage ''processes'' are ''ps'', ''top'' and ''kill''. | |||
==ps== | |||
''ps'' displays information about a selection of the active processes. | |||
Usage examples: | |||
#Show all processes owned by user zsolt.<pre>$ ps -au zsolt</pre> | |||
#Show all processes in full format owned by user zsolt.<pre>$ ps -fau zsolt</pre> | |||
#Show all (every) running processes.<pre>$ ps -ef</pre> | |||
Example of output: | |||
<pre> | |||
$ ps aux | |||
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND | |||
root 1 0.0 0.0 10560 668 ? Ss Oct21 0:47 init [3] | |||
root 2 0.0 0.0 0 0 ? S Oct21 0:10 [kthreadd] | |||
root 3 0.0 0.0 0 0 ? S Oct21 2:20 [ksoftirqd/0] | |||
root 5 0.0 0.0 0 0 ? S Oct21 0:00 [kworker/u:0] | |||
root 6 0.0 0.0 0 0 ? S Oct21 1:12 [migration/0] | |||
root 7 0.0 0.0 0 0 ? S Oct21 0:59 [migration/1] | |||
root 8 0.0 0.0 0 0 ? S Oct21 0:00 [kworker/1:0] | |||
root 9 0.0 0.0 0 0 ? S Oct21 2:13 [ksoftirqd/1] | |||
root 10 0.0 0.0 0 0 ? S Oct21 10:10 [kworker/0:1] | |||
… | |||
# USER - user owning the process | |||
# PID - process ID | |||
# CPU - CPU usage | |||
# MEM - memory usage | |||
# STAT - process status | |||
# START - process was started on | |||
# TIME - effective CPU time used by the process | |||
</pre> | |||
Process state codes: | |||
*D - Uninterruptible sleep (usually IO) | |||
*R - Running or runnable | |||
*S - Interruptible sleep, waiting for an event to complete | |||
*T - Stopped | |||
*Z - Defunct ("zombie") process | |||
==top== | |||
''top'' command displays a dynamic and real-time view of the running system. It shows processes and their status, CPU load, memory usage, etc. | |||
Order, size of information displayed for tasks are all user configurable and can be made persistent across restarts. | |||
Pressing "q" while ''top'' is running will exit the program.<br> | |||
"?" shows a short help and again "q" will quit the short help and return to the program.<br> | |||
"<" and ">" changes sort field.<br> | |||
"W" writes current view settings to startup file. | |||
<pre>$ top | |||
asks: 404 total, 1 running, 402 sleeping, 1 stopped, 0 zombie | |||
Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st | |||
Mem: 16146M total, 15020M used, 1126M free, 0M buffers | |||
Swap: 0M total, 0M used, 0M free, 10180M cached | |||
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ PPID COMMAND | |||
1 root 20 0 10560 668 532 S 0 0.0 0:47.10 0 init | |||
2 root 20 0 0 0 0 S 0 0.0 0:10.01 0 kthreadd | |||
3 root 20 0 0 0 0 S 0 0.0 2:20.57 2 ksoftirqd/0 | |||
5 root 20 0 0 0 0 S 0 0.0 0:00.02 2 kworker/u:0 | |||
6 root RT 0 0 0 0 S 0 0.0 1:12.76 2 migration/0 | |||
7 root RT 0 0 0 0 S 0 0.0 0:59.45 2 migration/1 | |||
8 root 20 0 0 0 0 S 0 0.0 0:00.00 2 kworker/1:0 | |||
9 root 20 0 0 0 0 S 0 0.0 2:13.76 2 ksoftirqd/1 | |||
10 root 20 0 0 0 0 S 0 0.0 10:10.87 2 kworker/0:1 | |||
11 root RT 0 0 0 0 S 0 0.0 0:59.99 2 migration/2 | |||
12 root 20 0 0 0 0 S 0 0.0 0:00.00 2 kworker/2:0 | |||
13 root 20 0 0 0 0 S 0 0.0 2:27.12 2 ksoftirqd/2 | |||
14 root RT 0 0 0 0 S 0 0.0 0:55.41 2 migration/3 | |||
</pre> | |||
==kill== | |||
''kill'' command sends signal to running processes. | |||
'''Note''': Please pay attention to PID (process ID) number, because you may stop another process then wanted. | |||
Usage examples: | |||
#Send terminate signal to process with PID 20011, allow process to clean-up. Terminate signal can be caught and interpreted or ignored by the process.<pre>$ kill 20011</pre> | |||
#Send terminate signal to process with PID 20011. Signal can not be caught or ignored.<pre>$ kill -9 20011</pre> | |||
[[Category:Hexagon]] | [[Category:Hexagon]] | ||
[[Category:Fimm]] | [[Category:Fimm]] | ||
Latest revision as of 18:29, 8 November 2015
What is Unix CLI?
Unix CLI is actually the shell, a text interface to the operating system. Shell interprets and executes your commands and their arguments by passing them to the operating system.
There are different shell variants, like bash, csh, ksh, sh, tcsh, etc., probably the most popular being bash. The name “bash” is an acronym for “Bourne Again SHell” and is an enhanced replacement for sh. In this short introductory we are going to cover bash.
Bash
Bash is an interactive and scriptable shell. Supports name expansion, wildcards and typing shortcuts.
Note: Interaction with bash is case-sensitive just like everything in Unix system.
Some useful shortcuts:
TAB, UP, DOWN, Ctrl+R, Esc+., Ctrl+C,!<command>
Popular commands
man
man is the system's manual pager. It is an interface to on-line manuals. man has it's own manual pager.
$ # Show default man page for top command $ man sleep $ # Show all manual pages for top command, successively $ man -a sleep
Whenever you are uncertain what a command does, what options and argument can be passed to, man is your friend.
apropos
apropos searches the manual page names and descriptions for the specified keyword.
$ apropos shell bash (1) - GNU Bourne-Again SHell bash [sh] (1) - GNU Bourne-Again SHell capsh (1) - capability shell wrapper chroot (1) - run command or interactive shell with special root directory chsh (1) - change your login shell …
pwd
pwd prints the full name of the current working directory.
$ pwd /tmp/test/course2015
cd
cd changes the current working directory to the specified directory name. If no directory is specified than cd changes current working directory to users home directory. cd supports relative and absolute paths. Additionally shortcuts like ~ (points to user's home directory).
$ pwd /tmp/test/course2015 $ cd bin $ pwd /tmp/test/course2015/bin $ cd ../../src $ pwd /tmp/test/src $ cd ~/bin $ pwd /home/lorand/bin $ # Change back to previous directory $ cd -
mkdir
mkdir command creates directories if they do not exist.
$ mkdir build
To create a whole directory structure, use "-p" option.
$ mkdir -p ~/build/wrf/3.7
ls
ls command lists content of the current or the specified directory. One of the most used options for ls is "-l", which will generate a long list format.
$ ls GConf libegroupwise-1.2.so.13 libospf.so.0.0.0 ImageMagick-6.4.3 libegroupwise-1.2.so.13.0.1 libossp-uuid.so.16 Mcrt1.o libelf-0.152.so libossp-uuid.so.16.0.22 PackageKitDbusTest.py libelf.so.0 libp11.so.1 … $ ls -l total 806992 drwxr-xr-x 3 root root 4096 Mar 11 2012 GConf drwxr-xr-x 4 root root 4096 Nov 18 2014 ImageMagick-6.4.3 -rw-r--r-- 1 root root 1358 Oct 8 20:07 Mcrt1.o -rw-r--r-- 1 root root 3383 May 29 2013 PackageKitDbusTest.py -rw-r--r-- 1 root root 4267 Oct 8 20:07 Scrt1.o …
"-a" will list all files and folders, even hidden ones.
Note: Files, directories starting with "." character are treated as hidden files.
$ pwd /work/shared/course2015/intro/bash $ ls -l total 0 $ ls -la total 33 drwxr-xr-x 2 lorand root 24576 Oct 27 10:10 . drwxr-xr-x 3 lorand root 4096 Oct 27 10:07 .. -rw-r--r-- 1 lorand root 1177 Oct 27 10:10 .bashrc -rw-r--r-- 1 lorand root 849 Oct 27 10:10 .vimrc
Filename expansion is also supported with the usage of special characters, so called wildcards "?" or "*". "?" replaces one character, while "*" replaces any number of characters.
$ ls -a .*rc .bashrc .vimrc
"-t" option sorts listing by modification time, while "-r" can reverse sorting order.
$ ls -lt total 8 drwxr-xr-x 3 lorand root 4096 Oct 27 10:07 intro drwxr-xr-x 4 floan root 4096 Oct 21 12:10 openMP $ ls -ltr total 8 drwxr-xr-x 4 floan root 4096 Oct 21 12:10 openMP drwxr-xr-x 3 lorand root 4096 Oct 27 10:07 intro
cp
cp command copies files and directories. Copies source (might be one or more files/directories) to destination. When multiple files/directories are specified as source, the destination has to be a directory.
$ # copy messages file to /backup folder $ cp /var/log/messages /backup $ # copy messages and maillog files to /backup folder $ cp /var/log/messages /var/log/maillog /backup $ # copy all C source to network mounted home folder $ # preserving timestamps and permissions, be verbose $ cp -a -v ~/development/my_project/*.c /mnt/nfs/projects/sources/ $ # copy all new or modified files to destination $ cp -u -v ~/development/my_project/*.c /mnt/nfs/projects/sources/ $ # copy all new or modified files to destination $ # be verbose and interactive - ask what to do $ cp -u -i -v ~/development/my_project/*.c /mnt/nfs/projects/sources/
mv
mv moves source (might be one or more files/directories) to destination. Same as for cp command, when multiple files/directories are specified as source, the destination has to be a directory. mv command is also used for renaming a file. NB: Irreversible!
$ # interactively move python files to python_files/ subdirectory $ mv -i *.py python_files/ $ # move python files and force overwrite if similar file exists $ # be verbose $ mv -f -v *.py python_files/
rm
rm command removes files and directories. rm will not remove directories by default unless recursive - -r - option is specified. NB: Irreversible! It is advisable to create an alias for rm command to "rm -i" (see alias command).
$ # interactively remove all object files in current directory $ rm -i ./*.o $ # force removal of ~/tmp/ directory and all of it's contents $ rm -rf ~/tmp/
find
find searches for files in a directory hierarchy. find command evaluates the given options from left to right, thus adding options in correct order is important. For example -maxdepth option must be appended before -name option, otherwise it will be ignored.
Find all the files whose name is hello.txt in a current working directory.
$ find . -name hello.txt ./hello.txt
Find all the files under /home directory with name hello.txt.
$ find /home -name hello.txt /home/hello.txt
Find all the files whose name is hello.txt and contains both capital and small letters in /home directory.
$ find /home -iname hello.txt ./hello.txt ./Hello.txt
Find all directories whose name is Hello in current working directory
$ find . -type d -name Hello ./Hello
To find a single file called hello.txt and remove it in current directory.
$ find . -type f -name "hello.txt" -exec rm -f {} \;
scp
scp command securely copies between hosts on the network. It uses ssh, thus providing same authentication and security mechanism. File names can be relative or absolute and may contain user and host specification. Some useful options are:
- -P - port number to connect to on remote host
- -p - preserve modification times, access times, and modes from the original file
- -r - copy recursively
Recursively copy examples folder to my home on hexagon.
scp -r ~/examples/ lorand@hexagon.hpc.uib.no:
Copy python files from hexagon, preserving modification and access times to /scratch/development.
scp -p lorand@hexagon.hpc.uib.no:~/development/my_project/*.py /scratch/development
rsync
rsync is a file copying tool. It can copy locally and to/from another host. It has a delta-transfer algorithm, reducing network traffic by sending only the differences between source and destination. By default it uses ssh but it can be configured to use different remote shells.
Useful options for rsync:
- -v - increase verbosity
- -a - archive mode
- -u - update, exclude newer files on the destination
- -l - copy symlinks
- -H - copy hard links
- -p - preserve permissions
- -n - dry-run, no changes are made
- -z - compress
Transfer all files matching the pattern *.c from the current directory to your src directory on hexagon.
$ rsync -t *.c hexagon:~/src/
Recursively transfer all files from the WRF/output directory to your local machine. The files are transferred in "archive" mode, ensuring that symbolic links, permissions, ownerships are preserved. Be verbose and use compression. Note: please pay attention to trailing "/" in the examples below.
$ # create output directory if not exist $ rsync -avz lorand@hexagon:~/WRF/output /data/WRF $ # sync only files and do not create output directory $ rsync -avz lorand@hexagon:~/WRF/output/ /data/WRF
Recursively transfer my project folder to the backup. Exclude vim backup files and skip newer files in the local folder.
$ rsync -avuz --exclude '*~' hexagon:project/ /backup/hexagon
cat
cat command concatenates files and prints on the standard output.
$ # concatenate motd $ cat /etc/motd Rocks Compute Node Rocks 6.1 (Emerald Boa) Profile built 16:14 10-Oct-2015
less
less allows paging trough text, one screenful at a time. Allows forward and backward movements in the text, line by line, or page by page. Useful commands in less are:
- "/" searches forward
- "?" searches backward
- "-I" switching between case-sensitive/case-insensitive searches
- "q" quits less
grep
grep and egrep prints lines matching a pattern. egrep is the same as grep -E.
Print lines containing "student".
$ grep "student" applicants.txt student Don Joe student Navn Navnesen
Print lines containing "student", but be case-insensitive.
$ grep -i "student" applicants.txt Student Neve Sincs student John Doe STudent Anonymous student Navn Navnesen StudEnT Hans Hansen
Print lines not containing "Navnesen" and "Hansen".
$ grep -v -E "Navnesen| Hansen" applicants.txt Student Neve Sincs student John Doe STudent Anonymous
List files, without showing their content, containing "alias".
$ grep -Rl mpif90 . ./efg-3.3.1-20130426-cray/external/io_mcel/configure.wrf.example ./efg-3.3.1-20130426-cray/arch/archive_configure.defaults ./efg-3.3.1-20130426-cray/arch/configure_old.defaults
tail
tail outputs the last part (by default last 10 lines) of the file.
$ # show last 25 lines from messages file $ tail -25 /var/log/messages $ # follow file output, output will continue until it is $ # broke with Ctrl+C $ tail -f /var/log/messages
head
head command prints the first 10 lines of each file. For more than one file given, head will list the first 10 lines for each file, preceding with the file name. -n option overrides the default the number of lines.
$ head -1 /etc/motd Welcome to hexagon.hpc.uib.no, a Cray XE6m-200 system operated by UiB. $ ps x | head -2 PID TTY STAT TIME COMMAND 14065 ? S 0:00 sshd: lorand@pts/9
awk
awk is a pattern scanning and processing language. Assuming the following example file with tab separated fields.
$ cat bergen-weather-statistics.txt jan 3.4° 516.1 mm 21.8 m/s feb 3.4° 246.4 mm 17.5 m/s mar 5.2° 349.0 mm 17.7 m/s apr 6.2° 140.9 mm 11.2 m/s mai 8.7° 240.5 mm 13.4 m/s jun 11.4° 129.8 mm 12.9 m/s jul 14.4° 108.2 mm 10.9 m/s aug 16.5° 192.4 mm 12.3 m/s sep 13.5° 121.9 mm 12.3 m/s okt 9.6° 167.1 mm 12.1 m/s
Print the monthly precipitation and add header.
$ awk 'BEGIN {print "MONTH\tPRECIPITATION\n---------------------";} \
{print $1, "\t", $3;}' bergen-weather-statistics.txt
MONTH PRECIPITATION
---------------------
jan 516.1
feb 246.4
mar 349.0
apr 140.9
mai 240.5
jun 129.8
jul 108.2
aug 192.4
sep 121.9
okt 167.1
Additionally to the above example, write a footer and append the total sum of monthly precipitation.
$ awk 'BEGIN {print "MONTH\tPRECIPITATION\n---------------------";} \
{print $1, "\t", $3;} {SUM+=$3;} END {print "---------------------\nTOTAL\t",SUM,"mm";}' \
bergen-weather-statistics.txt
MONTH PRECIPITATION
---------------------
jan 516.1
feb 246.4
mar 349.0
apr 140.9
mai 240.5
jun 129.8
jul 108.2
aug 192.4
sep 121.9
okt 167.1
---------------------
TOTAL 2212.3 mm
List maximum wind speed for each month and calculate average wind speed.
$ awk 'BEGIN {print "MONTH\tMAX. WINDSPEED\n---------------------";} {print $1, "\t", $5;}\
{SUM+=$5;} END {print "---------------------\nAVG.\t", SUM/NR,"m/s";}' bergen-weather-statistics.txt
MONTH MAX.WINDSPEED
---------------------
jan 21.8
feb 17.5
mar 17.7
apr 11.2
mai 13.4
jun 12.9
jul 10.9
aug 12.3
sep 12.3
okt 12.1
---------------------
AVG. 14.21 m/s
sed
sed is a stream editor for filtering and transforming text. It is used to perform basic text transformations on an input, which can be either text or pipe.
sed syntax is sed 'PATTERN/REGEX/REPLACEMENT/FLAGS' file_name. Some common usage examples are listed below.
Read applicants.txt file and replace "student" with "employee".
$ cat applicants.txt Student Neve Sincs student John Doe STudent Anonymous student Navn Navnesen StudEnT Hans Hansen $ sed 's/student/employee/' < applicants.txt Student Neve Sincs employee John Doe STudent Anonymous employee Navn Navnesen StudEnT Hans Hansen
Same as above but be case-insensitive.
$ sed 's/student/employee/i' < applicants.txt employee Neve Sincs employee John Doe employee Anonymous employee Navn Navnesen employee Hans Hansen
In the above examples the edited text was written to stdout. Let's make the changes in-place.
$ cat applicants.txt Student Neve Sincs student John Doe STudent Anonymous student Navn Navnesen StudEnT Hans Hansen $ sed 's/student/employee/i' -i applicants.txt $ cat applicants.txt employee Neve Sincs employee John Doe employee Anonymous employee Navn Navnesen employee Hans Hansen
Replace "Navn" with "Name", then do it globally.
$ sed 's/Navn/Name/' < applicants.txt employee Neve Sincs employee John Doe employee Anonymous employee Name Navnesen employee Hans Hansen $ sed 's/Navn/Name/g' < applicants.txt employee Neve Sincs employee John Doe employee Anonymous employee Name Nameesen employee Hans Hansen
Eliminate comments.
$ cat applicants.txt #Student Neve Sincs student John Doe #STudent Anonymous student Navn Navnesen StudEnT Hans Hansen $ sed -e 's/#.*//' applicants.txt student John Doe student Navn Navnesen StudEnT Hans Hansen
Eliminate empty lines.
$ cat applicants.txt #Student Neve Sincs student John Doe #STudent Anonymous student Navn Navnesen StudEnT Hans Hansen $ sed -e '/^$/d' applicants.txt #Student Neve Sincs student John Doe #STudent Anonymous student Navn Navnesen StudEnT Hans Hansen
Eliminate both comments and empty lines.
$ sed -e 's/#.*//;/^$/d' applicants.txt student John Doe student Navn Navnesen StudEnT Hans Hansen
alias
alias command defines or lists aliases. Command itself without any arguments will list existing aliases.
$ lc -bash: lc: command not found $ alias lc='ls -l --color' $ lc -rwxr-xr-x 1 root root 5008747 Oct 13 13:35 helloworld_c -rw------- 1 lorand root 0 Oct 13 13:36 helloworld_c.err -rw------- 1 lorand root 2068 Oct 13 13:38 helloworld_c.out $ alias alias ..='cd ..' alias ...='cd ../..' alias build='cd /work/$USER/build' alias cd..='cd ..' alias dir='ls -l' alias lc='ls -l --color'
screen
screen is a full screen window manager, which allows you to create a window with a shell within it. It allows you to run your program even after logging out from the server or in case of network connection interruption. You will be able to pick up your work from where you left it.
Commands within the screen window consist of Ctrl+A plus one more character. Character D detaches from current screen. ? shows help.
Create a window named "test" and enable logging.
$ screen -S test -L
List existing screens.
$ screen -ls There are screens on: 31400.test2 (Detached) 31264.test1 (Detached) 2 Sockets in /var/run/uscreens/S-lorand.
Reattach to screen "test2".
$ screen -r test2
Attach to a non-detached screen (multi-display mode).
$ screen -x test1
Variables
Variables are referenced by $ sign. There are some special variables:
- $0 - name of the script
- $1, $2, ... , ${10}, ${11}, ... - are positional arguments
- $* - all arguments
- $# - number of arguments
User variables
User variables are defined by user and can not be the type of special variables or environment variables, however they can be overwritten.
$ myvar="Hello World" $ echo $myvar Hello World
Environment variables
Environment variables are initialized by system-wide configuration files. They can be appended or overwritten by user configuration files. Some of the most common environment variables are:
- PATH
- USER
- HOME
- LD_LIBRARY_PATH
- MANPATH
- CFLAGS
- LDFLAGS
List my current environment.
$ env
Adjust PATH to include local "bin" directory. Note: items in the PATH environment variable are ":" separated and processed from left to right.
$ echo $PATH /bin:/sbin:/usr/bin:/usr/sbin $ export PATH=~/bin:$PATH $ echo $PATH /home/lorand/bin:/bin:/sbin:/usr/bin:/usr/sbin
Remove CFLAGS variable.
$ echo $CFLAGS -march=native -O2 -pipe $unset CFLAGS $ echo $CFLAGS
Conditionals
if
In bash we have the if and the case conditionals.
Expressions used with if:
- -f - true if file exists and is a regular file
- -d - true if file exists and is a directory
- -s - true if file exists and it's size is greater then 0
- -x - true if file exists and is executable
- -z - true if the length of string is zero
- -nt- true if file1 is newer than file2 "[ file1 -nt file2 ]"
- -ot- true if file1 is older than file2 "[ file1 -ot file2 ]"
- == - true if the strings are equal
- != - true if the strings are not equal
Arithmetic expressions used with if:
- -eq - it equal to
- -ne - not equal to
- -lt - less then
- -le - less then or equal to
- -gt - greater then
- -ge - greater then or equal to
Note: arithmetic expressions have the form "[ arg1 OPERATOR arg2 ]", where "arg1" and "arg2" are integers.
Expressions in if conditional can be combined:
- ! - true if expression is false "[ ! exp ]"
- -a - true if both expressions are true "[ exp1 -a exp2 ]"
- -o - true if any of the expressions is true "[ exp1 -o exp2 ]"
Usage examples:
1.
# check if backup folder exists
if [ -d /backup ]; then
echo "Found /backup folder"
# check if our music-rsync script is present and executable
if [ -x /backup/music-rsync ]; then
echo "Found music-rsync script"
else
echo "music-rsync script not found, backup aborted"
fi
else
# create /backup folder
mkdir /backup
echo "/backup folder was created"
fi
2.
# check if process time is greater or equal to 1hour
if [ $PTIME -ge 3600 ]; then
# notify me if this is first occurence
if [ ! -f /home/lorand/.email_control ]; then
# control file is not present, we are sending email
send_email()
# create control file, we do not want to SPAM
touch /home/lorand/.email_control
fi
fi
case
case is helpful when more complex conditionals are needed and nested if conditionals becomes to complex. case conditional has the form:
case EXPRESSION in
CASE1)
COMMAND-LIST;;
CASE2)
COMMAND-LIST;;
...
CASEn)
COMMAND-LIST;;
esac
Usage example:
case "$1" in
start)
run_experiment
send_email("experiment started")
;;
stop)
send_email("stopping experiment")
stop_experiment
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
;;
esac
Loop
for
for is a looping command with similar syntax to C for statement.
for usage examples:
for NR in 1 2 3 4 5
do
echo "Hello student $NR"
done
for FILE in $(ls)
do
echo $FILE
done
while
while command will repeatedly execute commands as long as the condition is met. while usage example.
#!/bin/bash c=1 while [ $c -le 5 ] do echo "Welcone $c times" (( c++ )) done
Another example for while: script will repeat until it gets empty string.
#!/bin/csh set yname="foo" while ( $yname != "" ) echo -n "Enter your name : " set yname = $< if ( $yname != "" ) then echo "Hi, $yname" endif end
until
until loop example: script will print integers from 0 to 9 then quit.
#!/bin/sh a=0 until [ ! $a -lt 10 ] do echo $a a=`expr $a + 1` done
Redirections and pipe
- stdin - standard input
- stdout - standard output
- stderr - standard error
- > - redirect stdout (to a file)
- >> - redirect stdout to a file, w/o overwriting it (append)
- < - stdin (from file)
- 2> - redirect stderr
- 2>&1 - redirect both stdout and stderr
- | - pipe commands
Examples:
- Save a list of files in "/usr/local/bin" to "/tmp/executables".
$ ls /usr/local/bin > /tmp/executables
- Append the list of files in "/usr/local/sbin" to "/tmp/executables", without overwriting previous list.
$ ls /usr/local/sbin >> /tmp/executables
- Save stdout and stderr from our "make" command to "compile-TODAY.log" file, where TODAY is generated too.
$ make 2>&1 compile-$(date +%F).log
Control operators
- - separate sequence of commands
- & - execute command in the background
- && - AND operator, return from all commands must be true
- || - OR operator, any command returning true will result in true
Examples:
- Run "make" then "make install", type both commands on one line.
$ make; make install
- Run "make" and if make executes successfully then "make install". This command will make sure that "make" did completed successfully.
$ make && make install
- Start scp in the background so we can continue to work.
$ scp -p -r myfiles/ mylocal_pc:/backup &
File system
Important directories typically present on a UNIX system:
- / - root directory of the entire filesystem
- /bin - contains the most essential UNIX commands
- /etc - configuration files
- /home - user files
- /lib - libraries
- /usr - user programs
- /usr/bin - non-essential command binaries
- /usr/include - standard location for C include files
- /usr/lib - standard libraries
- /usr/sbin - non-essential system binaries, daemons
- /sbin - essential system binaries, sysadmin commands
- /tmp - temporary files
- /var - administrative files, log files
- /dev - essential devices, i.e. /dev/null
- /proc - virtual filesystem providing process and kernel information
Security structure
On UNIX, each file, directory or device has three permission groups. The first permission group describes owners’ permissions; the second one describes (UNIX) groups’ permissions; and the third group describes permissions related to any other user (not owner and not member of UNIX group). To each file, directory and device there are additional security bits, such as set-GID and sticky-bit attached. set-GID bit allows a newly created file to inherit the same group as directory has. The sticky-bit prevents any other user then owner or root from deleting a file within that directory.
Example of file permissions:
$ ls -ld /bin drwxr-xr-x 2 root root 4096 Jun 5 11:42 /bin $ ls -l /bin/ls -rwxr-xr-x 1 root root 106120 Apr 16 2015 /bin/ls
Ownerships and permissions
File, directory and device ownerships and permissions can be managed (amongst others) using chown, chgrp and chmod commands.
chown
chown command changes owner (and user) of files and directories.
Usage examples:
- Change owner of file to "zsolt" user.
$ chown zsolt /bin/ls
- Change owner of file to "zsolt", group to "surf" and execute it recursively on the backup directory.
$ chown zsolt:surf -R /backup
chgrp
chgrp command changes group owner of files and directories.
Usage example:
Change group of file to "wheel" group.
$ chgrp wheel ~/bin/reset.py
chmod
chmod command changes the file mode bits. Mode bits can symbolic or numeric (octal number) representations.
- read “r” or “4”
- write “w” or “2”
- execute “x” or “1”
Usage examples:
- Remove write permissions from "other" group, do it in symbolical mode.
$ chmod o-w /usr/local/bin/transform.pl
- Allow read, write, execute to owner; allow read, execute to group; deny all permissions to others; do it in octal mode.
$ chmod 750 /usr/local/bin/transform.pl
Processes
A process is an instance of an executed program, containing the program code and it's current state.
A process has: an image of the machine code to be executed; uses a region of (virtual) memory address; file descriptors allocated to the process; security attributes (owner and the process' set of permissions); and finally it's state. The memory region owned by the process contains the code to be executed, process specific data such as input and output. It also contains a call stack to keep track of active subroutines and a heap to hold intermediate computation data generated during run time.
Processes can have following states:
- created
- running
- waiting
- stopped
- terminated
There are parent and child processes and on a UNIX machine init is the parent process of the whole system. Each process has an ID number and assigned priority.
Useful commands to monitor and manage processes are ps, top and kill.
ps
ps displays information about a selection of the active processes.
Usage examples:
- Show all processes owned by user zsolt.
$ ps -au zsolt
- Show all processes in full format owned by user zsolt.
$ ps -fau zsolt
- Show all (every) running processes.
$ ps -ef
Example of output:
$ ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 10560 668 ? Ss Oct21 0:47 init [3] root 2 0.0 0.0 0 0 ? S Oct21 0:10 [kthreadd] root 3 0.0 0.0 0 0 ? S Oct21 2:20 [ksoftirqd/0] root 5 0.0 0.0 0 0 ? S Oct21 0:00 [kworker/u:0] root 6 0.0 0.0 0 0 ? S Oct21 1:12 [migration/0] root 7 0.0 0.0 0 0 ? S Oct21 0:59 [migration/1] root 8 0.0 0.0 0 0 ? S Oct21 0:00 [kworker/1:0] root 9 0.0 0.0 0 0 ? S Oct21 2:13 [ksoftirqd/1] root 10 0.0 0.0 0 0 ? S Oct21 10:10 [kworker/0:1] … # USER - user owning the process # PID - process ID # CPU - CPU usage # MEM - memory usage # STAT - process status # START - process was started on # TIME - effective CPU time used by the process
Process state codes:
- D - Uninterruptible sleep (usually IO)
- R - Running or runnable
- S - Interruptible sleep, waiting for an event to complete
- T - Stopped
- Z - Defunct ("zombie") process
top
top command displays a dynamic and real-time view of the running system. It shows processes and their status, CPU load, memory usage, etc. Order, size of information displayed for tasks are all user configurable and can be made persistent across restarts.
Pressing "q" while top is running will exit the program.
"?" shows a short help and again "q" will quit the short help and return to the program.
"<" and ">" changes sort field.
"W" writes current view settings to startup file.
$ top
asks: 404 total, 1 running, 402 sleeping, 1 stopped, 0 zombie
Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 16146M total, 15020M used, 1126M free, 0M buffers
Swap: 0M total, 0M used, 0M free, 10180M cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ PPID COMMAND
1 root 20 0 10560 668 532 S 0 0.0 0:47.10 0 init
2 root 20 0 0 0 0 S 0 0.0 0:10.01 0 kthreadd
3 root 20 0 0 0 0 S 0 0.0 2:20.57 2 ksoftirqd/0
5 root 20 0 0 0 0 S 0 0.0 0:00.02 2 kworker/u:0
6 root RT 0 0 0 0 S 0 0.0 1:12.76 2 migration/0
7 root RT 0 0 0 0 S 0 0.0 0:59.45 2 migration/1
8 root 20 0 0 0 0 S 0 0.0 0:00.00 2 kworker/1:0
9 root 20 0 0 0 0 S 0 0.0 2:13.76 2 ksoftirqd/1
10 root 20 0 0 0 0 S 0 0.0 10:10.87 2 kworker/0:1
11 root RT 0 0 0 0 S 0 0.0 0:59.99 2 migration/2
12 root 20 0 0 0 0 S 0 0.0 0:00.00 2 kworker/2:0
13 root 20 0 0 0 0 S 0 0.0 2:27.12 2 ksoftirqd/2
14 root RT 0 0 0 0 S 0 0.0 0:55.41 2 migration/3
kill
kill command sends signal to running processes.
Note: Please pay attention to PID (process ID) number, because you may stop another process then wanted.
Usage examples:
- Send terminate signal to process with PID 20011, allow process to clean-up. Terminate signal can be caught and interpreted or ignored by the process.
$ kill 20011
- Send terminate signal to process with PID 20011. Signal can not be caught or ignored.
$ kill -9 20011