Shell scripts: variables, conditions, loops and regular expressions

A shell script is a text file holding the commands you would otherwise type into the terminal. Bash reads it top to bottom and runs one line after the other. That is the whole idea – and it is enough to turn a task you repeat every day into one command that takes parameters, checks its input and can be reused by other scripts.

This session covers the language: variables, arguments, conditions, loops, functions, regular expressions and redirection. The next one, Shell scripting in practice, combines all of it into real tasks, and your shell-script assignment is built from exactly these pieces.

The one thing to remember: bash is a language of commands. A condition is a command whose exit code is 0 or not, a function returns a status and prints its result, and before any command runs, the shell rewrites the line you wrote. Most surprising behaviour follows from these three facts.

Your first script

#!/bin/bash
# A comment: everything after # is ignored
echo "Hello world"

#!/bin/bash

The shebang. Always the very first line: it tells the system which interpreter runs the file.

# …

A comment. The shell ignores everything after the # on that line.

echo

Prints its arguments to the standard output – the terminal, unless redirected.

A new file is not executable. Give it the x (execute) permission once, then run it. After chmod +x, ls -l hello.sh shows -rwxr-xr-x in the first column – the permission bits from Session 1.

chmod +x hello.sh     # once per file: give it the execute (x) permission
./hello.sh            # run it from the current directory

Three ways to run a script

./hello.sh            # needs x; new process, interpreter from the shebang
bash hello.sh         # no x needed; new process, bash chosen by you
source hello.sh       # no x needed; runs in YOUR shell (same as: . hello.sh)

The first two start a new process; source (or its short form, a single dot) runs the lines inside the shell you are typing into (Fig. 1). That difference shows up as soon as a script sets a variable or changes directory: in a child process the change disappears when the script ends, with source it stays.

Variables and quoting

name="student"           # no spaces around =, no $ on the left
echo "$name"             # $ to read the value
echo "${name}_backup"    # braces mark where the name ends
today=$(date +%F)        # command substitution: the output becomes the value
sum=$(( 3 + 4 ))         # arithmetic
unset name               # from here on, "$name" is an empty string

• —No spaces around =. name = "x" runs a command called name with two arguments.
• —No $ when assigning, $ when reading.
• —Names use letters, digits and _, and do not start with a digit.
• —An undefined variable is silently an empty string – no error. A typo in a name is a bug bash will not report.

What the shell does to a line before running it

This is the key to quoting. When you press Enter, the command does not get your text. The shell first substitutes variables, then command substitutions, then splits the line into words at spaces, then expands patterns like *.txt into matching file names (Fig. 2). Only the finished words reach the command.

echo "Hello $USER"       # Hello student
echo 'Hello $USER'       # Hello $USER
echo "Today: $(date +%F)"  # Today: YYYY-MM-DD

The rule that follows: almost always write variables in double quotes, "$file". Without them a file name with a space becomes two arguments, and an empty value disappears entirely – which turns [ $a == x ] into the syntax error [ == x ].

Environment variables

echo "User: $USER, home: $HOME"
echo "Dice: $(( RANDOM % 6 + 1 ))"
env                      # list every environment variable

Script arguments

Words after the script name arrive as numbered variables (Fig. 3). Quotes on the command line decide what counts as one argument – the same word splitting as before.

./args.sh "Ada Lovelace" 1815 London

echo "Script name: $0"
echo "First: $1"
echo "Count: $#"
for arg in "$@"; do      # "$@" keeps every argument as one word
    echo "Argument: $arg"
done

Script name: ./args.sh
First: Ada Lovelace
Count: 3
Argument: Ada Lovelace
Argument: 1815
Argument: London

Conditions: if and test

if [ "$n" -lt 10 ]; then
    echo "one digit"
elif [ "$n" -lt 100 ]; then
    echo "two digits"
else
    echo "three or more digits"
fi

The spaces inside [ … ] are not style. [ is a command – another name for test – and everything up to ] are its arguments. Like every command it finishes with an exit code, and that code is all if looks at (Fig. 4).

[ 5 -lt 10 ]; echo $?     # 0  -> true
[ 5 -gt 10 ]; echo $?     # 1  -> false
test 5 -lt 10; echo $?    # 0  -> [ is just another name for test

So any command can be a condition. grep -q prints nothing and only reports through its exit code – exactly what an if needs:

if grep -q "error" log.txt; then
    echo "The log contains an error"
fi

Operators

Combine conditions with && (and), || (or) and ! (not) inside [[ ]], for example [[ ! -f "$file" ]] or [[ $ext == "jpg" || $ext == "png" ]].

Which brackets?

if (( n % 2 == 0 )); then     # inside (( )) no $ is needed and < > work
    echo "$n is even"
fi

In a #!/bin/bash script, [[ ]] is the comfortable default. Learn [ ] too – most existing scripts use it.

Reading input

read -r -p "Enter username: " username
read -r -s -p "Enter password: " password   # -s: typing is not shown
echo                                        # -s also hides the Enter, so break the line

-r keeps backslashes as typed (use it every time), -p prints a prompt, -s hides what is typed. Arguments are better for anything another script might pass in; read is for interactive questions.

Working with strings

Bash can cut and change strings itself, without calling another program. All of these are forms of ${…}:

str="ELTE IK University"
echo "${#str}"          # 18                  length
echo "${str:5:2}"       # IK                  2 characters from index 5
echo "${str:8}"         # University          from index 8 to the end
echo "${str: -10}"      # University          last 10 (the space before - matters)
echo "${str/IK/TTK}"    # ELTE TTK University first match replaced
echo "${str//E/e}"      # eLTe IK University  every match replaced
echo "${str^^}"         # ELTE IK UNIVERSITY  upper case
echo "${str,,}"         # elte ik university  lower case

first="ELTE "
second="IK"
both="${first}${second}"   # ELTE IK
both+=" University"        # ELTE IK University

Braces matter when text follows the name: "$file_backup" reads a variable called file_backup, "${file}_backup" appends _backup to $file. The default-value form is the shortest way to make an argument optional:

n=${1:-10}        # the first argument, or 10 if there is none

Loops

count=1
while [ "$count" -le 5 ]; do
    echo "$count"
    ((count++))
done

count=1
until [ "$count" -gt 3 ]; do     # runs while the condition is FALSE
    echo "$count"
    ((count++))
done

for fruit in apple pear banana; do echo "$fruit"; done   # a list of words
for i in {1..5}; do echo "$i"; done                      # a range
for file in *.sh; do echo "Script: $file"; done          # files (globbing)

for (( i=10; i>0; i-- )); do
    echo -n "$i "      # -n: no newline
done
echo

for i in {1..10}; do
    if (( i == 3 )); then continue; fi   # skip the rest of this round
    if (( i == 6 )); then break; fi      # leave the loop
    echo -n "$i "
done                                     # prints: 1 2 4 5

One loop you will write in almost every task processes a file line by line. The redirection after done feeds the file into the loop, and each read takes the next line:

while read -r line; do
    echo "Line: $line"
done < numbers.txt

Session 4 takes this loop apart – splitting lines into fields, the last line without a newline, and why piping into it loses your variables.

Functions

rectangle_area() {
    local area=$(( $1 * $2 ))   # local: exists only inside the function
    echo "Area: $area"
}

rectangle_area 10 20            # call without parentheses: Area: 200

• —Define a function before the line that calls it.
• —Call it by name, without parentheses; arguments follow, separated by spaces.
• —Inside, the function reads its own arguments as $1, $2, $#, "$@" – the script's arguments are hidden while it runs.
• —Without local, every variable is global: the rest of the script sees it and can overwrite it.

The surprise is how a function gives something back. It has two separate channels (Fig. 6): what it prints, which the caller captures with $( ), and its exit status, a number from 0 to 255 that if can branch on.

square() {
    echo $(( $1 * $1 ))         # the RESULT goes to stdout
}
is_even() {
    (( $1 % 2 == 0 ))           # the STATUS: 0 if true, 1 if false
}

result=$(square 7)              # 49
if is_even 4; then echo "4 is even"; fi

Regular expressions

A regular expression describes the shape of text you are looking for. grep, sed, awk and bash's [[ … =~ … ]] all use them. Most characters match themselves; the special ones (. * [ ( ? + ^ $ \) need a backslash to be taken literally – \. is a real dot. Try patterns on regex101, which explains every piece.

Characters and classes

Inside [ ] most special characters lose their meaning: [.,] is a dot or a comma, no backslash needed. grep -E does not know \d from other languages – write [0-9].

Anchors and repetition

Groups and alternatives

Parentheses make a group, so a repetition applies to all of it: (ab)+ matches ab, abab. The bar chooses: (apple|pear|plum). A group also remembers what it matched, and sed can reuse it as \1, \2 – here to swap two words:

echo "apple pear" | sed -E 's/(\w+) (\w+)/\2 \1/'    # pear apple

Reading a real pattern

The example script line_counter.sh counts the lines of a file that hold a single number ending in 1, with an optional sign and an optional decimal part. Read the pattern left to right, one piece at a time (Fig. 7).

grep -cE '^[+-]?([0-9]+[.,])?[0-9]*1$' "$1"

grep options you will use

grep -E '^[0-9]+$' numbers.txt      # lines that are only digits
grep -cE '^[0-9]+$' numbers.txt     # how many such lines
grep -iv 'error' log.txt            # lines WITHOUT "error", any case

In bash's own test it is the other way round: the pattern after =~ must not be quoted, or bash compares it as plain text.

if [[ $times =~ ^[0-9]+$ ]]; then     # pattern NOT quoted
    echo "a non-negative integer"
fi

Standard streams and redirection

Every process starts with three open streams, each identified by a number, its file descriptor: 0 is stdin (input), 1 is stdout (normal output), 2 is stderr (error messages). On a terminal 1 and 2 both end up on the screen, so they look the same – until you redirect one of them (Fig. 8).

ls > out.txt              # stdout to a file (overwrite)
ls >> out.txt             # stdout appended
sort < names.txt          # stdin from a file
ls nope 2> err.txt        # stderr to a file
ls nope 2>/dev/null       # stderr thrown away
ls . nope > all.txt 2>&1  # both into the same file (order matters)

Why error messages go to stderr

echo "File not found: $1" >&2   # an error message: stderr, not stdout
exit 1                          # and a non-zero exit code

>&2 means "send this echo's stdout to where descriptor 2 points". The & is essential: >2 would create a file called 2. The payoff shows when someone saves your script's output:

$ ./line_counter.sh nope.txt > result.txt
File not found: nope.txt        <- on the screen; result.txt stays clean

The same holds for pipes and $( ): count=$(./line_counter.sh nope.txt) captures only stdout, so an error message never ends up in count looking like a result. Pipes themselves – cmd1 | cmd2, stdout of one into stdin of the next – are covered in Session 2.

Putting it together: a robust script

The example scripts of this session share a shape worth copying: check everything at the top, stop at the first problem with a message on stderr and a non-zero exit code, then do the work knowing the input is valid. Here is the head of line_writer.sh, which prints a text a given number of times:

if [ "$#" -ne 2 ]; then
    echo "Usage: $0 <number of times> <text>" >&2
    exit 1
fi
times=$1
text=$2
if [[ ! $times =~ ^[0-9]+$ ]]; then
    echo "Error: '$times' is not a non-negative integer" >&2
    exit 1
fi

dir_tester.sh does the same with input from read: it refuses an empty name, then creates the directory only if it does not exist yet (mkdir -p also creates missing parent directories).

read -r -p "Enter directory name: " dir
if [ -z "$dir" ]; then
    echo "Directory name cannot be empty" >&2
    exit 1
fi
if [ -d "$dir" ]; then echo "Directory exists"; else mkdir -p "$dir"; fi

Exit codes

Every command and script ends with an exit code: 0 for success, 1–255 for some kind of failure. Your scripts should follow the same convention, because other scripts – and if – rely on it.

./line_writer.sh 1 test; echo $?    # 0
./line_writer.sh;        echo $?    # 1

Common mistakes and debugging

When a script misbehaves

bash -n script.sh      # syntax check only, nothing runs
bash -x script.sh      # print every command, with the values filled in
set -x                 # inside a script: tracing on ...
set +x                 # ... and off again

bash -x is the quickest way to see what the shell really ran – every line after expansion, prefixed with +:

$ bash -x line_writer.sh 2 hi
+ '[' 2 -ne 2 ']'
+ times=2
+ text=hi
+ [[ 2 =~ ^[0-9]+$ ]]
+ (( i=0 ))
+ (( i<times ))
+ echo hi
hi

For a second opinion, paste the script into ShellCheck. It finds missing quotes, unused variables and most of the table above, and explains each warning.

Practice

Write each one as a script with a shebang, run it with good and bad input, and check $? after a failure. Try first, open the hint when stuck.

name=${1:-$USER}

$ ./calc.sh 7 3
Sum:        10
Difference: 4
Product:    21

if [ "$#" -ne 2 ]; then
    echo "Usage: $0 <number1> <number2>" >&2
    exit 1
fi
if [[ ! $1 =~ ^[+-]?[0-9]+$ || ! $2 =~ ^[+-]?[0-9]+$ ]]; then ...

if [ -f "$path" ]; then ...
elif [ -d "$path" ]; then ...
else ...

for (( i=2; i<=n; i+=2 )); do
    echo -n "$i "
done

factorial() {
    local result=1 i
    for (( i=2; i<=$1; i++ )); do result=$(( result * i )); done
    echo "$result"
}
result=$(factorial "$1")

12.1
-13.321
123
321
asd
something
11

grep -E '^-[0-9]+$' numbers.txt

-7          match
-42         match
-13.321     no – the dot is not a digit
-0.5        no
+5          no – the sign must be a minus
12          no

while true; do
    if ! read -r -p "Number (0 to finish): " number; then break; fi
    if [[ ! $number =~ ^[+-]?[0-9]+$ ]]; then echo "Not an integer" >&2; continue; fi
    (( number == 0 )) && break
    # ... add the number to the running sum
done

Summary

• —Start every script with #!/bin/bash, make it executable once with chmod +x, run it with ./. source runs it in your own shell instead of a child process.
• —Before a command runs, the shell substitutes variables and $( ), splits words and expands globs. Double-quote your variables.
• —Arguments are $1 … $9, ${10}, their count is $#, all of them "$@".
• —if runs a command and branches on its exit code; [ is the test command, so it needs spaces. 0 means true.
• —Functions print their result (capture it with $( )) and return only a status.
• —Quote regex patterns for grep in single quotes; leave the pattern after =~ unquoted.
• —Errors go to stderr with >&2, followed by exit 1. Results go to stdout.