Shell scripting in practice: input, errors, case and pipelines

In the previous session you met the pieces of a shell script one by one: variables, arguments, if, loops, functions, regular expressions and redirection. This session puts them together. The tasks build on each other – a file copy with error handling, a word search, a pipeline that finds the most common value in a column, and finally an interactive menu that uses everything at once.

Before the tasks come a few new tools you need for them: read from places other than the keyboard, the case statement, the filters cut, sort and uniq, and the habits that make a script robust – checking the input first, writing errors to stderr, and exiting with a code that says what happened.

The pattern behind every task: check the input at the top and stop with a message on stderr if anything is wrong; then do the work, ideally as a pipeline of small filters you built and tested one stage at a time in the terminal.

Getting input: arguments and read

A script gets data from the user in one of two ways (Fig. 1). Arguments are written after the script name when it starts – ./search.sh menu.txt wine – and are fixed from then on. read takes one line while the script is running. Use arguments for anything another script or a test might pass in (file names, options, values); use read for genuinely interactive programs – a menu, a question, a password.

Copy the arguments into named variables at the top of the script. $2 means nothing to a reader three screens later; $word does. ${3:-01} gives a default when the argument is missing, and "$@" walks all arguments, each as one word even if it contains spaces.

file=$1
word=$2
month=${3:-01}       # default value when there is no third argument

for arg in "$@"; do  # every argument, each as its own word
    echo "Argument: $arg"
done

read doesn't care where its line comes from. Always give it -r (keep backslashes as they are); -p prints a prompt. If it gets several variable names, it splits the line at whitespace: the first word goes into the first variable, and the rest of the line into the last one.

# from the keyboard
read -r -p "Enter a word: " word

# from a file, line by line
while read -r line; do
    echo "Line: $line"
done < menu.txt

# from another command's output, line by line
find main_folder -type f | while read -r file; do
    echo "File: $file"
done

# from a variable (here string): first word → count, the rest → illness
result="      3 headache"
read -r count illness <<< "$result"

The <<< operator is the here string: it hands one string to a command's standard input as if it were a one-line file. It belongs to a small family of input redirections:

read -r first < file.txt                  # <    from a file
cat << EOF                                # <<   from a block written in the script
line one
line two
EOF
read -r count illness <<< "$result"       # <<<  from one string (here string)

The pipe-and-read trap

The obvious way to split a variable is to pipe it into read: echo "$result" | read -r count illness. It runs without an error – and afterwards both variables are empty. The reason is how pipes work: bash starts each side of a pipe as a separate process. The read on the right runs in a child shell (a subshell), sets its variables, and exits. The variables die with it (Fig. 2). The process tree from Session 2 explains why: a child process can't change its parent's variables.

result="      3 headache"

echo "$result" | read -r count illness
echo "$count / $illness"           #  /           (empty – read ran in a subshell)

read -r count illness <<< "$result"
echo "$count / $illness"           # 3 / headache

The same trap catches a while read loop on the right side of a pipe. Printing inside the loop works fine – but a counter or flag you change in the loop is gone after done:

n=0
find main_folder -type f | while read -r file; do
    ((n++))
done
echo "$n"                          # 0 – the loop counted in a subshell

n=$(find main_folder -type f | wc -l)            # fix 1: let a filter count

while read -r file; do ((n++)); done < <(find main_folder -type f)
                                   # fix 2: process substitution, loop stays here

Fail early, fail loudly

The rule for every script in this course: check the input at the top, and stop at the first problem – with a message on standard error (>&2) and a non-zero exit code (exit 1). Everything below the checks can then assume the input is fine, so it stays short and free of nested ifs (Fig. 3).

Here is the opening most of today's scripts share. The usage text lives in a function because it is printed from more than one place; usage >&2 sends the function's entire output to stderr.

#!/bin/bash

usage() {
    echo "Usage: $0 <file> <word>"
}

if [ "$#" -ne 2 ]; then
    usage >&2                # the function's whole output goes to stderr
    exit 1
fi

file=$1
word=$2

if [ ! -f "$file" ]; then
    echo "Error: file '$file' does not exist" >&2
    exit 1
fi

# from here on: the real work, on input that passed every check

Test the command, not $?

if runs a command and branches on its exit code – 0 means true, anything else false. [ … ] is just one such command; cp, mkdir and grep work the same way, so there is no need to inspect $? separately. grep is especially useful here: it exits with 0 if it found a match, 1 if it didn't, and 2 on an error such as a missing file. With -q it prints nothing and only sets the code.

if cp "$source" "$target"; then
    echo "Copied"
else
    echo "Error: copy failed" >&2
    exit 1
fi

if grep -q "$word" "$file"; then   # -q: print nothing, only set the exit code
    echo "Found"
fi

read can fail too

When the user presses Ctrl+D, the terminal signals end of input and read returns a non-zero code. In a while true menu loop that ignores this, the script spins forever printing the menu. Put the read into the condition and leave the loop when it fails:

while true; do
    if ! read -r -p "Your choice: " choice; then
        echo                # Ctrl+D: end of input – leave the loop
        break
    fi
    # … handle $choice …
done

The case statement

When one value has to be compared with several fixed options – command-line switches, menu choices – an if … elif … elif chain gets long. case says the same thing more directly. It compares the value with each pattern from top to bottom and runs the first branch that matches, and only that one (Fig. 4).

case "$1" in
    -d|--date)              # | inside a pattern: either one matches
        date
        ;;
    -w)
        whoami
        ;;
    -l) ls -l "$HOME" ;;    # a short branch fits on one line
    *)                      # anything else – always the last branch
        echo "Error: unknown option '$1'" >&2
        exit 1
        ;;
esac

The patterns are glob patterns, the same wildcards you use for file names – not regular expressions:

Filters and the counting pipeline

A filter reads standard input and writes standard output, so filters chain with | (Session 2 introduced them). Besides grep, sort, tr and wc, today's tasks need these:

The practice file patients.txt has one patient per line: name, date of the visit, illness.

Anna Kovacs,2023-10-13,headache
Peter Szabo,2023-10-13,headache
Eva Toth,2023-10-01,headache
Mark Varga,2023-10-20,cold
Julia Balogh,2023-10-22,cold
Daniel Farkas,2023-09-01,stomach ache
Zoe Papp,2023-09-19,headache
Adam Lukacs,2023-11-07,cold
Nora Fekete,2023-11-15,headache
Bence Kiss,2023-11-21,cold
Lili Molnar,2023-05-30,leg pain
Gergo Simon,2023-05-04,leg pain
Rita Horvath,2023-05-24,headache
Tamas Racz,2023-05-11,leg pain

Which illness is the most common? This is the classic pipeline of the course, and the way to write it is one stage at a time, looking at the output after each. First keep only the illness column:

cut -d ',' -f 3 patients.txt

headache
headache
headache
cold
cold
stomach ache
…

Sort it so equal lines sit next to each other, then let uniq -c count each group:

cut -d ',' -f 3 patients.txt | sort | uniq -c

      4 cold
      6 headache
      3 leg pain
      1 stomach ache

Sort by the count, largest first, and keep the top line:

cut -d ',' -f 3 patients.txt | sort | uniq -c | sort -rn | head -n 1

      6 headache

The result is a line like 6 headache, with leading spaces. To print it nicely, split it with the here string from earlier: read -r count illness <<< "$result" – the spaces disappear, the number goes into count, and the rest of the line, even if it has several words like stomach ache, into illness.

Walking through files

To visit the files in one directory, combine for with a glob pattern. Quote the variable part and leave the * outside the quotes, so the shell still expands it:

for file in "$dir"/*.txt; do
    if [ ! -f "$file" ]; then
        continue           # no match: $file holds the literal text "$dir/*.txt"
    fi
    echo "$file"
done

The -f check is not decoration. When no file matches, bash does not give an empty list – it leaves the pattern as it is, so the loop body runs once with the literal text as the "file name":

$ for f in empty_dir/*.txt; do echo "got: $f"; done
got: empty_dir/*.txt

The * only looks into the directory you name, not into subdirectories. For a whole tree, let find list the files and read its output line by line – keeping the pipe trap above in mind:

find "$dir" -type f | while read -r file; do
    echo "$file"
done

Two small commands help when you build names, for example for a backup folder: basename keeps the last part of a path, and date +%F prints today's date as YYYY-MM-DD. Inside double quotes, $( … ) drops a command's output right into a string:

basename /home/student/tasks/main_folder     # main_folder
date +%F                                     # today, as YYYY-MM-DD
name="$(basename "$dir")_backup_$(date +%F)" # main_folder_backup_YYYY-MM-DD

Never read and write the same file

"Convert the file to uppercase" sounds like one line: tr … < f.txt > f.txt. It leaves you with an empty file. Redirections are set up by the shell before the command starts, and > opens its file by truncating it to zero bytes. By the time tr reads f.txt, there is nothing left to read (Fig. 6).

tr '[:lower:]' '[:upper:]' < "$file" > "$file"        # wrong: the file ends up empty

tr '[:lower:]' '[:upper:]' < "$file" > "$file.tmp" &&
    mv "$file.tmp" "$file"                           # right: temp file, then replace

The && matters: the original is only replaced if tr succeeded. The same applies to every filter – sort f > f, grep … f > f – always go through a temporary file.

Filters or loops?

Every task in this session can be solved two ways. The Unix way chains ready-made filters – grep, cut, sort, uniq, find, tr – each doing one small job. The programming way reads the file line by line in a while read loop and works with variables, ifs, counters and arrays, the way you would in C, Java or Python. Both are correct; knowing both means you are never stuck.

The loop's equivalent of cut is read itself: set IFS (the characters read splits at) to a comma for that one command, and each field lands in its own variable (Fig. 7).

while IFS=, read -r name date illness || [ -n "$name" ]; do
    illness=${illness%$'\r'}                   # drop a Windows line ending
    [ -z "$illness" ] && continue               # skip empty lines
    [ "${date:5:2}" != "$month" ] && continue   # the month filter, without grep
    echo "$name: $illness"
done < "$file"

Each line of that loop handles something the filters did for free:

• —|| [ -n "$name" ] – if the last line has no newline at the end, read still fills the variables but returns false, and without this the last line is silently lost.
• —${illness%$'\r'} – a file saved on Windows ends every line with an extra \r, and "headache\r" is not equal to "headache". Session 5 covers line endings in depth.
• —[ -z "$illness" ] && continue – skip empty lines instead of counting an empty illness.

Counting per illness (what sort | uniq -c did) needs two parallel arrays: illnesses[i] holds a name and counts[i] its count, and the same index ties them together. For each line, search the name in the first array; add it if it's new, or increment its count:

index=-1
for (( i=0; i<${#illnesses[@]}; i++ )); do
    if [ "${illnesses[i]}" == "$illness" ]; then
        index=$i
        break
    fi
done

if [ "$index" -eq -1 ]; then
    illnesses+=("$illness")      # first time: add it with a count of 1
    counts+=(1)
else
    (( counts[index]++ ))        # seen before: increment its count
fi

Common mistakes

Practice

The tasks build on each other: 1–4 wrap a single command in a script with proper checks, 5 adds case, 6 a pipeline, 7 a loop over files, 8 combines 5 and 6 with functions, and 9 is an interactive menu that uses all of it. Work in the tasks folder:

tasks/
├── source.txt            a few lines to copy (task 1)
├── menu.txt              one dish per line (tasks 2 and 9)
├── all_lower.txt         lowercase words (task 4)
├── patients.txt          name,YYYY-MM-DD,illness per line (tasks 6, 8, 9)
└── main_folder/          text1.txt, text3.txt (tasks 3 and 7)
    └── sub_folder/       text2.txt

Rules for every task:

• —check the number of arguments; if it's wrong, print a Usage: … line
• —error messages go to stderr (>&2) and the script stops with exit 1
• —a missing file or directory is an error too
• —quote every variable: "$file"

$ ./copy.sh source.txt target.txt
Destination file 'target.txt' does not exist, creating it.
Content copied from 'source.txt' to 'target.txt'.

$ ./copy.sh nope.txt target.txt
Error: source file 'nope.txt' does not exist

if [ ! -e "$destination" ]; then
    echo "Destination file '$destination' does not exist, creating it."
    touch "$destination"
fi
if cp "$source" "$destination"; then
    echo "Content copied from '$source' to '$destination'."

Salad Wine
Pasta wine
Pizza
Hamburger
Winegum pudding
Salmon
Carp Wine

$ ./search_word.sh menu.txt wine
Salad Wine
Pasta wine
Carp Wine

$ ./search_word.sh menu.txt cheese
No line contains the word 'cheese'.

if ! grep -iw "$word" "$file"; then
    echo "No line contains the word '$word'." >&2
    exit 1
fi

shopt -s nocasematch                     # == ignores case from here on
while IFS= read -r line || [ -n "$line" ]; do
    read -r -a words <<< "$line"         # split the line into an array of words
    for w in "${words[@]}"; do
        [[ $w == "$word" ]] && { echo "$line"; ((found++)); break; }
    done
done < "$file"

$ ./find_files_with_word.sh main_folder banana
main_folder/text1.txt
main_folder/text3.txt
main_folder/sub_folder/text2.txt

find "$directory" -type f | while read -r file; do
    if grep -q "$word" "$file"; then
        echo "$file"
    fi
done

search_dir() {
    local entry
    for entry in "$1"/*; do
        if [ -d "$entry" ]; then
            search_dir "$entry"              # the function calls itself for subfolders
        elif [ -f "$entry" ] && contains_word "$entry"; then
            echo "$entry"
        fi
    done
}

$ ./copy.sh all_lower.txt test.txt     # work on a copy first
$ ./uppercase.sh test.txt
$ cat test.txt
APPLE
PEAR
BANANA
ORANGE
PINE

[ -w "$file" ] || { echo "Error: '$file' is not writable" >&2; exit 1; }
tr '[:lower:]' '[:upper:]' < "$file" > "$file.tmp"
mv "$file.tmp" "$file"

$ ./switches.sh -w
student

$ ./switches.sh -x
Error: unknown option '-x'
Usage: ./switches.sh -d | -w | -l
  -d  print the current date and time
  -w  print the name of the logged in user
  -l  list the home directory of the user

usage() {
    echo "Usage: $0 -d | -w | -l"
    echo "  -d  print the current date and time"
}
    …
    *) echo "Error: unknown option '$1'" >&2; usage >&2; exit 1 ;;

$ ./most_common_illness.sh patients.txt 10
Most common illness in month 10: headache (3 patient(s))

$ ./most_common_illness.sh patients.txt 02
No patients in month 02.

[[ $month =~ ^(0[1-9]|1[0-2])$ ]] || { echo "Error: month must be 01-12" >&2; exit 1; }

result=$(grep -E ",[0-9]{4}-$month-[0-9]{2}," "$file" | cut -d ',' -f 3 |
         sort | uniq -c | sort -rn | head -n 1)
read -r count illness <<< "$result"

best=0
for (( i=1; i<${#counts[@]}; i++ )); do
    (( counts[i] > counts[best] )) && best=$i
done
echo "${illnesses[best]} (${counts[best]} patient(s))"

$ ./backup.sh main_folder
Copied: main_folder/text1.txt
Copied: main_folder/text3.txt
2 file(s) backed up to 'main_folder_backup_YYYY-MM-DD'.

backup_dir="$(basename "$directory")_backup_$(date +%F)"
mkdir -p "$backup_dir"
for file in "$directory"/*.txt; do
    [ -f "$file" ] || continue
    cp "$file" "$backup_dir" && echo "Copied: $file" && ((count++))
done

$ ./patients_query.sh patients.txt -i headache
Anna Kovacs
Peter Szabo
…

$ ./patients_query.sh patients.txt -c
      6 headache
      4 cold
      3 leg pain
      1 stomach ache

file=$1; option=$2; value=$3       # file is global: every function sees it

case "$option" in
    -i) [ -z "$value" ] && { echo "Error: -i needs an illness" >&2; exit 1; }
        list_patients_by_illness "$value" ;;
    -l) list_illnesses ;;
    # … -m, -c, -h, and *) for the rest
esac

grep -i ",$illness$" "$file" | cut -d ',' -f 1

===== MENU =====
1) Current date and time
2) Logged in user and home directory
3) Number of .txt files in the current directory
4) List of illnesses (patients.txt)
5) Search a word in menu.txt
0) Exit
Your choice: 7
Error: invalid choice '7'
…
Your choice: 0
Bye! You ran 1 operation(s).

while true; do
    print_menu
    if ! read -r -p "Your choice: " choice; then echo; break; fi
    case "$choice" in
        1) show_date ;;
        0) break ;;
        *) echo "Error: invalid choice '$choice'" >&2; continue ;;
    esac
    ((operations++))               # skipped by continue, so invalid input isn't counted
done

Further practice

Summary

here string

<<< "$s" – one string as a command's standard input

subshell

a child copy of the shell; each side of a pipe runs in one, and its variables die with it

guard clause

a check at the top that ends the script on bad input

exit status

0 for success, 1–255 for failure; what if and && test

filter

a command that reads stdin and writes stdout, so it fits into a pipe

glob pattern

wildcards like * and [0-9], used by file names and by case

IFS

the characters read splits a line at; IFS=, turns read into a CSV splitter

• —Arguments are fixed at start; read takes a line at run time, from the keyboard, a file, a pipe or a here string.
• —Variables set on the right side of | vanish. Use <<<, a filter, or done < … instead.
• —Check input first; errors go to stderr with exit 1; put commands like cp and grep -q straight into if.
• —case runs the first matching branch; end branches with ;; and put *) last.
• —Most-common-value: cut | sort | uniq -c | sort -rn | head -n 1 – built and checked one stage at a time.
• —Guard glob loops with -f; never redirect into the file you are reading – use a temp file and mv.
• —Every filter has a loop equivalent; the loop needs care with the last line, empty lines and \r.