Text processing and a key–value store

These two sessions are practice. The tasks prepare you for the practical test, and almost all of them use the same handful of tools again and again: the line-reading loop from last session, regular expressions, string operations and arrays. One tool is new – the associative array, a key–value store, which turns "count how often each thing occurs in this file" into a few lines.

The way to use this page: read the toolkit sections once, then try each task yourself before you open its hint. The hint shows the approach and the lines that matter; the rest of the script is yours to write.

The pattern behind every task: read the file safely, clean each line, decide with a pattern, then print, count or store. Only the middle two steps change from task to task.

The toolkit: one loop, split inside

Every file-reading solution starts from the universal pattern of the previous session. Paste it and rewrite only the processing part:

while IFS= read -r line || [ -n "$line" ]; do
    line=${line%$'\r'}             # drop a Windows \r
    if [ -z "$line" ]; then         # skip empty lines
        continue
    fi

    # --- process the line ---
done < "$file"

This time we don't split fields in the while header with IFS=',' read. Instead we split the line inside the loop with a here string. The loop header then stays identical in every script, and the cleaning (\r, empty lines) happens once, before the split.

IFS=',' read -r name date illness <<< "$line"   # split at the commas
read -r word1 word2 rest <<< "$line"            # split at spaces

read puts one field into each variable, and the rest of the line into the last one. That is why the swap-words task uses a third, unused rest variable: without it, a line with three words would leave two of them in word2.

$ read -r first second <<< "one two three four"; echo "[$second]"
[two three four]
$ read -r first second rest <<< "one two three four"; echo "[$second]"
[two]

Regular expressions in a script

[[ $line =~ pattern ]] is true when the line matches the extended regular expression; [[ ! $line =~ pattern ]] negates it. These are the elements the tasks need:

The anchors are the part beginners drop, and the result is a check that lets almost everything through (Fig. 1).

A decimal-number check, read piece by piece (Fig. 2):

if [[ $line =~ ^[+-]?[0-9]*\.?[0-9]+$ ]]; then    # 23, -12.2, +7, .5 – yes
    echo "$line"                                  # 33.3o, 12., asd2 – no
fi

Regex or glob?

To decide whether a word is somewhere in the line you don't need a regex. With == and !=, [[ ]] uses glob patterns – the same wildcards as for file names, where * is any text:

if [[ $line == *"$word"* ]]; then                    # the line contains $word
if [[ $line != *alma* && $line != *körte* ]]; then  # contains neither

"$word" stays quoted inside the pattern, so special characters in it match literally.

The same with grep

Every loop-based filter has a one-line grep version. The loop wins when you need to do more per line (count, transform, write to two files); grep wins when filtering is all you do.

String operations

Bash can cut strings without calling any external program. The ones these tasks use:

Walking the digits of a number combines the length with one-character slices (Fig. 3). Treat the number as a string: this is also how you reverse it without any extra command.

for (( i=0; i<${#number}; i++ )); do
    digit=${number:i:1}
    sum=$(( sum + digit ))
done

tr works on whole character classes; sed on patterns:

sed walks the file by itself, so replacing a word needs no loop. It does need care with the output file:

sed "s/$old/$new/g" "$file" > "$file.tmp"
mv "$file.tmp" "$file"

The same applies to a loop that rewrites its own input: done < "$file" > "$file.tmp", then mv.

Arrays

An array keeps all the lines (or words) in memory, which you need whenever the output order differs from the input order – reversing a file, for instance.

lines=()                          # empty array
lines+=("$line")                  # append
echo "${#lines[@]}"               # number of elements
echo "${lines[0]}"                # first element
read -r -a words <<< "$line"      # the words of a line into an array

for (( i=${#lines[@]}-1; i>=0; i-- )); do   # back to front
    echo "${lines[i]}"
done

The last index is ${#lines[@]}-1, because indexing starts at 0. Starting the backwards loop at ${#lines[@]} prints an empty element first.

Key–value store: associative arrays

An ordinary array is indexed by numbers. An associative array – called a map, dictionary or hash in other languages – is indexed by any string, and each key has one value. For counting it is ideal: the key is what you count, the value is how many.

declare -A counts                      # create it – the -A is required

counts["apple"]=3                      # set a value
counts["apple"]=$(( ${counts["apple"]:-0} + 1 ))   # increment; a missing key starts at 0

echo "${counts["apple"]}"              # one value:        4
echo "${!counts[@]}"                   # all KEYS (the !), in no particular order
echo "${#counts[@]}"                   # number of keys

${counts["apple"]:-0} reads "the value, or 0 if the key doesn't exist yet" – that is what makes the first increment work. Counting words then looks like this:

declare -A counts
for word in "${words[@]}"; do
    counts["$word"]=$(( ${counts["$word"]:-0} + 1 ))
done

for word in "${!counts[@]}"; do
    echo "$word: ${counts["$word"]}"
done

plum: 1
pear: 2
apple: 3

Finding the maximum means walking the keys and remembering the best one so far:

max_count=0
max_word=""
for word in "${!counts[@]}"; do
    if [ "${counts["$word"]}" -gt "$max_count" ]; then
        max_count=${counts["$word"]}
        max_word=$word
    fi
done

Composite keys

To count by two things at once – month and illness – glue them into one key with a separator, key="$month:$illness", and take it apart later with the two cutting expansions.

Worked example: messy data

Task 13 below puts everything together, on a file that is dirty in every way a real export is: a header line, spaces around the commas, punctuation after the illness (headache., headache!), mixed case, an empty and a broken line, CRLF line endings and no final newline. The goal: for every month, the most common illness and how many patients had it. (The file itself is in Hungarian – fejfájás is headache – the figure uses English.)

The cleaning step is two lines – spaces and punctuation out, everything lowercase:

date=$(echo "$date" | tr -d '[:space:]')
illness=$(echo "$illness" | tr -d '[:space:][:punct:]' | tr '[:upper:]' '[:lower:]')

The solution has three phases: clean and count per month:illness key; find the maximum per month by walking the keys and keeping two more associative arrays, max_count[month] and max_illness[month]; and print sorted, because the key order is random.

Practice

For every task: check the arguments and that the file exists, error messages go to standard error, and the script stops with exit 1. Read files with the universal pattern. Tasks 5 and 12 rewrite their input file – try them on a copy (cp replace.txt test.txt).

23
11
12.2
-12.2

if [[ $line =~ ^[+-]?[0-9]*\.?[0-9]+$ ]]; then
    echo "$line"
fi

There are 2 lines in the file that contain only letters.

if [[ $line =~ ^[[:alpha:][:space:]]+$ ]]; then
    count=$(( count + 1 ))
fi
# one line: grep -c -E '^[[:alpha:][:space:]]+$' letters.txt

> "$destination"
while ...; do
    if [[ $line != *"$word"* ]]; then
        echo "$line" >> "$destination"
    fi
done < "$source"

if [[ $line != *alma* && $line != *körte* && $line != *"$word"* ]]; then
    echo "$line"
fi

$ ./replace.sh test.txt alma barack
Word 'alma' replaced with 'barack' in 'test.txt'.

sed "s/\b$old\b/$new/g"     # GNU sed: whole words only, "almafa" stays

12 → 21
21 → 12
56 → 65
90 → 09

if [[ $number =~ ^[0-9]{2}$ ]]; then
    echo "${number:1:1}${number:0:1}"
fi

asd1 asd2  →  asd2 asd1
egy ketto  →  ketto egy

read -r word1 word2 rest <<< "$line"
echo "$word2 $word1"

Warning: skipping invalid number: 'asd'
Sum of numbers in the file: 20

if [[ ! $line =~ ^[0-9]+$ ]]; then
    echo "Warning: skipping invalid number: '$line'" >&2
    continue
fi
sum=$(( sum + line ))

Enter a number: 1234
Reversed number: 4321

read -r -p "Enter a number: " number
for (( i=${#number}-1; i>=0; i-- )); do
    reversed="$reversed${number:i:1}"
done

Enter a number: Sum of digits: 10

printf '%s\n' "${reversed[@]}" > "$file"   # only after ALL lines are read

Warning: invalid line skipped: 'Név, Dátum, Betegség'
Warning: invalid line skipped: 'Hibás Sor'
Month 05: lábfájás (3 patient(s))
Month 09: gyomorpanaszok (2 patient(s))
Month 10: fejfájás (3 patient(s))
Month 11: megfázás (3 patient(s))

key="$month:$illness"
counts["$key"]=$(( ${counts["$key"]:-0} + 1 ))
...
for key in "${!counts[@]}"; do
    month=${key%%:*}
    illness=${key#*:}
    if [ "${counts["$key"]}" -gt "${max_count["$month"]:-0}" ]; then
        max_count["$month"]=${counts["$key"]}
        max_illness["$month"]=$illness
    fi
done

More practice

The inputs contain the usual traps – CRLF, no final newline, empty lines, extra spaces. Start every solution from the universal pattern.

clean=$(echo "$line" | tr -d ' ' | tr '[:upper:]' '[:lower:]')
# reverse $clean into $reversed (task 10), then:
[ "$clean" == "$reversed" ] && echo "$line"

read -r -a words <<< "$line"
out=""
for (( i=${#words[@]}-1; i>=0; i-- )); do
    out="$out${words[i]} "
done
echo "${out% }"                     # drop the trailing space

for word in "${banned[@]}"; do
    line=${line//"$word"/***}
done
[ "$line" != "$original" ] && changed=$(( changed + 1 ))

number=$(echo "$line" | tr -d '[:space:]')
[[ $number =~ ^[0-9]+$ ]] || { echo "Warning: line $n: '$number'" >&2; continue; }

if [ "${#line}" -gt "$max_len" ]; then
    max_len=${#line}
    longest=$line
fi

case "$2" in
    -n) numeric_lines "$file" ;;
    -t) text_lines "$file" ;;
    -w) lines_with_word "$file" "$3" ;;
    *)  usage; exit 1 ;;
esac

IFS=',' read -r name count date <<< "$line"
name=$(echo "$name" | sed 's/^[[:space:]]*//; s/[[:space:]]*$//')   # trim the ends only
count=$(echo "$count" | tr -d '[:space:]')
visits["$name"]=$(( ${visits["$name"]:-0} + count ))

Common mistakes

anchor

^ and $ – the start and end of the line in a regex.

glob

A wildcard pattern (*, ?, […]), as in file names; used by == inside [[ ]].

here string

<<< "$line" – feeds one string to a command such as read.

associative array

An array indexed by strings, created with declare -A.

composite key

Two values glued into one key, e.g. 10:headache.

Summary

• —Anchor every validating regex: ^…$. Never quote it inside [[ =~ ]].
• —"Contains a word" is a glob: [[ $line == *"$word"* ]].
• —Strings are arrays of characters: ${#s} and ${s:i:1} cover reversing and digit sums.
• —Never redirect into the file you are reading – temporary file, then mv.
• —Count with declare -A: clean the key first, ${counts[$k]:-0} + 1, walk "${!counts[@]}" for the maximum, sort for ordered output.