# Text processing and a key–value store > A practice block for the practical test: anchored regular expressions, glob tests, string slicing, tr and sed, arrays – and associative arrays for counting per key, with a worked example on messy data. Source: https://laszlonemes.com/university/computer-systems/text-processing · Author: László Nemes Session 6 · Computer Systems A practice block for the practical test: anchored regular expressions, glob tests, string slicing, tr and sed, arrays – and associative arrays for counting per key, with a worked example on messy data. 19 min read `Anchored regex with =~` `Regex vs glob` `String slicing and cutting` `sed with a temporary file` `declare -A counting` `Composite keys` These two sessions are practice. The tasks prepare you for the practical test, and almost all of them use the same handful of tools again and again: the line-reading loop from last session, regular expressions, string operations and arrays. One tool is new – the **associative array**, a key–value store, which turns "count how often each thing occurs in this file" into a few lines. The way to use this page: read the toolkit sections once, then try each task *yourself* before you open its hint. The hint shows the approach and the lines that matter; the rest of the script is yours to write. After this session you can - test a line against a regular expression with `[[ $line =~ … ]]` – anchored, so the whole line has to match - choose between a regex and a glob pattern (`*"$word"*`) for "does the line contain …" - take strings apart with `${#s}`, `${s:i:1}`, `${s%%…}`, `${s#…}` and clean them with `tr` - store lines in an array and walk it backwards - count occurrences per key with `declare -A`, and find the maximum > **The pattern behind every task:** read the file safely, clean each line, decide with a pattern, then print, count or store. Only the middle two steps change from task to task. ## The toolkit: one loop, split inside Every file-reading solution starts from the universal pattern of the previous session. Paste it and rewrite only the processing part: ```shell while IFS= read -r line || [ -n "$line" ]; do line=${line%$'\r'} # drop a Windows \r if [ -z "$line" ]; then # skip empty lines continue fi # --- process the line --- done < "$file" ``` This time we don't split fields in the `while` header with `IFS=',' read`. Instead we split the line **inside** the loop with a here string. The loop header then stays identical in every script, and the cleaning (`\r`, empty lines) happens once, before the split. ```shell IFS=',' read -r name date illness <<< "$line" # split at the commas read -r word1 word2 rest <<< "$line" # split at spaces ``` `read` puts one field into each variable, and **the rest of the line into the last one**. That is why the swap-words task uses a third, unused `rest` variable: without it, a line with three words would leave two of them in `word2`. ```text $ read -r first second <<< "one two three four"; echo "[$second]" [two three four] $ read -r first second rest <<< "one two three four"; echo "[$second]" [two] ``` ## Regular expressions in a script `[[ $line =~ pattern ]]` is true when the line **matches** the extended regular expression; `[[ ! $line =~ pattern ]]` negates it. These are the elements the tasks need: | Pattern | Meaning | Used in | | --- | --- | --- | | `^` … `$` | Start and end of the line: the **whole** line has to match, not just a part of it | every task | | `[0-9]+` | At least one digit | tasks 8–11 | | `[0-9]{2}` | Exactly two digits | task 6 | | `[0-9]{4}-[0-9]{2}-[0-9]{2}` | A date like `YYYY-MM-DD` | task 13 | | `[+-]?` | Optional sign – `?` means 0 or 1 of the previous item | task 1 | | `[0-9]*\.?[0-9]+` | Decimal number. `\.` is a literal dot; a bare `.` matches any character | task 1 | | `[[:alpha:]]`, `[[:space:]]` | POSIX classes: any letter, any whitespace. Combined: `[[:alpha:][:space:]]` | task 2 | The anchors are the part beginners drop, and the result is a check that lets almost everything through (Fig. 1). > **Fig. 1** · Without anchors, `=~` asks "is there a match somewhere in the line?" > > *Diagram:* Four lines tested against an integer pattern. Without the anchors, one digit anywhere in the line is a match, so asd2, 12.2 and x 33 y are wrongly accepted. With ^ and $ the whole line must be digits, so only 23 matches. A decimal-number check, read piece by piece (Fig. 2): > **Fig. 2** · A regex is a sequence of small claims about the line, read left to right. > > *Diagram:* The decimal number pattern read left to right: start of line, optional sign, any number of digits, an optional literal dot, at least one digit, end of line. 23, -12.2, +7, .5 and 0.25 match; 33.3o, 12., asd2, 1.2.3 and a lone plus sign don't. ```shell if [[ $line =~ ^[+-]?[0-9]*\.?[0-9]+$ ]]; then # 23, -12.2, +7, .5 – yes echo "$line" # 33.3o, 12., asd2 – no fi ``` Common mistake **Don't quote the regex** Inside `[[ =~ ]]`, a quoted right-hand side is matched as **literal text**, not as a regex. Write the pattern unquoted, or put it in a variable and use that unquoted. ```shell [[ $line =~ "^[0-9]+$" ]] # WRONG: quoted, so it looks for the literal text ^[0-9]+$ [[ $line =~ ^[0-9]+$ ]] # right pattern='^[0-9]+$' [[ $line =~ $pattern ]] # also right: an unquoted variable is a regex ``` ### Regex or glob? To decide whether a word is *somewhere in* the line you don't need a regex. With `==` and `!=`, `[[ ]]` uses glob patterns – the same wildcards as for file names, where `*` is any text: ```shell if [[ $line == *"$word"* ]]; then # the line contains $word if [[ $line != *alma* && $line != *körte* ]]; then # contains neither ``` `"$word"` stays quoted inside the pattern, so special characters in it match literally. ### The same with grep Every loop-based filter has a one-line `grep` version. The loop wins when you need to do more per line (count, transform, write to two files); `grep` wins when filtering is all you do. | With a loop | With grep | | --- | --- | | `[[ $line =~ re ]]` + `echo` | `grep -E 're' file` | | `[[ ! $line =~ re ]]` + `echo` | `grep -v -E 're' file` | | `[[ $line =~ re ]]` + counter | `grep -c -E 're' file` | | `[[ $line != *"$word"* ]]` + `echo` | `grep -v "$word" file` | ## String operations Bash can cut strings without calling any external program. The ones these tasks use: | Expression | Meaning | Example (s="1234") | | --- | --- | --- | | `${#s}` | Length | `4` | | `${s:i:1}` | The character at index i (from 0) | `${s:0:1}` → `1`, `${s:3:1}` → `4` | | `${s//a/b}` | Replace every `a` with `b` (one `/`: only the first) | `${s//1/X}` → `X234` | | `${s%$'\r'}` | Remove a trailing `\r` | | | `${key%%:*}` | The part before the first colon (cuts the longest :* from the end) | `10:headache` → `10` | | `${key#*:}` | The part after the first colon (cuts the shortest *: from the start) | `10:headache` → `headache` | Walking the digits of a number combines the length with one-character slices (Fig. 3). Treat the number as a **string**: this is also how you reverse it without any extra command. > **Fig. 3** · Indexes run from 0 to `${#number}` − 1; the direction of the loop decides the task. > > *Diagram:* The string 1234 has length 4 and its characters sit at indexes 0 to 3. Walking i from 0 up and taking one character at a time gives the digit sum 10; walking i from 3 down to 0 builds the reversed string 4321. ```shell for (( i=0; i<${#number}; i++ )); do digit=${number:i:1} sum=$(( sum + digit )) done ``` `tr` works on whole character classes; `sed` on patterns: | Command | Effect | | --- | --- | | `tr -d '[:space:]'` | Delete every space, tab and `\r` – for numbers | | `tr -d '[:punct:]'` | Delete punctuation (`. , ! ? ;` …) | | `tr '[:upper:]' '[:lower:]'` | Lowercase (reliable for unaccented letters) | | `sed 's/^[[:space:]]*//; s/[[:space:]]*$//'` | Trim only the **ends**; inner spaces stay (`" Hanna Molnar "` → `Hanna Molnar`) | | `sed "s/$old/$new/g"` | Replace in every line; `g` = every occurrence in the line, not just the first | `sed` walks the file by itself, so replacing a word needs no loop. It does need care with the output file: ```shell sed "s/$old/$new/g" "$file" > "$file.tmp" mv "$file.tmp" "$file" ``` > **Fig. 4** · The shell sets up `>` before the command runs – that is why the temporary file is not optional. > > *Diagram:* Redirecting a command's output into its own input file empties the file before the command reads it, so the content is lost. Writing to a temporary file and renaming it over the original with mv keeps the data. The same applies to a loop that rewrites its own input: `done < "$file" > "$file.tmp"`, then `mv`. ## Arrays An array keeps all the lines (or words) in memory, which you need whenever the output order differs from the input order – reversing a file, for instance. ```shell lines=() # empty array lines+=("$line") # append echo "${#lines[@]}" # number of elements echo "${lines[0]}" # first element read -r -a words <<< "$line" # the words of a line into an array for (( i=${#lines[@]}-1; i>=0; i-- )); do # back to front echo "${lines[i]}" done ``` The last index is `${#lines[@]}-1`, because indexing starts at 0. Starting the backwards loop at `${#lines[@]}` prints an empty element first. ## Key–value store: associative arrays An ordinary array is indexed by numbers. An **associative array** – called a map, dictionary or hash in other languages – is indexed by any string, and each key has one value. For counting it is ideal: the key is *what* you count, the value is *how many*. > **Fig. 5** · Same idea, different index: numbers for an array, strings for `declare -A`. > > *Diagram:* An ordinary array is indexed by numbers 0, 1, 2. An associative array, created with declare -A, is indexed by strings. Counting words: each word is a key, and its value goes up by one every time the word appears. ```shell declare -A counts # create it – the -A is required counts["apple"]=3 # set a value counts["apple"]=$(( ${counts["apple"]:-0} + 1 )) # increment; a missing key starts at 0 echo "${counts["apple"]}" # one value: 4 echo "${!counts[@]}" # all KEYS (the !), in no particular order echo "${#counts[@]}" # number of keys ``` `${counts["apple"]:-0}` reads "the value, or 0 if the key doesn't exist yet" – that is what makes the first increment work. Counting words then looks like this: ```shell declare -A counts for word in "${words[@]}"; do counts["$word"]=$(( ${counts["$word"]:-0} + 1 )) done for word in "${!counts[@]}"; do echo "$word: ${counts["$word"]}" done ``` output – note the order ```text plum: 1 pear: 2 apple: 3 ``` **Finding the maximum** means walking the keys and remembering the best one so far: ```shell max_count=0 max_word="" for word in "${!counts[@]}"; do if [ "${counts["$word"]}" -gt "$max_count" ]; then max_count=${counts["$word"]} max_word=$word fi done ``` ### Composite keys To count by two things at once – month *and* illness – glue them into one key with a separator, `key="$month:$illness"`, and take it apart later with the two cutting expansions. > **Fig. 6** · `%%` cuts from the end, `#` cuts from the start; the pattern says where. > > *Diagram:* A composite key glues two values with a separator: 10:headache. The expansion with %% and :* cuts from the end and keeps the month, 10. The expansion with # and *: cuts from the start and keeps the illness, headache. Good to know **Order and availability** The keys come back in no particular order. If the output must be sorted, pipe the keys through `sort` and loop over that: ```shell for month in $(printf '%s\n' "${!max_count[@]}" | sort); do echo "Month $month: ${max_illness["$month"]}" done ``` Associative arrays exist in every bash on the course server, on Linux and in Git Bash. The very old bash that macOS ships by default doesn't have them – install a current bash there. ## Worked example: messy data Task 13 below puts everything together, on a file that is dirty in every way a real export is: a header line, spaces around the commas, punctuation after the illness (`headache.`, `headache!`), mixed case, an empty and a broken line, CRLF line endings and no final newline. The goal: for every month, the most common illness and how many patients had it. (The file itself is in Hungarian – `fejfájás` is headache – the figure uses English.) > **Fig. 7** · Clean first, then count. Without cleaning, headache, headache. and Headache are three different keys. > > *Diagram:* One messy record on its way to the result: the raw line with spaces, punctuation, mixed case and a carriage return is split at the commas, its fields are cleaned, the date is validated, the month plus illness key is counted, and finally the most common illness is picked per month. The cleaning step is two lines – spaces and punctuation out, everything lowercase: ```shell date=$(echo "$date" | tr -d '[:space:]') illness=$(echo "$illness" | tr -d '[:space:][:punct:]' | tr '[:upper:]' '[:lower:]') ``` The solution has three phases: **clean and count** per `month:illness` key; **find the maximum per month** by walking the keys and keeping two more associative arrays, `max_count[month]` and `max_illness[month]`; and **print sorted**, because the key order is random. Common mistake **Look at the raw data before you count** Run `cat -A` on the input first. Every invisible difference – a trailing space, a `\r`, a capital letter – becomes a separate key, and the counts quietly split. In your assignment **Counting in your shell assignment** Most assignments contain a "how many per …" or "which is the most …" question. Decide the key first – exactly what makes two records "the same"? – then normalise every field that goes into it (trim, strip punctuation, lowercase) **before** it becomes a key. A pipeline like `cut | sort | uniq -c | sort -rn | head -n 1` is fine for one question; once you need it per month or per person, the associative array is cleaner. ## Practice For every task: check the arguments and that the file exists, error messages go to standard error, and the script stops with `exit 1`. Read files with the universal pattern. Tasks 5 and 12 rewrite their input file – try them on a copy (`cp replace.txt test.txt`). Task 1 **Lines that are numbers** Print the lines of a file that contain only a number. Think of real numbers too: sign and decimal point. Input: `numbers_mixed.txt`. ```text 23 11 12.2 -12.2 ``` **Approach:** The decimal pattern from Fig. 2. `33.3o` and `asd2` fail because of the anchors. `-12.2` is the last line, without a newline – if it's missing, check your loop header. ```shell if [[ $line =~ ^[+-]?[0-9]*\.?[0-9]+$ ]]; then echo "$line" fi ``` Task 2 **Count the lines made of letters** Count how many lines contain only letters (and spaces). Input: `letters.txt`. ```text There are 2 lines in the file that contain only letters. ``` **Approach:** One bracket expression with two classes; `asd3asd` fails because of the digit. ```shell if [[ $line =~ ^[[:alpha:][:space:]]+$ ]]; then count=$(( count + 1 )) fi # one line: grep -c -E '^[[:alpha:][:space:]]+$' letters.txt ``` Task 3 **Copy, leaving out a word** Copy a file to another file, leaving out every line that contains a given word. Arguments: source, destination, word. If the destination doesn't exist, create it (and say so). **Approach:** A glob test, the output emptied before the loop. One-line version: `grep -v "$word" "$source" > "$destination"`. ```shell > "$destination" while ...; do if [[ $line != *"$word"* ]]; then echo "$line" >> "$destination" fi done < "$source" ``` Task 4 **Exclude several words** Print the lines that contain neither `alma` (apple) nor `körte` (pear), nor the word given as an argument. Input: `fruits.txt`. **Approach:** Three `!=` conditions joined with `&&`. ```shell if [[ $line != *alma* && $line != *körte* && $line != *"$word"* ]]; then echo "$line" fi ``` Task 5 **Replace a word in the file** Replace every occurrence of a word with another one, in the file itself. Arguments: file, old word, new word. ```text $ ./replace.sh test.txt alma barack Word 'alma' replaced with 'barack' in 'test.txt'. ``` **Approach:** The only task that needs no loop: `sed` with a temporary file and `mv` (Fig. 4). Plain `sed` also replaces inside words (`almafa` becomes `barackfa`); with GNU sed, `\b` limits it to whole words: ```shell sed "s/\b$old\b/$new/g" # GNU sed: whole words only, "almafa" stays ``` Task 6 **Reverse two-digit numbers** A file has two-digit numbers, one per line. Print each with its digits swapped. ```text 12 → 21 21 → 12 56 → 65 90 → 09 ``` **Approach:** Validate with `^[0-9]{2}$`, then glue the two characters in the other order. Linux has `rev` for this, but Git Bash doesn't – slicing works everywhere. ```shell if [[ $number =~ ^[0-9]{2}$ ]]; then echo "${number:1:1}${number:0:1}" fi ``` Task 7 **Swap two words** Every line of a file has two words. Swap them in every line. ```text asd1 asd2 → asd2 asd1 egy ketto → ketto egy ``` **Approach:** Split with a here string; the third variable catches anything extra. ```shell read -r word1 word2 rest <<< "$line" echo "$word2 $word1" ``` Task 8 **Sum the numbers in a file** A file has one number per line. Add them up; warn about lines that aren't numbers. ```text Warning: skipping invalid number: 'asd' Sum of numbers in the file: 20 ``` **Approach:** Validate, warn to standard error, `continue`; add up the rest. ```shell if [[ ! $line =~ ^[0-9]+$ ]]; then echo "Warning: skipping invalid number: '$line'" >&2 continue fi sum=$(( sum + line )) ``` Task 9 **Copy the non-numeric lines** A file mixes numbers and text. Copy every line that is **not** purely a number to a new file. **Approach:** The mirror image of task 1: the same kind of pattern with `!`. One line: `grep -v -E '^[0-9]+$' in.txt > out.txt`. Task 10 **Reverse a number** Read a number from the keyboard with `read` and print it reversed. ```text Enter a number: 1234 Reversed number: 4321 ``` **Approach:** Handle it as a string: walk the characters from the back and append them to a new string. That also keeps leading zeros – `1200` becomes `0021`. Validate the input first. ```shell read -r -p "Enter a number: " number for (( i=${#number}-1; i>=0; i-- )); do reversed="$reversed${number:i:1}" done ``` Task 11 **Digit sum** Read a number and print the sum of its digits. It should also work with piped input (`echo 1234 | ./digit_sum.sh`). ```text Enter a number: Sum of digits: 10 ``` **Approach:** The forward loop of Fig. 3. Check the input is all digits, or the arithmetic fails on a letter. Task 12 **Reverse the lines of a file** Reverse a file in place – the last line becomes the first. Use an array as temporary storage. **Approach:** Read every line into an array, build the reversed array (or print backwards), and only write the file when **all** lines have been read. ```shell printf '%s\n' "${reversed[@]}" > "$file" # only after ALL lines are read ``` Task 13 **Most common illness per month** Each line of `patients_messy.txt` is `Name, YYYY-MM-DD, illness` – with every kind of dirt described in the worked example. For every month, print the most common illness and the number of patients. Warn about invalid lines. ```text Warning: invalid line skipped: 'Név, Dátum, Betegség' Warning: invalid line skipped: 'Hibás Sor' Month 05: lábfájás (3 patient(s)) Month 09: gyomorpanaszok (2 patient(s)) Month 10: fejfájás (3 patient(s)) Month 11: megfázás (3 patient(s)) ``` **Approach:** Follow Fig. 7. The month is `${date:5:2}`. The header and the broken line fail the date check, so they need no special case. ```shell key="$month:$illness" counts["$key"]=$(( ${counts["$key"]:-0} + 1 )) ... for key in "${!counts[@]}"; do month=${key%%:*} illness=${key#*:} if [ "${counts["$key"]}" -gt "${max_count["$month"]:-0}" ]; then max_count["$month"]=${counts["$key"]} max_illness["$month"]=$illness fi done ``` ## More practice The inputs contain the usual traps – CRLF, no final newline, empty lines, extra spaces. Start every solution from the universal pattern. Task 14 **Palindromes** Print the lines of `palindromes.txt` that read the same backwards, ignoring spaces and case (`Never odd or even`). Expected: 6 lines, the last one without a newline in the file. **Approach:** Normalise first, then reuse the reversing loop of task 10. ```shell clean=$(echo "$line" | tr -d ' ' | tr '[:upper:]' '[:lower:]') # reverse $clean into $reversed (task 10), then: [ "$clean" == "$reversed" ] && echo "$line" ``` Task 15 **Reverse the words in every line** Task 7 for any number of words: in every line of `word_lines.txt`, reverse the word order. Expected first lines: `harom ketto egy`, `korte alma`, `e d c b a`. **Approach:** Split the line into an array with `read -r -a`, then walk it backwards. ```shell read -r -a words <<< "$line" out="" for (( i=${#words[@]}-1; i>=0; i-- )); do out="$out${words[i]} " done echo "${out% }" # drop the trailing space ``` Task 16 **Censor banned words** `banned_words.txt` has one banned word per line (with Windows line endings). Replace every banned word in `story.txt` with `***`, write the result to `censored.txt`, and print how many lines changed. Expected: 3. **Approach:** Read the banned words into an array – stripping the `\r`, or not a single replacement happens. Then replace each word in each line, and compare with the original line. ```shell for word in "${banned[@]}"; do line=${line//"$word"/***} done [ "$line" != "$original" ] && changed=$(( changed + 1 )) ``` Task 17 **Digit sums from a file** `numbers_digits.txt` has one number per line, with spaces, invalid and empty lines. Write `number: digit sum` for each valid number to `digit_sums.txt`, then the number with the largest digit sum. Warn about invalid lines with their line number. Expected: 6 valid numbers, 2 invalid lines, `Max: 99 (18)`. **Approach:** Clean, validate, then the digit loop of task 11 plus a running maximum. ```shell number=$(echo "$line" | tr -d '[:space:]') [[ $number =~ ^[0-9]+$ ]] || { echo "Warning: line $n: '$number'" >&2; continue; } ``` Task 18 **Longest line, longest word** Print the longest line and the longest word of `text.txt`, with their lengths. Expected: a 59-character line and a 22-letter word. **Approach:** Two running maximums: one over lines, one over the words of each line (`read -r -a`). ```shell if [ "${#line}" -gt "$max_len" ]; then max_len=${#line} longest=$line fi ``` Task 19 **Filter with options** Filter a file (first argument) by an option: `-n` numeric lines only, `-t` text lines only, `-w ` lines containing the word. Unknown option: usage text and `exit 1`. Expected on `numbers_and_text.txt`: `-n` → 2 lines, `-t` → 5, `-w asd` → 4. **Approach:** Wrap tasks 1, 9 and 3 into functions and dispatch with `case`. ```shell case "$2" in -n) numeric_lines "$file" ;; -t) text_lines "$file" ;; -w) lines_with_word "$file" "$3" ;; *) usage; exit 1 ;; esac ``` Task 20 **The most frequent patient** Lines of `visits.txt` are `Name, visits, last date`; a name can appear in **several** lines, and its visits add up. Print who visited most, and how many times. Watch the spaces and the CRLF endings. Expected: `Kiss Lajos - 12`. **Approach:** Sum per name in an associative array, then find the maximum. Trim only the ends of the name – the space inside it belongs to it. ```shell IFS=',' read -r name count date <<< "$line" name=$(echo "$name" | sed 's/^[[:space:]]*//; s/[[:space:]]*$//') # trim the ends only count=$(echo "$count" | tr -d '[:space:]') visits["$name"]=$(( ${visits["$name"]:-0} + count )) ``` ## Common mistakes | Wrong | Right | Why | | --- | --- | --- | | `[[ $line =~ [0-9]+ ]]` | `[[ $line =~ ^[0-9]+$ ]]` | Without anchors `asd2` matches too. | | `[[ $line =~ "^[0-9]+$" ]]` | `[[ $line =~ ^[0-9]+$ ]]` | Quoted, the pattern is literal text. | | `[[ $line == "$word" ]]` for "contains" | `[[ $line == *"$word"* ]]` | Without `*` only an identical line matches. | | regex on a Windows file | first `line=${line%$'\r'}` | The `\r` stops `$` from matching. | | `sed … "$file" > "$file"` | temporary file, then `mv` | `>` empties the file before `sed` starts. | | `for (( i=${#a[@]}; i>=0; i-- ))` | `for (( i=${#a[@]}-1; i>=0; i-- ))` | The last index is length − 1. | | counting by string key without `declare -A` | `declare -A counts` first | Otherwise it is a numeric array and every string key lands on index 0. | | `for key in ${!counts[@]}` | `for key in "${!counts[@]}"` | Unquoted, `Lajos Kiss` splits into two keys. | | `echo "$x" \| rev` | `${x:i:1}` in a loop | `rev` isn't available everywhere (Git Bash). | - **anchor** – `^` and `$` – the start and end of the line in a regex. - **glob** – A wildcard pattern (`*`, `?`, `[…]`), as in file names; used by `==` inside `[[ ]]`. - **here string** – `<<< "$line"` – feeds one string to a command such as `read`. - **associative array** – An array indexed by strings, created with `declare -A`. - **composite key** – Two values glued into one key, e.g. `10:headache`. ## Summary - Anchor every validating regex: `^…$`. Never quote it inside `[[ =~ ]]`. - "Contains a word" is a glob: `[[ $line == *"$word"* ]]`. - Strings are arrays of characters: `${#s}` and `${s:i:1}` cover reversing and digit sums. - Never redirect into the file you are reading – temporary file, then `mv`. - Count with `declare -A`: clean the key first, `${counts[$k]:-0} + 1`, walk `"${!counts[@]}"` for the maximum, `sort` for ordered output.