Reading and writing files; line endings

Most shell tasks – in class, in the practical test and in your assignment – read a file, do something with every line and write the result somewhere. The loop itself is short. What breaks it is the file: a last line without a newline, a few empty lines at the end, or a file that was saved on Windows. All three are invisible in an editor and all three change what your script does.

Here is the symptom in one line. A file holds a single word, banana, and wc -l says it has 0 lines. By the end of this session you will know why, how to see it, and the one loop pattern that handles all of these files correctly.

The whole session in one sentence: a line is the text before a newline character, and Unix tools only count \n as a newline – so a file that ends without one, or uses \r\n, is not what your loop thinks it is.

Reading and writing files

Redirections

A command reads standard input and writes standard output. Redirections point those streams at files instead of the terminal. You have met some of them already; these are the ones file processing needs.

echo "apple" > fruits.txt     # fruits.txt: apple
echo "pear"  > fruits.txt     # fruits.txt: pear         (apple is gone)
echo "plum" >> fruits.txt     # fruits.txt: pear, plum   (appended)

The two combine into the standard shape for writing a result file: empty it once before the loop, then append inside the loop. Without the first line, running the script twice would leave the first run's output in the file, with the second run's output after it.

> "$output_file"                     # empty it once, before the loop

while IFS= read -r line || [ -n "$line" ]; do
    echo "$line" >> "$output_file"   # append inside the loop
done < "$input_file"

Line by line

For real files, this is the loop header to use. Every part of it has a job:

while IFS= read -r line || [ -n "$line" ]; do
    ...
done < "$file"

When a line is made of fields, such as Anna Kiss;87, set IFS to the separator and give read one variable per field. If there are more fields than variables, the rest of the line goes into the last variable.

while IFS=';' read -r name points || [ -n "$name" ]; do
    echo "$name: $points"
done < grades.txt

What is a line?

In a text file a line is ended by a newline character. Which character that is depends on the system:

The names come from typewriters: the carriage return pushes the carriage back to the start of the line, the line feed rolls the paper up by one line. Windows kept both; Unix systems kept only the line feed. Bash and the Unix tools – read, grep, wc – treat \n as the end of a line, and in a well-formed file the last line ends with one too.

The four test files of this session (numbers_*.txt) contain the same three numbers – 10, 20, 30 – and still behave differently, because their bytes differ (Fig. 2).

Plain cat doesn't show any of this. Use cat -et: it prints $ at every newline, ^M for a carriage return and ^I for a tab. A last line without $ has no newline. (On Linux cat -A is the same.) For the raw bytes, use od -c.

cat -et file.txt             # $ at every line end, ^M for \r, ^I for a tab
od -c file.txt               # every byte, \n and \r included
tail -c 1 file.txt | od -c   # only the last byte

$ cat -et numbers_ok.txt
10$
20$
30$
$ cat -et numbers_crlf.txt
10^M$
20^M$
30^M$
$ cat -et numbers_no_newline.txt
10$
20$
30$ 

Trap 1: no newline after the last line

This is the most common one. Windows Notepad and several editors don't put a newline at the end of the file, and in a script echo -n or printf '%s' produce the same. Three things go wrong:

Fig. 3 walks through the loop on the file 20\n30. The second read gets 30 and returns failure in the same step. With a plain while read condition that failure ends the loop and 30 is lost; the || [ -n "$line" ] part says "unless something was read".

wc -l < numbers_no_newline.txt     # 2 – counts \n characters
grep -c '' numbers_no_newline.txt  # 3 – counts lines

The third symptom destroys data, so it gets its own picture. A registry file stores one name,phone record per line; its last line has no newline; your script appends a new record.

The fix checks the last byte first. tail -c 1 prints it, and command substitution $( ) strips a trailing newline from what it captures – so the result is empty exactly when the file ends in \n.

# $( ) drops a trailing \n, so the result is empty exactly when the file ends in \n
if [ -n "$(tail -c 1 "$file")" ]; then
    echo >> "$file"                   # add the missing newline first
fi
echo "$name,$phone" >> "$file"

Trap 2: extra empty lines

read reads empty lines as well, so the loop runs more times than there are data lines. The count and the average come out wrong, a number check reports "invalid line", a copy script copies blank lines. The fix is to skip them:

if [ -z "$line" ]; then
    continue
fi

Do it after stripping the carriage return (next section): on a Windows file an "empty" line still contains \r, and -z would not see it as empty.

Trap 3: Windows line endings (CRLF)

When a file made on Windows is read on Linux, read cuts at \n and the \r stays at the end of the line. You can't see it in the terminal, but every comparison sees it (Fig. 5).

$ ./sum.sh numbers_crlf.txt
./sum.sh: line 4: 10: syntax error: invalid arithmetic operator (error token is "\r")

$ ./script.sh                         # the script itself was saved with CRLF
-bash: ./script.sh: /bin/bash^M: bad interpreter: No such file or directory

Where it comes from: the file was created on Windows (Notepad, a CSV export from Excel), or git on Windows converted line endings on checkout (core.autocrlf). The course repository has a .gitattributes file that keeps .sh and .txt files LF everywhere – and leaves the deliberately broken test files alone.

line=${line%$'\r'}               # inside the loop: drop one \r from the end
tr -d '\r' < in.txt > out.txt     # the whole file at once
dos2unix file.txt                 # the dedicated tool, where it is installed

Try it: the naive loop on all four files

The loop you would write first, without any of the fixes:

count=0; sum=0
while read -r number; do
    count=$(( count + 1 ))
    sum=$(( sum + number ))
done < "$file"

The empty-lines file sums correctly only by luck: an empty variable counts as 0 in arithmetic. The count – and any average – is still wrong.

The universal pattern

Put the three fixes together and you get the loop to start every file-processing script from. Copy it, and only rewrite the "process" part.

while IFS= read -r line || [ -n "$line" ]; do
    line=${line%$'\r'}             # 1. drop a Windows \r
    if [ -z "$line" ]; then         # 2. skip empty lines
        continue
    fi

    # 3. --- process the line ---
    echo "$line"
done < "$file"

Three common variations:

• —Fields: while IFS=';' read -r name points || [ -n "$name" ]. The \r is now in the last field, so strip it there: points=${points%$'\r'}.
• —Numbers: number=$(echo "$line" | tr -d '[:space:]') removes spaces, tabs and the \r in one step.
• —Output file: > "$output_file" before the loop, echo "…" >> "$output_file" inside.

And the checks that come before the loop in every task of this session:

if [ "$#" -ne 1 ]; then
    echo "Usage: $0 <file>" >&2
    exit 1
fi
if [ ! -f "$1" ]; then
    echo "Error: file '$1' does not exist" >&2
    exit 1
fi

LF

\n, line feed – the line ending on Linux and macOS.

CR

\r, carriage return – on Windows it comes before every LF.

CRLF

\r\n, the Windows line ending.

truncate

Empty a file to 0 bytes – what > does before writing.

here string

<<< "$var" – one string as a command's input.

Practice

For every task: check the number of arguments and that the input file exists, write error messages to standard error and stop with exit 1. Tasks 1–5 are general scripting warm-ups; 6–9 read and write files and include the traps on purpose. Run your script on every numbers_*.txt file, and compare with cat -et.

$ ./factorial.sh 5
5! = 120
$ ./factorial.sh 21
Error: the number must be at most 20

result=1                            # 0! = 1 falls out for free
for (( i=2; i<=n; i++ )); do
    result=$(( result * i ))
done

while read -r word || [ -n "$word" ]; do
    [ "$word" == "end" ] && break
    words+=("$word")
done
printf '%s\n' "${words[@]}" | sort

$ ./sum.sh 4 -2 10
Sum: 12
$ ./sum.sh 1 x
Error: 'x' is not an integer

if [ "$1" == "--help" ]; then        # before the argument count check
    echo "Usage: $0 <n1> [n2 ... n10]"
    exit 0                            # help that was asked for is not an error
fi
for n in "$@"; do
    [[ $n =~ ^-?[0-9]+$ ]] || { echo "Error: '$n' is not an integer" >&2; exit 1; }
    sum=$(( sum + n ))
done

> odd_lines.txt; > even_lines.txt
while IFS= read -r line || [ -n "$line" ]; do
    n=$(( n + 1 ))
    if (( n % 2 == 1 )); then echo "$line" >> odd_lines.txt
    else                      echo "$line" >> even_lines.txt; fi
done < "$file"

$ ./registry.sh -a "Dan Szabo" 06301112222
Added: Dan Szabo,06301112222
$ ./registry.sh -s szab
Dan Szabo: 06301112222

grep -q "^$name," "$registry" && { echo "Error: '$name' exists" >&2; exit 1; }

grep -v "^$name," "$registry" > "$registry.tmp"     # delete = keep everything else
mv "$registry.tmp" "$registry"

grep -c '' "$registry"                              # count, not wc -l
cut -d ',' -f 1 "$registry" | sort -u               # names, sorted, once each

File:                 numbers_no_newline.txt
Lines (wc -l):        2
Lines (read loop):    3
Ends with newline:    NO
Empty lines:          0
Lines ending in CRLF: 0

if [[ $line == *$'\r' ]]; then          # the line still ends in \r
    crlf=$(( crlf + 1 ))
fi
[ -z "$(tail -c 1 "$file")" ] && ends="yes" || ends="NO"

$ ./sum_numbers.sh numbers_mixed.txt result.txt
Warning: line 6 is not an integer, skipped: 'abc'
Warning: line 8 is not an integer, skipped: '1.5'
Sum: 133

number=$(echo "$line" | tr -d '[:space:]')   # spaces, tabs and \r in one go
[ -z "$number" ] && continue
if [[ ! $number =~ ^-?[0-9]+$ ]]; then
    echo "Warning: line $line_number is not an integer, skipped: '$number'" >&2
    continue
fi

$ cat -et clean.txt
First line$
Second line, had leading spaces$
Fourth line, had an empty line before$
Fifth line$

tr -d '\r' < "$input_file" | while read -r line || [ -n "$line" ]; do
    [ -n "$line" ] && echo "$line"     # read without IFS= trims both ends
done > "$output_file"

name;points;grade
Anna Kiss;87;5
Bela Nagy;45;1
Carol Toth;92;5
…
---
Students: 7
Average: 64
Best: Carol Toth (92)

while IFS=';' read -r name points || [ -n "$name" ]; do
    line_number=$(( line_number + 1 ))
    [ "$line_number" -eq 1 ] && continue                  # header
    points=$(echo "$points" | tr -d '[:space:]')          # the \r sits in the LAST field
    ...
done < "$input_file"

More practice

Every task reads from a file and writes to a file, and the input files contain this session's traps. Recreate the output file on every run – running twice must give the same result.

if [ -z "$line" ]; then
    echo >> "$output_file"      # keep the empty line, without a number
    continue
fi
number=$(( number + 1 ))
echo "$number: $line" >> "$output_file"

while IFS=';' read -r product quantity price || [ -n "$product" ]; do
    quantity=$(echo "$quantity" | tr -d '[:space:]')
    price=$(echo "$price" | tr -d '[:space:]')
    if [[ ! $quantity =~ ^[0-9]+$ ]] || [[ ! $price =~ ^[0-9]+$ ]]; then
        echo "line $line_number: $product" >> errors.txt
        continue
    fi

read -r date time level message <<< "$line"
case "$level" in
    INFO)    echo "$line" >> info.log ;;
    WARNING) echo "$line" >> warning.log ;;
    ERROR)   echo "$line" >> error.log ;;
esac

count=${#names[@]}
[ "${#emails[@]}" -gt "$count" ] && count=${#emails[@]}
for (( i=0; i<count; i++ )); do
    echo "${names[i]};${emails[i]}" >> "$output_file"   # a missing element is just ""
done

while IFS=, read -r current_name current_phone || [ -n "$current_name" ]; do
    if [ "$current_name" == "$name" ]; then echo "$name,$phone"
    else echo "$current_name,$current_phone"; fi
done < "$registry" > "$registry.tmp"
mv "$registry.tmp" "$registry"

Common mistakes

Summary

• —> empties then writes, >> appends. Empty the output once before the loop, append inside it.
• —A line ends with \n. Windows adds a \r before it; some files have no \n after the last line.
• —cat -et shows the truth: $ = \n, ^M = \r, no $ at the end = no final newline.
• —Start every file loop from the universal pattern: || [ -n "$line" ], strip \r, skip empty lines.
• —Test on a normal file, an unterminated one and a CRLF one – on the Linux server, not only in Git Bash.