Processes, signals and filters

In the first session you typed a command, waited, and got the prompt back. This session looks at what happens in between. Every command you start becomes a process: a running program with its own number, its own parent, and a state the kernel keeps track of. You can list processes, pause them, push them into the background and send them signals – and a script can decide what to do when a signal arrives.

The second half is about filters: small programs that read lines, change them and write lines. Connected with pipes, a handful of them answer questions that would take a page of code in Java. This is the style the practical test expects, so it is worth getting fluent in it now.

The model in one breath: a running program is a process, the kernel shares the CPU between processes and passes them signals, and a pipe turns one process's output into the next one's input.

Processes and PIDs

Unix and Linux are multitasking systems: many programs run at once, even on a machine with a single core. They do it by time-sharing. The kernel lets one process run for a few milliseconds, then switches to the next. The switching is so fast that it looks simultaneous (Fig. 1). With several cores, some processes really do run in parallel – one per core – but there are always far more processes than cores, so the switching never stops.

To tell processes apart, the kernel gives each one a PID (process ID), an integer. Two copies of the same program – two people running vim on the server – are two processes with two PIDs. The first user-space process gets PID 1 (on most distributions it is systemd); every other process is started by an existing one, its parent, whose PID it remembers as its PPID. That makes all processes one big tree (Fig. 2).

PIDs are not a counter that grows forever. When the numbers reach a maximum they wrap around, and the PID of a process that has ended can be handed out again. So a PID identifies a process only while it runs – don't save one in a file and trust it next week. The maximum is in a file under /proc, the directory where the kernel shows its process information as files:

cat /proc/sys/kernel/pid_max

process

A running instance of a program: code, memory, open files and a state, managed by the kernel.

PID

Process ID – the number the kernel gives a process while it runs.

PPID

Parent process ID – the PID of the process that started this one.

kernel

The core of the operating system: it schedules processes, manages memory and talks to the hardware.

Looking at processes: ps, top, htop

ps takes a snapshot of the process list. Without options it shows only the processes of your current terminal – usually just your shell and ps itself:

  PID TTY          TIME CMD
 4321 pts/3    00:00:00 bash
 4402 pts/3    00:00:00 ps

For everything on the machine use ps aux. It adds the owner, the CPU and memory share, the state and the start time. Combined with grep, it is the quickest way to find the PID of one program:

ps                          # the processes of this terminal
ps -f                       # full format: adds the owner (UID) and the parent (PPID)
ps aux                      # every process on the system, with CPU, memory and state
ps aux | grep sleep         # find the PID of one program
cat /proc/sys/kernel/pid_max   # the largest PID before the numbers wrap around

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.1 168000 11800 ?        Ss   08:00   0:12 /sbin/init
student   4321  0.0  0.0  10100  5200 pts/3    Ss   10:02   0:00 -bash
student   4398  0.0  0.0   5500   900 pts/3    S    10:05   0:00 sleep 300
student   4402  0.0  0.0  11800  3400 pts/3    R+   10:05   0:00 ps aux

The STAT column

The first letter of STAT is the state of the process. Most processes on a server are sleeping almost all the time – they wait for a key press, a network packet or a timer – and only a few are running at any moment (Fig. 3).

Extra characters after the letter add detail: s session leader (your login shell), l multi-threaded, + in the foreground of its terminal, < high priority, N low priority ("nice" to the others), L pages locked in memory. So Ss is a sleeping session leader, and R+ is the ps you just ran.

top and htop

ps is a photo; top is a live view that refreshes every few seconds. The header shows the load, the CPU and memory use of the whole machine; below it the processes are sorted by CPU use, so the one that makes the server slow is at the top. Press q to quit. htop shows the same in colour, with bars per CPU core, mouse support and scrolling – use it if the system has it installed.

What a program does: time and strace

Write time before a command, and after the command ends you get three numbers:

time ls
time sleep 2

$ time sleep 2

real    0m2.003s
user    0m0.001s
sys     0m0.002s

• —real – wall-clock time from start to end, as a stopwatch would measure it.
• —user – CPU time spent in the program's own code.
• —sys – CPU time the kernel spent working for the program (opening files, writing output).

The sleep 2 example is the interesting one: two seconds of real time, almost zero CPU. A sleeping process costs nothing – the S state at work. When real is much larger than user + sys, the program was waiting (for the disk, the network, you); when they are close, it was computing.

strace: the conversation with the kernel

A program cannot open a file, read the keyboard or print to the screen by itself. Only the kernel touches the hardware, so the program asks it through system calls. strace runs a command and prints every system call it makes, with the arguments and the return value (Fig. 4). It is a debugging tool: when a program fails with an unclear message, the trace shows which file it was looking for and what the kernel answered.

strace ls                       # every system call ls makes, on the terminal
strace -o trace.txt ls          # the same, written to trace.txt
strace -e trace=openat ls       # only one kind of call

execve("/usr/bin/ls", ["ls"], 0x7ffd5e1c /* 24 vars */) = 0
...
openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
getdents64(3, /* 5 entries */, 32768)   = 144
getdents64(3, /* 0 entries */, 32768)   = 0
close(3)                                = 0
write(1, "a.txt  b.txt  scripts\n", 22)  = 22
exit_group(0)                           = ?

Foreground and background

By default a command runs in the foreground: the shell waits for it, you get no prompt, and what you type – including Ctrl+C – goes to that command. Put an & at the end and it runs in the background: the shell starts it and gives you the prompt back at once.

The shell calls each command it manages a job and numbers them per terminal. When you start a background job it prints two numbers: [1] is the job number, valid only in this shell, and the second is the PID, valid in the whole system. Job commands take %1; kill and ps take the PID (and kill accepts %1 as well).

$ sleep 300 &
[1] 4398
$ sleep 400
^Z
[2]+  Stopped                 sleep 400
$ jobs
[1]-  Running                 sleep 300 &
[2]+  Stopped                 sleep 400
$ bg %2
[2]+ sleep 400 &
$ fg %1
sleep 300

Signals and kill

A signal is a short notification the kernel delivers to a process: something happened, or the process should do something. Signals come from the keyboard (Ctrl+C), from other processes (kill), or from the kernel itself (a division by zero, a timer that expired). Each one has a name and a number; kill -l lists them all.

Every signal has a default action – for most of them, ending the process. A program may install its own handler instead, or ignore a signal. Two signals are special: SIGKILL and SIGSTOP never reach the program's code. The kernel acts on them directly, which is exactly why they always work (Fig. 6).

The command for sending signals is called kill, for historical reasons – most signals it sends do not have to kill anything. The syntax is kill -signal PID, and without a signal it sends SIGTERM:

kill -l               # every signal this system knows
kill 4398             # SIGTERM (15): "please terminate" – the polite default
kill -15 4398         # the same, by number
kill -TERM 4398       # the same, by name
kill -1 4398          # SIGHUP
kill -9 4398          # SIGKILL: the kernel ends it, no cleanup – the last resort
kill %1               # a job number works as well as a PID

trap: a script that reacts to signals

A shell script is a process too, so it receives signals. With the built-in trap you say what the shell should run when a given signal arrives, instead of the default action. The command goes in single quotes; the signal can be given by name (INT, SIGINT) or by number (2):

trap 'echo ELTE!' INT        # Ctrl+C now prints ELTE! instead of ending the shell
trap '' INT                  # ignore SIGINT completely
trap - INT                   # back to the default action (same as: trap - 2)

The practical use is cleaning up. A script that writes a temporary file should delete it even when someone presses Ctrl+C halfway through. $$ expands to the PID of the running shell, which makes a file name that no other copy of the script will use at the same time:

tmp="/tmp/report.$$"                    # $$ = the PID of this shell: a unique name
trap 'rm -f "$tmp"; exit 1' INT TERM    # Ctrl+C or kill: delete the file, then stop

sort data.txt > "$tmp"
# ... work with "$tmp" ...
rm -f "$tmp"

Note the exit inside the trap. Without it the handler would run, and then the script would carry on from where it was interrupted – the signal has been "handled", so nothing ends it.

Midnight Commander

mc is a two-panel file manager that runs inside the terminal, in the tradition of Norton Commander. Each panel shows a directory; you copy, move, view and edit files between them with the function keys. It is handy for looking around an unfamiliar directory tree on the server, and a panel can also show a remote file system.

mc            # start Midnight Commander
# Tab         switch between the two panels
# F3 / F4     view / edit the selected file
# F5 / F6     copy / move to the other panel
# F10         quit

On your own Debian or Ubuntu machine it installs with sudo apt install mc. On the course server you have no sudo – use it if it is there, and don't let it replace the commands: the test is about commands and scripts.

Filters and pipes

A filter is a program that reads lines from its standard input, does one thing to them, and writes the result to its standard output: grep keeps some lines, sort reorders them, tr changes characters, wc counts. Most filters also accept a file name, but the point is that they work on a stream.

The pipe | connects two processes: the standard output of the left one becomes the standard input of the right one. Both run at the same time, and the lines flow through as they are produced. Error messages go to a third stream, standard error, which the pipe does not carry (Fig. 7). The full story of the three streams and redirection is in the next session; here you need only |, plus < (read standard input from a file) and > (write standard output to a file).

Counting and listing

wc -l counts lines, so ls | wc -l counts the entries of the current directory: when its output goes into a pipe, ls prints one name per line. find . -type d lists every directory under the current one, recursively; grep -v inverts the match and keeps the lines that do not contain the pattern. who lists the logged-in sessions, one per line, with the user name in the first column.

sort: text or numbers

sort orders lines alphabetically, sort -r in reverse. "Alphabetically" means character by character, which surprises everyone the first time they sort numbers (Fig. 8). Add -n to compare the values as numbers.

tr: replacing characters

tr translates characters: every character of the first set becomes the matching character of the second. It reads only standard input – it takes no file name – so the file goes in with < and the result comes out with >:

tr '[:lower:]' '[:upper:]' < file.txt > output.txt

tr never modifies file.txt. It writes a new stream, and the shell puts that stream into output.txt.

grep: lines, matches and whole lines

grep keeps the lines that contain a pattern. Three options change what is counted, and mixing them up gives a wrong number that looks right. With this fruits.txt:

apple pear apple
pear apple
apple

grep -cx 'apple' fruits.txt            # 1 – lines that are exactly "apple"
grep -c 'apple' fruits.txt             # 3 – lines that contain "apple"
grep -o 'apple' fruits.txt | wc -l     # 4 – every occurrence, one per output line

-x requires the whole line to match, -c counts matching lines, and -o prints each match on its own line – the way to count occurrences rather than lines. Read the task: "how many lines are exactly apple" is -cx, "how many times does apple appear" is -o | wc -l.

A pattern can be a regular expression. With -E, ^ means the start of the line, $ the end, [0-9] one digit, + one or more of the previous item, and ? an optional item. So these keep the lines that are integers:

grep -E '^[0-9]+$' numbers.txt          # 0, 12, 123
grep -E '^-?[0-9]+$' numbers.txt        # ... and -5, -42
grep -E '^[+-]?[0-9]+$' numbers.txt     # ... and +10

Without ^ and $, 10a would match too – it contains digits. Regular expressions get a full treatment in the next session; to experiment, paste a pattern into a regex tester such as regex101.com.

Practice

Do these on the course server. The first three are about processes and signals, the rest are filters. Each filter task is a one-liner – the hint shows it, but try first, and check the output.

sleep 300 &              # the shell prints [1] and the PID
ps aux | grep 'sleep 300'
kill 4398                # use the PID you saw – SIGTERM
jobs                     # [1]+  Terminated   sleep 300

sleep 400                # foreground – then press Ctrl+Z
jobs                     # [1]+  Stopped
bg %1                    # continues in the background
fg %1                    # back in front – now Ctrl+C ends it

trap 'echo ELTE!' INT
sleep 30                 # press Ctrl+C: ELTE! appears
trap - INT

ls | wc -l
ls -A | wc -l            # hidden entries too, without . and ..

find . -type d -not -path '*/.*' | grep -v 'temp'

who
who | cut -d ' ' -f 1 | sort -u     # first column only, each name once

10
2
30
4

sort file.txt
sort -r file.txt
sort -n numbers.txt

tr '[:lower:]' '[:upper:]' < file.txt > output.txt
cat output.txt

grep -cx 'apple' fruits.txt

0
12
-5
+10
12.5
abc
10a

grep -E '^[0-9]+$' numbers.txt > filtered.txt

Summary

• —A running program is a process with a PID and a parent (PPID); the processes form a tree rooted at PID 1.
• —ps aux lists everything, top/htop watch it live; STAT tells you whether a process runs (R), waits (S, D), is stopped (T) or a zombie (Z).
• —time separates waiting from computing; strace shows the system calls a program makes.
• —&, Ctrl+Z, jobs, bg and fg move jobs between foreground, stopped and background. Job numbers are per shell, PIDs are system-wide.
• —Signals are sent with kill: SIGTERM first, SIGKILL last. Ctrl+C is SIGINT; Ctrl+D is end of input, not a signal.
• —trap runs your command when a signal arrives – use it to delete temporary files named with $$.
• —Filters read stdin and write stdout; | chains them. sort -n for numbers, grep -x for whole lines, grep -o to count occurrences.