Bash – Catching errors within a piped command chain

When chaining commands in Bash using the pipe ‘|’ separator, the commands execute in sequence, with the output of the first command being used as the input to the second command. But what happens if an error occurs when executing a command within a piped list of commands?

When chaining commands in Bash using the pipe ‘|’ separator, the commands execute in sequence, with the output of the first command being used as the input to the second command.

When running a single Bash command, it is possible to detect if an error has occurred by examining the ‘$?’ built-in Bash variable directly after the command has completed. A zero ‘0’ return value means success, a non-zero return value means that an error has occured.

For example, trying to write output to a new file in the root of the file system.

bash ~$ echo "Create a file with one line in it" > /new-file.txt
-bash: /new-file.txt: Permission denied
bash ~$ echo $?
1

Whereas writing a file to your home path will succeed as expected.

bash ~$ echo "Create a file with one line in it" > ~/new-file.txt
bash ~$ echo $?
0

But what happens if an error occurs when executing a command within a piped list of commands? Well in this case, the default is for the last command to set the return value for the entire command chain.

bash ~$ cat /new-file.txt | wc
cat: /new-file.txt: No such file or directory
       0       0       0
bash ~$ echo $?
0

This can be overridden by setting the pipefail Bash option.

bash ~$ set -o pipefail
bash ~$ cat /new-file.txt | wc
cat: /new-file.txt: No such file or directory
       0       0       0
bash ~$ echo $?
1

This can be extremely useful if you regularly chain commands together, as the return value can be used to display a message to the user indicating an issue has occurred within the command chain.

Linux command pipeline using ‘|’

When using pipelines, the output from the first command is treated as the input to the second command, the output of the second command is treated as the input to the third command, and so on.

When writing bash scripts, I often connect a series of commands together into a pipeline, where the commands are separated by the pipe '|' character.
When using pipelines, the output from the first command is treated as the input to the second command, the output of the second command is treated as the input to the third command, and so on.
This can be useful in a number of situations, such as when you need to process the output of a command further before displaying or assigning to a variable.
For example, given a file containing a sequence of numbers

bash ~$ cat numbers.txt
2250
2262
1
1
1
15379
15379
1
16112
16121

We can find the numbers in the file with the largest distribution as follows

bash ~$ sort -n numbers.txt | \
          uniq -c | \
          sort -rn | \
          head
141 2
 69 1685
 59 1
 53 2950
 11 1902
  4 2870
  4 2132
  3 9151
  3 4345
  3 1796

Where we first sort the contents of the file, using -n to sort them numerically, then pipe that output into the uniq command with the -c option to count the unique values, then sort again, this time with -rn for reverse numeric order, and finally take the first 10 entries in the output (10 is the default number of lines that head will return.)