Concatenating Commands with Pipes and Substitution

Motivation

In the last Example we have seen how we can redirect the input and output of commands from and to files. Now often enough we only produce output of one command for the purpose of feeding it to another program and we are not interested in the intermediate outputs.

In other instances we have the output of one program that we want to substitute on the command line as arguments to an other program.

With these tools we are extremely flexible and can create powerful commands.

Pipes

If we want the output of one program as the input of another we can put a | vertial bar (in this context called pipe symbol) in between them:

$ ls -l | wc -l 

So the above example would run the ls -l command that lists the files and directories in our current directory and then count the lines. So we know how many of those are there.

$ find . | grep -i doc | grep -v cache 

The find . would list all files and directories in all sub-directories form the current directory on (the dot denotes the current directory). Then the output is passed to the grep command. Grep filters out only lines that correspond to that string (actually it can do regular-expression) and only lets those through. The -i says that it should match case-insensitive. Finally the output is passed to another grep command: This time with the option -v wich inverts the match: That filters out all matching lines. In our case any file or directory name that has the word cache in it.

One specially useful tool for using with pipe is less. less is the successor of more. If the output of a command is very long and would scroll over several pages you want to page through it one page at a time.

$ find /usr/bin/ | less

The find lists all the files and directories below /usr/bin which is usually a very long list. With less you can conveniently scroll through the output. Pressing the space bar brings you to the next page. By pressing q you exit the program. Less has many more functions. See: less

Here is a list of a few commands that are useful in pipes:

$ sort     # sort lines alphabetically
$ sort -u  # unique - avoid duplicates
$ sort -n  # sort numerically - the first part of the line is interpreted as number
$ cut -b 7-15  # only display the characters 7 to 15 from each line
$ cut -f 3,5 -d ","  # display fields 3 and 5. the fields of each line are separated with , 
$ rev      # revert each line from back to start
$ grep bla # filter out lines that contain "bla"
$ wc -l    # count lines
$ uniq     # avoid duplicates (assumes already sorted with sort)
$ uniq -u  # only display unique lines. 
$ tr x u   # replaces each x with an u
$ tr A-Z a-z # converts uppercase to lowercase

Substituting the output of one command in the command line of another

Sometimes we want the output of one command not as input but as argument to an other command. This can be done in most shells by using backticks ` command ` or in bash the more readable $( command ). I will use the $( command ) syntax here.

$ date +%Y-%m-%d_%H%M
2020-03-21_18:34

The date -I prints out the date in this format. Now lets assume you have a webcam attached to your computer and you have the fswebcam then you can give a filename and it fetches an image from the webcam and stores it in that file. Now you want to fetch an image every minute and store it in a filename that comes from the output of your date program. So you could run:

$ fswebcam $(date +%Y-%m-%d_%H%M).jpg

Exercises

• Play around with the examples above.

• A riddle for you: Lets assume you organize a conference and you have a list of about 1000 participants who have registered. The list is in a file reg.txt and has one email address in each line. Also you have an other file with the name unreg.txt of people who have later canceled their registration (so their address is still in reg.txt) Now you want to create third file with the name conf.txt where all addresses from reg.txt are stored if they have not unregistered (so their email does not show up in unreg.txt). Now think of a way to generate the conf.txt with only using the sort and uniq commands listed above.