(Note that the parentheses are part of the syntax; you type them literally.) Here, generate_data and generate_more_data represent arbitrary commands, including pipelines, that produce streams of data. The awk program processes each stream in turn, not realizing that the data is coming from multiple sources. This is shown graphically in Figure 7-1.a.

Figure 7-1. Process substitution for both input and output data streams

Process substitution may also be used for output, particularly when combined with the tee(1) program, which sends its input to multiple output files and to standard output. For example:

generate_data | tee >(sort | uniq > sorted_data) \
                    >(mail -s 'raw data' joe) > raw_data

This command uses tee to (1) send the data to a pipeline that sorts and saves the data, (2) send the data to the mail program to user joe, and (3) redirect the original data into a file. This is represented graphically in Figure 7-1.b. Process substitution, combined with tee, allows you to create nonlinear data graphs, freeing you from the straight "one input, one output" paradigm of traditional Unix pipes.

Process substitution is only available on Unix systems that support the /dev/fd/N special files for named access to already open file descriptors. (This is different from the use of /dev/fd/N described earlier in this chapter, where the shell itself interprets the pathname. Here, because external commands must be able to open files in /dev/fd, the feature must be directly supported by the operating system.) Most modern Unix systems, including GNU/Linux, support this feature. Like brace substitution, it must be enabled at compile time, and may not be available in your version of ksh. As with brace expansion, it is enabled by default when ksh93 is compiled from source code.

We've touched upon command-line processing (see Figure 7-2) throughout this book; now is a good time to make the whole thing explicit.[101] Each line that the shell reads from the standard input or a script is called a pipeline; it contains one or more commands separated by zero or more pipe characters (|). For each pipeline it reads, the shell breaks it up into commands, sets up the I/O for the pipeline, and then does the following for each command:

[101] Even this explanation is slightly simplified to elide the most petty details, e.g., "middles" and "ends" of compound commands, special characters within [[...]] and ((...)) constructs, etc. The last word on this subject is the reference book The New KornShell Command and Programming Language by Morris Bolsky and David Korn, published by Prentice-Hall.

Figure 7-2. Steps in command-line processing

1. Splits the command into tokens that are separated by the fixed set of metacharacters: space, TAB, newline, ;, (, ), <, >, |, and &. Types of tokens include words, keywords, I/O redirectors, and semicolons.

2. Checks the first token of each command to see if it is a keyword with no quotes or backslashes. If it's an opening keyword (if and other control-structure openers, function, {, (, ((, or [[), the command is actually a compound command. The shell sets things up internally for the compound command, reads the next command, and starts the process again. If the keyword isn't a compound command opener (e.g., is a control-structure "middle" like then, else, or do, an "end" like fi or done, or a logical operator), the shell signals a syntax error.

3. Checks the first word of each command against the list of aliases. If a match is found, it substitutes the alias's definition and goes back to Step 1; otherwise, it goes on to Step 4. This scheme allows recursive aliases; see Chapter 3. It also allows aliases for keywords to be defined, e.g., alias aslongas=while or alias procedure=function.

4. Substitutes the user's home directory ($HOME) for the tilde character (~) if it is at the beginning of a word. Substitutes user's home directory for ~user.[102]

   [102] Two obscure variations on this: the shell substitutes the current directory ($PWD) for ~+ and the previous directory ($OLDPWD) for ~-.

   Tilde substitution occurs at the following places:

   • As the first unquoted character of a word on the command line

   • After the = in a variable assignment and after any : in the value of a variable assignment

   • For the word part of variable substitutions of the form ${variable op word} (see Chapter 4)

5. Performs parameter (variable) substitution for any expression that starts with a dollar sign ($).

6. Does command substitution for any expression of the form $(string) or `string`.

7. Evaluates arithmetic expressions of the form $((string)).

8. Performs process substitution, if that feature is compiled into the shell and your system supports /dev/fd.

9. Performs brace expansion, if that feature is compiled into the shell.

10. Takes the parts of the line that resulted from parameter, command, and arithmetic substitution and splits them into words again. This time it uses the characters in $IFS as delimiters instead of the set of metacharacters in Step 1.

    Normally, successive multiple input occurrences of characters in IFS act as a single delimiter, which is what you would expect. This is true only for whitespace characters, such as space and TAB. For non-whitespace characters, this is not true. For example, when reading the colon-separated fields of /etc/passwd, two successive colons delimit an empty field. For example:

    IFS=:
    while read name passwd uid gid fullname homedir shell
    do
    

         ...
    done < /etc/passwd
    

    To get this behavior with whitespace-delimited fields (for example, where TAB characters delimit each field), put two successive instances of the delimiter character into IFS.

    ksh ignores any inherited (environment) value of IFS. Upon startup, it sets the value of IFS to the default of space, TAB, and newline.

11. Performs filename generation, a.k.a. wildcard expansion, for any occurrences of *, ?, and [ ] pairs. It also processes the regular expression operators that we saw in Chapter 4.

12. Uses the first word as a command by looking up its location according to the rest of the list in Chapter 4, i.e., as a special built-in command, then as a function, then as a regular built-in command, and finally as a file in any of the directories in $PATH.

13. Runs the command after setting up I/O redirection and other such things.

That's a lot of steps -- and it's not even the whole story! But before we go on, an example should make this process clearer. Assume that the following command has been run:

Although this list of steps is fairly straightforward, it is not the whole story. There are still two ways to subvert the process: by quoting, and by using the advanced command eval.

You can think of quoting as a way of getting the shell to skip some of the 13 steps above. In particular:

• Single quotes ('...') bypass everything through Step 11, including aliasing. All characters inside a pair of single quotes are untouched. You can't have single quotes inside single quotes, even if you precede them with backslashes.[103]

  [103] However, as we saw in Chapter 1, '\'' (i.e., single quote, backslash, single quote, single quote) acts pretty much like a single quote in the middle of a single-quoted string; e.g., 'abc'\''def' evaluates to abc'def.

• Double quotes ("...") bypass steps 1 through 4, plus steps 8 through 11. That is, they ignore pipe characters, aliases, tilde substitution, wildcard expansion, process substitution, brace expansion, and splitting into words via delimiters (e.g., spaces) inside the double quotes. Single quotes inside double quotes have no effect. But double quotes do allow parameter substitution, command substitution, and arithmetic expression evaluation. You can include a double quote inside a double-quoted string by preceding it with a backslash (\). You must also backslash-escape $, ` (the archaic command substitution delimiter), and \ itself.

Table 7-8 contains some simple examples that show how these work; they assume the statement dave=bob was run and user fred's home directory is /home/fred.

If you are wondering whether to use single or double quotes in a particular shell programming situation, it is safest to use single quotes unless you specifically need parameter, command, or arithmetic substitution.

Task 7-5 is a more advanced example of command-line processing that should give you deeper insight into the overall process.

Recall from Chapter 4 that we found a simple way to set up the prompt string PS1 so that it always contains the current directory: PS1='($PWD)-> '.

One problem with this setup is that the resulting prompt strings can get very long. One way to shorten them is to substitute tilde notation for users' home directories. This cannot be done with a simple string expression analogous to the above. The solution is somewhat complicated and takes advantage of the command-line processing rules.

The basic idea is to create a "wrapper" around the cd command, as we did in Chapter 5, that installs the current directory with tilde notation as the prompt string. We will see how to make this wrapper function shortly. The code we need to insert tilde notation is complicated in its own right; we develop it first.

We start with a function that, given a pathname as argument, prints its equivalent in tilde notation if possible. In order to write this function, we assume that we already have an associative array named tilde_ids, in which the subscripts are home directories and the values are user names. Thus, print ${tilde_ids[/home/arnold]} would print the value arnold. Here's the function, named tildize:

function tildize {
    # subdir of our home directory
    if [[ $1 == $HOME* ]]; then
        print "\~${1#$HOME}"
        return 0
    fi

    # loop over homedirs trying to match current dir
    typeset homedir
    for homedir in ${!tilde_ids[*]}; do
        if [[ $1 == ${homedir}?(/*) ]]; then
            print "\~${tilde_ids[$homedir]}${1#$homedir}"
            return 0
        fi
    done
    print "$1"
    return 1
}

The first if clause checks if the given pathname is under the user's home directory. If so, it substitutes tilde (~) for the home directory in the pathname and returns.

If not, we loop over all the subscripts in tilde_ids, comparing each one to our current directory. The test matches home directories by themselves or with some other directory appended (the ?(/*) part.) If a user's home directory is found, ~user is substituted for the full home directory in the given pathname, the result is printed, and the function exits.

Finally, if the for loop exhausts all users without finding a home directory that is a prefix of the given pathname, tildize simply echoes back its input.

Now, how do we create the tilde_ids array? We use the function init_tilde_db. It should be called once, from the .profile file when we log in. The tilde_ids array must be explicitly declared as an associative array using typeset -A:

# tilde_ids[] is global associative array
# mapping directories to user names
typeset -A tilde_ids

function init_tilde_db {
    typeset user homedir    # local vars
    awk -F: '{ print $1, $6 }' /etc/passwd |
        while read user homedir; do
            if [[ $homedir != / ]]; then
                tilde_ids[$homedir]=$user
            fi
        done
}

We use the awk utility to extract the first and sixth fields of the file /etc/passwd, which contain user IDs and home directories, respectively.[104] In this case, awk acts like cut. The -F: is analogous to -d:, which we saw in Chapter 4, except that awk prints the values on each line separated by spaces, not colons (:).

[104] In large multi-machine environments, you may need to use something like ypcat passwd | awk ... or niscat passwd.org_dir | awk ... to get the same information. Check with your system administrator.

awk's output is fed into a while loop that checks the pathname given as argument to see if it contains some user's home directory. (The conditional expression eliminates "users" like daemon and root, whose home directories are root and therefore are contained in every full pathname.)

In fact, you'd be right.[105] But there's a hidden cost here. The function is run every time that the shell prints the prompt. Even if all you do is hit ENTER, the shell runs the tildize function. If there are lots of users on your system, the shell loops through all of the home directories, each time. To avoid this, we write a cd function that only updates the prompt when we actually change directories. The following code should go into your .profile or environment file, along with the definition of tilde_ids and tildize:

[105] This doesn't work in ksh88, though.

As we saw in Chapter 5, writing a function with the same name as a built-in command looks pretty strange at first glance. But, following the POSIX standard, the Korn shell distinguishes between "special" built-in commands and regular built-in commands. When the shell looks for commands to execute, it finds functions before it finds regular built-in commands. cd is a regular built-in command, so this works. Within the function, we use the the cleverly named command command to actually get at the real cd command.[106] The statement command cd "$@" passes the function's arguments on to the real cd in order to change the directory. (As a side note, the shell defines an alias command='command ', which allows you to use command with aliases.)

[106] As mentioned earlier, command is not a special built-in. Woe be to the shell programmer who writes a function named command!

When you log in, this code sets PS1 to the initial current directory (presumably your home directory). Then, whenever you enter a cd command, the function runs to change the directory and reset the prompt.

Of course, the function tildize can be any code that formats the directory string. See the exercises at the end of this chapter for a couple of suggestions.

We have seen that quoting lets you skip steps in command-line processing. Then there's the eval command, which lets you go through the process again. Performing command-line processing twice may seem strange, but it's actually very powerful: it lets you write scripts that create command strings on the fly and then pass them to the shell for execution. This means that you can give scripts "intelligence" to modify their own behavior as they are running.

Now consider eval $listpage instead of just $listpage. When the shell gets to the last step, it runs the command eval with arguments ls, |, and more. This causes the shell to go back to Step 1 with a line that consists of these arguments. It finds | in Step 2 and splits the line into two commands, ls and more. Each command is processed in the normal (and in both cases trivial) way. The result is a paginated list of the files in your current directory.

Now you may start to see how powerful eval can be. It is an advanced feature that requires considerable programming cleverness to be used most effectively. It even has a bit of the flavor of artificial intelligence, in that it enables you to write programs that can "write" and execute other programs.[109] You probably won't use eval for everyday shell programming, but it's worth taking the time to understand what it can do.

[109] You could actually do this without eval, by printing commands to a temporary file and then "sourcing" that file with . filename. But that is much less efficient.

As a more interesting example, we'll revisit Task 4-1, the very first task in the book. In it, we constructed a simple pipeline that sorts a file and prints out the first N lines, where N defaults to 10. The resulting pipeline was:

sort -nr $1 | head -${2:-10}

The first argument specifies the file to sort; $2 is the number of lines to print.

Now suppose we change the task just a bit so that the default is to print the entire file instead of 10 lines. This means that we don't want to use head at all in the default case. We could do this in the following way:

if [[ -n $2 ]]; then
    sort -nr $1 | head -$2
else
    sort -nr $1
fi

In other words, we decide which pipeline to run according to whether or not $2 is null. But here is a more compact solution:

eval sort -nr \$1 ${2:+"| head -\$2"}

The last expression in this line evaluates to the string | head -\$2 if $2 exists (is not null); if $2 is null, then the expression is null too. We backslash-escape dollar signs (\$) before variable names to prevent unpredictable results if the variables' values contain special characters like > or |. The backslash effectively puts off the variables' evaluation until the eval command itself runs. So the entire line is either:

eval sort -nr \$1 | head -\$2

if $2 is given or:

eval sort -nr \$1

if $2 is null. Once again, we can't just run this command without eval because the pipe is "uncovered" after the shell tries to break the line up into commands. eval causes the shell to run the correct pipeline when $2 is given.

Next, we'll revisit Task 7-3 from earlier in this chapter, the start function that lets you start a command in the background and save its standard output and standard error in a logfile. Recall that the one-line solution to this task had the restriction that the command could not contain output redirectors or pipes. Although the former doesn't make sense when you think about it, you certainly would want the ability to start a pipeline in this way.

eval is the obvious way to solve this problem:

function start {
    eval "$@" > logfile 2>&1 &
}

The only restriction that this imposes on the user is that pipes and other such special characters must be quoted (surrounded by quotes or preceded by backslashes).

Task 7-6 is a way to apply eval in conjunction with various other interesting shell programming concepts.

How does make do this? Simple: it compares the modification times of the input and output files (called sources and targets in make terminology), and if the input file is newer, make reprocesses it.

Now suppose that we write a shell function called makecmd that reads and executes a single construct of this form. Assume that the makefile is read from standard input. The function would look like the following code.

This function reads the line with the target and sources; the variable colon is just a placeholder for the :. Then it checks each source to see if it's newer than the target, using the -nt file attribute test operator that we saw in Chapter 5. If the source is newer, it reads, prints, and executes the commands until it finds a line that doesn't start with a TAB or it reaches end-of-file. (The real make does more than this; see the exercises at the end of this chapter.) After running the commands, it breaks out of the for loop, so that it doesn't run the commands more than once. (It isn't necessary to strip the initial TAB from the command. The shell discards the leading whitespace automatically.)