14.2.1. Debuggers

Several interactive debuggers are available for Linux. The de facto standard debugger is gdb, which we just covered in detail.

14.2.2. Profiling and Performance Tools

There are several utilities out there that allow you to monitor and rate the performance of your program. These tools help you locate bottlenecks in your code — places where performance is lacking. These tools also give you a rundown on the call structure of your program, indicating what functions are called, from where, and how often (in other words, everything you ever wanted to know about your program, but were afraid to ask).

gprof is a profiling utility that gives you a detailed listing of the running statistics for your program, including how often each function was called, from where, the total amount of time that each function required, and so forth.

In order to use gprof with a program, you must compile the program using the -pg option with gcc. This adds profiling information to the object file and links the executable with standard libraries that have profiling information enabled.

Having compiled the program to profile with -pg, simply run it. If it exits normally, the file gmon.out will be written to the working directory of the program. This file contains profiling information for that run and can be used with gprof to display a table of statistics.

As an example, let's take a program called getstat, which gathers statistics about an image file. After compiling getstat with -pg, and run it:

papaya$ getstat image11.pgm > stats.dat 
papaya$ ls -l gmon.out 
-rw-------   1 mdw      mdw         54448 Feb  5 17:00 gmon.out 
papaya$

Indeed, the profiling information was written to gmon.out.

To examine the profiling data, we run gprof and give it the name of the executable and the profiling file gmon.out:

papaya$ gprof getstat gmon.out

If you do not specify the name of the profiling file, gprof assumes the name gmon.out. It also assumes the executable name a.out if you do not specify that, either.

gprof output is rather verbose, so you may want to redirect it to a file or pipe it through a pager. It comes in two parts. The first part is the "flat profile," which gives a one-line entry for each function, listing the percentage of time spent in that function, the time (in seconds) used to execute that function, the number of calls to the function, and other information. For example:

Each sample counts as 0.01 seconds. 
  %   cumulative   self              self     total            
 time   seconds   seconds    calls  ms/call  ms/call  name     
 45.11     27.49    27.49       41   670.51   903.13  GetComponent 
 16.25     37.40     9.91                             mcount 
 10.72     43.93     6.54  1811863     0.00     0.00  Push 
 10.33     50.23     6.30  1811863     0.00     0.00  Pop 
  5.87     53.81     3.58       40    89.50   247.06  stackstats 
  4.92     56.81     3.00  1811863     0.00     0.00  TrimNeighbors

If any of the fields are blank in the output, gprof was unable to determine any further information about that function. This is usually caused by parts of the code that were not compiled with the -pg option; for example, if you call routines in nonstandard libraries that haven't been compiled with -pg, gprof won't be able to gather much information about those routines. In the previous output, the function mcount probably hasn't been compiled with profiling enabled.

[53]Always a possibility where this author's code is concerned!

The second part of the gprof report can help us here. It gives a detailed "call graph" describing which functions called other functions and how many times they were called. For example:

index % time    self  children    called     name 
                                                 <spontaneous> 
[1]     92.7    0.00   47.30                 start [1] 
                0.01   47.29       1/1           main [2] 
                0.00    0.00       1/2           on_exit [53] 
                0.00    0.00       1/1           exit [172]

The first column of the call graph is the index: a unique number given to every function, allowing you to find other functions in the graph. Here, the first function, start, is called implicitly when the program begins. start required 92.7% of the total running time (47.30 seconds), including its children, but required very little time to run itself. This is because start is the parent of all other functions in the program, including main; it makes sense that start plus its children require that percentage of time.

The call graph normally displays the children as well as the parents of each function in the graph. Here, we can see that start called the functions main, on_exit, and exit (listed below the line for start). However, there are no parents (normally listed above start); instead, we see the ominous word <spontaneous>. This means that gprof was unable to determine the parent function of start; more than likely because start was not called from within the program itself but kicked off by the operating system.

Skipping down to the entry for GetComponent, or function-under-suspect, we see the following:

index % time    self  children    called     name 
                0.67    0.23       1/41          GetFirstComponent [12] 
               26.82    9.30      40/41          GetNextComponent [5] 
[4]     72.6   27.49    9.54      41         GetComponent [4] 
                6.54    0.00 1811863/1811863     Push [7] 
                3.00    0.00 1811863/1811863     TrimNeighbors [9] 
                0.00    0.00       1/1           InitStack [54]

The parent functions of GetComponent were GetFirstComponent and GetNextComponent, and its children were Push, TrimNeighbors, and InitStack. As we can see, GetComponent was called 41 times — one time from GetFirstComponent and 40 times from GetNextComponent. The gprof output contains notes that describe the report in more detail.

GetComponent itself requires more than 27.49 seconds to run; only 9.54 seconds are spent executing the children of GetComponent (including the many calls to Push and TrimNeighbors!). So it looks as though GetComponent and possibly its parent GetNextComponent need some tuning; the oft-called Push function is not the sole cause of the problem.

gprof also keeps track of recursive calls and "cycles" of called functions and indicates the amount of time required for each call. Of course, using gprof effectively requires that all code to be profiled is compiled with the -pg option. It also requires a knowledge of the program you're attempting to profile; gprof can tell you only so much about what's going on. It's up to the programmer to optimize inefficient code.

One last note about gprof: running it on a program that calls only a few functions — and runs very quickly — may not give you meaningful results. The units used for timing execution are usually rather coarse — maybe one one-hundredth of a second — and if many functions in your program run more quickly than that, gprof will be unable to distinguish between their respective running times (rounding them to the nearest hundredth of a second). In order to get good profiling information, you may need to run your program under unusual circumstances — for example, giving it an unusually large data set to churn on, as in the previous example.

Use of calls is simple: you tell it the names of the source files to map out, and a function-call tree is displayed. For example:

papaya$ calls scan.c 
    1   level1 [scan.c] 
    2           getid [scan.c] 
    3                   getc 
    4                   eatwhite [scan.c] 
    5                           getc 
    6                           ungetc 
    7                   strcmp 
    8           eatwhite [see line 4] 
    9           balance [scan.c] 
   10                   eatwhite [see line 4]

By default, calls lists only one instance of each called function at each level of the tree (so that if printf is called five times in a given function, it is listed only once). The -a switch prints all instances. calls has several other options as well; using calls -h gives you a summary.

14.2.3. Using strace

14.2.4. Using Valgrind

Valgrind is a replacement for the various memory-allocation routines, such as malloc, realloc, and free, used by C programs, but it also supports C++ programs. It provides smarter memory-allocation procedures and code to detect illegal memory accesses and common faults, such as attempting to free a block of memory more than once. Valgrind displays detailed error messages if your program attempts any kind of hazardous memory access, helping you to catch segmentation faults in your program before they happen. It can also detect memory leaks — for example, places in the code where new memory is malloc'd without being free'd after use.

Valgrind is not just a replacement for malloc and friends. It also inserts code into your program to verify all memory reads and writes. It is very robust and therefore considerably slower than the regular malloc routines. Valgrind is meant to be used during program development and testing; once all potential memory-corrupting bugs have been fixed, you can run your program without it.

For example, take the following program, which allocates some memory and attempts to do various nasty things with it:

#include <malloc.h> 
int main( ) { 
  char *thememory, ch; 
 
  thememory=(char *)malloc(10*sizeof(char)); 
   
  ch=thememory[1];     /* Attempt to read uninitialized memory */ 
  thememory[12]=' ';   /* Attempt to write after the block */ 
  ch=thememory[-2];    /* Attempt to read before the block */ 
}

To find these errors, we simply compile the program for debugging and run it by prepending the valgrind command to the command line:

owl$ gcc -g -o nasty nasty.c
owl$ valgrind nasty
=  =18037=  = valgrind-20020319, a memory error detector for x86 GNU/Linux.
=  =18037=  = Copyright (C) 2000-2002, and GNU GPL'd, by Julian Seward.
=  =18037=  = For more details, rerun with: -v
=  =18037=  =
=  =18037=  = Invalid write of size 1
=  =18037=  =    at 0x8048487: main (nasty.c:8)
=  =18037=  =    by 0x402D67EE: _ _libc_start_main (in /lib/libc.so.6)
=  =18037=  =    by 0x8048381: _ _libc_start_main@@GLIBC_2.0 (in /home/kalle/tmp/nasty)
=  =18037=  =    by <bogus frame pointer> ???
=  =18037=  =    Address 0x41B2A030 is 2 bytes after a block of size 10 alloc'd
=  =18037=  =    at 0x40065CFB: malloc (vg_clientmalloc.c:618)
=  =18037=  =    by 0x8048470: main (nasty.c:5)
=  =18037=  =    by 0x402D67EE: _ _libc_start_main (in /lib/libc.so.6)
=  =18037=  =    by 0x8048381: _ _libc_start_main@@GLIBC_2.0 (in /home/kalle/tmp/nasty)
=  =18037=  =
=  =18037=  = Invalid read of size 1
=  =18037=  =    at 0x804848D: main (nasty.c:9)
=  =18037=  =    by 0x402D67EE: _ _libc_start_main (in /lib/libc.so.6)
=  =18037=  =    by 0x8048381: _ _libc_start_main@@GLIBC_2.0 (in /home/kalle/tmp/nasty)
=  =18037=  =    by <bogus frame pointer> ???
=  =18037=  =    Address 0x41B2A022 is 2 bytes before a block of size 10 alloc'd
=  =18037=  =    at 0x40065CFB: malloc (vg_clientmalloc.c:618)
=  =18037=  =    by 0x8048470: main (nasty.c:5)
=  =18037=  =    by 0x402D67EE: _ _libc_start_main (in /lib/libc.so.6)
=  =18037=  =    by 0x8048381: _ _libc_start_main@@GLIBC_2.0 (in /home/kalle/tmp/nasty)
=  =18037=  =
=  =18037=  = ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)
=  =18037=  = malloc/free: in use at exit: 10 bytes in 1 blocks.
=  =18037=  = malloc/free: 1 allocs, 0 frees, 10 bytes allocated.
=  =18037=  = For a detailed leak analysis,  rerun with: --leak-check=yes
=  =18037=  = For counts of detected errors, rerun with: -v

For each memory violation, Valgrind reports an error and gives us information on what happened. The actual Valgrind error messages include information on where the program is executing as well as where the memory block was allocated. You can coax even more information out of Valgrind if you wish, and, along with a debugger such as gdb, you can pinpoint problems easily.

You may ask why the reading operation in line 7, where an initialized piece of memory is read has not led Valgrind to emit an error message. This is because Valgrind won't complain if you pass around initialized memory, but it still keeps track of it. As soon as you use the value (e.g., by passing it to an operating system function or by manipulating it), you receive the expected error message.

owl$ valgrind --leak-check=yes --show-reachable=yes nasty
...
=  =18081=  = ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)
=  =18081=  = malloc/free: in use at exit: 10 bytes in 1 blocks.
=  =18081=  = malloc/free: 1 allocs, 0 frees, 10 bytes allocated.
=  =18081=  = For counts of detected errors, rerun with: -v
=  =18081=  = searching for pointers to 1 not-freed blocks.
=  =18081=  = checked 4029376 bytes.
=  =18081=  =
=  =18081=  = definitely lost: 0 bytes in 0 blocks.
=  =18081=  = possibly lost:   0 bytes in 0 blocks.
=  =18081=  = still reachable: 10 bytes in 1 blocks.
=  =18081=  =
=  =18081=  = 10 bytes in 1 blocks are still reachable in loss record 1 of 1
=  =18081=  =    at 0x40065CFB: malloc (vg_clientmalloc.c:618)
=  =18081=  =    by 0x8048470: main (nasty.c:5)
=  =18081=  =    by 0x402D67EE: _ _libc_start_main (in /lib/libc.so.6)
=  =18081=  =    by 0x8048381: _ _libc_start_main@@GLIBC_2.0 (in /home/kalle/tmp/nasty)
=  =18081=  =
=  =18081=  = LEAK SUMMARY:
=  =18081=  =    possibly lost:   0 bytes in 0 blocks.
=  =18081=  =    definitely lost: 0 bytes in 0 blocks.
=  =18081=  =    still reachable: 10 bytes in 1 blocks.
=  =18081=  =

14.2.5. Interface Building Tools

A number of applications and libraries let you easily generate a user interface for your applications under the X Window System. If you do not want to bother with the complexity of the X programming interface, using one of these simple interface-building tools may be the answer for you. There are also tools for producing a text-based interface for programs that don't require X.

The classic X programming model has attempted to be as general as possible, providing only the bare minimum of interface restrictions and assumptions. This generality allows programmers to build their own interface from scratch, as the core X libraries don't make any assumptions about the interface in advance. The X Toolkit Intrinsics (Xt) provides a rudimentary set of interface widgets (such as simple buttons, scrollbars, and the like), as well as a general interface for writing your own widgets if necessary. Unfortunately this can require a great deal of work for programmers who would rather use a set of premade interface routines. A number of Xt widget sets and programming libraries are available for Linux, all of which make the user interface easier to program.

Before you start developing with XView or Motif, a word of caution is in order. XView, which was once a commercial product of Sun Microsystems, has been dropped by the developers and is no longer maintained. Also, while some people like the look, the programs written with XView look very nonstandard. Motif, on the other hand, is still being actively developed (albeit rather slowly), but it also has some problems. First, programming with Motif can be frustrating. It is difficult, error-prone, and cumbersome since the Motif API was not designed according to modern GUI API design principles. Also, Motif programs tend to run very slowly. For these reasons, you might want to consider one of the following:

Many programmers are finding that building a user interface, even with a complete set of widgets and routines in C, requires much overhead and can be quite difficult. This is a question of flexibility versus ease of programming: the easier the interface is to build, the less control the programmer has over it. Many programmers are finding that prebuilt widgets are adequate enough for their needs, so the loss in flexibility is not a problem.

One of the problems with interface generation and X programming is that it is difficult to generalize the most widely used elements of a user interface into a simple programming model. For example, many programs use features such as buttons, dialog boxes, pull-down menus, and so forth, but almost every program uses these widgets in a different context. In simplifying the creation of a graphical interface, generators tend to make assumptions about what you'll want. For example, it is simple enough to specify that a button, when pressed, should execute a certain procedure within your program, but what if you want the button to execute some specialized behavior the programming interface does not allow for? For example, what if you wanted the button to have a different effect when pressed with mouse button 2 instead of mouse button 1? If the interface-building system does not allow for this degree of generality, it is not of much use to programmers who need a powerful, customized interface.

14.2.6. Revision Control Tools — RCS

Why bother with revision control? Many large projects require some kind of revision control in order to keep track of many tiny complex changes to the system. For example, attempting to maintain a program with a thousand source files and a team of several dozen programmers would be nearly impossible without using something like RCS. With RCS, you can ensure that only one person may modify a given source file at any one time, and all changes are checked in along with a log message detailing the change.

When you check in a file with RCS, revisions are added to the RCS file, and the original file is deleted by default. In order to access the original file, you must check it out from the RCS file. When you're editing a file, you generally don't want someone else to be able to edit it at the same time. Therefore, RCS places a lock on the file when you check it out for editing. Only you, the person who checked out this locked file, can modify it (this is accomplished through file permissions). Once you're done making changes to the source, you check it back in, which allows anyone working on the project to check it back out again for further work. Checking out a file as unlocked does not subject it to these restrictions; generally, files are checked out as locked only when they are to be edited but are checked out as unlocked just for reading (for example, to use the source file in a program build).

RCS automatically keeps track of all previous revisions in the RCS file and assigns incremental version numbers to each new revision that you check in. You can also specify a version number of your own when checking in a file with RCS; this allows you to start a new "revision branch" so that multiple projects can stem from different revisions of the same file. This is a good way to share code between projects but also to assure that changes made to one branch won't be reflected in others.

Here's an example. Take the source file importrtf.c, which contains our friendly program:

#include <stdio.h> 
 
int main(void) { 
  printf("Hello, world!"); 
}

papaya$ ci importrtf.c 
importrtf.c,v  <--  importrtf.c 
enter description, terminated with single '.' or end of file: 
NOTE: This is NOT the log message! 
>> Hello world source code 
>> . 
initial revision: 1.1 
done 
papaya$

The RCS file importrtf.c,v is created, and importrtf.c is removed.

papaya$ co -l importrtf.c 
importrtf.c,v  -->  importrtf.c 
revision 1.1 (locked) 
done 
papaya$

will check out importrtf.c (from importrtf.c,v) and lock it. Locking the file allows you to edit it, and to check it back in. If you only need to check the file out in order to read it (for example, to issue a make), you can leave the -l switch off of the co command to check it out unlocked. You can't check in a file unless it is locked first (or if it has never been checked in before, as in the example).

papaya$ ci -l importrtf.c 
importrtf.c,v  <--  importrtf.c 
new revision: 1.2; previous revision: 1.1 
enter log message, terminated with single '.' or end of file: 
>> Changed printf call 
>> . 
done 
papaya$

This automatically checks out the file, locked, after checking it in. This is a useful way to keep track of revisions even if you're the only one working on a project.

If you use RCS often, you may not like all those unsightly importrtf.c,v RCS files cluttering up your directory. If you create the subdirectory RCS within your project directory, ci and co will place the RCS files there, out of the way from the rest of the source.

In addition, RCS keeps track of all previous revisions of your file. For instance, if you make a change to your program that causes it to break in some way and you want to revert to the previous version to "undo" your changes and retrace your steps, you can specify a particular version number to check out with co. For example:

papaya$ co -l1.1 importrtf.c 
importrtf.c,v  -->  importrtf.c 
revision 1.1 (locked) 
writable importrtf.c exists; remove it? [ny](n): y  
done 
papaya$

checks out version 1.1 of the file importrtf.c. You can use the program rlog to print the revision history of a particular file; this displays your revision log entries (entered with ci) along with other information such as the date, the user who checked in the revision, and so forth.

/* $Header: /work/linux/running4/safarixml/RCS/ch14.xml,v 1.1 2002/09/20 20:51:50 sierra Exp sierra
$ */

in the source file, co will replace it with an informative line about the revision date, version number, and so forth, as in:

/* $Header: /work/linux/hitch/programming/tools/RCS/rcs.tex 
       1.2 1994/12/04 15:19:31 mdw Exp mdw $ */

(We broke this line to fit on the page, but it is supposed to be all on one line.)

Other keywords exist as well, such as $Author: jhawks $, $Date: 2002/12/16 20:28:32 $, and $Log: ch14.xml,v $

Many programmers place a static string within each source file to identify the version of the program after it has been compiled. For example, within each source file in your program, you can place the line:

static char rcsid[  ] = "\@(#)$Header: /work/linux/running4/safarixml/RCS/ch14.xml,v 1.3 2002/09/24 
15:30:14 andrews Exp ssherman $;

papaya$ what importrtf 
importrtf: 
        $Header: /work/linux/hitch/programming/tools/RCS/rcs.tex 
                      1.2 1994/12/04 15:19:31 mdw Exp mdw $ 
papaya$

what picks out strings beginning with the characters @(#) in a file and displays them. If you have a program that has been compiled from many source files and libraries, and you don't know how up-to-date each component is, you can use what to display a version string for each source file used to compile the binary.

14.2.7. Revision Control Tools — CVS

CVS, the Concurrent Versioning System, is more complex than RCS and thus perhaps a little bit oversized for one-person projects. But whenever more than one or two programmers are working on a project or the source code is distributed over several directories, CVS is the better choice. CVS uses the RCS file format for saving changes, but employs a management structure of its own.

By default, CVS works with full directory trees. That is, each CVS command you issue affects the current directory and all the subdirectories it contains, including their subdirectories and so on. You can switch off this recursive traversal with a command-line option, or you can specify a single file for the command to operate on.

When you as a developer have checked out a local directory tree, all the files are writable. You can make any necessary changes to the files in your personal workspace. When you have finished local testing and feel sure enough of your work to share the changes with the rest of the programming team, you write any changed files back into the central repository by issuing a CVS commit command. CVS then checks whether another developer has checked in changes since you checked out your directory tree. If this is the case, CVS does not let you check in your changes, but asks you first to take the changes of the other developers over to your local tree. During this update operation, CVS uses a sophisticated algorithm to reconcile ("merge") your changes with those of the other developers. In cases in which this is not automatically possible, CVS informs you that there were conflicts and asks you to resolve them. The file in question is marked up with special characters so that you can see where the conflict has occurred and decide which version should be used. Note that CVS makes sure conflicts can occur only in local developers' trees. There is always a consistent version in the repository.

14.2.8. Patching Files

patch is a program that makes context-dependent changes in a file in order to update that file from one version to the next. This way, when your program changes, you simply release a patch file against the source, which the user applies with patch to get the newest version. For example, Linus Torvalds usually releases new Linux kernel versions in the form of patch files as well as complete source distributions.

A nice feature of patch is that it applies updates in context; that is, if you have made changes to the source yourself, but still wish to get the changes in the patch file update, patch usually can figure out the right location in your changed file to which to apply the change. This way, your versions of the original source files don't need to correspond exactly to those against which the patch file was made.

/* hello.c version 1.0 by Norbert Ebersol */ 
#include <stdio.h>  
 
int main( ) { 
  printf("Hello, World!"); 
  exit(0); 
}

Let's say you were to update this source, as in the following:

/* hello.c version 2.0 */ 
/* (c)1994 Norbert Ebersol */ 
#include <stdio.h>  
 
int main( ) { 
  printf("Hello, Mother Earth!\n"); 
  return 0; 
}

If you want to produce a patch file to update the original hello.c to the newest version, use diff with the -c option:

papaya$ diff -c hello.c.old hello.c > hello.patch

This produces the patch file hello.patch that describes how to convert the original hello.c (here, saved in the file hello.c.old) to the new version. You can distribute this patch file to anyone who has the original version of "Hello, World," and they can use patch to update it.

[56]The output shown here is from the last version that Larry Wall has released, Version 2.1. If you have a newer version of patch, you will need the -- verbose flag to get the same output.

papaya$ patch < hello.patch 
Hmm...  Looks like a new-style context diff to me... 
The text leading up to this was: 
-------------------------- 
|*** hello.c.old        Sun Feb  6 15:30:52 1994 
|--- hello.c    Sun Feb  6 15:32:21 1994 
-------------------------- 
Patching file hello.c using Plan A... 
Hunk #1 succeeded at 1. 
done 
papaya$

patch warns you if it appears as though the patch has already been applied. If we tried to apply the patch file again, patch would ask us if we wanted to assume that -R was enabled — which reverses the patch. This is a good way to back out patches you didn't intend to apply. patch also saves the original version of each file that it updates in a backup file, usually named filename~ (the filename with a tilde appended).

In many cases, you'll want to update not only a single source file, but also an entire directory tree of sources. patch allows many files to be updated from a single diff. Let's say you have two directory trees, hello.old and hello, which contain the sources for the old and new versions of a program, respectively. To make a patch file for the entire tree, use the -r switch with diff:

papaya$ diff -cr hello.old hello > hello.patch

Now, let's move to the system where the software needs to be updated. Assuming that the original source is contained in the directory hello, you can apply the patch with:

papaya$ patch -p0 < hello.patch

The -p0 switch tells patch to preserve the pathnames of files to be updated (so that it knows to look in the hello directory for the source). If you have the source to be patched saved in a directory named differently from that given in the patch file, you may need to use the -p option without a number. See the patch(1) manual page for details about this.

14.2.9. Indenting Code

Take this terribly formatted source:

double fact (double n) { if (n=  =1) return 1; 
else return (n*fact(n-1)); } 
int main ( ) { 
printf("Factorial 5 is %f.\n",fact(5)); 
printf("Factorial 10 is %f.\n",fact(10)); exit (0); }

Running indent on this source produces the relatively beautiful code:

#include <math.h> 
 
double 
fact (double n) 
{ 
  if (n =  = 1) 
    return 1; 
  else 
    return (n * fact (n - 1)); 
} 
void 
main ( ) 
{ 

  printf ("Factorial 5 is %f.\n", fact (5)); 
  printf ("Factorial 10 is %f.\n", fact (10)); 
  exit (0); 
}

papaya$ indent -troff importrtf.c | groff -mindent

produces troff code and formats it with groff.

Finally, indent can be used as a simple debugging tool. If you have put a } in the wrong place, running your program through indent will show you what the computer thinks the block structure is.