10. Auditing and Logging

Contents:
The Basic Log Files
The acct/pacct Process Accounting File
Program-Specific Log Files
Per-User Trails in the Filesystem
The UNIX System Log (syslog) Facility
Swatch: A Log File Tool
Handwritten Logs
Managing Log Files

After you have established the protection mechanisms on your system, you will need to monitor them. You want to be sure that your protection mechanisms actually work. You will also want to observe any indications of misbehavior or other problems. This process of monitoring the behavior of the system is known as auditing . It is part of a defense-in-depth strategy: to borrow a phrase from several years ago, you should trust, but you should also verify.

UNIX maintains a number of log files that keep track of what's been happening to the computer. Early versions of UNIX used the log files to record who logged in, who logged out, and what they did. Newer versions of UNIX provide expanded logging facilities that record such information as files that are transferred over the network, attempts by users to become the superuser, electronic mail, and much more.

Log files are an important building block of a secure system: they form a recorded history, or audit trail , of your computer's past, making it easier for you to track down intermittent problems or attacks. Using log files, you may be able to piece together enough information to discover the cause of a bug, the source of a break-in, and the scope of the damage involved. In cases where you can't stop damage from occurring, at least you will have some record of it. Those logs may be exactly what you need to rebuild your system, conduct an investigation, give testimony, recover insurance money, or get accurate field service performed.

But beware: Log files also have a fundamental vulnerability. Because they are often recorded on the system itself, they are subject to alteration or deletion. As we shall see, there are techniques that may help you to mitigate this problem, but no technique can completely remove it unless you log to a different machine.

Locating to a different machine is actually a good idea even if your system supports some other techniques to store the logs. Consider some method of automatically sending log files to a system on your network in a physically secured location. For example, sending logging information to a PC or Apple Macintosh provides a way of storing the logs in a machine that is considerably more difficult to break into and disturb. We have heard good reports from people who are able to use "outmoded" 80486 or 80386 PCs as log machines. For a diagram of such a setup, see Figure 10.1 .

Figure 10.1: Secure logging host.

10.1 The Basic Log Files

Most log files are text files that are written line by line by system programs. For example, each time a user on your system tries to become the superuser by using the su command, the su program may append a single line to the log file sulog , which records whether the su attempt was successful or not.

Different versions of UNIX store log files in different directories. The most common locations are:

/usr/adm	Used by early versions of UNIX
/var/adm	Used by newer versions of UNIX , so that the /usr partition can be mounted read-only
/var/log	Used by some versions of Solaris, Linux, BSD , and free BSD to store log files

Within one of these directories (or a subdirectory in one of them) you may find variants of some or all of the following files:

acct or pacct	Records commands run by every user
aculog	Records of dial-out modems (automatic call units)
lastlog	Logs each user's most recent successful login time, and possibly the last unsuccessful login too
loginlog	Records bad login attempts
messages	Records output to the system's "console" and other messages generated from the syslog facility
sulog	Logs use of the su command
utmp [1]	Records each user currently logged in
utmpx	Extended utmp
wtmp [2]	Provides a permanent record of each time a user logged in and logged out. Also records system shutdowns and startups
wtmpx	Extended wtmp
vold.log	Logs errors encountered with the use of external media, such as floppy disks or CD-ROMS
xferlog	Logs FTP access

[1] Most versions of UNIX store the utmp file in the /etc directory .

[2] Early versions of System V UNIX stored the w tmp file in the /etc directory

The following sections describe some of these files and how to use the UNIX syslog facility.

Many UNIX systems allow the administrator to enable a comprehensive type of auditing (logging) known as " C2 audit." This is so-named because it is logging of the form specified by U.S. Department of Defense regulations to meet the certification at the C2 level of trust. These regulations are specified in a document called the Trusted Computer System Evaluation Criteria (often referred to as the "Orange Book" in the "Rainbow Series").

C2 auditing generally means assigning an audit ID to each group of related processes, starting at login. Thereafter, certain forms of system calls performed by every process are logged with the audit ID. This includes calls to open and close files, change directory, alter user process parameters, and so on.

Despite the mandate for the general content of such logging, there is no generally accepted standard for the format. Thus, each vendor that provides C2-style logging seems to have a different format, different controls, and different locations for the logs. If you feel the need to set such logging on your machine, we recommend that you read the documentation carefully. Furthermore, we recommend that you be careful about what you log so as not to generate lots of extraneous information, and that you log to a disk partition with lots of space.

The last suggestion, above, reflects one of the biggest problems with C2 audit: it can consume a huge amount of space on an active system in a short amount of time. The other main problem with C2 audit is that it is useless without some interpretation and reduction tools, and these are not generally available from vendors - the DoD regulations only require that the logging be done, not that it be usable! Vendors have generally provided only as much as is required to meet the regulations and no more.

Only a few third-party companies provide intrusion detection or audit-analysis tools: Stalker, from Haystack Laboratories, is one such product with some sophisticated features. Development of more sophisticated tools is an ongoing, current area of research for many people. We hope to be able to report something more positive in a third edition of this book.

In the meantime, if you are not using one of these products, and you aren't at a DoD site that requires C2 logging, you may not want to enable C2 logging (unless you like filling up your disks with data you may not be able to interpret). On the other hand, if you have a problem, the more logging you have, the more likely you will be able to determine what happened. Therefore, review the documentation for the audit tools provided with your system if it claims C2 audit capabilities, and experiment with them to determine if you want to enable the data collection.

10.1.1 lastlog File

UNIX records the last time that each user logged into the system in the lastlog log file. This time is displayed each time you log in:

login: ti password: 
books2sell
 Last login: Tue Jul 12 07:49:59 on tty01

This time is also reported when the finger command is used:

% finger tim Login name: tim           In real life: Tim Hack Directory: /Users/tim     Shell: /bin/csh Last login Tue Jul 12 07:49:59 on tty01 No unread mail No Plan. %

Some versions of System V UNIX display both the last successful login and the last unsuccessful login when a user logs into the system:

login: tim password: 
books2sell
 Last successful login for tim : Tue Jul 12 07:49:59 on tty01 Last unsuccessful login for tim : Tue Jul 06 09:22:10 on tty01

Teach your users to check the last login time each time they log in. If the displayed time doesn't correspond to the last time they used the system, somebody else might have been using their account. If this happens, the user should immediately change the account's password and notify the system administrator.

Unfortunately, the design of the lastlog mechanism is such that the previous contents of the file are overwritten at each login. As a result, if a user is inattentive for even a moment, or if the login message clears the screen, the user may not notice a suspicious time. Furthermore, even if a suspicious time is noted, it is no longer available for the system administrator to examine.

One way to compensate for this design flaw is to have a cron -spawned task periodically make an on-disk copy of the file that can be examined at a later time. For instance, you could have a shell file run every six hours to do the following:

mv /var/adm/lastlog.3 /var/adm/lastlog.4 mv /var/adm/lastlog.2 /var/adm/lastlog.3 mv /var/adm/lastlog.1 /var/adm/lastlog.2 cp /var/adm/lastlog /var/adm/lastlog.1

This will preserve the contents of the file in six-hour periods. If backups are done every day, then they will also be preserved to the backups for examination later.

If you have saved copies of the lastlog file, you will need a way to read the contents. Unfortunately, there is no utility under standard versions of UNIX that allows you to read one of these files and print all the information. Therefore, you need to write your own. The following Perl script will work on SunOS systems, and you can modify it to work on others.[3]

[3] The layout of the lastlog file is usually documented in an include file such as /usr/include/lastlog.h

Example 10.1: Script that Reads lastlog File.

#!/usr/local/bin/perl $fname = (shift || "/var/adm/lastlog"); $halfyear = 60*60*24*365.2425/2; # pedantry abounds setpwent; while (($name, $junk, $uid) = getpwent) { 		$names{$uid} = $name; } endpwent;  open(LASTL, $fname); for (uid = 0; read(LASTL, $record, 28); $uid++) { 	(	$time, $line, $host) = unpack('l A8 A16', $record); 		next unless $time;  		$host = "($host)" if $host; 		($sec, $min, $hour, $mday, $mon, $year) = localtime($time);     $year += 1900 + ($year < 70 ? 100 : 0); 		print $names{$uid}, $line, "$mday/$mon"; 		print (time - $time > $halfyear) ? "/$year" : "  "$hour:$min"; 		print "   $host\n"; } close LASTL;

This program starts by checking for a command-line argument (the "shift"); if none is present, it uses the default. It then calculates the number of seconds in half a year, to be used to determine the format of the output (logins more than six months in the past are printed differently). Next, it builds an associative array of UIDS to login names.

After this initialization, the program reads a record at a time from the lastlog file. Each binary record is then unpacked and decoded. The stored time is decoded into something more understandable, and then the output is printed.

While the lastlog file is designed to provide quick access to the last time that a person logged into the system, it does not provide a detailed history recording the use of each account. For that, UNIX uses the wtmp log file.

10.1.2 utmp and wtmp Files

UNIX keeps track of who is currently logged into the system with a special file called /etc/utmp . This is a binary file that contains a record for every active tty line, and generally does not grow to be more than a few kilobytes in length (at the most). A second file, /var/adm/wtmp , keeps track of both logins and logouts. This file grows every time a user logs in or logs out, and can grow to be many megabytes in length unless it is pruned.

In Berkeley-derived versions of UNIX , the entries in the utmp and wtmp files contain:

Name of the terminal device used for login
Username
Hostname that the connection originated from, if the login was made over a network
Time that the user logged on

In System V UNIX , the wtmp file is placed in /etc/wtmp and is also used for accounting. The AT&T System V.3.2 utmp and wtmp entries contain:

Username
Terminal line number
Device name
Process ID of the login shell
Code that denotes the type of entry
Exit status of the process
Time that the entry was made

The extended wtmpx file used by Solaris, IRIX and other SVR4 UNIX operating systems, includes the following:

Username (32 characters instead of 8)
inittab id (indicates the type of connection; see Appendix C, UNIX Processes )
Terminal name (32 characters instead of 12)
Device name
Process ID of the login shell
Code that denotes the type of entry
Exit status of the process
Time that the entry was made
Session ID
Unused bytes for future expansions
Remote host name (for logins that originated over a network)

UNIX programs that report the users that are currently logged into the system ( who, whodo, w, users, and finger ), do so by scanning the /etc/utmp file. The write command checks this file to see if a user is currently logged in, and determines which terminal he is.

The last program, which prints a detailed report of the times of the most recent user logins, does so by scanning the wtmp file.

The ps command gives you a more accurate account of who is currently using your system than the who , whodo , users , and finger commands because under some circumstances, users can have processes running without having their usernames appear in the /etc/utmp or /var/adm/wtmp files. (For example, a user may have left a program running and then logged out, or used the rsh command instead of rlogin .)

However, the commands who , users , and finger have several advantages over ps :

They often present their information in a format that is easier to read than the ps output.
They sometimes contain information not present in the ps output, such as the names of remote host origins.
They may run significantly faster than ps .

NOTE: The permissions on the /etc/utmp file are sometimes set to be writable by any user on many systems. This is ostensibly to allow various user programs that create windows or virtual terminals to show the user as "logged" into each window without requiring superuser privileges. Unfortunately, if this file is world-writable, then any user can edit her entry to show a different username or terminal in a who listing - or to show none at all. (She can also edit anybody else's entry to show whatever she wants!) It also is the source of an old security hole that keeps reappearing in various vendors' releases of UNIX : by changing the terminal name in this file to that of a sensitive file, an attacker can get system programs that write to user terminals (such as wall and biff ) to overwrite the target with selected output. This can lead to compromise or damage.

If your system has a mode -rw-rw-rw- /etc/utmp file, we recommend that you change it to remove the write access for nonowners. Then complain to your vendor for not fixing such an old and well-known flaw.

10.1.2.1 su command and /etc/utmp and /var/adm/wtmp files

When you use the su command (see "su: Changing Who You Claim to Be" in Chapter 4, Users, Groups, and the Superuser ), it creates a new process with both the process's real UID and effective UID altered. This gives you the ability to access another user's files, and run programs as the other user.

However, while su does not change your entry in the /etc/utmp or the /var/adm/wtmp files, the finger command will continue to display the account to which you logged in, not the one that you su 'ed to. Many other programs as well - such as mail - may not work properly when used from within a su subshell, as they determine your username from the /etc/utmp entry and not from the real or effective UID .

Note that different versions of the su command have different options available that allow you to reset your environment, run a different command shell, or otherwise modify the default behavior. One common argument is a simple dash, as in " su - user ". This form will cause the shell for user to start up as if it were a login shell.

Thus, the su command should be used with caution. While it is useful for quick tests, because it does not properly update the utmp and wtmp files, it can cause substantial confusion to other users and to some system utilities.

10.1.3 last Program

Every time a user logs in or logs out, UNIX makes a record in the file wtmp . The last program displays the contents of this file in an understandable form.[4] If you run last with no arguments, the command displays all logins and logouts on every device. last will display the entire file; you can abort the display by pressing the interrupt character (usually CTRL-C ):

[4] On some SVR4 systems you can use the " who -a " command to view the contents of the wtmp file Check your documentation to see which command version you would use on your system.

% last dpryor    ttyp3    std.com          Sat Mar 11 12:21 - 12:24  (00:02) simsong   ttyp2    204.17.195.43    Sat Mar 11 11:56 - 11:57  (00:00) simsong   ttyp1    204.17.195.43    Sat Mar 11 11:37   still logged in dpryor    console                   Wed Mar  8 10:47 - 17:41 (2+06:53) devon     console                   Wed Mar  8 10:43 - 10:47  (00:03) simsong   ttyp3    pleasant.cambrid Mon Mar  6 16:27 - 16:28  (00:01) dpryor    ftp      mac4             Fri Mar  3 16:31 - 16:33  (00:02) dpryor    console                   Fri Mar  3 12:01 - 10:43 (4+22:41) simsong   ftp      pleasant.cambrid Fri Mar  3 08:40 - 08:56  (00:15) simsong   ttyp2    pleasant.cambrid Thu Mar  2 20:08 - 21:08  (00:59) ...

In this display, you can see that five login sessions have been active since March 7th: simsong , dpryor , devon , dpyror (again), and simsong (again). Two of the users ( dpryor and devon ) logged on to the computer console. The main user of this machine is probably the user dpryor (in fact, this computer is a workstation sitting on dpryor 's desk.) The terminal name ftp indicates that dpryor was logged in for FTP file transfer. Other terminal names may also appear here, depending on your system type and configuration; for instance, you might have an entry showing pc-nfs as an entry type.

The last command allows you to specify a username or a terminal as an argument to prune the amount of information displayed. If you provide a username, last displays logins and logouts only for that user. If you provide a terminal name, last displays logins and logouts only for the specified terminal:

% last dpryor dpryor    ttyp3    std.com          Sat Mar 11 12:21 - 12:24  (00:02) dpryor    console                   Wed Mar  8 10:47 - 17:41 (2+06:53) dpryor    ftp      mac4             Fri Mar  3 16:31 - 16:33  (00:02) dpryor    console                   Fri Mar  3 12:01 - 10:43 (4+22:41) dpryor    ftp      mac4             Mon Feb 27 10:43 - 10:45  (00:01) dpryor    ttyp6    std.com          Sun Feb 26 01:12 - 01:13  (00:01) dpryor    ftp      mac4             Thu Feb 23 14:42 - 14:43  (00:01) dpryor    ftp      mac4             Thu Feb 23 14:20 - 14:25  (00:04) dpryor    ttyp3    mac4             Wed Feb 22 13:04 - 13:06  (00:02) dpryor    console                   Tue Feb 21 09:57 - 12:01 (10+02:04)

You may wish to issue the last command every morning to see if there were unexpected logins during the previous night.

On some systems, the wtmp file also logs shutdowns and reboots.

10.1.3.1 Pruning the wtmp file

The wtmp file will continue to grow until you have no space left on your computer's hard disk. For this reason, many vendors include shell scripts with their UNIX releases that zero the wtmp file automatically on a regular basis (such as once a week or once a month). These scripts are run automatically by the cron program.

For example, many monthly shell scripts contain a statement that looks like this:

# zero the log file cat /dev/null >/var/adm/wtmp

Instead of this simple-minded approach, you may wish to make a copy of the wtmp file first, so you'll be able to refer to logins in the previous month. To do so, you must locate the shell script that zeros your log file and add the following lines:

# make a copy of the log file and zero the old one rm /var/adm/wtmp.old ln /var/adm/wtmp /var/adm/wtmp.old cp /dev/null /var/adm/wtmp.nul mv /var/adm/wtmp.nul /var/adm/wtmp

Most versions of the last command allow you to specify a file to use other than wtmp by using the -f option. For example:

% last -f /var/adm/wtmp.old

Some versions of the last command do not allow you to specify a different wtmp file to search through. If you need to check this previous copy and you are using one of these systems, you will need to momentarily place the copy of the wtmp file back into its original location. For example, you might use the following shell script to do the trick:

#!/bin/sh mv /var/adm/wtmp /var/adm/wtmp.real mv /var/adm/wtmp.old /var/adm/wtmp last $* mv /var/adm/wtmp /var/adm/wtmp.old mv /var/adm/wtmp.real /var/adm/wtmp

This approach is not without its problems, however. Any logins and logouts will be logged to the wtmp.old file while the command is running.

10.1.4 loginlog File

If you are using a System V-based version of UNIX (including Solaris), you can log failed login attempts in a special file called /var/adm/loginlog .

To log failed login attempts, you must specifically create this file with the following sequence of commands:

# touch /var/adm/loginlog # chmod 600 /var/adm/loginlog # chown root /var/adm/loginlog

After this file is created, UNIX will log all failed login attempts to your system. A "failed login attempt" is defined as a login attempt in which a user tries to log into your system but types a bad password five times in a row. Normally, System V UNIX hangs up on the caller (or disconnects the Telnet connection) after the fifth attempt. If this file exists, UNIX will also log the fact that five bad attempts occurred.

The contents of the file look like this:

# cat /var/adm/loginlog simsong:/dev/pts/8:Mon Nov 27 00:42:14 1995 simsong:/dev/pts/8:Mon Nov 27 00:42:20 1995 simsong:/dev/pts/8:Mon Nov 27 00:42:26 1995 simsong:/dev/pts/8:Mon Nov 27 00:42:39 1995 simsong:/dev/pts/8:Mon Nov 27 00:42:50 1995 #