5.8.1.1 Start with a minimal clean operating system installation

Start with a clean operating system installation, straight from vendor distribution media. If you do this, you will know exactly what you're working with. You won't need to retrofit something that may already have problems. Using such a system will also make later work easier. Most vendor security patches you later obtain, as well as the vendor configuration instructions and other documentation, assume you're starting from an unmodified installation.

While you're installing the operating system, install as little as you can get away with. It's much easier to avoid installing items than it is to delete them completely later on. For that matter, once your operating system is minimally functional, it's not hard to add components if you discover you need them. Don't install any optional subsystems unless you know you will need them.

5.8.1.2 Fix all known system bugs

Get a list of known security patches and advisories for your operating system; work through them to determine which are relevant for your own particular system, and correct all of the problems described in the patches and advisories. You get this information from your vendor sales or technical support contacts, or from the user groups, newsgroups, or electronic mailing lists devoted to your particular platform.

A helpful list of contacts is available via anonymous FTP at the Firewalls mailing list archive:

ftp://ftp.greatcircle.com/pub/firewalls/vendor_security_contacts

In addition, be sure to get from the Computer Emergency Response Team Coordination Center ( CERT-CC ) any advisories relevant to your platform, and work through them. (For information on how to contact CERT-CC and retrieve its information, see the list of resources in Appendix A, Resources .)

5.8.1.3 Use a checklist

To be sure you don't overlook anything in securing your bastion host, use a security checklist. There are several excellent checklists around. Be sure to use one that corresponds to your own platform and operating system version.

Appendix A of Practical UNIX Security , referenced earlier, contains an extensive checklist that covers most UNIX platforms. More specific checklists for particular operating system releases are often available through the formal or informal support channels for those platforms; check with your vendor support contacts, or the user groups, newsgroups, or mailing lists that are devoted to the platform.

5.8.1.4 Safeguard the system logs

As a security-critical host, the bastion host requires considerable logging. The next step in building the bastion host is to make sure that you have a way of safeguarding the system logs for the bastion host. The system logs on the bastion host are important for two reasons:

• They're one of the best methods of determining if your bastion host is performing as it should be. If everything the bastion host does is logged (and it should be), you should be able to examine the logs to determine exactly what it's doing and decide if that's what it's supposed to be doing. Chapter 12 describes the use of system logs in maintaining your firewall.

• When (not if!) someday someone does successfully break in to the bastion host, the system logs are one of the primary mechanisms that determine exactly what happened. By examining the logs and figuring out what went wrong, you should be able to keep such a break-in from happening again.

Where should you put the system logs? On the one hand, you want the system logs to be somewhere convenient; you want them to be where they can be easily examined to determine what the bastion host is doing. On the other hand, you want the system logs to be somewhere safe; this will keep them from any possible tampering in case you need to use them to reconstruct an incident.

The solution to these seemingly contradictory requirements is to keep two copies of the system logs - one for convenience, the other for catastrophes.

System logs for convenience

The first copy of the system logs is the one you'll use on a regular basis to monitor the ongoing activity of the machine. These are the logs against which you run your daily and weekly automated analysis reports. You can keep these logs either on the bastion host itself or on some internal host.

The advantage of keeping them on the bastion host is simplicity: you don't have to set up logging to go to some other system, nor do you have to configure the packet filters to allow this. The advantage to keeping them on an internal host is ease of access: you don't have to go to the bastion host, which doesn't have any tools anyway, to examine the logs. Avoid logging in to the bastion host, in any case.

System logs for catastrophes

The second copy of the system logs is the one you'll use after a catastrophe. You can't use your convenience logs at a time like this. Either the convenience logs won't be available, or you won't be sure of their integrity any longer.

One of the simplest ways to create catastrophe logs is to attach a line printer to one of the bastion host's serial ports, and simply log a copy of everything to that port. There are some problems with this approach, though. First, you have to keep the printer full of paper, unjammed, and with a fresh ribbon. Second, once the logs are printed, there's not much you can do with them except look at them. Because they aren't in electronic form, you have no way to search or analyze them in an automated fashion.

A more effective way to create catastrophe logs is to connect a dedicated personal computer to a serial port on the bastion host, as a dropsafe logging device. Configure the PC in such a way that it boots up into a terminal program in "record" mode, and that every so often (every 100,000 bytes, for example), the log files are rotated and pruned so the system never runs out of disk space. In this way, anything the bastion host spits out to that serial port will be recorded on the personal computer's disk. Unless someone has physical access to the PC (the PC should not be connected to a network), the logs will remain safely out of reach. The advantage of this approach over the first method is that the data remains in electronic form. If you have to do searches and analyses on the data (after an incident, for example), you can retrieve it from the dropsafe machine.

If you have a write-once device available to you, use that device; doing so is probably technically easier, especially if your write-once device can emulate a filesystem. Be sure you can trust the write-once feature. Some magneto-optical drives are capable of both multiple-write and write-once operations, and keep track of the mode they're in via software. If the system is compromised, it may be possible to rewrite supposedly write-once media.

Some operating systems (notably BSD 4.4-Lite and systems derived from it, such as current releases of BSDI , Free BSD , and Net BSD ) support append-only files. These are not an advisable alternative to write-once media or a dropsafe machine. Even if you can trust the implementation of append-only files, the disk that they're on is itself writable, and there may be ways to access it outside of the filesystem, particularly for an intruder who wants to destroy the logs.

Figure 5.2 shows how you can connect a personal computer to your system to do logging.

Figure 5.2: Use a PC attached by a serial line to create catastrophe logs

Setting up system logs

On a UNIX system, logging is handled through syslog . The syslog daemon records log messages from various local and remote clients (programs with messages they want logged). Each message is tagged with facility and priority codes: the facility code tells syslog what general subsystem this message is from (for example, the mail system, the kernel, the printing system, the Usenet news system, etc.), and the priority code tells syslog how important the message is (ranging from debugging information and routine informational messages through several levels up to emergency information). The /etc/syslog.conf file controls what syslog does with messages, based on their facility and priority. A given message might be ignored, logged to one or more files, forwarded to the syslog daemon on another system, flashed onto the screens of certain or all users who are currently logged in, or any combination.

When you configure syslog to record messages to files, you could configure it to send all messages to a single file, or to split messages up to multiple files by facility and priority codes. If you split messages by facility and priority codes, each log file will be more coherent, but you'll have to monitor multiple files; you may have an easier time finding messages from a particular service. If you direct everything to a single file, on the other hand, you'll only have a single file to check for all messages, but that file will be much larger.

Many non- UNIX systems, particularly network devices such as routers, can be configured to log messages via syslog . If your systems have that capability, configuring them to so they all log to your bastion host provides a convenient way to collect all their messages in a single place.

Be aware that remote logging via syslog (e.g., from a router to your bastion host, or from your bastion host to some internal host) is not 100% reliable. For one thing, syslog is a UDP -based service, and the sender of a UDP packet has no way of knowing whether or not the receiver got the packet unless the receiver tells the sender ( syslog daemons don't confirm receipt to their senders). For another thing, even if syslog were TCP -based, you still couldn't absolutely depend on it not to lose messages; what if the receiving system was down or otherwise unavailable? This is one reason that it's important to have a locally attached dropsafe machine to reliably capture all syslog messages.

Despite its weaknesses, though, syslog is a useful service; you should make extensive use of it.

5.8.2.1 How are services managed?

On UNIX machines, most services are managed in one of two ways:

• By controlling when they start and who can use them

• By service-specific configuration files

There are two ways services get started on UNIX systems:

• At boot time from a machine's /etc/rc files

• On demand by the inetd daemon (which is itself started at boot time)

A few services - for example, Sendmail - can be configured to run under either or both mechanisms.

Services started by /etc/rc files

Services in the first category are designed to run indefinitely. They are started once (when the machine boots), and they are never supposed to exit. (Of course, sometimes they do exit, either because they're killed by a system administrator, or because they trip over a bug or some other error.) Servers are written in this way if they need to handle small transactions quickly, or if they need to "remember" information. Writing them in this way avoids the delays associated with starting a new copy of the server to handle each request made to it.

Servers of this kind are started from a UNIX system's /etc/rc files, which are shell scripts executed when the machine boots. Examples of servers typically started from /etc/rc files are those that handle NFS , SMTP , and DNS . In BSD -based versions of UNIX , there are customarily a few files in /etc with names that start with "rc." (for example /etc/rc.boot ). In System V-based versions of UNIX , there are customarily directories in /etc instead of files (for instance, /etc/rc.0.d ); the directories contain the various startup commands, each in its own little file.

In either case, you need to be careful to look at all of the startup scripts and all of the scripts they call, recursively. Usually more than one script is run in the process of bringing a system all the way up. On modern UNIX systems, those scripts often call others, sometimes through multiple levels of indirection. For example, you may find that a startup script calls another script to start up networking, and that one calls yet another script to start up file service. You may also find that startup scripts use mystical options to familiar commands (e.g., they often run ifconfig with little-used options that cause ifconfig to pick up configuration information from obscure places). Be sure you understand these options and that you replace any that tell the machine to pick up information about itself from the network (or from services it normally provides but that you are going to turn off).

Some versions of UNIX also have an additional process started from these files that is designed to restart other servers if they fail. If such a program exists on a system, it will try to start the other servers if they are removed from the startup files but not from its configuration file. Either turn off this program or be sure to remove from the program's configuration file any servers removed from the startup files. You'll notice the program when you work through the startup files.

Services started by inetd

Some servers are designed to be started "on demand," and to exit after they provide the requested service. Such servers are typically used for services that are requested infrequently; for services that aren't sensitive to delays in starting a new server from scratch; and for services that require a new server process to deal with each request (for example, Telnet or FTP sessions, where a separate server is used for each active session).

Servers of this kind are usually run from the inetd server. (The inetd server itself, because it runs indefinitely, is started from the /etc/rc files, as described in the previous section.) The inetd server listens for requests for services specified in the /etc/inetd.conf configuration file. When it hears such a request, it starts the right server to process the request.

5.8.2.2 How to disable services

If you disable a critical service, you must first make certain you have a way to boot the machine. This could be a second hard disk with a full root partition on it or a CD-ROM drive with the operating system install disk. You need to be ruthless; if you delete the wrong thing and can't reboot, at best you're going to be over-cautious about deleting things, and at worst you're going to end up with an unusable computer.

Second, you must save a clean copy of every file before you modify it. Even when you're just commenting things out, every so often your fingers slip, and you delete something you didn't mean to, or you change a critical character.

Once you've taken these precautions, walk through the startup files for your system, line by line, making sure you know exactly what each line does - including the command line options - and commenting out or deleting the lines that start services you don't need. In a perfect world, you would comment out every line, and then uncomment or rewrite the ones you need. Unfortunately, if you do this and the machine crashes in the middle, it will not reboot. It's rather a lot of trouble for a small increase in security over merely commenting out with a very free hand.

You will frequently see services that are started after a check for some configuration file. If you don't want the service to run, comment out the entire code block. Don't leave the code active simply because the configuration file doesn't currently exist and the service won't currently be started. Someone or something might create the configuration file some time in the future. Commenting out the entire thing is more secure and less risky.

Commenting out lines is preferable to removing them, because it leaves evidence of your intent. When you comment something out, add a comment about why you have commented it out. If you delete something, replace it with a comment about why you have deleted it. Make sure that the next person to look at the files knows that you got rid of things on purpose and doesn't helpfully "fix" it for you. If you comment out a call to another script, add a comment in that script indicating that it's not supposed to be started, and why. Renaming it or commenting out its contents are also good ways to help ensure that it won't accidentally reappear.

For every service that you leave enabled, apply the same line-by-line procedure to the service's configuration files. Obviously, you want to pay particular attention to inetd 's configuration file. On most systems, this file is called /etc/inetd.conf . (On other systems, this file might be called /etc/servers or something else; check your manual pages for inetd ). If you have a daemon-watcher and have decided to leave it on, its configuration files are also particularly important.

In general, you'll need to reboot your machine after you have changed the configuration files. The changes won't take effect until you do so.

After you have rebooted and tested the machine, and you are comfortable that the machine works without the disabled services, you may want to remove the executables for those services. If the executables are lying around, they may be started by somebody - if not you, some other system administrator, or an intruder. A few services may even be executable by nonroot users if they use nonstandard ports.

If you feel uncertain about removing executables, consider encrypting them instead. Do not use the standard UNIX crypt program; the encryption algorithm it uses is little more than a toy and can be trivially broken. Instead, use a more secure encryption program like snuffle or something that uses the DES or IDEA algorithm. Choose a secure key; if you forget the key, you're no worse off than if you'd deleted the files, but if an intruder gets the key, you're considerably worse off.

5.8.2.3 Which services should you leave enabled?

Certain services are essential to the operation of the machine, and you'll probably need to leave these enabled, no matter what else the machine is configured to do. On a UNIX system, these processes include:

init , swap , and page

The three kernel pseudo-processes used to manage all other processes

cron

Runs other jobs at fixed times, for housekeeping and so on

syslogd

Collects and records log messages from the kernel and other daemons

inetd

Starts network servers (such as telnetd and ftpd ) when such services are requested by other machines

In addition, you'll obviously need server processes for the services that you've decided to provide on your bastion host, e.g., real or proxy Telnet, FTP , SMTP , and DNS servers.

5.8.2.4 Which services should you disable?

You will want to disable all services except the ones you have decided to provide, and the supporting services necessary for those to run, as described above. You may not always know which services are the required support services, particularly because UNIX names tend to be cryptic and uninformative.

How do you know which services to disable?

There are three simple rules to apply:

• If you don't need it, turn it off.

• If you don't know what it does, turn it off (you probably didn't need it anyway).

• If turning it off causes problems, you now know what it does, and you can either turn it back on again (if it's really necessary) or figure out how to do without it.

NFS and related services

Start with NFS and related network services. You aren't going to need these. No internal machine should trust your bastion host enough to let the bastion host mount the internal machine's disks via NFS . Besides that, there probably won't be anything on the bastion host that you'll want to export via NFS . NFS is very convenient, but it's incredibly insecure.

NFS services are provided by a whole set of servers; the specific set of servers, and the names of the individual servers, varies slightly from one version of UNIX to the next. Look for these names or names like them:

• nfsd

• biod

• mountd

• statd

• lockd

• automount

• keyserv

• rquotad

• amd

Most of these services are started at boot time from the /etc/rc files, although some are started on demand by inetd . mountd is somewhat peculiar in that it is often started at boot time and is listed in the inetd configuration file, apparently so that it will be restarted if for some reason the copy that was started at boot time crashes.

Other RPC services

You should also disable other services based on the Remote Procedure Call ( RPC ) system. The most critical of these is NIS/YP , a service which is provided by the following servers:

• ypserv

• ypbind

• ypupdated

These servers are generally started at boot time from the /etc/rc files.

Also disable these RPC -based services:

• rexd (the remote execution service, started by inetd )

• walld (the "write all", or wall daemon, started by inetd )

All RPC -based services depend on a single service usually called portmap (on some machines it is known as rpcbind ). If you've disabled all of the RPC -based services, you can (and should) also disable the portmap service. How can you tell if you've disabled all the RPC -based services? Before disabling portmap , but after disabling what you think are the rest of the RPC -based services, reboot the machine and then issue a rpcinfo -p command. If the output of that command shows only entries for portmap itself, this means that no other RPC services are running. On the other hand, if the output shows that other RPC services are still running, you will need to investigate further to determine what and why. If you decide to provide any RPC -based services, you must also provide the portmap service. In that case, consider using Wietse Venema's replacement portmap , which is more secure than the versions shipped with most UNIX systems (see Appendix B for information on where to find it).

Booting services

Your bastion host should probably not provide booting services; nothing should trust the host enough to be willing to boot from it. This means that, in most cases, you should disable these services:

• tftpd

• bootd

• bootpd

BSD ‘r’ command services

These should all be disabled. The servers for these services are typically named rshd , rlogind, and rexecd , and are typically started by inetd . The remaining "r" services are based on these and will not run without them.

routed

Another server that your bastion host probably doesn't need is routed . This server is started at boot time from the /etc/rc files, listens to routing information broadcasts, and updates the kernel routing table based on what it hears.

You probably don't need routed on your bastion host, because your bastion host is probably located on the perimeter of your network, where routing should be fairly simple. A more secure approach is to create static routes pointing to your internal networks and a default route pointing to your Internet gateway router. You do this at boot time by adding appropriate "route add" commands to the /etc/rc files.

If you must do dynamic routing on your bastion host, obtain and use gated (see Appendix B for information on how to get it) rather than routed . gated understands the same routing protocols as routed (plus several more), but gated can be specifically told who to accept routing information from, while routed will accept information from anyone. Be sure that you set up the configuration files for gated to limit which hosts it listens to.

fingerd

The finger server supplies information about existing accounts and accounts on UNIX systems. This server is started on demand by inetd . The information provided by fingerd can be valuable to crackers; it tells them information about potential targets, such as:

• Which accounts exist. This tells them which accounts they should try to guess passwords for.

• Personal information about the people with accounts. This tells them what passwords to start guessing with.

• Which accounts are in use. This tells them which accounts should be avoided, at least until they're not in use.

• Which accounts haven't been used lately. This tells them which accounts are good targets for attack, because the owners probably won't notice that the accounts are being used.

On the other hand, Internet users often use finger (the program that talks to your fingerd daemon) quite legitimately. finger is helpful in locating email addresses and telephone numbers. Instead of simply disabling fingerd , you might want to replace it with a program that obtains information from a more basic source of contact information for your site; the information might include:

• Your main phone number

• Who to contact if they have questions about your site's products or services

• Sample email addresses if standardized aliases such as Firstname_Lastname are maintained for users at your site

• Who to contact in case of network or security problems involving your site

You can provide this kind of generic information to anybody who uses finger to check on your site, regardless of what specific information they've requested. The easiest way to accomplish this is to put the information in a file (for example, /etc/finger_info ) and then replace the part of the /etc/inetd.conf entry for fingerd that specifies the program to run with something like /bin/cat /etc/finger_info . Doing this causes the contents of the /etc/finger_info file to be returned to anyone contacting your fingerd server.

For example, here is the old /etc/inetd.conf line from Great Circle Associate's system:

finger stream tcp nowait nobody /usr/libexec/fingerd fingerd

and here is the new /etc/inetd.conf line:

finger stream tcp nowait nobody /bin/cat cat /etc/finger_info

and here are the contents of the /etc/finger_info file:

Great Circle Associates Phone: +1 415 962 0841 Email: [email protected]  For more information, or to report system problems, please  send email or call.

ftpd

If you're going to provide anonymous FTP service on your bastion host, you need to reconfigure the FTP server appropriately. You should replace the ftpd program with one more suited to providing anonymous FTP service than the standard ftpd programs shipped by most UNIX vendors. (See Chapter 8 for information about providing anonymous FTP service.)

If you're not going to provide anonymous FTP , you can probably disable your FTP server entirely; it's started on demand by inetd .

Even if you've disabled the FTP server on your bastion host, you can still use the FTP client program (typically called simply ftp ) on the bastion host to transfer files to and from other systems. You'll just have to do the work from the bastion host, instead of from the other systems.

Other services

There are lots of other services you probably don't need and should disable. Although the specific list depends on your own site's security policy and needs, and on the platform you're using, it should probably include the following:

• uucpd ( UUCP over TCP/IP )

• rwhod (sort of like fingerd , in that it tells you who's currently logged in on the system)

• lpd (the printer daemon)

5.8.2.5 Turning off routing

If you have a dual-homed host that is not supposed to be a router, you will need to specifically disable routing. In order to act as an IP router, a dual-homed host needs to accept packets that are addressed to other machines' IP addresses, and send them on appropriately. This is known as IP forwarding , and it's usually implemented at a low level in the operating system kernel. An IP -capable host with multiple interfaces normally does this automatically, without any special configuration.

Other machines have to know that the dual-homed host is a router in order to use it as such. Sometimes this is done simply by configuring those machines to always route packets for certain networks to the dual-homed host (this is called static routing ). More often, however, the dual-homed host is configured to broadcast its routing capabilities via a routing protocol such as Routing Information Protocol ( RIP ). Other machines hear these routing broadcasts and adjust their own routing tables accordingly (this is called dynamic routing ). This broadcast of routing information by the dual-homed host is usually done by an additional program (for example, routed or gated on a UNIX system), which often has to be turned on explicitly.

To use a dual-homed host as a firewall, you need to convert it to a nonrouting dual-homed host; you take a machine that has two network interfaces, and you configure it so it can't act as a router between those two interfaces. This is a two-step process:

1. Turn off any program that might be advertising it as a router; this is usually relatively straightforward.

2. Disable IP forwarding; this is considerably more difficult, and may require modifying the operating system kernel.

Fortunately, these days a number of UNIX vendors provide supported parameters for turning off IP forwarding. Even for vendors that don't, it's about as easy as kernel patches get on most machines: turning off IP forwarding necessitates only a change in the value of a single kernel variable. You need to consult your vendor to find out how to turn off IP forwarding on your machines.

Unfortunately, it's not always that easy to turn off all routing, particularly on BSD -based systems, such as Sun OS and Ultrix. On such systems, you can patch the kernel to turn off IP forwarding, but the IP source-routing option usually remains a security hole.

What is source routing ? Normal IP packets have only source and destination addresses in their headers, with no information about the route the packet should take from the source to the destination. It's the job of the routers in between the source and the destination to determine the most efficient route. However, source-routed IP packets contain additional information in the IP header that specifies the route the packet should take. This additional routing information is specified by the source host; thus the term source-routed.

When a router receives a source-routed packet, it follows the route specified in the packet, instead of determining the most efficient route from source to destination. The source-routing specification overrides the ordinary routing. Because of the way the routing code is implemented in most UNIX kernels, turning off IP forwarding does not disable forwarding of source-routed packets. It's implemented completely separately. Unlike IP forwarding, source routing is not generally an easily patchable option. It is possible to disable it (you can disable anything with persistence and/or source code), but it's much more difficult.

Source-routed packets can easily be generated by modern applications like the Telnet client that's freely available on the Internet as part of the BSD 4.4 release. Unless you block source-routed packets somewhere else, such as in a router between the dual-homed host and the Internet, source-routed packets can blow right past your dual-homed host and into your internal network.

Worse still, source routing goes both ways. Once source-routed packets make their way to an internal system, the system is supposed to reply with source-routed packets that use the inverse of the original route. The reply from your internal system back to the attacker will also blow right through your dual-homed host, allowing two-way connection through a firewall that was supposed to block all communications across it.

If you are not going to screen your dual-homed host, you will need to patch your operating system so that it rejects source-routed packets. Consult your vendor, and/or appropriate security mailing lists (discussed in Appendix A ) for information on how to do this on your platform. This is not the sort of thing most vendors' customer support structures deal with rapidly and painlessly; expect to have to dig to find what you need to know.

5.8.3.1 Using the TCP Wrapper package to protect services

The TCP Wrapper package, written by Wietse Venema, monitors incoming network traffic and controls network activity. It is a simple but very effective piece of publicly available software set up to run whenever certain ports (corresponding to certain services) are connected. TCP Wrapper provides simple access control list protection, as well as improved logging, for services that are started by inetd .

Using the TCP Wrapper package is easy. Here's what you do:

1. Install the package and set up a pair of simple access control files that define which hosts and networks are allowed to access which services.

2. Reconfigure your inetd to run the main TCP Wrapper program (called tcpd ) instead of the "real" server.

3. When a request for a service comes in, inetd starts tcpd , which evaluates the request against the TCP Wrapper configuration files. This program decides whether or not to log the request, and whether or not to carry out the request.

4. If tcpd decides that the request is acceptable, it starts the "real" server to process the request.

For example, if you want to allow Telnet connections from a specific host (172.16.1.2) to your machine, but deny Telnet connections from all other hosts, you would change the line for telnetd in your /etc/inetd.conf file to say something like:

telnet stream tcp nowait root /usr/local/libexec/tcpd telnetd

You would also need to create an /etc/hosts.allow file that tells the TCP Wrapper package (the tcpd program) which host to allow connections from:

telnetd : 172.16.1.2

And finally, you'd need to create an /etc/hosts.deny file to tell the TCP Wrapper package to deny all connections from all hosts by default, and to send email to root about each probe:

ALL : ALL : (/usr/local/etc/safe_finger -l @%h | \     /usr/ucb/Mail -s "PROBE %d from %c" root)&

Note that the /etc/hosts.deny file only applies to services protected by the TCP Wrapper package (that is, services for which you've configured inetd to run tcpd instead of the real server). If you don't tell inetd to run the TCP Wrapper package (the tcpd program) for a given service, then the TCP Wrapper package won't do anything regarding that service.

Despite its name, the TCP Wrapper package supports UDP -based services in addition to TCP -based services. Beware, however, that the TCP Wrapper package can only control when to start UDP -based servers; it cannot control access to those servers once they're started, and many UDP -based servers will hang around and continue to process requests for some period of time beyond the initial start-up request. Many eventually time out and exit, but once they've been started through a legitimate request, they're vulnerable to illegitimate requests.

5.8.3.2 Using netacl to protect services

The netacl component of the TIS FWTK (described in some detail in Chapter 7, Proxy Systems ) provides much the same capability as the TCP Wrapper package. To implement the same example as above (except for the ability to trace probes from unauthorized systems) using netacl , you would change the line for telnetd in your /etc/inetd.conf file to:

telnet stream tcp nowait root /usr/local/lib/netacl telnetd

Then, you would add the following lines to your FWTK netperm configuration file (wherever that is on your system):

netacl-telnetd: permit-hosts 172.16.1.2 -exec /usr/libexec/telnetd

5.8.4.1 Reconfigure and rebuild the kernel

The first step in this phase of building your bastion host is to rebuild the operating system kernel to remove kernel capabilities you don't need. This may sound intimidating, but it's generally a relatively straightforward operation; it needs to be, because you'll be using the same capabilities you'd use to install a new type of device on your system. Every UNIX system, as shipped, contains some form of configuration support (they range considerably in how kernel reconfiguration is supported and in what you can do). Besides reducing the size of your kernel (and thereby making more memory available for other purposes), rebuilding the kernel denies to attackers the chance to exploit these capabilities.

Some capabilities are particularly dangerous. In particular, you should probably remove the following capabilities or device drivers:

• NFS and related capabilities

• Anything that enables network sniffing, e.g., Network Interface Tap ( NIT ) or Berkeley Packet Filter ( BPF )

Although NIT and BPF are provided for testing and debugging purposes, they are frequently used by attackers. NIT and BPF are dangerous because they let the machine grab all packets off the Ethernet it's attached to, instead of only the packets addressed to it. Disabling these capabilities may prevent you from using the machine as a packet filtering system, so you may not be able to delete them in all architectures.

If your bastion host is a dual-homed host, this is the time to disable IP forwarding.

You have to be more careful when you disable kernel capabilities than when you disable services started by inetd or at boot time from the /etc/rc files (as described earlier). There are a lot of interdependencies between kernel capabilities. For this reason, it's sometimes hard to determine exactly what a given capability is used for. The consequences of disabling a capability that is actually needed can be severe, e.g., the new kernel might not boot.

Make sure you follow your vendor's instructions for building and installing new kernels. Always keep a backup copy of your old kernel. If you have a backup, you can boot from it if you find out that something is wrong with the new kernel. Be sure you know how to boot a backup kernel and look up the procedure before you try to boot onto the new kernel. Be sure the backup kernel is in the root partition; you can't boot from a kernel that isn't on the root partition.

When you can reboot the machine, go through the kernel configuration files the same way you went through the startup files, checking every single line to make certain that it's something you want. Again, watch for places where one configuration file contains another, and check your documentation to be sure that you've looked at all the configuration files that are consulted. Often there is one file for including device drivers and one or more for parameters; IP forwarding will be in the latter.

Once you've got a working kernel, you'll probably want to delete or encrypt your old "full function" kernel. Replace it with a backup copy of the working minimal kernel. Doing so will keep a cracker who somehow manages to break into your machine from simply using that old kernel to reboot, and thereby restore all of the services you so carefully disabled. For similar reasons, you'll probably also want to delete the files and programs needed to build a new kernel.

If your kernel uses loadable modules, it may be difficult to determine when they're used. You will want to delete or encrypt all the ones that you don't want used, but because they're not always explicitly loaded, you may not know which those are. Keeping an alternate boot medium handy, try moving them out of the directory for loadable modules. Run the machine through its paces before you finally remove or encrypt them.

Beware! Your vendor may have provided copies of "generic" kernels (which typically have every possible capability enabled) in unexpected locations for use during the installation of the machine and its (nonexistent) client machines. SunOS 4.x, for example, has such kernels in the /usr/stand directory. Poke around in all the directories where installation files are kept and all the directories for clients. The documentation generally tells you where client kernels are, but rarely tells you about the internals of the install process. Check the documentation for disaster recovery advice, which may helpfully tell you where to locate spare kernel images.

5.8.4.2 Remove nonessential programs

The next step is to remove all of the programs that aren't essential for day-to-day operation. If a program isn't there, an attacker can't exploit any bugs that it might contain. This is especially true for setuid/setgid programs, which are a very tempting target for an attacker. You should remove programs you normally think of as being essential. Remember that the bastion host is purely providing Internet services; it does not need to be a comfortable environment to work in.

Window systems and compilers are examples of major programs you can get rid of. Attackers find these programs very useful: window systems are fertile ground for security problems, and compilers can be used to build the attacker's own tools.

Before deleting programs like compilers, make sure you've finished using them yourself; make sure you've built, installed, and tested everything you're going to need on this machine, such as the tools for auditing the system (discussed later in this chapter).

Instead of simply deleting key tools you'd expect an attacker to use, such as the compiler, you might want to replace them with programs that raise an alarm (for example, sending email or tripping your pager) when someone tries to run them. You might even want to have the programs halt the system after raising the alarm, if you believe it's better for the machine to be down than under attack. This is a prime way to humiliate yourself, however; you yourself are probably the one person most likely to forget where you are to try to run a forbidden command. It's also a good way to set yourself up for denial of service attacks.

You'll want to do two scans looking for things to delete:

1. Walk through all the standard directories for binaries on your system (everything that's in root's path or in the default user path). If you're unsure whether a program is needed, turn off execute permission on it for a while (a few days) before you remove or encrypt it and see what happens. You may also want to run the machine for a while before you do the scan and check the access times on files to see if they've been used.

2. Use find to look for every file on the system that has the setuid or setgid bit turned on. The arguments to find differ radically from system to system, but you will probably want something like this:

   find / -type f \( -perm -04000 -o -perm -02000 \) -ls

3. Some versions of find provide special primitives for identifying setuid and setgid files.

5.8.4.3 Mount filesystems as read-only

Once you've got the bastion host configured, you don't want anybody (particularly an attacker) to be able to change the configuration. To guard against this happening, mount the filesystems on the bastion host as read-only if possible (particularly the filesystems that contain program binaries) to protect against tampering.

It's much better if you can use hardware write-protect; an attacker may be able to remount disks with write permission without getting physical access to the machine, but it's not going to do any good if the hardware write-protect on the disk is on. Most IPI and SMD disks have switches for this on their front panels. Many SCSI disks have a "write-disable" jumper you can set. If you find powering the disk down and removing it from the case unacceptable as a way to get write access, you could wire this jumper to an external switch on the drive enclosure.

You can't write-protect everything, of course. You have to provide a certain amount of writable filesystem space for things like scratch space, system logs, and the mail spool. You might be able to use a RAM disk for this; however, you'll have to be sure that your operating system supports this, that you have enough RAM , and that you think you can afford to lose the contents of the RAM disk (for example, electronic mail in transit between internal hosts and the Internet) whenever your machine reboots.

With most versions of UNIX , you'll also have to either provide writable disk space for memory swapping or turn off swapping. Many versions of UNIX do not allow you to turn off swapping; however, they will usually allow you to use a separate disk for swap space, and that disk can safely be left writable. Using a RAM disk will increase your memory usage to the point where you will probably need swap space.

Systems based on BSD 4.4-Lite (for instance, current releases of Net BSD , FreeBSD, and the BSDI product) have a new immutable attribute that can be set on a per-file basis. If a file is marked "immutable," the file cannot be changed, not even by root, unless the system is running in single-user mode. If your operating system provides this capability, use it to protect your programs and configuration files from tampering by an attacker. (We recommend that approach only if you cannot use hardware write protection, or an additional layer of security to use with hardware write protection. Because it's implemented in software, it is more likely to be compromisable.)

5.8.5.1 Auditing packages

Most auditing packages have two basic purposes:

• Checking for well-known security holes. These are holes that have been uncovered by system administrators, exploited by attackers in system break-ins, or documented in computer security books and papers.

• Establishing a database of checksums of all files on a system; doing this allows a system administrator to recognize future changes to files - particularly unauthorized changes.

There are several very good automated auditing packages freely available on the Internet. The three most commonly used are these:

• COPS  - The Computer Oracle and Password System, developed by Dan Farmer and Gene Spafford

• Tiger  - Developed as part of the TAMU package by Texas A&M University

• Tripwire  - Developed by Gene H. Kim and Gene Spafford

COPS and Tiger both check for well-known security holes. There is significant overlap in what COPS and Tiger check; however, they're both free, so it's a good idea to obtain and run both of them to get the best possible coverage. Tripwire is a filesystem integrity checker. It is strictly a tool for dealing with checksum databases; it is much better at this than either COPS or Tiger (which both have basic checksum database capabilities), but has no ability to check for well-known security holes. These packages are independent of each other; there's nothing to prevent you from using all three of them in combination on your bastion host, and that would probably be a good idea. Appendix B gives you information on how to get all three packages.

Because the well-known security holes tend to be somewhat operating system-specific, the effectiveness of the packages that check for these security holes is very dependent on which operating system you have, and which version of the operating system it is. If it's an operating system and version the package knows about, that's great. If it isn't, then the package has to grope around blindly, trying to guess what holes might exist. (Fortunately, attackers will usually have the same problem, if not to the same extent.)

Commercial packages that perform similar functions are starting to become available, but none yet have the complete capabilities and widespread acceptance of COPS , Tiger, and Tripwire.

5.8.5.2 Use the auditing packages

How do you use the various auditing packages to audit your system? The details of what you do depend upon which package you're using. (See the documentation provided with the packages for detailed instructions.) This section provides some general tips.

You will need to do some configuration. Don't just install the program, run it, and expect you'll get reasonable results. Expect to go through several iterations of running the auditing package, getting warnings, and reconfiguring your machine or the auditing package to get rid of warnings. When you get warnings, you have to decide whether the auditing package is wrong, or you are. There will be some cases where the right thing to do is to turn off checks, but it shouldn't be your automatic response.

Once you've used the tools described in the previous section to create your initial baseline, store a copy of the tools and these initial audit results somewhere safe. Under no circumstances should you store the only copy of the baseline or the tools on the bastion host. Prepare for the worst: if someone were to break into the bastion host and tamper with the only copy of the baseline audit, this would compromise your ability to use the audit later on to detect illicit changes on the system. If intruders can change the auditing software, it doesn't matter whether they can change the baseline; they could simply set up the auditing software to reproduce the baseline. Keeping a copy of the baseline audit on a floppy disk or magnetic tape that's locked up someplace safe is a good way to protect against such a compromise. Preferably, you don't want an intruder to even read the audit results; why tell them what you expect the system to look like and what files you aren't watching?

Periodically, e.g., daily or weekly, depending on your own site's needs and capabilities, audit the machine once again and compare the new audit to the baseline. Make sure you can account for any differences you find. Ideally, you should automate this periodic re-audit so it happens regularly and reliably. Unfortunately, this is easier said than done. Arranging for automatic audits that can't be defeated by "replay" attacks can be a neat trick. In a replay attack, an attacker who has compromised your auditing system simply sends you a recording of a prior good audit whenever your system invokes the automatic auditing capability. The most practical defense against this is to run your automated auditing system often enough that it's unlikely an attacker could break in, discover the auditing system, and subvert it (covering his tracks) before the next audit runs. This suggests that you should run an audit at least daily.

5.8.5.3 About checksums for auditing

Checksums are very helpful in auditing. An intruder who changes a program or configuration file will almost certainly correct the modification dates afterwards, so you can't use these dates as a reliable index. Comparing every file to a baseline copy avoids that problem, but takes a lot of time and requires that you store a copy of every single file, effectively doubling your storage requirements. Checksums are probably your best bet.

A checksum is a number calculated from the contents of the file that will change if the file is changed. Checksum calculation is time-consuming, but not as time-consuming as reading everything twice to do a bit-by-bit compare. In addition, storing the checksums takes up much less space than storing the entire file. Checksums are not full representations of the file, however, and every checksum algorithm has cases where it will give the same checksum for two different files. The better the checksum algorithm, the less likely it is that files with the same checksum resemble each other in any other way.

For example, the common UNIX spell command uses a hashing algorithm that shares this property with checksums. spell will relatively reliably detect misspelled English words, but will often happily accept complete garbage, because random character strings may happen to have the same hashes as good words. This represents a problem only if you are likely to type "qzx" when you meant to type "the". Similarly, a good checksum algorithm may come up with the same number for /bin/login and a document containing the text of "Jabberwocky," but it won't come up with the same number for /bin/login and any other executable capable of logging people in (especially a modified copy of /bin/login ).

However, the standard UNIX checksum programs ( /bin/sum , for example) don't have this property. Those checksum programs use simple cyclic redundancy counter ( CRC ) algorithms designed to catch bit errors during data transfers. They can't keep up with the latest tools crackers use to subvert traditional checksum programs. Crackers now have programs that manipulate the unused bytes in a file (particularly an executable binary file) to make the checksum for that file come out to whatever they want it to be. They can make a modified copy of /bin/login that produces the same checksum, and sum will not be able to detect any difference

For real security, you need to use a "cryptographic" checksum algorithm like MD5 or Snefru; these algorithms produce larger and less predictable checksums that are much more difficult to spoof. The COPS , Tiger, and Tripwire auditing packages described above all include and use such algorithms in place of the normal UNIX checksum programs.