Proper Filesystem Layout

There are two main schools of thought when it comes to filesystem layout: One group says it's best to segregate as much as possible, putting everything on its own filesystem; the other group says that one should have only one or two filesystems, putting everything together on communal filesystems.

The goal of this article is to convince you to fall somewhere in the middle. Increasing or reducing the number of filesystems on a server is often a good idea, but should only be done for legitimate reasons.

General notes

Having a small number of filesystems is usually a good idea for a number of reasons. Somewhat trivially, it means fewer filesystems to monitor for free space. It also provides more leeway for sizing mistakes, because if everything is broken out into separate filesystems and you were incorrect in your forecast of how much space each filesystem would need, you might find yourself in the unhappy circumstance of having a nearly full filesystem on a server which also has a several nearly empty ones.

Speaking of filesystems filling up, this is one of the reasons for increasing the number of filesystems: Some forms of Unix behave very badly when certain filesystems, such as /tmp and / fill up. Separating these filesystems from directories which tend to grow, such as /var, can save you a lot of headaches. Aside from the operating system, when filesystem gets full, it generally means something will stop working correctly because an app is locked waiting to write data, or logs or other data cannot be written and thus are lost.

In laying out both your operating system and your application filesystems, there are a couple of factors you should always keep in mind. Probably the most important of these is backups. It is unlikely that you are designing a server that doesn't need some kind of backup, and proper filesystem design can greatly simplify the process. Some backup software can only backup entire partitions, which makes your granularity on backup rules relatively large. If you need a tighter granularity for backups, the only way to achieve that might be to break your data into separate filesystems.

It isn't immediately obvious, but filesystem layout can also have a significant impact on system performance. The first cylinders of a disk -- the cylinders at the outer edge -- are spinning the fastest and thus provide the best performance, but you also need to keep in mind the parts of the disk where the disk head will spend the most time.

If your system uses a lot of swap, which is usually the second partition, and you have a filesystem that does a lot of reads and/or writes in the last partition on the disk, then your disk head will constantly be shuttling back and forth across your disk, ignoring all of the data in the middle, but working madly at the edges. This will seriously degrade your server performance and should be avoided at all costs.

Operating system filesystems

There are five main system filesystems, which we can cover relatively easily because the guidelines on how to partition them don't change much from system to system. These are /, /usr, /var, /tmp, and /home.

/tmp

It is easiest to discuss /tmp, because it is most often a virtual filesystem stored in unused swap and contains only transient files. As such, you are not losing any real disk space, and data integrity isn't so important. However, because most systems set /tmp to be as large as the entire swap allotment, filling /tmp generally means running out of swap space, and this is a very bad thing. At least some systems, such as Solaris, allow /tmp to be restricted to a size smaller than swap, which can save you from application outages and server reboots.

/var

It is also relatively easy to lay out /var, because it will always grow and thus should be placed at least somewhat apart. Including /var on another partition, such as /, can have disastrous consequences when an app has serious problems or there is a hardware failure, because gigabytes of logs can be generated very quickly. Filling /var just means you can't log anymore, but filling / can mean service interruptions.

The most difficult aspect of /var is sizing, because it is often difficult to forecast how much log space you will need. In these heady days of 9-GB disks being considered small, you should start with a 1-GB /var partition and go up from there. For a web or email server with a lot of logging, you probably want significantly more. In fact, if you find yourself with extra space after partitioning a system, it is probably a good idea to throw that space into /var. Space on /var can also be saved by using a network logging server to store your logs.

/usr

In this case, /usr may or may not include /usr/local. Most open-source Linux systems depend heavily on /usr/local, but many commercial Unix variants use it sparingly or not at all. What makes /usr unique is that it is composed entirely of extremely important files that will only be modified during an upgrade or a patch application. If your use of /usr/local fits this bill, then include it with /usr, otherwise consider separating it.

In general, the only good reasons for separating /usr are either that your / partition has all of your apps on it or, my favorite reason, you want to mount it read-only for security reasons -- after all, most of the files modified by a root kit reside in /usr. If this does not suit your needs, then either include it on / or with /usr/local, depending on the characteristics of your installation. One benefit of separating /usr from everything else is that you can size it very closely, leaving only about 15 percent for upgrade/patching room because it is very unlikely to grow.

/

Although it is tempting to think of / as the catch-all filesystem and make it large enough to handle that duty, you might find it more appropriate to make it small. Application filesystems will be discussed later, but without /var or /usr, / is now relatively small. In fact, one of your goals when sizing the root partition should be to keep it as small as possible, hopefully under 100 MB. Because the major components of / are /etc, /dev, the kernel, and some statically linked binaries, this should be relatively easy to accomplish.

Unlike /usr, the / partition usually must be mounted read/write, because it is where system configuration information, device filesystems, and the /proc filesystem -- if it exists on your system -- reside. For some systems, it is possible to get / to the point where it does not need to be written to, but you have to know your system very well to do this, and it is probably not a good idea even then.

Similarly to /usr, though, good filesystem layout means you can size your / partition very closely; on Solaris it is relatively easy to get a / parition of only around 50 MB of data, so a 75-MB partition will easily last the life of the server.

/home

Whether /home counts as part of the operating system or part of the application depends on what the server is being used for. If you have a web server that will only have administrators logging into it, then /home can safely be included on / because it should be small and should not change much. But if the server will be a file server for the LAN, then /home often becomes part of the application. You can even remove /home entirely and replace it with NFS-mounted home directories, which may or may not be appropriate depending on your application and network design, and how closely up-time is linked to the existence /home (because NFS-mounting /home adds another point of failure).

Application partitions

Servers are most often provisioned for a single application or service, and it is definitely preferable to do so. As such, it is usually easy and often makes sense to put that application on a separate partition. Also, many servers put the operating system on internal disks and all application data on an array of some kind, which clearly eliminates the possibility of collocating that data on the same disk as your operating system.

Applications are where filesystem layout becomes most difficult, though -- whether you have one or many, either on local disk or an array. In some cases, such as high-end databases like Oracle, the application vendor will have specific requirements as to filesystem layout, which leaves where to physically locate each partition on disk as your only decision -- that's another topic entirely. Most other applications provide much more leeway in filesystem layout, meaning your decisions are more difficult. Generally speaking, it is best to put your application on a separate filesystem from your operating system. But this is not always possible with limited disk space or many different applications, and with services such as NIS+ that use essentially no disk space there is no point in doing this.

If your application requires a lot of space, then put that space in a separate filesystem, especially if there will be a lot of non-logging disk access. If the application will be doing significant amounts of logging, place the logs in /var if at all possible, especially if that is the only real I/O the application requires.

If you are planning on running many applications on a single server, plan to segregate them but only up to a point: If you have 50 very similar services and none of them require much space or I/O, then collocate them, but if you have 5 very different services with disparate space and I/O needs, then it makes sense to give them their own filesystems.

Some applications turn what would otherwise be operating system filesystems into application directories, such as when an application resides in /usr/local or when a mail server spools into /var. Some applications prefer to be in /opt, and some commercial Unix variants prefer to place both operating system and application data in /opt -- this is especially true with HP-UX. In these cases, it is best to break the application portion out onto a separate filesystem, such as /usr/local/apache or /var/spool. If you cannot do that, treat the entire filesystem like an application filesystem rather than an operating system filesystem. This will simplify your backup rules and make data loss and service interruption easier to avoid.

Sometimes makes sense to separate an application from its data. Web server configuration files and binaries don't change very often, but the data that goes with them is usually quite dynamic. Databases also often separate the application from the data. For this type of decision, you can treat the separate pieces as though they were separate applications: If they have very disparate I/O and space needs, they should be separated; but if they behave similarly, then it makes sense to keep them together.

Conclusion

Hopefully this has helped provide some basic guidelines for how to partition your all-important data. Because of the large disks most servers ship with these days, we usually don't have the space constraints that traditionally plagued system administrators, but improperly setting up a server can still come back to haunt you. More and more operating systems are also providing comprehensive volume management, which allows a system administrator to recover from poor disk-space forecasting. However, no one wants to try to resize volumes and reallocate data on a live server, and it often can only provide stop-gap solutions because of how the filesystems are laid out on disk. The absolute best aid in figuring out how to partition a system is a historical analysis of the application for which you are provisioning space, but such analyses are depressingly rare.

In the absence of a historial analysis of the system, just follow these guidelines, always pay attention to how your various filesystems will be used, and never forget that underlying it all is actual hardware -- if you ignore the hardware itself, whether in sizing, purchasing, or how you spread the partitions across it, the best filesystem layout in the world won't keep your server running smoothly.

Luke A. Kanies is a Unix system administrator who is more interested in the operating system than what is being done with it. He is currently working as a contractor and researcher, trying to make a better sysadmin.

oreillynet.com Copyright © 2003 O'Reilly & Associates, Inc.