Storage Soup: Unslung and Filesystems, Formatting, and Partitioning
This article pertains to:
This article sets out to provide the basic background one needs in order to effectively select and manage disk drives and flash memory devices on an NSLU2 running Unslung.
Disk devices are arranged in a fairly simple structure: they have some number of blocks of data, and given a block number, one can read the corresponding block of data, or write the block of data. This is clearly very efficient, but not very usable. The closest interface on Unslung to get to this level of the storage device is to open the raw device itself (/dev/sda, for example), and manipulate it directly. There's little one can do to control a disk device - when it's plugged in, the device is queried, and its characteristics and size are automatically recorded by the operating system.
A filesystem is a structure we impose on the disk device so that we can have an easier and more flexible way of referring to the data contained on it. The filesystem design determines what sort of features are available, and how they behave. For example, a simple filesystem may lack the data structures to record things like the name or ID of the owner of a file, other security information. Also, the structure of the filesystem will determine how it performs for various operations. Certain filesystem are better suited than others for certain types of data manipulation.
In order to work with a filesystem on Unslung, one needs several things. First we need to know where to find the disk drive itself. We also need to know where we wish to look to find the filesystem that will be on the additional drive (i.e. the Unslung equivalent of the Windows "D:" nomenclature). Finally, we need to be able to tell the computer which filesystem type it should use or expect to find on the disk -- as previously outlined, there are different filesystem types that need to be considered.
Filesystems are usually put on a disk by a process called "formatting" the drive. Formatting is a destructive process -- all existing data is erased in order to ready the drive for storing data in the new filesystem format.
Note: What the NSLU2 refers to as "formatting" is a bit of a misnomer; the NSLU2 does much more than just "format" the disk. And to make matters more "interesting", if the NSLU2 determines that the disk already appears to be formatted, it will call it formatted, but not actually perform the operation. This "feature" has led to many a frustrated user who has quite reasonably expected that "reformatting" the drive from the NSLU2 Web GUI will actually do a "format" operation and re-initialize the disk's filesystem.
Just as we subdivide projects into smaller projects, houses into rooms, books into chapters, etc, we find it useful in many cases to subdivide a single raw disk drive into smaller units. These units are termed "partitions". Each partition is just a subset of the disk device. By convention, a partition table is written to the disk at a known location, and this allows different types of computers to understand how the disk is divided up. There are limits to how many partitions one might reasonably chop a disk device into, and unfortunately different ideas at different times led to some very unfortunate and confusing structures in the partition table. Most users will not have to worry about this, except that it might be important to note that fewer partitions are always easier to deal with; once a single disk gets beyond 4 partitions, something called an "extended partition" happens, and it isn't pretty. It doesn't work very well on Unslung, either.
Partitioning a drive is a complex and error-prone operation, and is best left to automated tools to handle. The exception is small devices like USB flash disks -- these devices are almost alway partitioned as a single partition which covers the entire device.
Putting It Together:
So a disk is organized into just a sequential list of blocks, and each block holds a specific amount of data. For convenience, the disk is subdivided into smaller chunks called partitions. And filesystems are applied to the individual partitions to make them usable. Putting all the details in place, we can begin to make some sense of the different types of filesystems and where and when we might expect to use them.
The Microsoft FAT filesystem is ancient. It's extremely limited. It makes very poor use of large partitions. It cannot handle large, large files (>2GB). It has no provisions for file ownership, or any other type of security. On the other hand, it is very space efficient for small partitions. Its simplicity makes it reasonably fast. And because of its simplicity and partly because of how long it's been around, it has become almost ubiquitious. If a computer can handle a filesystem, it can probably read a FAT disk. So it's really no surprise that the filesystem on the typical flash memory stick is almost always a basic FAT filesystem.
NTFS is Microsoft's answer to the limitations of FAT. It works very well on large partitions, it has extensive facilities for file-system security, it is reasonably fast. It carries a significant overhead, however, which makes it less suitable for small partitions and devices. But most significantly, it is proprietary. As of the time of this writing, the NTFS filesystem format has not been completely reverse-engineered, so there are no "industrial strength" open-source drivers that can write an NTFS filesystem. The result of this is that support for NTFS is limited to mainly Microsoft Windows computers, and some companies who license a commercial implementation of an NTFS driver (which is what Linksys did for the NSLU2 in the R63 firmware release).
The EXT3 filesystem is the Linux filesystem of choice. It works well on many different sizes of partitions and devices, handles all the needs of the Linux security model, is fast, is reasonably simple, and is freely available. Unfortunately, not very many Windows systems can read or write to an EXT3 filesystem. (This is perhaps a good point to note yet again that what the NSLU2 identifies as a "Formatted (ext3)" disk in its web-based GUI is not *just* a disk with an EXT3 filesystem -- in order for the NSLU2 to recognize the disk it must be partitioned very specifically, and the EXT3 filesystem on the partitions must contain specific files. So the "Formatted (ext3)" indication is really a bit of a misnomer on the part of Linksys. See WhatIsANativeUnslungDisk for more specific information on the topic.)
Swap space is not really a filesystem. It's actually just a way to designate a partition on the disk drive as a place to use for storage when main memory overflows. Windows machines use a swap file for this purpose. Linux systems can also use a swap file, but its more efficient to use a partition directly. One of the partitions required to be present for the NSLU2 to identify a disk a "Formatted (ext3)" is a partition marked as a swap partition.