Chapter 8. File Contents

8.0.1 Basic Operations

Treating files as unstructured byte streams necessarily governs what you can do with them. You can read and write sequential, fixed-size blocks of data at any location in the file, increasing its size if you write past the current end. Perl uses an I/O library that emulates C's stdio(3) to implement reading and writing of variable-length records like lines, paragraphs, and words.

The <FH> operator returns undef on error or when end of the file is reached, so use it in loops like this:

while (defined ($line = <DATAFILE>)) {
    chomp $line;
    $size = length($line);
    print "$size\n";                # output size of line
}

Because this operation is extremely common in Perl programs that process lines of text, and that's an awful lot to type, Perl conveniently provides some shorter aliases for it. If all shortcuts are taken, this notation might be too abstract for the uninitiated to guess what it's really doing. But it's an idiom you'll see thousands of times in Perl, so you'll soon get used to it. Here are increasingly shortened forms, where the first line is the completely spelled-out version:

while (defined ($line = <DATAFILE>))       { ... }
while ($line = <DATAFILE>)                 { ... }
while (<DATAFILE>)                         { ... }

In the second line, the explicit defined test needed for detecting end-of-file is omitted. To make everyone's life easier, you're safe to skip that defined test, because when the Perl compiler detects this situation, it helpfully puts one there for you to guarantee your program's correctness in odd cases. This implicit addition of a defined occurs on all while tests that do nothing but assign to one scalar variable the result of calling readline, readdir, or readlink. As <FH> is just shorthand for readline(FH), it also counts.

while (<DATAFILE>) {
    chomp;
    print length( ), "\n";             # output size of line
}

In scalar context, <FH> reads just the next line, but in list context, it reads all remaining lines:

@lines = <DATAFILE>;

Each time <FH> reads a record from a filehandle, it increments the special variable $. (the "current input record number"). This variable is reset only when close is called explicitly, which means that it's not reset when you reopen an already opened filehandle.

undef $/;
$whole_file = <FILE>;               # "slurp" mode

% perl -040 -e '$word = <>; print "First word is $word\n";'

The digits after -0 are the octal value of the single character to which $/ is to be set. If you specify an illegal value (e.g., with -0777), Perl will set $/ to undef. If you specify -00, Perl will set $/ to "". The limit of a single octal value means you can't set $/ to a multibyte string; for instance, "%%\n" to read fortune files. Instead, you must use a BEGIN block:

% perl -ne 'BEGIN { $/="%%\n" } chomp; print if /Unix/i' fortune.dat

Use print to write a line or any other data. The print function writes its arguments one after another and doesn't automatically add a line or record terminator by default.

print HANDLE "One", "two", "three"; # "Onetwothree"
print "Baa baa black sheep.\n";     # Sent to default output handle

8.0.2. Newlines

All systems use the virtual "\n" to represent a line terminator, called a newline. There is no such thing as a newline character; it is a platform-independent way of saying "whatever your string library uses to represent a line terminator." On Unix, VMS, and Windows, this line terminator in strings is "\cJ" (the Ctrl-J character). Versions of the old Macintosh operating system before Mac OS X used "\cM". As a Unix variant, Mac OS X uses "\cJ".

Operating systems also vary in how they store newlines in files. Unix also uses "\cJ" for this. On Windows, though, lines in a text file end in "\cM\cJ". If your I/O library knows you are reading or writing a text file, it will automatically translate between the string line terminator and the file line terminator. So on Windows, you could read four bytes ("Hi\cM\cJ") from disk and end up with three in memory ("Hi\cJ" where "\cJ" is the physical representation of the newline character). This is never a problem on Unix, as no translation needs to happen between the disk's newline ("\cJ") and the string's newline ("\cJ").

Terminals, of course, are a different kettle of fish. Except when you're in raw mode (as in system("stty raw")), the Enter key generates a "\cM" (carriage return) character. This is then translated by the terminal driver into a "\n" for your program. When you print a line to a terminal, the terminal driver notices the "\n" newline character (whatever it might be on your platform) and turns it into the "\cM\cJ" (carriage return, line feed) sequence that moves the cursor to the start of the line and down one line.

Even network protocols have their own expectations. Most protocols prefer to receive and send "\cM\cJ" as the line terminator, but many servers also accept merely a "\cJ". This varies between protocols and servers, so check the documentation closely!

The important notion here is that if the I/O library thinks you are working with a text file, it may be translating sequences of bytes for you. This is a problem in two situations: when your file is not text (e.g., you're reading a JPEG file) and when your file is text but not in a byte-oriented ASCII-like encoding (e.g., UTF-8 or any of the other encodings the world uses to represent their characters). As if this weren't bad enough, some systems (again, MS-DOS is an example) use a particular byte sequence in a text file to indicate end-of-file. An I/O library that knows about text files on such a platform will indicate EOF when that byte sequence is read.

8.0.3. I/O Layers

You may specify I/O layers when you open the file:

open($fh, "<:raw:utf8", $filename);                  # read UTF-8 from the file
open($fh, "<:encoding(shiftjis)", $filename);        # shiftjis japanese encoding
open(FH,  "+<:crlf", $filename);                     # convert between CRLF and \n

binmode($fh, ":raw:utf8");
binmode($fh, ":raw:encoding(shiftjis)");
binmode(FH,  "<:raw:crlf");

Because binmode pushes onto the stack of I/O layers, and the facility for removing layers is still evolving, you should always specify a complete set of layers by making the first layer be :raw as follows:

binmode(HANDLE, ":raw");                     # binary-safe
binmode(HANDLE);                             # same as :raw
binmode(HANDLE, ":raw :utf8");               # read/write UTF-8
binmode(HANDLE, ":raw :encoding(shiftjis)"); # read/write shiftjis

8.0.4. Advanced Operations

$rv = read(HANDLE, $buffer, 4096)
        or die "Couldn't read from HANDLE : $!\n";
# $rv is the number of bytes read,
# $buffer holds the data read

To write a fixed-length record, just use print.

truncate(HANDLE, $length)           or die "Couldn't truncate: $!\n";
truncate("/tmp/$$.pid", $length)    or die "Couldn't truncate: $!\n";

$written = syswrite(DATAFILE, $mystring, length($mystring));
die "syswrite failed: $!\n" unless $written =  = length($mystring);
$read = sysread(INFILE, $block, 256, 5);
warn "only read $read bytes, not 256" if 256 != $read;

$pos = sysseek(HANDLE, 0, 1);       # don't change position
die "Couldn't sysseek: $!\n" unless defined $pos;

These are the basic operations available to you. The art and craft of programming lies in using these basic operations to solve complex problems such as finding the number of lines in a file, reversing lines in a file, randomly selecting a line from a file, building an index for a file, and so on.