2.3.1. Basic Document Fetch

$document = get("http://www.suck.com/daily/2001/01/05/1.html");

If the document can't be fetched, get( ) returns undef. Incidentally, if LWP requests that URL and the server replies that it has moved to some other URL, LWP requests that other URL and returns that.

With LWP::Simple's get( ) function, there's no way to set headers to be sent with the GET request or get more information about the response, such as the status code. These are important things, because some web servers have copies of documents in different languages and use the HTTP language header to determine which document to return. Likewise, the HTTP response code can let us distinguish between permanent failures (e.g., "404 Not Found") and temporary failures ("505 Service [Temporarily] Unavailable").

Even the most common type of nontrivial web robot (a link checker), benefits from access to response codes. A 403 ("Forbidden," usually because of file permissions) could be automatically corrected, whereas a 404 ("Not Found") error implies an out-of-date link that requires fixing. But if you want access to these codes or other parts of the response besides just the main content, your task is no longer a simple one, and so you shouldn't use LWP::Simple for it. The "simple" in LWP::Simple refers not just to the style of its interface, but also to the kind of tasks for which it's meant.

2.3.2. Fetch and Store

$status = getstore("http://www.suck.com/daily/2001/01/05/1.html",
                   "/tmp/web.html");

There are two problems with this. The first is that the document is now stored in a file instead of in a variable where you can process it (extract information, convert to another format, etc.). This is readily solved by reading the file using Perl's built-in open( ) and <FH> operators; see below for an example.

The other problem is that a status code by itself isn't very useful: how do you know whether it was successful? That is, does the file contain a document? LWP::Simple offers the is_success( ) and is_error( ) functions to answer that question:

$successful = is_success(status);
$failed     = is_error(status);

If the status code status indicates a successful request (is in the 200-299 range), is_success( ) returns true. If status is an error (400-599), is_error( ) returns true. For example, this bit of code saves the BookTV (CSPAN2) listings schedule and emits a message if Gore Vidal is mentioned:

use strict;
use warnings;
use LWP::Simple;
my $url  = 'http://www.booktv.org/schedule/';
my $file = 'booktv.html';
my $status = getstore($url, $file);
die "Error $status on $url" unless is_success($status);
open(IN, "<$file") || die "Can't open $file: $!";
while (<IN>) {
  if (m/Gore\s+Vidal/) {
    print "Look!  Gore Vidal!  $url\n";
    last;
  }
}
close(IN);

2.3.3. Fetch and Print

$status = getprint(url);

The document is printed to the currently selected output filehandle (usually STDOUT). In other respects, it behaves like getstore( ). This can be very handy in one-liners such as:

% perl -MLWP::Simple -e "getprint('http://cpan.org/RECENT')||die" | grep Apache

2.3.4. Previewing with HEAD

The return value of head( ) depends on whether you call it in scalar context or list context. In scalar context, it is simply:

$is_success = head(url);

If the server answers the HEAD request with a successful status code, this returns a true value. Otherwise, it returns a false value. You can use this like so:

die "I don't think I'll be able to get $url" unless head($url);

Regrettably, however, some old servers, and most CGIs running on newer servers, do not understand HEAD requests. In that case, they should reply with a "405 Method Not Allowed" message, but some actually respond as if you had performed a GET request. With the minimal interface that head( ) provides, you can't really deal with either of those cases, because you can't get the status code on unsuccessful requests, nor can you get the content (which, in theory, there should never be any).

In list context, head( ) returns a list of five values, if the request is successful:

(content_type, document_length, modified_time, expires, server)
    = head(url);

Currently, that program prints the following, when run:

http://us.a1.yimg.com/us.yimg.com/i/ww/m5v9.gif
That image/gif document is 5611 bytes long.
It was modified 251207569 seconds ago; that's about 2907 days ago, at Thu Apr 14 18:00:00 1994!

http://hooboy.no-such-host.int/
Couldn't get http://hooboy.no-such-host.int/

http://www.yahoo.com
That text/html document is ??? bytes long.
I don't know when it was last modified.

http://www.ora.com/ask_tim/graphics/asktim_header_main.gif
That image/gif document is 8588 bytes long.
It was modified 62185120 seconds ago; that's about 720 days ago, at Mon Apr 10 12:14:13 2000!

http://www.guardian.co.uk/
That text/html document is ??? bytes long.
I don't know when it was last modified.

http://www.pixunlimited.co.uk/siteheaders/Guardian.gif
That image/gif document is 4659 bytes long.
It was modified 24518302 seconds ago; that's about 284 days ago, at Wed Jun 20 11:14:33 2001!

Incidentally, if you are using the very popular CGI.pm module, be aware that it exports a function called head( ) too. To avoid a clash, you can just tell LWP::Simple to export every function it normally would except for head( ):

use LWP::Simple qw(!head);
use CGI qw(:standard);

If not for that qw(!head), LWP::Simple would export head( ), then CGI would export head( ) (as it's in that module's :standard group), which would clash, producing a mildly cryptic warning such as "Prototype mismatch: sub main::head ($) vs none." Because any program using the CGI library is almost definitely a CGI script, any such warning (or, in fact, any message to STDERR) is usually enough to abort that CGI with a "500 Internal Server Error" message.