 |  |
| |
What the Test Harness Does
Writing Tests with Test::Simple
Writing Tests with Test::More
Conditional Tests
More Complex Tests (Multiple Test Scripts)
Testing Things That Write to STDOUT and STDERR
Exercise
As briefly described in Chapter 13, a distribution contains a testing facility invoked from make test. This testing facility permits a module author to write and run tests during development and maintenance and the ultimate module installer to verify that the module works in the new environment.
Why have tests during development? One emerging school of thought states that the tests should be written first, even before the module is created, as a reflection of the module's specification. Of course, the initial test run against the unwritten module will show nearly complete failure. However, as functionality is added, proper functionality is verified immediately. (It's also handy to invoke the tests frequently as you code to make sure you're getting closer to the goal, not breaking more things.)
Certainly, errors may be found in the test suite. However, the defect rate for tests are usually far lower than the defect rate for complex module code; if a test fails, it's usually a good indication that there's more work to be done.
But even when Version 1.0 of the module is finally shipped, there's no need to abandon the test suite. Unless You code the mythical "bug-free module," there will be bug reports. Each bug report can (and should) be turned into a test.[92] While fixing the bug, the remaining tests prevent regression to a less functional version of the code—hence the name regression testing.
[92]If you're reporting a bug in someone else's code, you can generally assume that sending them a test for the bug will be appreciated. A patch would be appreciated even more!
Then there's always the 1.1 or 2.0 releases to think about. When you want to add functionality, start by adding tests.[93] Because the existing tests ensure your upward compatibility, you can be confident that your new release does everything the old release did, and then some.
[93]And writing the documentation at the same time, made easier by Test::Inline, as you'll see later.
Good tests also give small examples of what you meant in your documentation, in case your writing isn't clear.[94] Good tests also give confidence to the installer that this code is portable enough to work on both your system and his system, including all stated and unstated dependencies.
[94]Many modules we've used from the CPAN were documented more by test examples than by the actual POD. Of course, any really good example should be repeated in your module's POD documentation.
Testing is an art. Dozens of how-to-test books have been written and read, and often ignored. Mostly, it's important to remember everything you have ever done wrong while programming (or heard other people do), and then test that you didn't do it again for this project.
Test things that should break (throw exceptions or return false values) as well as things that should work. Test the edges. Test the middle. Test one more or one less than the edge. Test things one at a time. Test many things at once. If something should throw an exception, make sure it didn't also negatively affect the state of the world before it threw the exception. Pass extra parameters. Pass insufficient parameters. Mess up the capitalization on named parameters. Throw far too much data at it. Throw far too little. Test what happens for undef. And so on.
For example, suppose that you want to test Perl's sqrt function, which calculates square roots. It's obvious that you need to make sure it returns the right values when its parameter is 0, 1, 49, or 100. It's nearly as obvious to see that sqrt(0.25) should come out to be 0.5. You should also ensure that multiplying the value for sqrt(7) by itself gives something between 6.99999 and 7.00001.[95] You should make sure that sqrt(-1) yields a fatal error and that sqrt(-100) does too. See what happens when you request sqrt(&test_sub( )), and &test_sub returns a string of "10000". What does sqrt(undef) do? How about sqrt( ) or sqrt(1,1)? Maybe you want to give your function a googol: sqrt( '1' . '0' x 100 ). Because this function is documented to work on $_ by default, you should ensure that it does so. Even a simple function such as sqrt should get a couple of dozen tests; if your code does more complex tasks than sqrt does, expect it to need more tests, too. There are never too many tests.
[95]Remember, floating-point numbers aren't always exact; there's usually a little roundoff. Feel free to write your tests to require more precision than this test implies but don't require more precision than you can get on another machine!
If you write the code and not just the tests, think about how to get every line of your code exercised at least once for full code coverage. (Are you testing the else clause? Are you testing every elsif case?) If you aren't writing the code or aren't sure, use the code coverage facilities.[96]
[96]Basic code coverage tools such as Devel::Cover are found in the CPAN.
Check out other test suites. The Perl distribution itself comes with thousands of tests, designed to verify that Perl compiles correctly on your machine in every possible way. Michael Schwern earned the title of "Perl Test Master" for getting the Perl core completely tested, and, still constantly beats the drum for "test! test! test!" in the community.
In summary, please write tests. Let's see how this is done.
Tests are usually invoked (either for the developer or the installer) using make test. The Makefile invokes the test harness, which eventually gets around to using the Test::Harness module to run the tests.
Each test lives in a separate .t file in the t directory at the top level of the distribution. Each test is invoked separately, so an exit or die terminates only that test file, not the whole testing process.
The test file communicates with the test harness through simple messages on standard output. The three most important messages are the test count, a success message, and a failure message.
An individual test file consists of one or more tests. These tests are numbered as small integers starting with one. The first thing a test file must announce to the test harness (on STDOUT) is the expected test number range, as a string 1..n. For example, if there are 17 tests, the first line of output should be:
1..17
followed by a newline. The test harness uses the upper number here to verify that the test file hasn't just terminated early. If the test file is testing optional things and has no testing to do for this particular invocation, the string 1..0 suffices.
After the header, individual successes and failures are indicated by messages of the form ok N and not ok N. For example, here's a test of basic arithmetic. First, print the header:
print "1..4\n"; # the header
Now test that 1 plus 2 is 3:
if (1 + 2 == 3) {
print "ok 1\n"; # first test is OK
} else {
print "not ok 1\n"; # first test failed
}You can also print the not if the test failed.[97]
[97]On some platforms, this may fail unnecessarily. For maximum portability, print the entire string of ok N or not ok N in one print step.
Don't forget the space!
print "not " unless 2 * 4 == 8; print "ok 2\n";
You could perhaps test that the results are close enough (important when dealing with floating-point values):
my $divide = 5 / 3; print "not " if abs($divide - 1.666667) > 0.001; # too much error print "ok 3\n";
Finally, you may want to deal with potential portability problems:
my $subtract = -3 + 3; print +(($subtract eq "0" or $subtract eq "-0") ? "ok 4" : "not ok 4"), "\n";
As you can see, there are many styles for writing the tests. In ancient Perl development, you saw many examples of each style. Thanks to Michael Schwern and chromatic and the other Perl Testing Cabal members, you can now write these much more simply, using Test::Simple.
|  | |
| 13.11. Exercise |  | 14.2. Writing Tests with Test::Simple |
Copyright © 2003 O'Reilly & Associates. All rights reserved.