Boost Test Library: The Test Execution Monitor

Introduction
Usage
Implementation
Compilation
Configuration
Examples and tests

See also: Unit Test Framework

Introduction

How should a test program report errors? Displaying an error message is an obvious possibility:

if( something_bad_detected )
  std::cout << "something bad has been detected" << std::endl;

But that requires inspection of the program's output after each run to determine if an error occurred. Since test programs are often run as part of a regression test suite, human inspection of output to detect error messages is too time consuming and unreliable. Test frameworks like GNU/expect can do the inspections automatically, but are overly complex for simple testing.

A better simple way to report errors is for the test program to return EXIT_SUCCESS (normally 0) if the test program completes satisfactorily, and EXIT_FAILURE if an error is detected. This allows a simple regression test script to automatically and unambiguous detect success or failure. Further appropriate actions such as creating an HTML table or emailing an alert can be taken by the script, and can be modified as desired without having to change the actual C++ test programs.

A testing protocol based on a policy of test programs returning EXIT_SUCCESS or EXIT_FAILURE does not require any supporting tools; the C++ language and standard library are sufficient. The programmer must remember, however, to catch all exceptions and convert them to program exits with non-zero return codes. The programmer must also remember to not use the standard library assert() macro for test code, because on some systems it results in undesirable side effects like a message requiring manual intervention.

The Boost Test Library's Test Execution Monitor combines the powers of the Test Tools and the Execution Monitor to automates those tasks. It provides a main() function which calls a user-supplied test_main() function. The library supplied main() relieves users from messy error detection and reporting duties. Users could use the Test Tools to perform complex validation tasks.

The Test Execution Monitor is intended for fairly simple test programs or to dig a problem in an existent production code. The Program Execution Monitor may be more suitable to monitor production (non-test) programs, cause it does not affect your program performance. The Unit Test Framework may be, for several reasons, more suitable for complex test programs:

you will be able to separate test on several test cases. The Unit test framework will produce the pass/fail statistic for each test case separately
if any of your test cases fails with exception if does not affect the rest of test
you will be able to run specific test case by it's name
separation onto test cases allows you clearly see which part of your test module is dedicated to which part of tested functionality
extra options for configuration

Usage

Let's take a look on the following simple test program:

#include <my_class.hpp>

int main( int, char* [] )
{
    my_class test_object( "qwerty" );

    return test_object.is_valid() ? EXIT_SUCCESS : EXIT_FAILURE;
}

There are several issues with above test.

You need to convert is_valid result in proper result code.
Would exception happened in test_object construction of method is_valid invocation, the program will crash.
You wont see any output, would you run this test manually.

The Test Execution Monitor solve all these issues. To integrate with it above program needs to be changed to:

#include <my_class.hpp>
#include <boost/test/test_tools.hpp>

int test_main( int, char* [] )  // note the name!
{
    my_class test_object( "qwerty" );

    BOOST_CHECK( test_object.is_valid() );

    return 0;
}

Now, you not only receive uniform result code, even in case of exception, but also nicely formatted output from BOOST_CHECK tool, would you choose to see it. Is there any other ways to perform checks? The following example test program shows six different ways to detect and report an error in the add() function.

#include <boost/test/test_tools.hpp>

int add( int i, int j ) { return i+j; }

int test_main( int, char *[] )             // note the name!
{
    // six ways to detect and report the same error:
    BOOST_CHECK( add( 2,2 ) == 4 );        // #1 continues on error
    BOOST_REQUIRE( add( 2,2 ) == 4 );      // #2 throws on error
    if( add( 2,2 ) != 4 )
      BOOST_ERROR( "Ouch..." );            // #3 continues on error
    if( add( 2,2 ) != 4 )
      BOOST_FAIL( "Ouch..." );             // #4 throws on error
    if( add( 2,2 ) != 4 ) throw "Oops..."; // #5 throws on error

    return add( 2, 2 ) == 4 ? 0 : 1;       // #6 returns error code
}

Approach #1 uses the BOOST_CHECK tool, which displays an error message on std::cout that includes the expression that failed, the source file name, and the source file line number. It also increments an error count. At program termination, the error count will be displayed automatically by the Test Execution Monitor.

Approach #2 using the BOOST_REQUIRE tool, is similar to #1, except that after displaying the error, an exception is thrown, to be caught by the Test Execution Monitor. This approach is suitable when writing a explicit test program, and the error would be so severe as to make further testing impractical. BOOST_REQUIRE differs from the C++ Standard Library's assert() macro in that it is always generated, and channels error detection into the uniform Test Execution Monitor reporting procedure.

Approaches #3 and #4 are similar to #1 and #2 respectively, except that the error detection is coded separately. This is most useful when the specific condition being tested is not indicative of the reason for failure.

Approach #5 throws an exception, which will be caught and reported by the Test Execution Monitor. This approach is suitable for both production and test code, in libraries or not. The error message displayed when the exception is caught will be most meaningful if the exception is derived from std::exception, or is a char* or std::string.

Approach #6 uses a return value to inform the caller of the error. This approach is particularly suitable for integrating existing test code with the test tools library. Although it works fine with the Program Execution Monitor or the Test Execution Monitor libraries, and is very useful for running existing code under them, most C++ experts prefer using exceptions for error reporting.

Implementation

The Test Execution Monitor implements a special case of the framework usage. Accordingly it shares most of the implementation with the Unit Test Framework. The Test Execution Monitor supplies the separate function main() in libs/test/src/test_main.cpp.

Check compilation instructions to see how to build standalone library for this component.

Configuration

The Test Execution Monitor uses the same configuration mechanisms as the Unit Test Framework do. For the list of supported parameters and their description see the Unit Test Framework parameters description.

Examples and tests

test_exec_example
test_exec_fail1
test_exec_fail2
test_exec_fail3
test_exec_fail4