Building Test Cases

Whenever possible packages should be shipped with one or more test cases. This allows users to check that all packages function correctly in their particular configuration and on their target, which may be custom hardware unavailable to the package developer. The component framework needs to provide a way of building such test cases. For example, if a makefile system is used then there could be a make tests target to build the test cases, or possibly a make check target to build and run the test cases and process all the results. Unfortunately there are various complications.

Not every test case will be applicable to every configuration. For example if the user has disabled the C library's CYGPKG_LIBC_STDIO component then there is no point in building or running any of the test cases for that component. This implies that test cases need to be associated with configuration options somehow. It is possible for the test case to use one or more #ifdef statements to check whether or not it is applicable in the current configuration, and compile to a null program when not applicable. This is inefficient because the test case will still get built and possibly run, even though it will not provide any useful information.

Many packages involve direct interaction with hardware, for example a serial line or an ethernet interface. In such cases it is only worthwhile building and running the test if there is suitable software running at the other end of the serial line or listening on the same ethernet segment, and that software would typically have to run on the host. Of course the serial line in question may be hooked up to a different piece of hardware which the application needs to talk to, so disconnecting it and then hooking it up to the host for running some tests may be undesirable. The decision as to whether or not to build the test depends not just on the eCos configuration but also on the hardware setup and the availability of suitable host software.

There are different kinds of tests, and it is not always desirable to run all of them. For example a package may contain a number of stress tests intended to run for long periods of time, possibly days or longer. Such tests should certainly be distinguished somehow from ordinary test cases so that users will not run them accidentally and wonder how long they should wait for a pass message before giving up. Stress tests may also have dependencies on the hardware configuration and on host software, for example a network stress test may require lots of ethernet packets.

In the current implementation of the component framework these issues are not yet addressed. Instead there is only very limited support for building test cases. Any package can define a calculated configuration option of the form CYGPKG_<package-name>_TESTS, whose value is a list of test cases. The calculated property can involve an expression so it is possible to adapt to a small number of configuration options, but this quickly becomes unwieldy. A typical example would be:

        cdl_option CYGPKG_UITRON_TESTS {
            display "uITRON tests"
            flavor  data
            no_define
            calculated { "tests/test1 tests/test2 tests/test3 \
                tests/test4 tests/test5 tests/test6 tests/test7 \
                tests/test8 tests/test9 tests/testcxx tests/testcx2 \
                tests/testcx3 tests/testcx4 tests/testcx5 \
                tests/testcx6 tests/testcx7 tests/testcx8 \
                tests/testcx9 tests/testintr" }
            description   "
This option specifies the set of tests for the uITRON compatibility layer."
        }

This implies that there is a file tests/test1.c or tests/test1.cxx in the package's directory. The commands that will be used to build the test case will take the form:

    $(CC) -c $(INCLUDE_PATH) $(CFLAGS) -o <build path>/test1.o \
         <source path>/tests/test1.c
    $(CC) $(LDFLAGS) -o <install path>/tests/test1 <build_path>/test1.o

The variables $(CC) and so on are determined in the same way as for custom build steps. The various paths and the current directory will depend on the exact build system being used, and are subject to change. As usual the sources in the component repository are treated as a read-only resources, intermediate files live in the build tree, and the desired executables should end up in the install tree.

Each test source file must be self-contained. It is not possible at present to build a little per-package library that can be used by the test cases, or to link together several object files to produce a single test executable. In some cases it may be possible to #include source code from a shared file in order to avoid unnecessary code replication. There is no support for manipulating compiler or linker flags for individual test cases: the flags that will be used for all files are $(CFLAGS) and $(LDFLAGS), as per custom build steps. Note that it is possible for a package to define options of the form CYGPKG_<PACKAGE-NAME>_LDFLAGS_ADD and CYGPKG_<PACKAGE-NAME>_LDFLAGS_REMOVE. These will affect test cases, but in the absence of custom build steps they will have no other effect on the build.