3 Writing Test Suites
3.1 Support for test suite authors
The ct module provides the main interface for writing
test cases. This includes:
-
Functions for printing and logging
-
Functions for reading configuration data
-
Function for terminating a test case with error reason
-
Function for adding comments to the HTML overview page
-
Tracing of line numbers in the test suite, i.e. if a test
case fails, the last 10 executed line numbers are displayed
Please turn to the reference manual for the ct
module for details about these functions.
The CT application also includes other modules named
ct_<something> that
provide support for the use of communication mechanisms such as
rpc, snmp, ftp, telnet etc in test suites.
3.2 Test suites
A test suite is an ordinary Erlang module that contains test
cases. It is recommended that the module has a name on the form
*_SUITE.erl. Otherwise, the directory function in CT will
not be able to locate the modules (per default).
ct.hrl shall be included in all test suite files.
Each test suite module must export the function all/0
which returns the list of all test cases in that module.
3.3 Init and end per suite
Each test suite module may contain the functions
init_per_suite/1 and end_per_suite/1.
If it exists, init_per_suite is called as the first
testcase of the suite. It typically contains initiation which is
common for all test cases in the suite, and which are only to be
performed once.
end_per_suite is called as the last test case of the
suite. This function should clean up after init_per_suite.
The argument to init_per_suite is Config, the
same as the argument to all test cases. init_per_suite can
modify this parameter with information that the other test cases
need.
If init_per_suite fails, all test cases in the test
suite will be skipped, including end_per_suite
It is possible in end_per_suite to check if the last executed
test case was successful or not (which consequently may determine
how cleanup should be performed). This is done by reading the value
tagged with tc_status from Config. The value is either
ok, {failed,Reason}, or {skipped,Reason}.
3.4 Init and end per test case
Each test suite module can contain the functions
init_per_testcase/2 and end_per_testcase/2.
If it exists, init_per_testcase is called before each
test case in the suite. It typically contains initiation which
must be done for each test case.
end_per_testcase/2 is called after each test case is
completed, giving a possibility to clean up.
The first argument to these functions is the name of the test
case. This can be used to do individual initiation and cleanup for
each test cases.
The second argument is called
Config. init_per_testcase/2 may modify this
parameter or just return it as is. Whatever is retuned by
init_per_testcase/2 is given as Config parameter to
the test case itself.
The return value of end_per_testcase/2 is ignored by the
test server (with exception of the
save_config
tuple).
If init_per_testcase crashes, the test case itself is
skipped and end_per_testcase is never called.
3.5 Test cases
The smallest unit that the test server is concerned with is a
test case. Each test case can in turn test many things, for
example make several calls to the same interface function with
different parameters.
It is possible to put many or few tests into each test
case. How many things each test case does is of course up to the
author, but here are some things to keep in mind:
Using many small test cases tend to result in extra and often
duplicated code as well as slow test execution because of
large overhead for initializations and cleanups. Lots of duplicated
code results in high maintenance cost and bad readability.
Larger test cases make it harder to tell what went wrong if it
fails, and large portions of test code will be skipped if a
specific part fails. Also, readability and maintainability suffers
when test cases become too extensive.
The test case function takes one argument, Config, which
contains configuration information such as data_dir and
priv_dir. See Data and
Private Directories for more information about these.
Note
The test case function argument Config should not be
confused with the information that can be retrieved from
configuration files (using ct:get_config/[1,2]). The Config argument
should be used for runtime configuration of the test suite and the
test cases. A configuration file should contain data related to the
SUT (system under test). These two types of config data are handled
differently!
All Config items can be extracted using the
?config macro, e.g PrivDir = ?config(priv_dir,Config).
If the test case function crashes or exits, it is considered a
failure. If it returns a value (no matter what actual value) it is
considered a success. An exception to this rule is the return value
{skip,Reason}. If this is returned, the test case is considered
skipped and gets logged as such.
If the test case returns the tuple {comment,Comment},
Comment is printed out in the overview log (this is equal to
calling ct:comment(Comment)).
3.6 Test case info function
For each test case function there can be an additional function
with the same name but with no arguments. This is the test case
info function. The test case info function is expected to return a
list of tagged tuples that specifies various properties regarding the
test case.
The following tags have special meaning:
-
timetrap
-
Set the maximum time the test case is allowed to take. If
the timetrap time is exceeded, the test case fails with
reason timetrap_timeout. Note that init_per_testcase
and end_per_testcase are included in the timetrap time.
-
userdata
-
Use this to specify arbitrary data related to the testcase. This
data can be retrieved at any time using the ct:userdata/3
utility function.
-
silent_connections
-
Please see the
Silent Connections
chapter for details.
-
require
-
Use this to specify configuration variables that are required by the
test case. If the required configuration variables are not
found in any of the test system configuration files, the test case is
skipped.
It is also possible to give a required variable a default value that will
be used if the variable is not found in any configuration file. To specify
a default value, start by adding a normal require-tuple to the list for the
variable in question and then let the key-value definition for the variable
follow. The definition should have the same form as in the configuration file.
Example:
my_testcase() ->
[{require, unix_telnet, {unix, [telnet, username, password]}},
{unix_telnet, {unix, [{telnet, "durin.du.uab"},
{username, "alladin"},
{password, "sesame"}]}}].
Note
Giving a required variable a default value can result
in that a test case is always run. For many types
of configuration values this is not a desired behavior.
If timetrap and/or require is not set specifically for
a particular test case, default values specified by the suite/0
function are used.
Other tags than the ones mentioned above will simply be ignored by
the test server.
Example:
reboot_node() ->
[
{timetrap,{seconds,60}},
{require,interfaces},
{userdata,
[{description,"System Upgrade: RpuAddition Normal RebootNode"},
{fts,"http://someserver.ericsson.se/test_doc4711.pdf"}]}
].
3.7 Test suite default data
The suite/0 function can be used in a test suite
module to set the default values for the timetrap and
require tags. If a test case info function also specifies
any of these tags, the default value is overruled. See above for
more information.
Other options that may be specified with the suite defaults list are:
Example:
suite() ->
[
{timetrap,{minutes,10}},
{require,global_names},
{userdata,[{info,"This suite tests database transactions."}]},
{silent_connections,[telnet]},
{stylesheet,"db_testing.css"}
].
3.8 Data and Private Directories
The data directory (data_dir) is the directory where the
test module has its own files needed for the testing. The name
of the data_dir is the the name of the test suite followed
by "_data". For example,
"some_path/foo_SUITE.beam" has the data directory
"some_path/foo_SUITE_data/".
The priv_dir is the test suite's private directory. This
directory should be used when a test case needs to write to
files. The name of the private directory is generated by the test
server, which also creates the directory.
Note
You should not depend on current working directory for
reading and writing data files since this is not portable. All
scratch files are to be written in the priv_dir and all
data files should be located in data_dir. If you do need
to use the current working directory, you must set it explicitly with
file:set_cwd/1 for each individual test case before use.
(The Common Test server sets current working directory to the test case
log directory at the start of every case).
3.9 Execution environment
Each test case, including init_per_testcase and
end_per_testcase is executed by a dedicated Erlang process. The
process is spawned when the test case starts, and terminated when
the test case is finished.
init_per_suite and end_per_suite are separate
test cases and will execute on their own processes.
The default time limit for a test case is 30 minutes, unless a
timetrap is specified either by the test case info function
or by the suite/0 function.
3.10 Illegal dependencies
Even though it is highly efficient to write test suites with
the Common Test framework, there will be mistakes in the
test suites. Noted below are some of the more frequent
dependency mistakes from our experience with running the
Erlang/OTP test suites.
-
Depending on current directory, and writing there:
This is a common error in test suites. It is assumed that
the current directory is the same as what the author used as
current directory when the test case was developed. Many test
cases even try to write scratch files to this directory. If
the current directory has to be set to something in
particular, use file:set_cwd/1 to set it. And
use the data_dir and priv_dir to locate data and
scratch files.
-
Depending on the Clearcase (file version control tool)
paths and files:
The test suites are stored in Clearcase but are not
(necessarily) run within this environment. The directory
structure may vary from test run to test run.
-
Depending on execution order:
There is no way of telling in which order the test cases
are going to be run, so a test case can't depend on a server
being started by a test case that runs "before". This has to
be so for several reasons:
The user may specify the order at will, and maybe some
particular order is better suited sometimes. Secondly, if the
user just specifies a test directory, the order the suites are
executed will depend on how the files are listed by the operating
system, which varies between systems. Thirdly, if a user
wishes to run only a subset of a test suite, there is no way
one test case could successfully depend on another.
-
Depending on Unix:
Running unix commands through unix:cmd or os:cmd are likely
not to work on non-unix platforms.
-
Nested test cases:
Invoking a test case from another not only tests the same
thing twice, but also makes it harder to follow what exactly
is being tested. Also, if the called test case fails for some
reason, so will the caller. This way one error gives cause to
several error reports, which is less than ideal.
Functionality common for many test case functions may be implemented
in common help functions. If these functions are useful for test cases
across suites, put the help functions into common help modules.
-
Failure to crash or exit when things go wrong:
Making requests without checking that the return value
indicates success may be ok if the test case will fail at a
later stage, but it is never acceptable just to print an error
message (into the log file) and return successfully. Such test cases
do harm since they create a false sense of security when overviewing
the test results.
-
Messing up for following test cases:
Test cases should restore as much of the execution
environment as possible, so that the following test cases will
not crash because of execution order of the test cases.
The function end_per_testcase is suitable for this.
common_test 1.3.4
Copyright © 1991-2008
Ericsson AB