HatMaker SystemTest Example Description.
Example Overview
The HatMaker System Test example shows how to use the MetaBoss Testing Framework.
The underlying requirement of the example is quite simple:
- Prior to coding of the CRM system we needed to have a CRM system simulator
so our GUI Client developers can work in parallel with middleware developers and
our Testers can (again in parallel) play with the system and prepare test cases.
- After the real implementation of the CRM system was coded we needed to test it against
requirements using prepared test cases.
- When subsequent releases of the CRM were made we needed to regression test these releases
to ensure that the system has not changed its behaviour in unexpected spots.
In this document we will show how these goals can be achieved. Please note that
the MetaBoss documentation pack also contains the Testing Framework User Gude and the Test Schema Reference
which should help you to better understand this example.
All files directly related to the HatMaker System Test example are located in the
${metaboss_home}/examples/HatMaker/SystemTest subdirectory. Some notable files and
directories are:
- ${metaboss_home}/examples/HatMaker/SystemTest/test.xml - Apache Ant file with
all procedures used in this example (eg. initialise database, run test, run simulator etc)
- ${metaboss_home}/examples/HatMaker/SystemTest/*.bat - various DOS batch files used to run
all sorts of test related procedures. The purpose of the particular batch file
is documented at the beginning of the file. One particular file - setenvironment.bat is used
to set the environment (eg. JAVA_HOME, ANT_HOME etc). You will need to review and possibly
modify this file prior to running the tests.
Please note that this system test example is a real-life working example in a sence that it
is used in MetaBoss product releases. Each time during release of MetaBoss we
automatically generate, build and test HatMaker in a number of configurations. At the end
we automatically verify that test results from all tested configurations do match and that there is no regression
(at least there is no unexpected regression).
All log files from the tests we run during MetaBoss product release are located in the
${metaboss_home}/examples/HatMaker/SystemTest/testlogs directory.
Using Simulator
This area is under construction
System Testing
System testing is performed by executing a collection of test scenarios against a particular enterprise configuration.
Consequently there are three key question: How to define test scenarios, How to define particular
enterprise configuration and How to run the test.
The test run is conducted by running Apache Ant with the ${metaboss_home}/examples/HatMaker/SystemTest/test.xml
script. This script file holds all the answers to the above questions. Some points to note are:
- Test scenarios are defined in series of XML files located in the ${metaboss_home}/examples/HatMaker/SystemTest/testscenario directory.
All xml files found in this directory (excluding the file named RegressionTestSpecimen.xml for the reasons we will explain latter)
will be executed in alphabetical order (in other words list of files will be sorted by name in alphabetical order)
- There are four test environment configurations provided in the example:
- Inprocess (not distributed) configuration using Oracle database server.
- Inprocess (not distributed) configuration using MySQL database server.
- J2EE Client (distributed) configuration using whatever database J2EE container is configured with
(we are using Oracle, but this is not important in this discussion).
- Inprocess (not distributed) configuration using XML files to provide
simulated data.
- The SQL scripts to initialise databases are generated during the build and stored in
${metaboss_home}/examples/HatMaker/SystemTest/dbscripts/oracle
directory for Oracle and ${metaboss_home}/examples/HatMaker/SystemTest/dbscripts/mysql
for MySQL. These scripts are used to automatically reinitialise database every time at the beginning of the test run.
Regression Testing
The first step in regression test preparation is to prepare a good specimen file. We do that by
taking a system test log file and transforming it in to a specimen file. The ${metaboss_home}/examples/HatMaker/SystemTest/specimenworkshop
subdirectory contains the example of procedure used to convert trusted test scenario log
into the specimen file. Some notable files are:
- ${metaboss_home}/examples/HatMaker/SystemTest/specimenworkshop/logToSpecimen.bat - a DOS batch file used to
run the conversion process.
- ${metaboss_home}/examples/HatMaker/SystemTest/specimenworkshop/goodlog.xml - The
trusted log file. This is a log file recorded as the result of the system test run which we find good enough to be
a specimen for regression testing. It was simply copied from the ${metaboss_home}/examples/HatMaker/SystemTest/testlogs
directory and renamed to goodlog.xml
- ${metaboss_home}/examples/HatMaker/SystemTest/specimenworkshop/logToSpecimen.xsl - The
stylesheet used to translate log files into the specimen files. Deals with removal of the
volatile fields or replacing of their values with regular expressions. In our case this template will
remove all PerformanceDocument elements and replace the values of the few timestamp fields with
regular expressions.
- ${metaboss_home}/examples/HatMaker/SystemTest/specimenworkshop/convert.xml - Apache Ant file with
the procedure used to convert the log file into the specimen file. It is very simple and only has
one build target - "convert". Inside this target we invoke the xslt transformation tool
which produces goodspecimen.xml by transforming goodlog.xml with use of logToSpecimen.xsl stylesheet.
- ${metaboss_home}/examples/HatMaker/SystemTest/specimenworkshop/goodspecimen.xml - The
converted log file which is now ready to be used as a specimen for the regression testing. You may wish to
check that all PerformanceDocument elements have disappeared and that the timestamp values have
in fact been replaced with the regular expressions.
The next step is to actually run the test. The procedure to run the regression test
is exactly the same as the one for system test with only one addition: parameter -specimenFile=
must be passed to the ScenarioRunner application and point to the specimen file. Please see
"runscenario.real.inprocess" build target inside ${metaboss_home}/examples/HatMaker/SystemTest/test.xml
for more details.
The final console output from the successful regression test run which has not detected any differences
between the current run and the specimen - should look like the example below:
|
