MetaBoss Testing Framework Guide.

System Testing.

Index

Overview

A system test is used to answer questions like "Does system behaviour corresponds to expected behaviour documented in use cases and system design documentation ?" and, if it does not, then "What are the differences between expected and actual outcomes ?". In order to fulfill its purpose the system test needs to be able to execute services with wide variety of different input parameter sets and compare outputs against expected outcomes.

The picture above shows what happens during the run of system test. In more detail:

1. XML files with Test Case definitions are read, parsed and executed one by one. These XML files contain definitions of which system needs to be called and what input parameters must be used. They also contain precondition definitions which may be used to exclude certain Test Cases from running if certain condition has been detected. We will look at the content of the Test Case files in more details in the separate chapter below.
2. Comprehensive log file is being saved with every bit of detail about services being executed. It includes things like services, their inputs, outputs, results of various condition checking, performance data and more.

Some definitions:

• Test Case - is a XML Element defining a particular transaction, which consists of one or more service invocations and possibly some conditions checking expressions. A Test Case Element is recursive and allows one Test Case to contain one or more smaller Test Cases inside. Test Case element may also contain "dynamic" references to other Test Cases. These references are defined using XSLT and XPath notation. These references are very important tool used to dynamically connect service invocations together - that is use results of previous service invocation as an input to the subsequent service invocations.
• Test Case File - is an XML file containing one or more TestCases.
• Test Scenario - is a sequence of Test Cases defined in one or more Test Case Files, which is executed "as one".
• Test Log File - is a XML file containing details about a single Test Scenario execution. It contains explicit historical information about what has happened (as opposed to Test Case File which contains XSLT and XPath expressions and is is essentally a program).
• Test Engine (or Test Runner) - is a Java application which reads Test Scenario, executes it and writes the Test Log. At the moment only command line version of the Test Runner is supplied with MetaBoss in ${metaboss_home}/lib/MetaBossSystemTester.jar (the main class is com.metaboss.sdlctools.applications.systemtester.ScenarioRunner). This is enough to run automatic unattended tests from within Ant scripts as well as to perform unit testing and even debugging of the service implementations. In the future releases we are planning to introduce the Interactive GUI version of the Test Runner.

Considering that typical enterprise service impacts the data stored on database, one of the first decisions in preparation of Test Scenarios is the decision what will be the content of the database at the starting point for the test scenario. The choices and their characteristics are:

• Start with empty database. This choice requires that early Test Cases in the test scenario (or alternatively the database scripts executed before the Test Scenario) populate database with some known data which then can be used by latter Test Cases. The benefit of this approach is the simplicity of writing Test Cases. This is because the database is always in the known state, so many input values can be specified as constants, which is simpler than dynamic values specified with XSLT. The disadvantage of this approach is that it requires services which are able to populate the database (i.e. CRUD or Administration type services or scripts). These facilities may not always be available or accessible. The other disadvantage is that running of tests under this scenario requires cleaning up of existing contents from the database and acquiring exclusive use of database.
• Start with database with any contents. This choice requires that all Test Cases be smart enough to account for any possible contents found in database and possible even make use of these contents. The benefit of this approach is that Test Scenario can be run on any database (in fact query only Test Scenarios can even be run on production systems to verify their health) and the exclusive use of database is not required. The major drawback is that the Test Cases themselves tend to be more complex as they have to cater for wider range of possibilities.

In practice the approach is usually a mixture of two choices presented above. The choice is even more complex in multisystem Test Scenarios with a number of systems and/or databases to consider. We favour the empty database approach due to its simplicity and outcome predictability. This approach, as we will see later, also makes it possible to reuse same Test Case Fiels in the regression testing.

Use of Special XML Elements to create dynamic test cases

One of the powerful features of the MetaBoss Testing Framework is the ability to use dynamic rendering of the Test Cases based on some dynamic values. The dynamic values are the values not known at the Test Case design time, such as:

• The current date or time related value. For example the Birth Registry System for Delivery Ward at the hospital might be designed to reject the Date Of Birth values older than two days. The Test Case for such system must be able to come up with the valid Date Of Birth value every time it runs.
• The environment property related value. This kind of dynamic value may be used to define test execution path or test data values at the test run time rather that at the test design time.
• The value derived from the results of running previous Test Cases. This type of dynamic value is probably used most often. Ability to have these values is the foundation of the Test Scenario mechanism, which allows to chain Test Cases in sequences with results of previous Test Cases able to be used in the subsequent ones.
• The value stored in the external data file.

The Special XML Elements dynamic rendering mechanism works as follows:

1. The Test Engine reads Test Cases, interprets them and invokes services.
2. At the same time comprehensive Test Log File (also in XML) is being contineously written. It contains information from original Test Cases as well as outputs obtained from the service calls and results of checking these outputs. So, at any point in time the Test Log document contains information about everything what has occurred so far in the test run.
3. When the Test Engine discovers an embedded Special XML Element inside the Test Case it performs action dictated by this element and uses the resulting value (if any) as if it was read from the original Test Case instead of the Special XML Element. In other words certain portions of the definition of the Test Case to be executed in the next step can be dynamically derived with help of these Special XML Elements.

Following Special XML Elements are currently supported:

• <ValueOf> This element is replaced with the value of environment property or result of XPath search through the Test Log accumulated up to the point when this element is processed (actual action depends on which attribute is specified inside this element). When it defines the XPath expression search - it actually works similar to the XSL Templates mechanism (described below in this gude) but it is much simpler to write.
• <ValueOfTimestamp> This element is replaced with the value of current time formatted as per format specification contained in its format attribute. It also offers facility to adjust the current time by specified number of days, minutes etc.
• <Include> This element is replaced with the content of the file specified in its filename attribute. The included file must be a valid XML file.

Use of XSL Templates to create dynamic test cases

As it was mentioned earlier, one of the powerful features of the MetaBoss Testing Framework is the ability to use dynamic rendering of the Test Cases based on some dynamic values. Previous chapter described how this can be done with use of special XML elements. This chapter will describe how similar outcome can be achieved with use of XSL Templates (XSL stands for eXtensible Stylesheet Language). Note that both methods (XSL Templates and special XML elements) are not mutually exclusive and their set of featues is not the same (there is some overlap though). The XSL Templates can be used to:

• Define series of service calls based on the series of input data sets.
• Define parts or whole of input data to the service call as a function of outputs obtained from the preceeding service call.
• Dynamically decide whether to execute a particular test case based on what happened in the test run prior to the decision making moment
• Dynamically validate the output from the service call and mark the test case as failed or succeeded.

The XSL Template dynamic rendering mechanism works as follows:

1. The Test Engine reads Test Cases, interprets them and invokes services.
2. At the same time comprehensive Test Log File (also in XML) is being contineously written. It contains information from original Test Cases as well as outputs obtained from the service calls and results of checking these outputs. So, at any point in time the Test Log document contains information about everything what has occurred so far in the test run.
3. When the Test Engine discovers an embedded XSL Template inside the Test Case it performs transformation of the Test Log with use of this XSL Template and uses the resulting XML document as if it was read from the original Test Case instead of the XSL Template. In other words any portion of the Test Case to be executed in the next step can be dynamically derived from the Test Log containing everything what has happened in all previous steps.

Test Case File Contents Description

The role of the Test Case File is to define execution sequence of operations, inputs to these operations and checks to be performed on the outputs from these operations.