|
Home · All Namespaces · All Classes · Grouped Classes · Modules · Functions | |
Contents:
The Test Plan defines the Quality Assurance procedures used to verify Qt Extended.
The purpose of the Test Plan is to describe in full the tests that are to take place, as acceptance of the product. The plan contains the complete set of criteria and verification protocols, which are to be employed in confirming that the product integration is complete (i.e., the functionality of the system is confirmed).
The intended readers for this document are:
The Test Plan applies to the current version of Qt Extended.
The Test Plan is intended to provide the vehicle for customers to confirm the functionality and completeness of the product. It is expected that the satisfaction of the complete series of criteria contained in this plan will signify successful functionality of the integrated product.
The goal is to deliver a product with no open bugs with a severity level of 1 and a minimal number of open bugs with severity level 2.
During the development process as well as during the release phase, Test Cases will be executed. The Test Cases are described in test documents which are (depending on the license type) part of the source package. Each Test Case also contains test specific criteria which decide upon Test Cases success of failure.
Each Test Case can have one of the following outcomes:
For each requirement, in principle, one or more Test Cases will exist.
Each Test Case is written as either a Unit test in C++ code (using QTestLib) or as an Integration/System test written in QtScript (using QtUiTest). The test plan provides an extract of the information from the test sources, and extends this with information from e.g. the requirements database.
Each Test Case described in the test plan contains the following fields:
| Field Name | Description |
|---|---|
| Requirement Summary | A summary of the requirement text. This information is extracted from the requirement database used for the development of Qt Extended. |
| Tested Application | The application that is tested. |
| Test Type | The type of test: Unit, Integration, System, Performance test. Additionally an indication can be given whether the test is a Manual or Automated test. |
| File Name | A reference to the actual file that contains the test case. The <basedir> reference points to the root directory where the source package is installed. |
Components can be tested by a combination of Unit, Integration and System tests.
For a System test the component is typically tested by launching the application in the test environment and by executing functionality of the component using the User Interface, and by verifying the outcome against an 'expected result'.
In cases where manual verification is required the expected outcome is specified in the testcase.
For an Integration test the component is typically tested by launching the component (if it has a GUI), or by launching other applications that make use of the tested component, inside the test environment and by executing functionality of the component using the User Interface. Where applicable, parts of the system may be replaced by stubs to partially insulate the component from it's environment.
A Unit test is typically used to test a single class of a component. The class-under-test is typically insulated from its environment by creating a standalone instance in a separate process, but where needed the class can interact with the environment.
Where possible all or parts of the test steps are automated, but in all cases there will be a manually executed 'Acceptance Test' for each GUI component in which the module will, at least, be verified against a standard checklist.
Tests are written on a highest priority first basis, and secondly on broad coverage.
At this stage we are aiming towards full functional coverage. Long term the goal is towards 100% code coverage. Note that only a subset of all tests are provided in the source package, and that the Test Plan and Test Results only mention the results for the actually shipped tests. Future releases may gradually include more tests.
Qt Extended is tested in a number of configurations, both in a Virtual Framebuffer as well as on reference devices. The public configurations used for this release are:
All automated tests are executed in a continuous development environment, and typically test results are available for each change to the code.
Manual tests are executed regularly, but certainly before each release.
A test report is included as part of the online reference documentation that comes with each source package and contains results for both manual and automated tests.
All tests are written with the use of QTestLib and/or QtUiTest.
Additional tools such as Valgrind may be used to do a deep analysis of test details/failures and tools such as LinkLint are used in the documentation process. These tools which will provide additional data used to improve the quality of the product but are not necessarily used in the Go/No-Go decision making process.
Each GUI component such as applications, games, settings and tools will need to be tested in a manual test. This test is in addition to any Manual/Automated tests that are testing specific parts of that component.
The checklist for all GUI components is as follows:
The performance target is to produce a quality product with a performance that is competitive in the market. This is going to be achieved by first looking at the different areas that effect the performance of Qt Extended, on actual devices, and then trying to measure these values to find out where performance needs to be improved.
The focus areas are:
Qt Extended is developed and tested with the following device parameters in mind:
| Property | Target Value |
|---|---|
| CPU | 200mhz ARM9 |
| RAM | 64MB |
| ROM | 64MB |
Deployment on devices with lower performance characteristics should be possible, but may require tweaking of the system.
Based on the target system and our goal to have a fast and responsive system we have defined the goals below. These are our initial targets only. Once we have reached these targets we will continue to improve and create a new set of targets to aim for.
| Target Type | Target Value |
|---|---|
| System Startup | The system must start in under 15 seconds (please note this does not include the OS startup time). |
| Applications | A relatively complex application must be able to start in under 2 seconds. |
| Multitasking | Three to four applications should be able to be run at any one time. |
| UI | A 'wait' cursor will be shown for actions that take more than 0.4 seconds and a progressbar + cancel option will be shown for actions that take more than 4 seconds. |
Qt Extended can be tested on devices as well as in a Virtual environment. Both cases have value and may provide data that would be hard or impossible to acquire 'on the other side'.
Once Qt Extended is installed in the Virtual environment or installed onto a device, no additional environment setup should be required that isn't already required by Qt Extended itself. It is important though that the environment is equal for each testrun, i.e. the same background processes are running for each test run, and that only background processes are running that are required.
Certain aspects of Qt Extended can only be tested if associated hardware modules are available on the device/desktop machine on which the test is executed. For instance:
Please refer to the manuals that come with these items to setup correctly.
http://snad.ncsl.nist.gov/nistnet/
https://helixcommunity.org/developers/testing/community_testing
$QPEDIR/image/lib/helix
For example:
In most cases, a test setup consists of:
Specific sub-domains for GCF testing are (but not limited to):
IMPORTANT: Tests that are expected to make use of a device's Phone modem need to be executed in a 'safe' environment that reduce the risk of interfering with public phone networks.
To test Qt Extended in a virtual environment, Qt Extended needs to be built for the desktop which is accomplished by configuring with the -qvfb command line:
<source-path>/configure -device foo -qvfb -add-module qtuitest [extra-configure-options]
bin/qbuild
bin/qbuild image
The -add-module qtuitest configure option is required to enable System testing if the device profile does not already enable the qtuitest module.
Once Qt Extended is built it can be executed using 'runqtopia', and tests can be executed as described in detail in the QtUiTest Manual.
To test Qt Extended on a device, Qt Extended needs to be built for that specific device:
<source-path>/configure -device foo -add-module qtuitest [extra-configure-options]
bin/qbuild
bin/qbuild image
The -add-module qtuitest configure option is required to enable System testing if the device profile does not already enable the qtuitest module.
After the build is finished the Qt Extended binaries then need to be copied over (flashed) onto the device. See the device manual for specific instructions on how to do this.
Once Qt Extended is flashed onto the device it usually needs to be restarted to activate the new version, after which tests can be executed as described in detail in the QtUiTest Manual.
Additional functionality that is required to run automated tests:
Tests are available for the following modules:
All requirements can also be traced in the Requirements Traceability Matrix
| Copyright © 2009 Nokia | Trademarks | Qt Extended 4.4.3 |