Running Autotests

Qt Creator supports both code based tests and build system based tests. Code based testing offers special handling for particular testing frameworks that strongly ties to the underlying code models or specialized parsers. Build system based testing is independent from any testing frameworks. It retrieves information directly from the underlying build system and uses it or even the build system as such to execute the respective tests.

Qt Creator integrates the following testing frameworks for unit testing applications and libraries:

• Boost.Test
• Catch2 test framework
• Google C++ Testing Framework
• Qt Test framework

In addition, Qt Creator offers build system based support for CTest.

You can use Qt Creator to create, build, and run code based tests for your projects.

Build System Based Tests

By default, Qt Creator does not handle build system based tests to avoid interference with code based parsers. To enable build system based tests, select the respective test tool in Preferences > Testing > General.

The detection of tests is usually much faster for build system based tests as this normally does not involve scanning or parsing. The information in the tests tree is usually more detailed when using code based tests.

If you enable both code based and build system based tests, you may duplicate tests inside the tests tree. See also Selecting Tests to Run.

Creating Tests

Use a wizard to create projects that have tests.

Creating Qt and Qt Quick Tests

To create a Qt or Qt Quick test:

1. Select File > New Project > Other Project > Auto Test Project > Choose to create a project with boilerplate code for a Qt test or a Qt Quick test.
2. In the Project and Test Information dialog, specify settings for the project and test:
   1. In the Test framework field, select Qt Test or Qt Quick Test.
   2. For a Qt test, select the GUI Application check box to create a Qt application.

      

   3. In the Test case name field, enter a name for the test case.
   4. For a Qt test, select the Requires QApplication check box to add the include statement for QApplication to the main.cpp file of the project.
   5. For a Qt Quick test, select the Generate setup code check box to execute C++ code before running any of the QML tests. The testing framework will call slots and invocable functions, as described in Executing C++ Before QML Tests.

      

   6. Select the Generate initialization and cleanup code checkbox to add functions to your test that the testing framework executes to initialize and clean up the test.
   7. In the Build system field, select the build system to use for building the project: qmake, CMake, or Qbs.

Qt Creator creates the test in the specified project directory. Edit the .cpp file to add private slots for each test function in your test. For more information about creating Qt tests, see Creating a Test.

Creating Google Tests

To create a Google test:

1. Select File > New Project > Other Project > Auto Test Project > Choose to create a project with boilerplate code for a Google test.
2. In the Project and Test Information dialog, specify settings for the project and test:
   1. In the Test framework field, select Google Test.
   2. In the Test suite name field, enter a name for the test suite.
   3. In the Test case name field, enter a name for the test case.
   4. Select the Enable C++ 11 check box to support C++ 11 features in the test.
   5. In the Google test repository field, select a directory that has a clone of the googletest repository.

      To use an installed Google C++ Testing framework instead, see Setting Up the Google C++ Testing Framework.

   6. In the Build system field, select the build system to use for building the project: qmake, CMake, or Qbs.

Qt Creator creates the test in the specified project directory. For more information about creating Google tests, see the Google Test Primer.

Creating Boost Tests

To build and run Boost tests, you must have the Boost.Test installed on the development host. Typically, the Boost installation includes it. You can download Boost from Boost.org.

If the compiler and build system can find the Boost libraries, you do not need to specify the include directory when creating the test.

To create a Boost test:

1. Select File > New Project > Other Project > Auto Test Project > Choose to create a project with boilerplate code for a Boost test.
2. In the Project and Test Information dialog, specify settings for the project and test:
   1. In the Test framework field, select Boost Test.
   2. In the Test suite name field, enter a name for the test suite.
   3. In the Test case name field, enter a name for the test case.
   4. In the Boost include dir (optional) field, enter the path to the directory that has files needed by Boost.Test, such as version.hpp and the test subfolder that contains the test header files.
   5. In the Build system field, select the build system to use for building the project: qmake, CMake, or Qbs.

Qt Creator creates the test in the project directory. For more information about creating Boost tests, see Boost.Test.

Creating Catch2 Tests

To build and run Catch2 tests, you can either install Catch2 libraries and headers or use the single include header file in the Catch2 repository.

If the compiler and build system can find the Catch2 headers automatically, you do not need to specify the include directory when creating the test.

To create a basic Catch2 test:

1. Select File > New Project > Other Project > Auto Test Project > Choose to create a project with boilerplate code for a Catch2 test.
2. In the Project and Test Information dialog, specify settings for the project and test:
   1. In the Test framework field, select Catch2.
   2. In the Test case name field, specify a name for the test case file.
   3. Select the Use Qt libraries check box to use a self-defined main function and set up the project to use Qt features.
   4. In the Catch2 include directory (optional) field, you may enter a path to the directory that has the Catch2 header files.
   5. In the Build system field, select the build system to use for building the project: qmake, CMake, or Qbs.

Qt Creator creates the test in the project directory. For more information about creating Catch2 tests, see Catch2.

Creating CTest Based Tests

CTest can execute tests for CMake based projects regardless of the test framework. You configure tests in the project file, usually, CMakeLists.txt. Basically, you enable testing for the project and register the test applications or even special commands.

enable_testing()
add_test(NAME test_example COMMAND test_example)

Add test_example as an executable before trying to register it as test. It may be any executable command including arguments. For detailed information on how to use CTest see Testing with CTest.

Setting Up the Google C++ Testing Framework

To build and run Google tests, install and configure the Google C++ Testing framework on the development host. You can either clone it from Git Hub or install it from an installation package.

To configure a project to use a cloned Google testing framework, edit the INCLUDEPATH variable in the project file (.pro) to include the source and include folders of Google Test's googletest and googlemock. Usually, you need to add the following subfolders:

• googletest
• googlemock
• googletest/include
• googlemock/include

Also, add the necessary files to the SOURCES variable. For example:

• googletest/src/gtest-all.cc
• googlemock/src/gmock-all.cc

To configure a project to use an installed Google testing framework package, add the following include paths to the .pro file:

• <googletest_install_path>/include/gtest
• <googletest_install_path>/include/gmock

Then add linker options to be able to find the libraries and to link against them. For example, for qmake based projects, you typically need to add the following values to the .pro file:

• LIBS += -lgtest -L<path_to_gtest_lib>
• LIBS += -lgmock -L<path_to_gmock_lib>

Building and Running Tests

To build and run tests:

1. Open a project that has tests.
2. In the Tests view, select the tests to run.
3. In the Test Results, select:
   • (Run All Tests) to run all tests.
   • (Run Selected Tests) to run the selected tests.
   • (Run Failed Tests) to re-run the tests which failed in the last run. Depending on the framework, this may select more tests if it is impossible to distinguish or to fully address the test.
   • (Run Tests for Current File) to run the tests in the file currently open in the code editor.

By default, Qt Creator builds a project before deploying and running it. To run all tests without building and deploying them again, select Run All Tests Without Deployment in the context menu. To run the selected tests without deployment, select Run Selected Tests Without Deployment.

The functions to run tests are also available in the context menu in the Tests view and in Tools > Tests.

If a test takes more than a minute to execute, the default timeout might stop the test execution. To increase the timeout, select Edit > Preferences > Testing > General.

Selecting Tests to Run

The Tests view shows all the tests found for the currently active test frameworks in the current project. Select the test cases to run.

If a Qt Quick test case does not have a name, it is marked Unnamed in the list. Run All Tests executes unnamed test cases. You cannot select or deselect them.

Qt Creator scans the project for tests when you open the project and updates the test list for the currently active test frameworks when you edit tests. To refresh the view, select Rescan Tests in the context menu.

To show or hide init and cleanup or data functions in the Tests view, select (Filter Test Tree), and then select Show Init and Cleanup Functions or Show Data Functions. Double-click a function in the list to open its source code in the code editor.

The Tests view lists test cases in alphabetic, case insensitive order. To list them in the order in which they appear in the source code, select (Sort Naturally).

Running and Debugging Tests from Code Editor

You can run and debug tests in the file currently open in the code editor. To run all tests in the open file, select Tools > Tests > Run Tests for Current File.

To run the test currently selected in the open file, select Run Test Under Cursor in the context menu.

To debug the currently selected test, select Debug Test Under Cursor in the context menu.

Specifying Test Settings

To customize the handling of tests, test frameworks, and test tools, select Edit > Preferences > Testing > General.

You can customize some settings at project level. To change settings for the current project instead of globally, select Projects > Project Settings > Testing.

In the Active Test Frameworks list, select tests for Qt Creator to handle. To improve the performance of full scans for tests, disable test frameworks you do not use.

To group related test cases for an active test framework, select the Group check box next to the framework name in the Active Test Frameworks list. By default, Qt Creator groups tests that are in the same directory.

Qt Creator omits internal messages and run configuration warnings for deduced configurations by default. To view them, deselect the Omit internal messages and Omit run configuration warnings check boxes.

By default, test result output shows a maximum of 100,000 characters. The view automatically scrolls to show the latest results. To display full results, deselect the Limit result output check box. To disable automatic scrolling, deselect the Automatically scroll results check box.

Set the maximum number of lines in the test result tooltip and description in Limit result description.

To group test results by the executable path that you use to run the tests, select Group results by application. This is useful if you have multiple test executables and run them all at once.

To automatically run tests after successfully building the current project, select them in Automatically run.

Sometimes, Qt Creator cannot deduce which executable or run configuration to use. If Qt Creator repeatedly asks you to select the tests to run when trying to execute tests, you can let it cache your choices and use them where appropriate. Qt Creator clears the cache when you switch to another project, close the current one, or select Reset Cached Choices.

Select the Process arguments check box to pass arguments to the test executable that you specify in the run configuration. This is an experimental feature that might cause the execution of the test executable to fail.

Specifying Settings for Running Qt Tests

The code inside a benchmark test is measured, and possibly also repeated several times in order to get an accurate measurement. This depends on the measurement back-end that you can select in the Benchmark Metrics group in Edit > Preferences > Testing > Qt Test: walltime, CPU tick counter, event counter, Valgrind Callgrind, and Linux Perf. For more information, see Creating a Benchmark.

To receive verbose output when running benchmarks, select the Verbose benchmarks check box.

To allow the debugger to interrupt Qt tests on assertions, select the Disable crash handler while debugging check box.

To record information about signals and slots in the test log, select the Log signals and slots check box.

To explicitly limit the maximum number of warnings in the test log, select the Limit warnings check box and set the limit. Set it to 0 if you want no limit at all. The default number is 2000.

To check for Qt Quick Tests that are derived from TestCase, select the Check for derived Qt Quick tests check box.

Specifying Settings for Running Google Tests

To specify settings for running Google tests, select Edit > Preferences > Testing > Google Test.

To run disabled tests, select the Run disabled tests check box.

To run several iterations of the tests, select the Repeat tests check box and enter the number of times to run the tests in the Iterations field. To make sure that the tests are independent and repeatable, run them in a different order each time by selecting the Shuffle tests check box. Set the seed for initializing the randomizer in the Seed field. The value 0 generates a seed based on the current timestamp.

To turn failures into debugger breakpoints, select the Break on failure while debugging check box.

To group Google tests by using a GTest filter, select GTest Filter in the Group mode field, and specify the filter in the Active filter field. For more information about GTest filters, see Running a Subset of the Tests.

Specifying Settings for Running Boost Tests

1. To specify settings for running Boost tests, select Edit > Preferences > Testing > Boost Test.

   

2. In the Log format field, select the error report format to specify the type of events to record in the test report.
3. In the Report level field, select the verbosity level of the test result report. Select No if you do not want a report.
4. Select the Randomize check box to execute the tests in a random order, using the seed specified in the Seed field for initializing the randomizer. The value 0 means no randomization, the value 1 uses the current time, and any other value generates a random seed.
5. Select the Catch system errors check box to catch system errors.
6. Select the Floating point exceptions check box to detect floating point exceptions.
7. Select the Detect memory leaks check box to detect memory leaks.

Specifying Settings for Running Catch2 Tests

1. To specify settings for running Catch2 tests, select Edit > Preferences > Testing > Catch Test.

   

2. Select the Show success check box to show succeeding expressions as well. By default Catch2 will print only fails.
3. Select the Break on failure while debugging check box to turn failures into debugger breakpoints.
4. Select the Skip throwing assertions check box to skip any assertion that test for throwing an exception.
5. Select the Visualize whitespace check box to turn whitespace into escape sequences.
6. Select the Warn on empty tests check box to get a warning when a test case does not check any assertion.
7. Select the Abort after check box to abort the test after the number of failures specified inside the spin box.
8. Select the Benchmark samples check box to specify the number of samples to collect while running benchmarks.
9. Select the Benchmark resamples check box to specify the number of resamples to use for statistical bootstrapping after benchmarking.
10. Select the Benchmark confidence interval check box to specify the confidence interval for statistical bootstrapping.
11. Select the Benchmark warmup time check box to specify the warmup time for each test before benchmarking starts.
12. Select the Disable analysis check box to disable statistical analysis and bootstrapping.

Specifying Settings for Running CTest-Based Tests

1. To specify settings for running CTest-based tests, select Edit > Preferences > Testing > CTest.

   

2. Select the Output on failure check box to show test specific output if a test fails. Contrary to the CTest default this is enabled by default.
3. Select Schedule random to execute the tests in random order.
4. Select Stop on failure to automatically stop the test execution on the first failing test.
5. In the Output mode field, select the verbosity level of the CTest output.

   Note: This only influences the output on the text display.

6. Select Repeat tests if you want to re-run tests under certain circumstances.
7. In the Repetition mode field, select the mode for re-running tests. Set the maximum count for repeating a test in the Count field.
8. Select Run in parallel to run the tests in parallel using the specified number of Jobs.
9. Select Test load to limit the parallel execution. CTest will not start a new test if it would cause the CPU load to pass the threshold set in Threshold.

Viewing Test Output

The Test Results view shows Qt and Qt Quick test results in XML format and other test results in plain text format.

Qt Test Output

XML can be parsed more easily and reliably than plain text. However, if a Qt or Qt Quick test crashes, it might not produce complete XML code that can be parsed, which might lead to information loss. You might see the lost information when viewing the results as plain text.

To view the results of Qt and Qt Quick tests as plain text, select Edit > Preferences > Testing > Qt Test, and then deselect the Use XML output check box. Then select the (Switch Between Visual and Text Display) button in Test Results to switch to the text display.

Summary of Messages

The following table lists the messages that Test Results displays:

To view only messages of a particular type, select (Filter Test Results), and then select the types of messages to show. To show all messages, select Check All Filters. To deselect all message types, select Uncheck All Filters.

Since Qt 5.4, you can add a BLACKLIST file for tests. It is mainly used internally by the Qt CI system.