Introduction to Testing

The Boost libraries are intended to be both reliable and portable. Every experienced programmer knows that means each library must be tested against a suitable number of test cases, on a range of platforms and compilers, and then tested again (regression tested) every time a change is made. The regression tests are run again before every public release.

There are three broad dimensions to Boost library testing:

  1. The actual test codes - unit tests, regression tests, new feature and bug tests. Refer to Writing Tests.

  2. Tests that are integrated with the primary build systems, B2 and CMake.

  3. Continuous Integration

The new library developer needs to consider all three, however the initial focus on a new library will be the first in this list. Or, to put it another way, "Quality assurance based on a wide range of targeted tests" as one of the key answers to Professor Hoare’s question:

"How did software get so reliable without proof?"

Define a Test Matrix

When you’re looking to submit a library to the Boost collection, it’s essential to ensure broad compatibility with various compilers, platforms, and configurations. Create a test matrix of what you intend to support, and document what you do not intend to support, and consider:

  1. Testing with multiple versions of each compiler. Popular compilers to consider include:

    • GCC (GNU Compiler Collection)

    • Clang

    • MSVC (Microsoft Visual C++)

      The Boost user base can be using older versions of these compilers, so strive for compatibility with a reasonable range of versions if possible.

      If your library depends on other Boost libraries or external libraries, ensure they are compatible with the compilers you are targeting. Be clear about any dependencies or prerequisites in your documentation.

      Be wary of non-standard compiler features or extensions. If you must use them, guard them with appropriate preprocessor checks. Boost provides its own set of configuration macros to help with this.

  2. Compatibility with various versions of the C++ Standard: C++11, C++14, C++17, C++20, and C++23. Some Boost libraries support many of the standards, while others target only more recent ones.

    Note

    Supporting C++03 is no longer considered good practice.

  3. Different operating systems, including Linux, Windows, macOS, and others like various BSDs.

  4. Different architectures: x86, x64, ARM, MIPS. Architecture can affect word size (usually 32 or 64 bit), endianness, memory alignment, inline assembly, cache sizes, latency, and other memory and performance issues.

When you have outlined your test matrix, study the predefined macros available to assist you, and make adjustments to your matrix appropriately.

Use Predefined Macros from Boost.Config

There are a set of macros in the Boost.Config library that can be used to identify compiler capabilities, platform specifics, and other configuration details. These macros are designed to smooth out differences between compilers and platforms, allowing for more portable code, for example:

Compiler Identification

  • BOOST_COMP_GNUC: Defined if the compiler is GCC.

  • BOOST_COMP_MSVC: Defined if the compiler is Microsoft Visual C++.

Compiler Version Checks

  • BOOST_COMP_MSVC >= BOOST_VERSION_NUMBER(19, 0, 0): Checks if the MSVC version is 19 or greater.

Platform Checks

  • BOOST_OS_LINUX: Defined if the operating system is Linux.

  • BOOST_ARCH_X86_64: Defined if the architecture is x86_64.

Standard Library Checks

  • BOOST_LIB_C_GNU: Defined if the C standard library is from GNU.

  • BOOST_LIB_STD_DINKUMWARE: Defined if the standard library is from Dinkumware (often associated with MSVC).

Feature Checks

  • BOOST_NO_CXX14_RETURN_TYPE_DEDUCTION: Defined if the compiler does not support return type deduction introduced in C++14.

  • BOOST_NO_CXX11_AUTO_DECLARATIONS: Defined if the compiler does not support auto declarations from C++11. For example:

    #include <boost/config.hpp>

// ...
#ifdef BOOST_NO_CXX11_AUTO_DECLARATIONS
// Use traditional type declaration
int x = 42;
#else
// Use C++11 auto
auto x = 42;
#endif
// ...

    Use this same coding structure for any of the other macros.

Regression Testing

Boost releases are run through regression tests which automatically generates compiler status HTML tables for various platforms. Unless otherwise indicated, the C++ Standard Library implementation is the one shipped with the compiler. Refer to Generating Library Status Tables.

Note

The HTML tables are not a good indication of a particular compiler’s compliance with the C++ Standard. The Boost libraries often contain workarounds which mask compiler deficiencies.

Some regression tests are run only occasionally, and might be relatively out-of-date. Check the date and revision in the column headings.

Next Steps

First, familiarize yourself with the Boost Test Policy.

Then, read the documentation for the following libraries, which support the writing of unit, feature and regression tests:

Start small and develop a good understanding of how these testing libraries work, before writing more than a few tests.

Advanced Testing

When you have a good understanding of the basic testing procedures, look into more advanced techniques, such as Fuzz Testing.

See Also

Edit this Page