Why choose Exercisix?

I was astonished recently to look at the "List of unit-testing frameworks" in Wikipedia. The number of frameworks for C++ is measured in tens! Naturally, a unit-testing framework should have a clear description of it's benefits to help the user make the choice. On the one hand, Exercisix always had a good manual page, this makes me sure everybody knows how to spell the tests. On the other hand, Exercisix is quite special, because it is tailored to a single particular task — test-driven development. So I feel the need to explain the design choices and describe tasks that are solved by the framework effectively.

What do I mean by test-driving the code

TDD is a fashionable thing and relies on unit-testing. No wonder now almost all testing frameworks claim they support TDD more or less. The unit-testing, however, is a broad term. For example, you may use unit-tests to develop your code, or you may use them to verify that the code you already have is working correctly. At present "TDD" and "unit-testing" are becoming buzzwords and different people attach different meaning to them.

When I say unit-test, I mean a test that examines the smallest unit of coded logic — a function. The material under examination is the pure logic of the function. For C++ it is conditionals, loops and function calls. Then, the function being tested is preferrably isolated from the rest of the code. This may or may be performed completely, but the amount of code that gets executed is always limited and known. External calls, like to the filesystem, network and libraries, never happen.

By test-aided development I mean using unit-tests for two things:

  1. Facilitate thinking. Writing tests before code makes you focus on small amounts of logic to be written. It also provides examples of how the code is to be used.
  2. Verify that the recently added code is consistent with what you already have. Expectedly, tests fail if inconsistency is introduced.

Ease of adding tests

Using TDD in a way described above means having many tests. Each function in the production code is a result of a test. Each argument to the function is a result of a test. Each conditional, loop or function call is a result of a test. To introduce an expression you need a test. For a complex expression you need two, three or more tests, depending on the complexity and your skill.

Then, tests must be simple. Very simple, so that you quickly understand what computation is performed by the code. Adding many tiny pieces of logic is only affordable if it is fast and easy, to keep overhead of adding tests small.

Exercisix makes every effort to minimize the work you do to add a test or start a suite:

  1. To start a suite you need to include the header file into your program, which becomes the executable containing tests. 
    #include <exercisix.hh>

// test cases go here
  2. To define a test case you need just write the body and prepend the TEST macro call before. All test cases are discovered by the framework automatically. 
    #include <exercisix.hh>
#include <plain-saw.hh> // testee

TEST("constructor initializes 'phase' to zero")
{
  auprops props(44100, 17);
  plain_saw saw(props);
  
  EXPECT( saw.phase, ==, 0.f );
}
  3. From coder's point of view, test cases are identified by their textual description. This frees you from boring task of inventing an identifier that is both descriptive, consists of alphanumeric characters only and fits into a line of code. In Exercisix, you write any text to identify the test, just like a string literal. The test case description may be split into several chunks and lines. 
    TEST("successive calls to render() start from the phase, "
     "calculated at the end of the previous call")
{

Terse test tools syntax

Then, test tools. In some frameworks they are called "assertions". To put stress on the fact that with Exercisix you first write the test and then the code, the basic test tool is called EXPECT. Since tests must be cheap to add, there is no room for attaching a text to each expectation (I use this word in the place of "assertion"). Of course, you may not even think of manually coding format strings to print the actual values if the code does not meet the expectation!

To make writing test cases a pleasure, Exercisix syntax used in test tools is close to natural. If you need to state that values of some expressions are equal, you write equality operator. In the case of failure the framework prints the name of the test case, points to the line, which contains the expectation, prints the expressions, the operation and computed values of expressions as well.

For the following code sample: 

#include <exercisix.hh>
#include <plain-saw.hh> // testee

TEST("constructor initializes 'phase' to zero")
{
  auprops props(44100, 17);
  plain_saw saw(props);
  
  EXPECT( saw.phase, ==, 0.f );
}
The framework output may look like this: 
test-plain-saw.cc:74:expectation failed

constructor initializes 'phase' to zero

operand1: saw.phase
operand2: 0.f
operator: ==
value1  : 10
value2  : 0
Assuming you actually initialize phase member variable with 10.

Brief reports

As explained above, in TDD unit-tests are executed merely to check whether your recent code changes are still consistent with your expectations. Since you work with small chunks of logic, you always know what to do if some test case fails — just remove recent changes.

You do not need to know how many test cases failed, because this will not provide any additional information. Your choice is between keeping the modifications and removing them. For this reason Exercisix does not bother you. If a single test case fails, then the execution stops and the details are printed. Similarly, if your changes meet expectations and are consistent with previously written code, then the name of the suite and a single word "SUCCESS" are printed:

test-plain-saw: SUCCESS

This eliminates the need for choosing among various reporting formats. The result is a boolean value. Do you prefer "Yes" in XML or in JSON? If your attention is actually necessary, Exercisix will clearly indicate that and your build system will stop with an error message.

The other use case for a detailed reporting is showing progress. Some frameworks print lots of messages like "staring test case..." and "test case finished". This is often seen as a way to feel more comfortable with slow tests. By default exercisix doesn't show any progress with running test cases. The real unit-tests are all about logic and computers are too fast to execute it slowly. Slow tests mean that something more than the logic is executed. Typically this is access to the filesystem or calls to external libraries. This is best fixed by separating logic of your application from external systems.

Fixtures

Test Fixture is all direct and indirect inputs of the function being tested. What do you naturally do if you need to prepare a value and pass it to a function as an argument? Right, declare a local variable to hold the value. Sometimes it is easier to use an expression embedded in the function call. For indirect outputs this approach is slightly complex, yet suitable.

Some unit-testing frameworks, however, provide specific features for preparing this evironment for function calls. Usually, you may name the fixture set-up code and then re-use it mulitple times. I do not see any value in it. Each test should result in small amount of code written by developer. Equal fixtures may only produce the code you already have. That's why fixture re-use is often applied together with the specific code in each test case, which tunes the fixture. This feels like a magic: environment necessary for a function call appears from nowhere and is tuned before executing the function.

My advice is to keep code simple and put all fixture setup code explicitly in the test case body. Having this it will be easier for you to notice duplication in setup logic and eliminate it by refactoring.

There is another argument for having fixture management in testing framework — for resource de-allocation. Test frameworks created with Java in mind offer facilities for fixture setup and clean-up. Shame on them! Unlike Java, C++ is capable of automatic resource management. In good C++ code you only need to declare variables necessary to call a function. The language will perform clean-up automatically at the end of test case execution. Exercisix supports good coding practices and I am reluctant to implement features for creating a kind of resource hell.

In a proper C++ managing resources for test fixtures looks like this: 

TEST("to_yacc(branch) adds helper rule to the specified grammar")
{
  int count; // dummy test double
  grammar g; // and the spy
             // that's all the fixture!
  
  to_yacc(branch(terminal("A", '*')), g, count);
  
  EXPECT( g.rules.size(), ==, 1u );
}

Suites

Test suites are a popular way of test organization. Many unit-test frameworks allow organizing test cases into test suites, these test suites into higher-level suites and higher-level suites belong by turn to a master suite. There is nothing like that in Exercisix. The source file name implicitly becomes the name of the suite. There is a simple rule: one source file — one suite.

Why there are no tools for more complex test suite organization? Because tests should be simple. There are no tests for tests. Complex test suite organization is often used to work around huge number of tests or slow tests. It's easy to get into a self-delusion and think that everything is under control when in fact test code complexity is high and production code is tightly coupled. Like a glossy paint, Exercisix reveals unevenness of your unit-tests by offering no more than a simple, suite-per-file organization.

Portability

The ultimate claim to fame for some frameworks is that they use reduced feature set of C++. This is advertized as a way to work in all development envirnoments possible, including those with 10-17 years old compilers and operating systems. To meet this goal framework authors reimplement many things typically found in the language or the standard library and package them with the framework itself. The motivation is that very old or scanty compilers are unable to provide either run-time type identification or templates or both. Hey folks! If your compiler can not provide features that were standardized in the previous century, how do you plan to compile your code? By keeping yourself reinventing the wheel over and over again?

Exercisix was developed with portability in mind. It does depend on the language features. Yes, it is a C++ framework and it actually requires an implementation of C++. No more, no less.

Conclusion

Exercisix as a cute tool for developing code using TDD. It provides detailed information when necessary and saves your time when you are en route. The inability to integrate Exercisix into your environment is also a useful fact: this means you are far from Lean development. In any way you have benefits of trying the framework.