• delaware

Use this one simple trick to never have bugs again

As a software engineer, you are fighting a never-ending battle against bugs that keep popping up in your projects. As bad as this is on small projects, on projects with a lot of people working on them, the number of bugs increases exponentially and oversight only gets harder. Software gurus keep shouting buzzwords like “test driven development” or “continuous integration” into the void. But as we will explain, the current best practice of having high test coverage is full of holes.

Unit tests & code coverage

Every serious coder would agree you need unit tests in order to verify that the code meets the set requirements. These tests often verify that corner cases should not give errors. But as nobody is perfect, these tests are neither: it is impossible to write unit tests that cover all possibly dangerous scenarios.

An often-used metric to determine how good the test set collection covers the different sections of the code is code coverage. This measure calculates the percentage of lines that are touched by the tests. But as it turns out, this is problematic approach: even a coverage of hundred percent does not mean that the software is bug-free. Code coverage is a great way to tell us how much of the code is tested, but it never explains how well the tests check that the software does what it is supposed to do.

So, if code coverage is a meaningless metric, what measures can we use to prove software has high quality? After all, everyone agrees you should not just take the engineer’s word for it. Two nice speakers of UAntwerpen, professor Serge Demeyer and his PhD student Ali Parsai, explained a possible way out of this mess: mutation analysis.

Mutation Analysis

The core idea of mutation analysis is the automatic injection of faults into the code which is to be tested. Then you run your unit tests on this infected code and you actually expect the tests to fail. If the tests do not fail, it means you wrote a test which has no relevant failure condition and is therefore effectively useless. The faults which are injected are really simple: for example, changing operators to similar operators with different semantics: changing a smaller than to a greater than operator.

A possible drawback of this approach is that mutation analysis is rather costly: the complete test suite has to be executed for the different possible changes in the code. Therefore, it takes a lot of time to complete and it is often not feasible to execute for every commit. A possible solution for this problem could be to run the mutation analysis every night, so that the results are available in the morning: the developers will be able to improve the software as soon as they come back at work.

By Thomas Delva (Master Computer Sciences, UGent) & Wim Vanparijs (Master Computer Sciences, VUB)