After an upgrade people are sometimes surprised to find that the next analysis of a project with no real changes shows a significant drop in coverage. Believe it or not, that really is a feature, not a bug, and it's called Executable Lines.
Executable Lines is a metric we created to count the number of lines in a file that could be executed by unit tests. Why does that matter to your coverage percentage? Because we're now using it as the denominator when we calculate total coverage. To understand why, imagine a project with two files of 100 lines each. File A is fully unit tested, and file B has no unit tests. The report we get from the coverage engine might (depending on the engine) only include file A, and show that the project is 100% covered. Before we introduced Executable Lines, we could only go by the coverage report, so we would also show that two-file project was shown as 100% covered, when coverage is really only 50%. With the introduction of executable lines, we're able to take files omitted from the coverage reports into account to show the real coverage numbers.
Finding those real numbers becomes even more important when you consider projects without any coverage at all. At the project level, the practical effect of introducing Executable Lines is that the value displayed goes from "-" (for no data) to "0%", which in real terms is no difference at all, since such projects probably already have quality gates that ignore coverage. But aggregate that project into a Portfolio or Application, and the impact of finally realizing that one of the projects in your aggregation has no tests can be significant.
You may be wondering at this point why we needed a new metric for this missing coverage calculation. Why not just use Lines of Code? Because not every line of code is executable. For instance, an import statement is a Line of Code, but not something that needs to be (or can be) covered by unit tests. The same can be said for class declarations, interface declarations, variable declarations, and so on. In fact, we've written up a developer guide to describe all the types of Lines of Code that are not Executable Lines.
Of course, nothing is without side effects. The first is that your SonarQube Server coverage percentage probably won't agree any more with the percentage reported by your coverage engine. That's because your coverage engine is only looking at part of the picture, and SonarQube Server is looking at all of it. The other side effect is that if you've explicitly configured your coverage engine to ignore certain parts of your code base, well... SonarQube Server is going to automatically re-incorporate them, so you'll have to configure the exclusions again on the SonarQube Server side, but that should be a one-time operation.
Assuming you're on a recent version of SonarQube Server, an upgrade of the relevant language analyzer means you'll automatically start seeing this behavior if the new analyzer version supplies the metric. The confusing part for some is that they didn't upgrade their analyzers; they upgraded the platform. But with every platform release we bundle the latest versions of the analyzers, so by doing one, you very often automatically do the other.
In case you're wondering, these are the analyzer versions that implement Executable Lines:
- SonarC# 6.0
- SonarCFamily 4.9
- SonarJava 4.4
- SonarJS 2.20
- SonarPHP 2.9.2
- SonarPLSQL 3.0
- SonarPython 1.9
- SonarSwift 2.1
Before you ask, Executable Lines is "coming soon" for the rest of the analyzers.