With the release of SonarQube Server 5.6, we introduced the SonarQube Server Quality Model, which pulls Bugs and Vulnerabilities out into separate categories to give them the prominence they deserve. Now we're tackling the other half of the job: "sane-itizing" rule severities, because not every bug is Critical.
Before the SonarQube Server Quality Model, we had no way of bringing attention to bugs and security vulnerabilities except to give them high severity ratings. So all rules with a Blocker or Critical severity were related to reliability (bugs) or security (vulnerabilities), and vice versa as a tautology. That made sense before the SonarQube Server Quality Model, but it doesn't now. Now, just being a Bug is enough to draw the right attention to an issue. Now, having every Bug or Vulnerability at the Blocker or Critical level is actually a distraction.
So we're fixing it. We've reclassified the severity on every single rule specification in the RSpec repository. The changes to existing reliability/bug rules are reflected in version 4.2 of the Java plugin, and future releases of Java and other languages should reflect the rest of the necessary changes. In some cases, the changes are significant (perhaps even startling), so it makes sense to explain the thinking.
The first thing to know is that the reclassifications are done based on a truth table:
For each rule, we first asked ourselves: What's the worst thing that can reasonably happen as a result of an issue raised by this rule, factoring in Murphy's Law without predicting Armageddon?
With the worst thing in mind, the rest is easy. For bugs we evaluate impact and severity with these questions:
Impact: Will the "worst thing" take down the application (either immediately or eventually), or corrupt stored data? If the answer is "yes", impact is high.
Likelihood: What is the probability the worst will happen?
For vulnerabilities, the questions are:
Impact: Could the exploitation of the vulnerability result in significant damage to your assets or your users?
Likelihood: What is the probability a hacker will be able to exploit the issue?
And for code smells:
Impact: Could the code smell lead a maintainer to introduce a bug?
Likelihood: What is the probability the worst will happen?
That's it. Rule severities are now transparent and easy to understand. And as these changes roll out in new versions of the language plugins, severity inflation should quickly become a thing of the past!