- This is not a bug, says the Verifier. You can't say it's a bug because it does not do what you want. It should be downgraded to feature request.
- This is the recommended replacement of the old tool, I retort, and as such it should do at least what the old tool did, plus the new stuff. I insist it is a bug.
- The manual does not mention feature X for the new tool, and then it is not a bug. It's a feature request"
The Verifier's reasoning is technically correct, and it is within the boundaries of his allowed action, so I see no point in arguing much more.
However, I can start a campaign of educating people on bug evaluation, in hope that these notes reach the ones in charge of writing the guidelines, so that this kind of requests is met more kindly next time.
So, what exactly is a Bug?I start by borrowing some definitions from Ron Patton's excellent book on Software testing. According to that, it's a bug when ...
(There is an online sample chapter, The realities of software testing containing the above definition)
- The software doesn’t do something that the product specification says it should do.
- The software does something that the product specification says it shouldn’t do.
- The software does something that the product specification doesn’t mention.
- The software doesn’t do something that the product specification doesn’t mention but should.
- The software is difficult to understand, hard to use, slow, or—in the software tester’s eyes—will be viewed by the end user as just plain not right.
Let's comment on each point.
Broken promise (positive)The software doesn’t do something that the product specification says it should do.
This is a no brainer. If you said that the editor would save my file when I hit the "save" button, and it doesn't, then it's a bug.
Broken promise (negative)The software does something that the product specification says it shouldn’t do.
This is as simple as the previous one. If you say that the program won't overwrite my painfully composed setup file and instead it silently restores the defaults, then it's a bug.
Collateral damageThe software does something that the product specification doesn’t mention.
Here we enter in tricky territory. There is an extra feature not mentioned in the manual. Sometimes you should be just happy and say thank you, for instance if your word processor can handle simple spreadsheet-like calculations in table cells. But if the same word processor makes a hidden copy of every document you edit for backup purposes, you may get quite angry. Either way, is anything of this sort happens, then it's a bug.
Forgotten specsThe software doesn’t do something that the product specification doesn’t mention but should.
At this point, we start bickering loudly. If your word processor's manual does not mention the "save" option, and such option is nowhere to be found in the menu and task bars, anyone faintly computer-literate would complain that this omission is inexcusable. If such a missing feature happens, then it's a bug.
Catch-all clauseThe software is difficult to understand, hard to use, slow, or—in the software tester’s eyes—will be viewed by the end user as just plain not right.
And now the bickering becomes open fight. You, the user, have the right to comfortable usage. The specifications warns you that you may get unpleasant side effects when using a killer application. You understand it and run it anyway, but it happens that the mentioned side effects are causing real damage. Guess what? It's a bug. No matter if it was in the docs. It should have been under the "known bugs" section. For instance, the docs may warn you that a system monitor, to keep you informed of the health of a database, will poll the server using a non-existing user. That sounds fine, but later, when your intrusion detection system rings you at 3am saying that the number of failed login attempts has reached the critical level and as a safety measure the database server was shut down, then, I can't help it, it's a bug.
This latest rule must not be used as an excuse to pass any request as a bug report. It's hard to set a clear boundary, but sometimes, without this fifth rule, many critical bugs would remain latent.