Limits of inspected-in quality

 “All necessary test cases, starting from requirements”

Quality has two major “sources”: built-in quality (build without defects) and inspected-in quality (test, find defects and fix them).

Poor built-in quality is – statistically speaking – a known and common problem in software development. In many cases, with product growing,  quality will decrease in time by accumulation of technical debt. There is a wide spread belief that we can improve the quality (even in difficult cases) based mostly on test & fix approach.  We will try to prove that is “mathematically” wrong and it is rather a wishful thinking.

(Traditional voice) We will test & fix, right? And we will do that in the most professional way. We will document very well the requirements. We will generate all the necessary test cases starting from requirements. We will use them to execute very professional tests and we will make all the significant need fixes for a good quality.

Yes, indeed, a professional way to generate the test cases will start from the requirements, will identify functional flows, will generate running scenarios and then the test cases considering input data, test conditions and expected results.

(Traditional) That should be great! Even TDD implement the test cases considering all these elements.

That is true, but there are difference: TDD offer something more. Anyway, you forgot something …

How to blow up traditional testing

Let consider a set of pretty complex set of functionality. The orthodox approach of testing will do that:

• (We have good analysts and good testers)
• Write good enough requirements
• Extract scenarios paths from functional flows and states transitions
• Generate test cases from scenarios considering also inputs, conditions and expected results
• Run the tests, find defects and fix them

That could take longer, but finally we will reach the quality goals, right?

NO!

Please consider this extra-scenario: what if we have too much technical debt. Let see some consequences. (used numbers are examples).

Scenarios –  If the requirements logic generates 100 scenarios, poor designed code could physically have 600 or more. How is that possible? It is pretty easy by accessing, for example, an undesired global context. If my scenario will access a global context (even only one global variable…), that will multiply the number of real test cases because these data are changed in an unexpected way (not specified in the requirements) by other flows. In fact, the global context will mix and multiplexes piece of functionalities that are not logically related. Duplicate code it is also a mighty enemy that create and multiply the test cases (“phantoms”, beyond the ones resulted from requirements). If we want to change a formula that was harcoded on each usage, how could really know a tester where was harcoded and how many times?

States – If the requirements logic suppose 50 states of the main entities & main logic, a poor designed code could physically have 500.  How is that possible? Again, that it is easy. For example, if the states transitions for one entity is not encapsulated, the code could be fragile enough to induce supplementary phantom states, because of poor written state transitions (one ways is to use only basic setters to realize these transitions).

Expected results – a poor design code will damage also the result display, log or persistence. The easiest way is to damage the displayed data: “catch” the errors and not display them, display a valid value for a null etc.

Let make a summary:

• We have 400 official test cases for 100 scenarios and 50 states transitions and that cover the requirements
• Physically, the poor designed system has more than 600 scenarios, 500 states transitions that will be covered by more than 2000 test cases

We will start testing – running 400 test case for a system that need 2000 test cases:

• We will fix the defects that belong to the 400 official test cases
• The testes will make some supplementary explorations and we will catch some of the defects from phantom test cases and phantom state transitions
• A lot of defects will remain unobserved, mostly from those “undocumented” test cases

(Traditional) Wait!! The team – developers, testers and analyst will not discover the problem? We will test more and will fix more defects!

Based on experience, that will almost never happening! There is almost zero chance to generate all the “undocumented” test cases only by pure exploration: the tester has no clue about where are most of the hidden, phantom test cases.

(Traditional) There should be a way to solve that!

There is one: when a tester discovers some phantom test cases, when the expected results are damaged then that tester must report this quality problem: “we have test-damaging defects, please analyze them, and fix the root cause”.

Test and fix cannot protect you from test-damaging defects

(Traditional) That sound good enough!

It is not! It is too late to discover at testing that we have a such poor code, and a such poor build-in quality that will affect the tests itself and will cost too much.

Some conclusions

There is nothing wrong to generate in an orthodox manner the test cases from requirements. Just that you also physically implement these requirements, in order to make the generated test cases effective. The developer must not have the “liberty” to implement phantom scenarios, phantom states, to mix data and flows in a way that was not specified by the requirements.

We need to physically implement the requirement

It is not effective, inefficient, unethical and unprofessional to build a system that cannot be tested – where the real system test cases are much more that requirements-generated test cases (and are practically impossible to generate on testing time).

Traditional way of trying to get the quality mostly by test and fix it is many case a cognitive bias (inadequate logic), that bring a spiral of undesired results. The Martin Fowler Technical Debt Quadrant logic it is applicable in this case.

What could be done?

We need to reduce as much is possible the impedance between requirements specification and physical implementation. Some examples:

• Physically “protect” the business. Separate the business aspects and do not duplicate
• Basic design principles: do not duplicate, do not access global context
• Physically “protect” the functional flows (separate: do not mix with other aspects)
• Physically “protect” the logic of timing sequences (see Uncle Bob “Clean Coders” videos)
• Physically “protect” the logical states transitions
• Prevent and fix test-damaging defects

What about TDD?

TDD is in the list of things that help us to physically implement the requirements. The ultimate requirements are the test cases, were TDD will physically implement the test cases. The only major limit of TDD is that almost never will implement all the test cases.

The ultimate requirements are the test cases

… but we almost never implement all test cases in auto-tests.

Anyway, from industry experience, the examples from the above list of are needed also to enable the TDD.