Tag Archives: Agile Design

Refactoring: strategy & collaborative work

Leave a comment Posted by on September 14, 2016

Introduction –  The extended definition of Refactoring contains also this part:  “Its heart is a series of small behavior preserving transformations. Each transformation (called a “refactoring”) does little, but a sequence of transformations can produce a significant restructuring. Since each refactoring is small, it’s less likely to go wrong. The system is kept fully working after each small refactoring, reducing the chances that a system can get seriously broken during the restructuring.”- Martin Fowler, at refactoring.com.

Note: this imaginary dialog it’s inspired from an intensive and extensive practice.

What if we will have to perform a Big Refactoring? 

Refactoring cannot be big, it is a special kind of redesign, performed in small steps. See the above definition again.

Reformulate: I need to do a lot of refactorings to clean a legacy code.  There is any best practice?    

You need a strategy?

Yes!

Well, agile and software engineering offer a lot of tactics, including the Martin Fowler refactoring sets but almost no strategy, except … you should work clean from the first time. Use Martin Fowler indications from the start (or early enough), also Uncle Bob Clean Code and Clean Architecture.

Hey! I already have a lot of legacy code debt to solve!

Ok! Let’s build a strategy: how do we start?

This was my question!

Refactoring supposed improving the design, while preserving the functionality. Tests included. Do you have good requirements specifications or a good set of automated tests?

Not in this case.

Then you should recover functionality knowledge from the code and put/perform incrementally some tests. Better: functionality should be explicitly represented in the code and should be testable. And remember: there are two kinds of functionality…

Two?

Yes, first, the one that is application-independent and represent the target domain (domain business rules) and the one that is application-depend aka flow of control.

I remember: that sound like Uncle Bob Clean Architecture.

Yes. You will need to be able to apply distinct tests to them, without mix them with other concerns such as UI, persistence, network and others. Anyway, where do I usually start? I will try to make the running scenarios very clear into the code and that mean the flow of control.

In English, please?

I want to clearly see these: were the event triggered by the system actors start and the end-to-end full path until return. More, I want to refactor to make this path clear enough.

How could be not clear? ­­­

Global context. If the functionality path chaotically accesses the global context, then we could have undesired intersections with other paths/scenarios, that will compromise both data and function. In the same time, we can decouple flow/orchestration from specialized concerns.

What we get?

We will have explicit representation of the functionality (with no undesired contacts with other flows), needed for tests (we can apply auto-tests on it). Also we will have the first entry points to the specialized parts that also could be <decorated> with some tests. Then we can apply tactical refactoring as we need.

And …the domain business rules?

Must be decoupled from other concerns and you have to dedicate them specialized design/test elements.

That’s all?

Almost. You need to test any redesign. Tests need knowledge about functionality. If some parts are missing, now it is the time to recover them in auto-tests (preferable) or in other form of specification.

How do I know that recovered requirements are correct?

You don’t. More, you should always suspect that spaghetti-like legacy code include many unobserved bugs. You should validate these functional requirements by intensive collaboration with your colleagues, with domain experts, customer and other stakeholders.

Do you have any idea about how to do that?    

Start with Pair Programming (refactor in pairs). Pairing is not enough, and you will probably need more people involved – use Model Storming: discuss the resulted functionality with more colleagues.

Model Storming?

Yes, it is an agile practice, part of Agile Modeling (and Disciplined Agile) and it was created to complement core practices from XP. Also, you should actively involve your stakeholders in validating the recovered functionality…. Active Stakeholder Participation, that it is another Agile Modeling recommended practices. And at the end you will have more free bonuses.

What bonuses?

Functionality it is easy to accurately read from code (seconds!) and your colleagues and your stakeholders will already have acquired the recovered functional knowledge.

Summary –  Refactoring for significant spaghetti legacy code need tests/testing. Usually, knowledge about functionality necessary for testing it is insufficient, so must be recovered from the code. An effective & proven way to do that is to apply Clean Architecture principles: decuple both domain rules and application specific flow of control (aka use cases). Anyway, legacy code with too much technical debt will contain a lot of bugs, so recovered functionality it is inaccurate and need to be validated.  Knowledge & expertise needed for validation it is distributed among team members, domain experts, customers and other stakeholders, so you need to work in a collaborative manner with all mentioned parts. There are some outstanding software engineering and agile practices that could help on this aspect:

Note: “need” and “necessary” are often use in above text, just because we have followed the logical path of necessary things for testing a redesigned legacy code.

Remember: A lot of technical debt ~ inaccurate functionality. To refactor & test, you must re-start the process & collaborative work from functional requirements acquisition.     

Practices , , , , , , , , , , , ,

Limits of inspected-in quality

1 Comment Posted by on April 4, 2016

 “All necessary test cases, starting from requirements”

Quality has two major “sources”: built-in quality (build without defects) and inspected-in quality (test, find defects and fix them).

Poor built-in quality is – statistically speaking – a known and common problem in software development. In many cases, with product growing,  quality will decrease in time by accumulation of technical debt. There is a wide spread belief that we can improve the quality (even in difficult cases) based mostly on test & fix approach.  We will try to prove that is “mathematically” wrong and it is rather a wishful thinking.

(Traditional voice) We will test & fix, right? And we will do that in the most professional way. We will document very well the requirements. We will generate all the necessary test cases starting from requirements. We will use them to execute very professional tests and we will make all the significant need fixes for a good quality.

Yes, indeed, a professional way to generate the test cases will start from the requirements, will identify functional flows, will generate running scenarios and then the test cases considering input data, test conditions and expected results.

(Traditional) That should be great! Even TDD implement the test cases considering all these elements.

That is true, but there are difference: TDD offer something more. Anyway, you forgot something …

How to blow up traditional testing

Let consider a set of pretty complex set of functionality. The orthodox approach of testing will do that:

  • (We have good analysts and good testers)
  • Write good enough requirements
  • Extract scenarios paths from functional flows and states transitions
  • Generate test cases from scenarios considering also inputs, conditions and expected results
  • Run the tests, find defects and fix them

That could take longer, but finally we will reach the quality goals, right?

NO!

Please consider this extra-scenario: what if we have too much technical debt. Let see some consequences. (used numbers are examples).

Scenarios –  If the requirements logic generates 100 scenarios, poor designed code could physically have 600 or more. How is that possible? It is pretty easy by accessing, for example, an undesired global context. If my scenario will access a global context (even only one global variable…), that will multiply the number of real test cases because these data are changed in an unexpected way (not specified in the requirements) by other flows. In fact, the global context will mix and multiplexes piece of functionalities that are not logically related. Duplicate code it is also a mighty enemy that create and multiply the test cases (“phantoms”, beyond the ones resulted from requirements). If we want to change a formula that was harcoded on each usage, how could really know a tester where was harcoded and how many times?

States – If the requirements logic suppose 50 states of the main entities & main logic, a poor designed code could physically have 500.  How is that possible? Again, that it is easy. For example, if the states transitions for one entity is not encapsulated, the code could be fragile enough to induce supplementary phantom states, because of poor written state transitions (one ways is to use only basic setters to realize these transitions).

Expected results – a poor design code will damage also the result display, log or persistence. The easiest way is to damage the displayed data: “catch” the errors and not display them, display a valid value for a null etc.

Let make a summary:

  • We have 400 official test cases for 100 scenarios and 50 states transitions and that cover the requirements
  • Physically, the poor designed system has more than 600 scenarios, 500 states transitions that will be covered by more than 2000 test cases

We will start testing – running 400 test case for a system that need 2000 test cases:

  • We will fix the defects that belong to the 400 official test cases
  • The testes will make some supplementary explorations and we will catch some of the defects from phantom test cases and phantom state transitions
  • A lot of defects will remain unobserved, mostly from those “undocumented” test cases

(Traditional) Wait!! The team – developers, testers and analyst will not discover the problem? We will test more and will fix more defects!

Based on experience, that will almost never happening! There is almost zero chance to generate all the “undocumented” test cases only by pure exploration: the tester has no clue about where are most of the hidden, phantom test cases.

(Traditional) There should be a way to solve that!

There is one: when a tester discovers some phantom test cases, when the expected results are damaged then that tester must report this quality problem: “we have test-damaging defects, please analyze them, and fix the root cause”.

Test and fix cannot protect you from test-damaging defects

(Traditional) That sound good enough!

It is not! It is too late to discover at testing that we have a such poor code, and a such poor build-in quality that will affect the tests itself and will cost too much.

Some conclusions

There is nothing wrong to generate in an orthodox manner the test cases from requirements. Just that you also physically implement these requirements, in order to make the generated test cases effective. The developer must not have the “liberty” to implement phantom scenarios, phantom states, to mix data and flows in a way that was not specified by the requirements.

We need to physically implement the requirement

It is not effective, inefficient, unethical and unprofessional to build a system that cannot be tested – where the real system test cases are much more that requirements-generated test cases (and are practically impossible to generate on testing time).

Traditional way of trying to get the quality mostly by test and fix it is many case a cognitive bias (inadequate logic), that bring a spiral of undesired results. The Martin Fowler Technical Debt Quadrant logic it is applicable in this case.

What could be done?

We need to reduce as much is possible the impedance between requirements specification and physical implementation. Some examples:

  • Physically “protect” the business. Separate the business aspects and do not duplicate
  • Basic design principles: do not duplicate, do not access global context
  • Physically “protect” the functional flows (separate: do not mix with other aspects)
  • Physically “protect” the logic of timing sequences (see Uncle Bob “Clean Coders” videos)
  • Physically “protect” the logical states transitions
  • Prevent and fix test-damaging defects

What about TDD?

TDD is in the list of things that help us to physically implement the requirements. The ultimate requirements are the test cases, were TDD will physically implement the test cases. The only major limit of TDD is that almost never will implement all the test cases.

The ultimate requirements are the test cases

… but we almost never implement all test cases in auto-tests.

Anyway, from industry experience, the examples from the above list of are needed also to enable the TDD.

Adaptive Design, Adaptive products, Home, Practices , , , ,

JIT – Just in time and Software Development

Leave a comment Posted by on January 18, 2015

(See also Part 2 – Two dimensions: Just in time and Envisioning)

JIT – solution for incertitude and complexity

Driven forces that introduces JIT Life-cycle in software development

  • Business side: often changes – it is too complex to perform (too much) ahead requirements gathering
  • Development side: software solutions are mostly design (instead of production) it is too complex to manage big chunks

As a consequence of the degree of incertitude and complexity for both requirements and solution, the life-cycle (planning) that suit better will have a JIT model. Agile development has adopted from the start such approach in its principles and practices: often and small releases, iterative development.

JIT approach it is a solution for dealing with incertitude and complexity.

JIT approach it is a solution for dealing with incertitude and complexity. It is similar with the mathematical approach to solve non-linear problems: feedback based approaches (control theory).  The main issue is that you cannot compute (in mathematics) something that is too complex. In software development that mean you cannot envision too much requirements, solution and plan because of incertitude and complexity.

You cannot “compute” something that is too complex

Agile is one of the development approaches that already use JIT for more aspects. We can observe that XP that use “small releases” approach, use also “simple design” principle/practice – they do not want to make guesses about possible solutions aspects, required by possible future change request.

Let reformulate some JIT aspects:

  • do not make guesses about incertitude (what is difficult of impossible to clarify)
  • do not try to “compute” too complex problems

Do not make guesses about incertitude

If these principles are not follow, we will have the same problems as in mathematics: huge deviations of the solution for small changes in the inputs. Translating to the software development that mean huge waste.

JIT and Agile

Some Agile principles and practices that already use JIT approach:

  • “Responding to change over following a plan” (Agile Manifesto – value)
  • “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.” (Agile Manifesto – principle)
  • “Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale” (Agile Manifesto – principle)
  • “Business people and developers must work together daily throughout the project.” (Agile Manifesto – principle)
  • “Make frequent small releases.” (XP rule)
  • “No functionality is added early.” (XP rule)
  • Simple design (XP Practice)
  • Model Storming (Agile Modeling / DAD – Disciplined Agile Delivery practice)
  • Document late, document continuously (Agile Modeling / DAD practice)
  • Active stakeholder participation (Agile Modeling / DAD practice)
  • Just Barely Good Enough (Agile Modeling / DAD practice)
  • Explicit support for JIT based life-cycles: Agile, Lean, Exploratory  (DAD – Disciplined Agile Delivery)
  • Inspect and Adapt (Scrum principle)

JIT – main difference versus manufacturing

We need to deal with the main difference versus manufacturing: JIT design versus JIT production. In manufacturing we repeat the solution (design) from the last life-cycle and in software development we need to find a new solution for the newer requirements (metaphor: the market request every time car with a new design). The major problem here is to integrate the previous simple design with the current simple design (there are not just additive). We need that:

  • The existent design must be easy to extend
  • Integration of “next” design must be quick, easy and clean

I have described a solution for this problem in a previous post, a solution that re-arrange some already know thinks from XP and Refactoring (as it was defined in the Martin Fowler book) –  an Adaptive Design based on this rules:

  • Use simple and clean design and then adapt for new changes (example of adapt “tool”: refactoring)
  • Use design practices that increase the adaptability (Refactoring, TDD, Clean Code, Clean Architecture)

JIT production from manufacturing it is based on responsiveness of a highly automated production. JIT design from software development it is more difficult to be automated, but we need to find that solution for responsiveness – it is mandatory to have an Adaptive Design.

Summary

  • JIT approach it is a solution for incertitude and complexity, that it is validated also in mathematics
  • Software development main problems are related to incertitude and complexity, that mean JIT approach could be useful in various ways
  • JIT rules: do not make guesses about incertitude, do not try to “compute” what it is too complex
  • There are many Agile values, principles and practice that are based on JIT approach
  • JIT Design require an Adaptive Design
Adaptive Design, Practices , , , , , , , , , ,

“Sending a probe”: alternative to stubs, mocks and service virtualization

3 Comments Posted by on December 15, 2014

Picard: Data, prepare a class one probe. Set sensors for maximum scan. I want every meter of Nelvana Three monitored. –  Star Trek The Next Generation, The Defector

Stubs – “used commonly as placeholders for implementation of a known interface, where the interface is finalized/known but the implementation is not yet known/finalized.”

Mock objects – “are simulated objects that mimic the behavior of real objects in controlled ways”

Service virtualization – “emulates only the behavior of the specific dependent components that developers or testers need to exercise in order to complete their end-to-end transactions. Rather than virtualizing entire systems, it virtualizes only specific slices of dependent behavior critical to the execution of development and testing tasks.”

NEW  – “Probe” –   An isolated feature/set of features (not just a mimic!) of a system, enhanced with testing support: flexible, configurable data and commands input  and enhanced evaluation/validation output.. Could be used for early integration tests for reducing risks and provide useful feedback. The probe can simulate different scenarios of using a “real”  feature sent in the remote environment and can send back useful feedback. The usage (and the integration) it is simulated and not the functionality!

The probe could be designed to integrate with other systems or to be standalone and just “explore” the environment (that rather mean integration with infrastructure systems) Stand alone probes, if are carefully used, can gather data also from production environment, without interfering with real applications and functionality.

Simple examples:

  • send a probe with the logging related feature, if for example the logging should use remote unavailable web (or others) services (available only in the integration environment and not available remote)
  • send a probe with a feature that send emails in integration environment specific context

Example with stubs, mocks and service virtualization

2
Description

  • we need to integrate our system, that shall contain features from A to L, with an external system
  • when features sets ABC and EFG will be ready, we want to start integration test and we will use stubs and mocks for features J and K, and service virtualization for feature L

Example with probes

ProbeFrom practical experience, I can tell that using “probes” you can get flexibility similar with unit testing in the integration environment, and for integration tests.

Description

  • we need to make some very early integration tests only for feature A (that it is realized also very early) because it is important and its integration suppose high risks and high incertitude
  • we will “send” the feature A realization in the integration environment and we will get a much feedback it is possible. For this purpose, feature A it is “decorated” with some facilities for: accepting flexible inputs, enhanced outputs/feedback, with others for adapt, if is necessary, to some external system and possible others stuff
  • we can run more test scenarios for feature A in target integration environment and gather the useful feedback
  • important: it is easier to change the probe specific decorations and to get quickly more other feedback then re-deploying more features together with too many needed stubs/mocks or services virtualizations.

The main trick it is the decoupling: we are isolating the test of <Feature A, Integration environment> with the benefits of getting from “decorations” more flexibility on tests (scenarios, input data) and a larger feedback channel. If the text context it is <Features A-B-C-D, Integration environment> we do not have these benefits for one single feature. The effort for building and using test cases with different usage scenarios and input data for only one feature it is much bigger in the last case.

Warnings:

  • “probe” testings are exploratory integration tests, for discovering problems or for investigating risks
  • “probe” based testing cannot be done if the design is not based on separation of concerns, SRP decoupling, functional cohesion and other similar  design principles.

One of the main concerns for Agile Architecture and Agile at scale is to perform the integration tests first / integration tests early. “Probe” based integration offer a flexible, opportunistic approach that could be included also in the spike solution category from XP.

Picard: Oh, it’s me, isn’t it? I’m the someone. I’m the one it finds! That’s what this launching is, a probe that finds me in the future. – “Star Trek: The Next Generation”, The Inner Light

Practices , , , ,

← Older posts

%d bloggers like this: