Roadmap to an agile design

BROKEN PROMISE

There is a huge imbalance between the Agile promise and what it is obtained with the usual set of Agile engineering practices.

 Why?

The common approach is to use the Agile methods such Scrum and XP as recipes or as a “closed universes”, by using only the engineering practices prescribed by the methods. Any other proven and outstanding practices are rather rarely used.

The standard set of practices – Simple Design, Refactoring, TDD – does not cover some significant problems:

• Refactoring is not effective & efficient if the first code is too dirty.
• None of these practices specify how to protect the business representation inside the software system against the changes related to technologies
• None of these practices can guarantee that changes in the customer business could be introduced in the software product with high efficiency/effectiveness

We need to go outside the methods and discover what the industry could offer for an Agile Design in order to keep that promise.

.

SUMMARY

Introduction – we need to go outside the recipes to put together all the industry strength, in order to fix the imbalance between the used practices and the Agile promise.

AGILE PROMISE – an Agile approach should offer good (sustainable, available, effective, efficient) results for adaptive problems/adaptive context (quick, often, even late) (See Agile Manifesto).

Engineering support – usually, in the best cases, this support does not go further than these three practices: simple design, refactoring and TDD.

What is missing – good enough support for efficiency/effectiveness on: keeping a clean code, business changes or technology changes. We need to define this support for gettting/maintaining an AGILE DESIGN.

The roadmap to an Agile Design – we need to find the minimal set of indispensable, context agnostic, design related practices to get/maintain an Adaptive Design and an Adaptive Product.

The “recipe” problem – the usual tendency is to use only the “inside methods” design practices and to ignore any other useful practices.

Aspects to be considered – the selected practices must cover capability of being adaptive:

• For design level and architecture level aspects;
• For process aspects that support envisioning of the initial design, cleanup of technical debt, quick quality check and adherence to business needs and their changes.
• For realizing the design principles that support building of adaptive products.

Indispensable practices for an Agile Design – by levels:

• DESIGN LEVEL
  • Clean Code, Refactoring, Applying Law Of Demeter, Programming Style
• ARCHITECTURE LEVEL
  • Clean Architecture, Applying Law Of Demeter
• PROCESS LEVEL
  • Architecture Envisioning, Look Ahead Modeling, Model Storming,  Refactoring, TDD, Simple Design, Non-Solo Development, Product Backlog, Continuous Integration
    

Indispensable principles for an Agile Design – Single Responsibility (of a design element), Separation of Concerns (in enhanced mode, with Dependency Inversion), Functional Cohesion, Law of Demeter.

Final thoughts: Any of these principles and practices are pillars on getting and maintained an Agile/Adaptive Design Even some of them are rarely used, are already defined and proved by major contributors to Agile knowledge and software industry. Also, my personal experience found the selected practices and principles very useful.

.

AGILE PROMISE

 The promise – Good and Adaptive

Agile approach main promise is to be always ready to respond quick, effective and efficient to often changes, even late, in customer business needs. Thant mean we have two kinds of promises:

• GENERIC – good results
  • always – sustainable and available
  • effective
  • efficient
• SPECIFIC – adaptive
  • quick
  • often
  • even late

If you have any doubts about these things, please review the Agile Manifesto (see [B-AM]) and the main original Agile methods as XP-Extreme Programming and Scrum. Can be easy observed that all the engineering and management practices are especially selected (in the principles and methods) to get good results, but for rather adaptive problems (Scrum explicitly declare that).

Engineering support

In order to keep that promise, the used engineering practices must be aligned to the declared goal. In my previous post, I have notified the necessity to build and maintain the software products as Adaptive Product in order to have and maintain the Agile capability. A significant aspect in getting this kind of product is to select and use all necessary design practices to get/maintain this capability: we need an Agile Design as an Adaptive Design.

If an Agile practitioner it is asked about this engineering support, the most probable answer is: “we are successfully using refactoring, test driven development and simple design”. All these are great practices and an engineering process that use them could be also great. The question is: if we are using all these practices we could still have significant problems with adaptability?

Unfortunately, the answer it is “yes” (See “Broken Promise”)

Another question: there are other agile practices that could solve these kinds of problems? The answer is “yes”, but are rarely used.

What is missing to keep the promise?

All the mandatory design practices necessary to keep the promise must be put together in a coherent set. Now, only XP have a set of core design practices, but is incomplete and does not cover all aspects of an adaptive design.

.

ROADMAP TO AN AGILE DESIGN

The roadmap – restoring the balance

We need to restore the balance – to find an Agile Design as an Adaptive Design that keep the Agile Promise. For that we need to find good enough set of design practices that could provide and cover the main design aspects.

We are looking for design related practices that:

• Are generic, not context dependent
• Indispensable for the “adaptive” property of the resulted software system
• Part of the industry knowledge and experience

Some comments about context dependency: a practice is generic if should be always used and considered and the only context-depended question is “how”.

The problem – methods as recipes

Among the known practices for an Agile design, just a subset are “inside methods” practices, that mean explicitly specified in the Agile methods as XP, Scrum, DAD (Disciplined Agile Delivery). Many other very interesting practices are described by the main contributors to the Agile knowledge, but not as part of some methods.

Practice	Proposed by	Inside method	Degree of usage*
Refactoring	XP	XP	Rather often
Simple Design	XP	XP	Rather often
TDD	XP, Scrum	XP	Sometime
Clean Code	Robert C. Martin	–	Rather rare
Clean Architecture	Robert C. Martin	–	Rather very rare
Non-Solo Development	Scott W. Ambler	DAD	Rather rare

Note (*) – Degree of usage is based on published statistics and by the observable number of references to these practices.

Unfortunately, in most cases, only “inside methods” have a larger adoption, because the Agile methods are used as “recipes” (… not in thein intention of the authors) or “closed universes”, where the practitioners avoid to assuming process responsibilities outside the methods border.

The consequence: some fundamental practices are rarely used and the resulted products are less adaptive because of their less adaptive design. The Agile promise is broken.

 .

SOLUTION – defining a set of indispensables practices

We need to find the set of indispensable practices for an Agile Design, but we need to be sure that we have enough coverage for all significant design-related aspects that will help us to keep the promise to be adaptive.

ASPECTS OF AN ADAPTIVE DESIGN – selected list:

• Design principles
  • DPs that contribute to an adaptive design
• Design level
  • Design context agnostic best practices to keep the product adaptive
  • Architecture context agnostic best practices to keep the product adaptive
• Design process
  • Cleanup of the technical debt
  • Fast quality checks
  • Adherence to business needs and changes (efficient and effective)
  • Knowledge management in adaptive context
  • Evolving solution and effects on project definition

INDISPENSABLE PRACTICES PER ASPECT – Then, for each aspect, we need to find the sine-qua-non (indispensable) practices to deal with that aspect. This is the proposed list:

Aspects category	Aspects	Practices
Design level	Adaptive design support	Clean Code, Refactoring,Programming Style, Law of Demeter (applying)
Architecture level	Adaptive design support	Clean Architecture, Law of Demeter (applying)
Process level	Pay the technical debt	Refactoring
	Fast quality check	TDD
	Adherence to business needs	Simple Design, Product Backlog, Clean Architecture
	Knowledge management	Non-Solo Development, Product Backlog
	Evolving solution, Project Definition	Architecture Envisioning, Look Ahead Modeling, Model Storming, Continuous integration

Observations:

• From similar practices are selected the ones with higher coverage.
• Practices are inter-dependent and each of them will contribute to all the aspects and will enhance the results of the others
• The selected practices must realize the selected design principles

INDISPENSABLE PRINCIPLES FOR AN ADAPTIVE DESIGN – These principles are part of the industry proven experience and are in the top recommendations for a good design a technical excellence. The other reason for including them in an agile related list is the contribution to the “adaptive” property of design.

Single Responsibility (of a design element): isolating responsibilities in functions, classes, components, services, sub-systems will make the system easy to build, change and maintain (… where the reasons are already part of the industry experience).

Separation of Concerns: that it is a higher level aspect of SR principle, where separation is rather considered at architectural level (layers, sub-systems) and the concerns are also high-level responsibilities (presentation, business, persistence, hardware interfaces, cross-cutting concerns). Each of these concerns could independently change and without decoupling them, the system will not be adaptive.

Functional Cohesion – that it is starting architectural/design decision that declares as main subjects of separation and decoupling:

• Business concerns from technological concerns
• Business concerns parts/functions from each other
• Could be applied at any design level sub-systems, components, classes, functions

Law of Demeter – See “Practice – Applying Law of Demeter”

It is impossible to explain that you have an Adaptive Design and you will keep the Agile promise and also do not respect these principles.

 .

PRACTICE – XP practices: this is how Agile works

The good part is that XP (see [B-XP]) it is an wonderful representation about how Agile fundaments are working. Here we will talk just about Simple Design, Refactoring and TDD. These practices are helping to keep the toughest agile promises from the Agile Manifesto:

• “…satisfy the customer through early and continuous delivery of valuable software.”
• “Welcome changing requirements, even late in development”

Each of these practices will have its fundamental role:

Simple design – XP: “always do the simplest thing that could possibly work next”.

• Request to focus only to current requirements, without trying to extrapolate the future ones. After a sequence of small releases, we will have a design properly adapted only to current business needs, not to hypothetically or obsolete ones (“junk stories”). Simple design means to be both focused and clean.

INDISPENSABLE – If Simple Design it is not used, then:

• Increasing technical debt and waste caused by guessing future needs instead of waiting the new business context to be relieved. Increasing amount of implemented junk stories

Refactoring – Martin Fowler: “Improving the design of an existing code” by “applying a series of small behavior-preserving transformations“

INDISPENSABLE – If Refactoring it is not used, then:

• The technical debt is accumulating in high amounts and the code will NOT be cleaned, easy to read, to understand and change. The productivity will decrease and the product will be more fragile until becoming completely unstable
• The previous simple design must be adapted to new requirements, and refactoring it is one of the most effective techniques

TDD – automatic tests, with high coverage at unit test level; the tests should be writing first.

INDISPENSABLE – If TDD it is not used, then:

• There is no other technique to support QA/QC concerns in dealing with changes that occur “late in development” and that could offer quick, effective and efficient safety net for refactoring necessary regression tests
• There is less separation of concerns

.

PRACTICE – Non-Solo Development (DAD)

XP introduce Pair Programming as essential Agile practices. The main meaning is to have “cooperative programming”. Unfortunately, this practice it is used too literally, only for direct coding activities. Yes, Agile has restored the importance of the coding in the overall development, but let think a little: what is the meaning of “Programming” from XP name? In fact, it is “Development”, where the effective programming/coding has fundamental importance. An XP programmer it is, in fact a multi-role developer involved also in planning, requirements, architecture, and design, coding and testing.

In DAD, Scott W. Ambler has introduced the term “Non-Solo Development” for a cooperative development approach that combines various kind of pair-development with solo-development.

INDISPENSABLE – If cooperative development it is not used, then:

• All the needed design practices cannot be applied coherently and consistent
• The needed design decisions will be rather under a minimum threshold of effectiveness and efficiency
• The skills and the knowledge of the team will not match to the necessities
• Agile process will not be enabled

.

PRACTICE – Product Backlog (Scrum)

The Product Backlog it is a very used practice, but its impact in the design is less known.

Note: An enhanced variant of the Product Backlog is the (Product) “Work Item List” from DAD.

INDISPENSABLE – If Product Backlog it is not used, then:

• The work will not be serialized and refined in clear increments before starting the main development effort (iteration level)
  • Consequence: Not enough support for realizing a Simple Design,
  • Consequence: Paying the Technical Debt (using Refactoring or other techniques) has poor support on planning and visibility;
  • Consequence: less support to effectively connect to change requests and customer feedback in order to get and maintain Adaptive Products.

.

PRACTICE – Architecture Envisioning (AM, DAD)

AM (Agile Modeling) make explicit the architectural concern that should be addressed early in the development as Architecture Envisioning. This is the place where should be taken also the initial decision about Clean Architecture and most significant decisions related to Refactoring and TDD. (See [B-SWA])

INDISPENSABLE – If is not used, then:

• The main technical risks will not be addressed early with all the resulted consequences
• The planning will have no real support , because the necessary work it is mostly unclear.
• The Clean Architecture rules cannot be applied (in the envisioning must be clarified how will be applied)
• Decisions related to significant refactoring and to high level decoupling needs related to TDD cannot be taken at the appropriator time

.

PRACTICE – Look Ahead Modeling (AM, DAD)

AM (Agile Modeling) make explicit the design risks and complexity concerns that should be in advance in the development (as a refining of Architecture Envisioning) , as Look Ahead Modeling. This is the place where should be taken also tin significant decisions related to necessary  Refactoring  and TDD . (See [B-SWA])

INDISPENSABLE – If is not used, then:

• The undesired complexity of the design will increase in the most sensible places
• The plan updates and progress tracking are affected because of not updated information about solution
• The TDD, Refactoring and Clean Code significant decisions will be affected

.

PRACTICE – Model Storming (AM, DAD)

AM (Agile Modeling) make explicit the design complexity concern that should be addressed JIT (Just-In-Time) in the development, when is needed,  as Model Storming . This is the place where should be taken also the possible decisions related to necessary  Refactoring  and TDD . (See [B-SWA])

INDISPENSABLE – If is not used, then:

• The undesired complexity of the design will increase in the most sensible places
• The TDD, Refactoring and Clean Code decisions will be affected

.

PRACTICE – Clean Code

The clean code rules are introduced by Robert C. Martin (see [B–CC]). Of course, there is an intersection and similarity with the refactoring rules introduced by Martin Fowler. The difference is rather at process level: Clean Code supposed also to write the proper code from the first time, combining with continuous cleaning and refactoring. In fact, we can combine Clean Code, Refactoring and other similar rules in a one single set where:

• Clean Code is what we can want to obtain as code state/attribute
• The first time code must be clean enough
• Refactoring is the operation to pay some technical debts and get closer to the Clean Code state

Clean Code as a practice should represent all the small design and design level best practices that are generic, context agnostic and address most generic causes of producing the technical debt.

Examples of subjects for clean code rules: meaningful names, functions and class creations, comments, unit tests, building systems, simple emerging design and others. Example of generic rules and (solved) problems: avoid feature envy, misplaced responsibility, make logical dependency physical.

INDISPENSABLE – If clean code rules are not used, then:

• Will be activated the causes that produce the most amount of technical debt – the code mess is systematically produced in huge amount.
• The code will be hard and very expensive to read, change and fix

.

PRACTICE – Clean Architecture

The Robert C. Martin concept of Clean Architecture (see [B–CA]) suppose that according to the fundamental design principle of Separation of Concerns we will take from the start some architectural decisions (because it is always useful and it is context agnostic):

• Functional cohesion – the system functions must be visible in its design: various business concerns are separated by design and the technological concerns from the business concerns. The non-functional aspects must be isolated and should be considered as details (!), because things as the used technologies could always change, but the representation of the core business must survive.
• Dependency injection – main way used to separate the various concerns

Alistair Cockburn proposes the similar concept of Hexagonal Architecture (see [B-HA]), as opposite to un-balanced architectures where only user-interface and persistence are driving the architecture, beyond the business part: “application communicates over ‘’ports’’ to external agencies”.

Koni Buhrer proposes the Universal Design Pattern (see [B-UDP]),, which gives little more details than the Hexagonal Architecture and could be a good approach to implement a Clean Architecture.

INDISPENSABLE – If clean architecture rules are not used, then:

• the core business is NOT protected from technological changes, because it is NOT decoupled from those aspects
• the core business is NOT realized using functional cohesion, the product will NOT have the “business DNA”, and cannot be an Adaptive Product (see my previous post), cannot be changed quick, efficient, effective and sustainable with customer business changes
• Successive changes related to business change will produce design and architectural level technical debt – affect the overall productivity and quality
• Similar undesired effects for successive changes caused by technologies aspects
• (dramatically) Decrease the effectiveness of TDD, because best criteria of decoupling are not applied and both code and tests will be change too often on business and technological changes.

.

PRACTICE – Applying Law of Demeter

The Law of Demeter (LoD) or principle of least knowledge – “The fundamental notion is that a given object should assume as little as possible about the structure or properties of anything else (including its sub-components), in accordance with the principle of “information hiding”.”

It was also subject of more research and has influence development of more patterns. For Demeter project, Karl Lieberherr created a so called “Adaptive Programming” based on this principle (see [B-AP]).

In our days, the Google Talks presentations bring back into public attention the significant advantages of this principle. Using this principle should be a specific way to create objects and functions, a specific way to share and distribute data and information between them: in any part of our code we should not use references to a global context.

Respect of this principle it is fundamental to make possible all other best practices: concerns decoupling and functional cohesion.

INDISPENSABLE – If Law of Demeter is not used, then:

• Separation of concerns, functional cohesion, clean code, and clean architecture are severely affected by breaking this rule. Consequences:
  • High amount of severe technical debt it is introduced (by using global context anywhere in the code)
  • The code fragility could dramatically increase (any change and fix could produce many new defects)

.

PROLOGUE (sic!)

I have selected these practices and principles for an Agile design from the intersection from what I know  from the industry experience and my personal experience. The starting point was that no Agile process method has included (from what I know) the Clean Code and Clean Architecture practices and, also, the principles aspect was ignored (exception: the above mentioned Adaptive Programming) .

When I tried to categorize the Refactoring and  Non-Solo Development,  I have added also the process aspect. Later, thanks to a feedback from Scott W. Ambler were added the missing Architecture Envisioning and design modeling practices.

I have tried to find past endeavors about:

• defining some practices as indispensable, where only Adaptive Programming have used a such logic
• unifying these indispensable practices for an Agile Design

I have found almost nothing, and the searches about “Adaptive Design”  found results only in furniture domain and was similar for “Agile Design“.

Better late than never… just yesterday (September 28) I have discovered a previous definition of an Agile Design:

http://agilemodeling.com/essays/agileDesign.htm

That it is part of Agile Modeling method (Scott W. Ambler) and contain: Architecture Envisioning, Iteration Modeling, Model Storming, Test First Design, Refactoring and Continuous Integration. Some comments: I think that Iteration Modeling is a sub-practice of Look Ahead Modeling (but indispensable by itself) and that Continuous Integration is also indispensable.

.

BIBLIOGRAPHY

[B- HA] –Hexagonal Architecture, Alistair Cockburn

• http://alistair.cockburn.us/Hexagonal+architecture

[B- UDP] –Universal Design Patterns

• http://www.ibm.com/developerworks/rational/library/content/RationalEdge/jan01/FromCraftToScienceJan01.pdf

[B- AP] – Using Law of Demeter. Adaptive Programming. Brad Appleton

• http://www.bradapp.com/docs/demeter-intro.html

[B- AM]- Agile Manifesto

• http://www.agilemanifesto.org/

[B –XP] – Extreme Programming

• http://www.extremeprogramming.org/rules.html

[B – CA] – Robert C. Martin (Uncle Bob) – Clean Architecture and Design-2012 COHAA The Path to Agility Conference

• http://www.youtube.com/watch?v=asLUTiJJqdE

[B – CC] – Robert C. Martin – Clean Code: A Handbook of Agile Software Craftsmanship, Prentice Hall, 2008

[B – SWA] – Scott W. Ambler – Agile Modeling

• http://www.agilemodeling.com/

Motto – The True corruption of Agile – Robert C. Martin

• http://blog.8thlight.com/uncle-bob/2014/03/28/The-Corruption-of-Agile.html