How to Make Agile Development Successful within a Fixed-Price Consulting Model

The purpose of this article is to discuss my experience in leveraging agile development approaches while delivering engagements working for a consulting firm. It should be noted that most of these engagements use fixed price contracts where scope, budget and schedule are defined well advanced of any detailed work. It is my opinion, that agile approaches and tools can still offer a lot of value on these kinds of projects.

What does it mean to develop using agile approaches?
While I assume that most readers are at least somewhat familiar with what it means to be agile I thought I would give a quick recap. The agile alliance defines the following manifesto:
Individuals and interactions over processes and tools

Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

The agile manifesto states that the items on the left of the statement are more important than the items on the right. However the manifesto clearly states that items on the right (including contract negotiation) still have value and are often a required part in any development project.
The agile manifesto further defines a couple of key principles that one should follow in order to truly claim that one is developing using agile methods. I've listed some of the most important ones below

1) early and continuous software delivery, with a preference towards small biweekly iterations.
2) acceptance and even welcoming of continually changing requirements
3) tight integration between the business and developers
4) giving motivated individuals the environment and support they need to be successful
5) a steady pace of software delivery that can be sustained indefinitely
6) face-to-face interaction is the best form of communication

While many of these tenants have become accepted by the larger developer community it's interesting to note that agile is often considered a "dirty" word among professional project managers and in many classical consulting organization in general.

fixed-price contracts: often a necessary evil when engaging with consultants
In order for any client to engage the development services of a consulting firm a contract must be created outlining the scope of the work. Often this is a fixed-price contract. Consultants typically work with client project managers to create a contract for an upcoming project, estimate the work, and determine the cost of the delivery in advance of any serious requirements or design work. Reasonably detailed contracts are required upfront to specify the nature of any engagement and the overall scope of any delivery. Needless to say, this environment can be perceived as not being conducive to an "agile approach" and as a result there's a fair amount of controversy around agile approaches at the firm that I work for.

Does an agile approach offer any benefit when following a contract model?
It is the opinion of many of the manager type resources within our firm that agile and fixed-price contracts are an absolute no-no. Typically our firm expresses a clear preference for structured, sequential, delivery methods. According to many in management, a highly structured approach supported by a robust and well defined change management process is the best way to manage any contractual risks to the engagement. Furthermore, management often equates agile development with "cowboy coding", and doesn't necessarily have the correct perception concerning the discipline required to make agile work. The problem with an overly structured approach, is typically things do change, scope does creep, and often an extensive, bureaucratic change management process only further complicates the situation. Far too often resources on the ground are asked to take up the slack. Often, the cost of change is not clearly perceived, quality begins to suffer, and the project is eventually impacted in some form or other.

My own experiences have led me to believe that an agile approach is crucial to the success of any development project that contains any technical or business ambiguity. I believe this represents the majority of most development projects, especially development projects that require outside consultant help. Typically, if the project was simple and risk-free, then clients would not be hiring consultants to manage the engagement in the first place.

The key to making agile work while also ensuring that the conditions of a particular contact between the consulting firm and the client are met is to make sure that agile approaches are supplemented with a suitable amount of structure that takes into account upfront pre-iterative activities such as initial planning, scoping, analysis and architecture.

What is interesting to note, is that agile and structured approaches do not need to be mutually exclusive. By being a little bit more liberal when interpreting explicit activities of some of the more structured approaches one is able to adapt and extend these approaches and execute them using an iterative and agile approach.

Listed below are set of best practices that have supported me well when applying agile approaches while still having to take into consideration factors such as budgeting, planning, and managing to explicit contract.

Attempt to structure contracts so that statements of work support an iterative approach
It is much easier to support an iterative approach when working against contracts that are smaller in duration. My experience has led me to believe that the ideal contract length is anywhere from two to four months. If contracts are any smaller you'll find that you're spending almost all of your time doing nothing but building contracts. You will also have very little room to make up for underestimated work within a particular contract. If contracts are any larger, you begin to open yourself to greater and greater risk of underestimating a particular piece of work.

Often master contracts are written for large engagements that can span several years. In this situation I found it helpful to negotiate with the client the ability to write separate requisitions of work against the master contract. These requisitions typically contain agreements for specific deliverables within a particular set of iterations. There will always be a situation where consulting firm will be forced to deliver work against a large multiyear contract. In these situations agile and iterative tools can still work to provide a internal baseline of progress and warning signals that the overall program might be in jeopardy well in advance of more typical waterfall like approaches.

Separate statements of work in sequence of perceived risk
While this might seem like an obvious statement, all too often I've been on contracts where statement of work have been written and executed in a typical waterfall like fashion as suggested by the order below.
1) requirements
2) design
3) implementation
4) testing

This approach is a completely naïve and does not take into account what risks should be mitigated in what order on a development project. While the specific risks on any particular project will vary depending on the situation, a more realistic structure for scheduling different statements of work is shown below.
1) Project Objectives - scope, business case, high-level cost, plan and schedule, conceptual architecture, POC
2) Project Definition - release planning, architecture, greater requirement stability, high risk prototyping and reference development
3) Project Construction-development in order of risk, code testing, detailed project documentation
4) Project transition-knowledge transfer, stakeholder product acceptance


I have frequently found it practical to include some degree of implementation work within each of these contracts. As an example, it is very difficult to know if the high-level conceptual architecture of the solution is feasible unless some portion of the architecture has already been implemented and proven. Frequently, it will be necessary to validate any conceptual architecture with a very simple strawman POC that ensures that all pieces of the solution are actually able to communicate with each other as intended.

Keep documentation (especially anything that could be considered "binding") at a high -level until it can be properly validated against an implementation reference
What this means is that requirements, domain models, architecture models, and any other project documentation that could be looked at as binding should be kept high-level and directional only until it can be verified with some kind of reference implementation. My experience has suggested that implementation has a massive effect on detailed design, almost a 100% delta on the documentation of the original solution. Where possible, I recommend keeping documentation at a high-level until a least some portion of it can be validated against a reference code base. Of course, this is not always possible and sometimes clients want documentation sooner, just be sure to communicate that all initial documentation needs to be validated with real implementation work, also make sure to be prepared to schedule the time to write the documentation at least twice.

Validate high-risk portions of the development effort prior to committing to engaging with the larger team
The majority of engagements that I involve myself in all contain some degree of technical risk. Examples of this type of risk are incorrect solution behavior, poor performance, poor reliability, or inaccurate estimation. Much of this risk can be effectively mitigated by initially deploying a small team of fairly senior resources in advance of the larger team. The sole purpose of this team is to address these technical risks. A senior team can use a combination of analysis and prototyping to resolve much of the technical and business related ambiguity prior to full-blown implementation. Once the mismatch between initial estimation, risk and actual solution is understood this information can be used to create a more finely tuned plan and accurate contract for later phases of engagement.

Use a pure agile development process to structure how work is conducted within each development team, but organize these teams and their efforts using more prescriptive milestone-based processes.
Agile development approaches are great for managing development teams on a micro/medium scale. They provide tools for estimating the real-time "velocity" of a development team, great techniques and tools for team collaboration, and place the proper emphasis on an automated test driven development method. What agile approaches don't typically do is describe how to use documentation, modeling, and planning to properly scope and coordinate multiple teams with each other. In my experience the following approaches supplement agile nicely to support larger engagements, especially with a bias towards a contracting model.
1) Explicit upfront documentation of the interfaces between teams and major systems
2) prototyping, testing, and documenting architecturally significant component in detail before a larger team begins development work using a untested platform or framework
3) documenting and testing nonfunctional requirements before architecture is considered to be properly validated
4) reasonably detailed functional requirements where continual business stakeholder involvement is not guaranteed
5) executing a explicit testing phase after the iterative development cycle

Long-term planning is still essential, even when developing using an agile approach, however planning detail should vary depending on the duration of the plan being used
In point of fact, agile development requires an awful lot of planning in order to be successful. Agile planning emphasizes the use of planning at different levels of details depending on how far out the plan goes. One approach, recommended by the excellent text "Agile Estimating and Planning" by Michael Cohn; is to create a set of product, release and iteration plans for a particular project, each at a different level of detail.

The product plan may span multiple years. The product plan describes the likely release schedule, as well as the contents and objectives of each release at a high level. Each release has a corresponding release plan. Typically, the release plan is only developed one or two months prior to the release actually been executed. The release plan defines how the release will be broken up into one or more iterations. Iteration deliverables (often called user stories, although I simply prefer the deliverables or features) are described at a high level, estimated, and allocated to a specific iteration. Iterations should be structured so that a small subset of system functionality is theoretically made available for the client to review. Each iteration is typically anywhere from two weeks to two months in duration, with a preference for the shorter side. Finally, each iteration is structured to support a more detailed activity-based iteration plan, however this planning is typically done at the beginning of the iteration or at most one iteration ahead of time.

Be flexible when deciding on how to adopt agile approaches
I found that one of the biggest inhibitors to adopting agile approaches is that agile evangelist can be somewhat overzealous/religious about agile in general. One of the interesting tenet of XP (a popular agile approach) is that "one must be extreme in applying all of XP practices". In my opinion, this is pure nonsense. There are many types of project environments that can benefit from many of the practices and approaches prescribed by the agile community without having to blindly follow every single principle to the letter. The question should not be "are you doing agile or not". Rather, the important question is "how can agile principles and practices benefit your current work environment?"

As an example, many agile approaches suggest that requirements be developed using "user stories", (an informal functional description of user and system behavior) and that all implementation work be based on developing code by realizing these user stories into code and allowing the common components, frameworks, and other architecture pieces to naturally evolve. This can be a very powerful approach, and I have always found that evolutionary forces have had a very beneficial impact on my architecture and design. However, larger scale systems developed in parallel by multiple teams typically require that some upfront architecture and design work be conducted. There is a middle ground where the right amount of modeling and/or documentation can be used to help direct and guide the development of one or more agile teams.

Furthermore, on a project that I was recently on, the team was directed to do nothing but develop a platform and framework for an SOA solution. This meant we had actually no capability to actually create "user stories" per se. Does this mean that we could not follow an agile approach? Of course not. Simply by being flexible we decided to designate the client architects as our "end users", and wrote user stories describing the features that are architects wanted the platform to do. We called our user stories "platform deliverables", assigned story points (calling them feature points), and developed and validated them with the client architects using an iterative approach. Examples of our platform deliverables were things like "Handle Service Request" or "Cache Service Request".

The point here is that one can adapt agile approaches to a surprising number of situations, one simply needs to be flexible, creative, and willing to treat the variety of approaches and methodologies out there, both non-agile and agile as tools, rather than as a prescriptive set of processes that should be followed "just because".