A Macro View of Agile

Despite the fact that the agile manifesto was signed over ten years ago agile adoption remains uneven.

One manifestation of this is how often I have clients ask me for even a simple overview of what I mean by agile.

I decided to put together this simple map that shows how the pieces fit together.






IMHO lean thinking and tools like A3 and Kanban tie all of the agile pieces together, and help organizations think and behave with agile principles in mind.


Supplementary agile practices like BDD and DDD integrate modeling with agile practices to help core agile team based approaches.

Core agile methods like XP are just that, and create the foundation for delivery excellence.

Technology focused movements like DevOps focus on upping the quality of from a technical perspective.

And while some will groan, RUP or UP provides some good lifecycle stages, roles etc,etc that can help scale agile.

Agile over RUP Part 4

RUP Principles

As I have said before, one of the best parts of RUP is the principles and best practices. Probably the fact that RUP has explicit principles that are clearly articulated makes it a better methodology that many of the SDLCs that I've seen put together, especially by those in the management consultant world. In fact most SDLC I have seen don't even bother having proper principles. Having a consumable set of principles means that development teams can pretty much guide work around whether they are following a principle, not whether they are following a specific detailed piece of processed material.

I have listed below the major RUP principles along with some minor alterations in order to make them more agile.

The RUP is a use case centric

RUP makes a big deal about how every single artifact in the process can be traced back to a particular use case. In RUP, use cases are a collection of scenarios grouped together according to their ability to help a specific user accomplish a specific goal. The idea of developing use cases and then hanging your design documents, plans, test, and anything else might need off of these use cases is a great idea.

Use case traceability is a complete waste of time...

Unfortunately RUP takes what is a reasonable approach and pushes it to the extremes of absurdity. RUP recommends that you follow what is known as "use case traceability". Use case traceability is the notion that you can have a database of every artifact created in your SDLC (including code) that tracks each artifact and their traceability to each use case and each subsequently developed artifact. In my experience, this is a colossal waste of time. By doing this, you are in effect trying to track a multidimensional, real-time, many to many relationship. What makes this even harder, is the only person capable of properly managing the database of artifacts is someone who has a reasonable understanding of the entire SDLC. This is most likely a very senior person on your team and he has better things to do than traceability management. More than likely, the person responsible for managing traceability is the person who drew the short straw in your project and is almost completely clueless about what he is actually managing.

But using use cases as a framework for managing a project is actually not a bad idea...

that being said, I almost always develop and maintain a use case hierarchy like the one prescribed by Alastair Cockburn. I do not always develop complete use cases, but I do use this hierarchy (usually stored in some kind of wiki) to act as an anchor to associate most of my other major SDLC elements. Of course in the agile world, there are a lot less SDLC artifacts so usually I associate individual use case nodes with things like:

• individual user stories
  

• test cases (using tools like FIT)
  

• simple models (as needed)
  

Traceability with working code is done by utilizing a revolutionary concept known as talking to the person responsible for developing or supporting the code. Relying on people seems to be something very scary in the manager world, but rather than focusing a whole bunch of work on doing complex traceability, try spending more work on making sure that there's always somebody who knows how your solution works.
The RUP Is Architecture Centric

The RUP also makes a big deal about being architecture centric. The RUP describes architecture as a filter on the various models necessary to build the system. (e.g. requirements model, design model, implementation model, deployment etc.) This filter represents a common vision of the system, it's common components, and unifying elements. The RUP spends a lot of time describing what architecture is and what it is and using terms that would probably baffle most of us, but the point is that according to RUP, architecture is something that needs to be considered throughout all aspects of developing software. While most people who follow agile probably recoil at the heavyweight definitions of architecture offered by the RUP, what is refreshing about RUP compared to most other methodologies is that RUP believes that architecture permeates all aspects of the system. Most other methods seem to put architecture "above and around" the actual implementation of the system. RUP on the other hand, believes that architecture is represented in its requirements, its design, its code, and how it's deployed. In other words architecture is more than a bunch of pretty diagrams with boxes and lines, and it doesn't cut architects who have no desire to be involved with implementation any slack.

Architecture is important, but keep it lightweight...

Where RUP tends to fail, is in the details. RUP has lots of advice on how to develop detailed use case models, detailed use case realizations and traceability matrices necessary to modeling and maintaining the "architecture" of the system.

Keeping a focus on architecture on any large scale project is important but in order for it to be maintainable it needs to be lightweight, and focused on value. What is valuable is going to change from project to project but in my experience make sure that effort is put into developing a set of architecture and coding standards, and that everyone be on the team is aware of them.

Everything else is gravy. (And gravy is not a good thing if you're trying to stay lean)
The RUP Is Iterative

In terms of principles that add value, this is a no-brainer to anybody who's trying to adopt agile. That being said, RUP does not offer very tangible advice on how long iteration should be, and how to manage what goes on in iteration. Most of the milestones, gating criteria and metrics are based around the larger scale phases.

While the RUP makes a big deal of iterating the best advice you are going to get around how to manage iterations are going to come from things like scrum or XP. Whenever anybody attempts to adopt RUP, the first thing I do is train them up on how to manage a project using iterations using approaches described by a really good text, Agile Estimating and Planning.

The RUP Is Model Driven

The RUP phases places a heavy emphasis on developing one or more diagrams/models to represent the system. When going through the RUP, one will encounter guidelines on how to develop use cases models, use case realization models, design models, deployment models, activity diagrams, etc.

In point of fact, it's probably a full-time job just to keep up with all of the UML diagrams recommended by RUP as well as the approach to using these diagrams within RUP. The biggest issue in using these models is one of maintenance. While not specifically saying so, RUP does imply that these models need to be developed, and kept up to date using reverse and forward engineering principles.

Models have A Lot Of Value, But Use Them for What They're Worth, They Aren't the Solution

Models are actually a great thing, they help us communicate, they help us understand, and they help us abstract implementation details that can take an awfully long time to learn. But models are expensive to create, and they are incredibly expensive to maintain. Models are also only so valuable if they are developed by the technical team in isolation of the business team. In order to be useful agile modeling best practices need to be used to supplement the RUP model driven approach. In short these practices consist of

• use collaborative modeling techniques (like CRC cards)
  

• don't be afraid to throwaway models
  

• focus models on interfaces between teams, complex business logic, and code that is currently being reused by multiple teams
  

• don't create a model without having an audience in mind
  

• don't create a model because your process says so
  

When Using RUP to Scale Agile Make Sure You Follow Agile Documentation Principle

Scott Ambler, has some great on how to develop documentation in an agile manner. In a nutshell, Scott suggests that you should only document when

• you are satisfying a specific project stakeholders
  

• you have a stakeholder who can help you build a table of contents and direct you on what he wants to see
  

• when there is specific business value
  

• above all, don't create a document because some piece of processed material says so.
  

So if you've actually made it to the bottom of this document, you should have a pretty good idea about how I, and you could scale agile using specific modified portions of RUP in a pragmatic fashion. Scott Amber also has some great posts on how to scale up agile with RUP
Hopefully, anybody reading this will also have a better understanding of the particular project take to implementing large-scale software development projects.

Agile over RUP Part 3

In my Previous postI mentioned that the rational unified process is slightly modified can offer good value to large-scale projects, in this submission I elaborate on some of the components of the process.

Product Lifecycle Phases and Skill-based Disciplines

One of the biggest differentiators of the Rational Unified Process is the way that the method uses a two-dimensional grid to categorize work into one or more phases, as well as a specific disciplines or skill sets of work.

The problem with many structured development processes that they categorize work along one dimension only, this dimension is usually based on skill sets. What this in effect means is that work is organized around completing requirements, then completing design, then completing development, etc. Given that the whole waterfall approach to methodology was first described as a anti-pattern almost 30 years ago, it's surprising how often I see software development methodologies popping up that prescribe to it, especially in the management consulting world. The RUP has at least has the good sense to realize that if you're going to break things up by phases, then create phases based on the natural lifecycle of a product.

While the RUP does categorize work into disciplines (requirements, design, etc.) the RUP explicitly states that this work from different disciplines can be conducted in parallel and that there is plenty of opportunity for the work to overlap. Furthermore, the RUP provides advice on how to break up phases into multiple iterations.

RUP Phases

a brief description of each of the RUP phases are as follows:

Inception

At the beginning of a project, there has to be a vision, a good idea that will benefit the business, and there has to be some method of generating the money. Agile doesn't talk about any of this, when you start an agile project, you're in requirements design and development. Somebody has to talk about putting together the business case, looking at the solution from an enterprise technology perspective, (should the solution is Java, or.Net, how about a package like PeopleSoft?) Stakeholders need to be lined up, and a high-level estimate of what the overall solution is going to cost needs to be put together. Someone asked all sorts think about organizational change and training. Many agile projects are doomed to failure (IMHO I don't have a stat of this or anything) if they don't spend it least a couple of weeks to a couple of months (depending on project size) working on inception. Think of inception as an iteration zero on steroids, and work is not only done by developers, although they do need to be heavily involved on the technical side.

Elaboration

Once the team has a general idea of what they are doing, and why they're doing it, the RUP recommends to start the SDLC process (i.e. requirements, design, development, test, deploy) in multiple iterations focusing specifically on technical uncertainties, scary requirements, and generally anything that makes the developers "stay up and shiver at night". Again, elaboration can be looked at as the enterprise version of a combination of multiple iteration zeros, but supplemented with a comprehensive, planned set of spiking. One of the fundamental pillars of RUP is that any large development project should contain a phase where a subset of a large development team can get together and experiment, prototype, and mitigate technical risk before applying a large-scale team to developing the entire solution. One interesting thing to note is that many companies adopting RUP confuse elaboration with design, in point of fact elaboration requires one or more complete iterations of requirements, design, develop, test and deploy to be considered complete. The difference between elaboration and construction is that elaboration is focused on mitigating technical risk.

Construction

Once the majority of technical risk or guiding a new platform, adopting some legacy code, or understanding some complex requirements have been mitigated through a number of completed development iterations. Management is supposedly able to magically deem that all architecture risk has been eliminated, and it's time to start a brick laying, brainless, assembly-line approach to completing the solution. Now that all risk has supernaturally disappeared, it's time to expand the team, and optimize the delivery approach so that is an effective, efficient manufacturing style process. (Excuse me while I laugh hysterically) As naïve as this viewpoint is to developing software is, what is even worse is that many organizations confuse construction with the phase where all software development is conducted. According to RUP construction still requires revisiting requirements, revisiting design, and of course developing and testing. The emphasis is that more development than requirements or design should take place.

Transition

At some point in time, the solution needs to be handed over to the client. Training needs to take place, consultants need to be replaced with counterparts within the organization and the solution needs to be entrenched within the organization. This is all completely reasonable, and is something to say doesn't really talk about, and in my opinion probably shouldn't. I'm not sure I see the value in having any SDLC try to provide advice on how to fundamentally deliver a software product to an organization, it's not that I find the transition phase to be incredibly important, I just think that the RUP provides extremely superficial advice, and waters down its own strong points by trying to do too much. I personally work for a consulting company that has an extremely strong organizational change department. And trust me, software process geeks really have no idea on how to approach this One. Again, the biggest mistake organizations make when adopting RUP is to confuse transition with testing and deployment, the two have nothing to do with each other. If anybody out there is really interested in how to tackle transition, I recommend reading the Heart of Change for a start.

Compared to other methodologies RUP phases make sense but...

Let's be honest, has anybody out there actually done any development work and said "hey we are done elaboration phase, it's time to start construction...". Well I have actually been on projects where we had a "elaboration team" which was responsible for putting together the architecture framework, setting up the technical foundation, and making sure that the platform was solid from a performance point of view, and could handle the various "complex" requirements. We then had a "construction team" which was scheduled to start several months after the elaboration team, which was supposed to use the "common design components", patterns and other pieces that the elaboration team that put together.

While this sounds great on paper, the reality is that the majority of elaboration work really started once the construction team had landed, it was only when they started using the "common components" to implement the "non-complex requirements" that the elaboration team really figured out how the common components should operate. In short, software development cannot be neatly broken out into elaboration and construction phases, what I see happening is that iterations tend to start out with what appear to be largely "elaboration style" activities. These early iterations tend to have more experimentation, prototyping, and spiking necessary to mitigate technical risk and figuring out the intricacies of whatever new technologies are being used on the project. Subsequent iterations tend to have less and less elaboration style work and more and more construction style work. That being said, every once in a while, a drastically new requirement comes into play, or the development team figures out a new approach that can drastically improve your overall solution but requires a dramatic rethinking of the way things are being done. In short, replace the RUP construction and elaboration phases with a development phase, and plan to have a decreasing but fluctuating amount of elaboration style activities in each development iteration.

In my next post on this topic, I complete this article by giving an overview of RUP principles and best practices and how to modify them to make them more agile.

Agile over RUP Part 2

This post is a continuation of my Previous poston my preferred development approach.

Reading about Agile Is Pretty Simple (which is a good thing...)

In my previous post on my preferred development approach, Agile over RUP, My Preferred Development Approach I touched upon the idea that mixing Agile with a more structured methodology like the Rational Unified Process (or the unified process in general) was a good way to combine what I see as a energetic, creative and real-time approach with a structured, more methodical framework that can help to organize large-scale projects.. I then went on to describe what agile means to me, and gave a summary of the agile best practices that I found useful on projects that I have been a part of.

In general, talking about agile is relatively straightforward. The practices are deceptively simple, and relatively easy to explain. The real difficulty in agile is in the implementation. I have probably been attempting to use "agile development" since early 2000, and I'm still probably learning at an incredible rate. I doubt that this rate of learning is going to decrease anytime soon, getting agile right is actually quite hard to do, but then again so is real success. IMHO this is all about what a good process should be, really easy to read, and incredibly hard to master.

Reading about the Rational Unified Process Is Pretty Complex (Which Isn't a Good Thing...)

The Rational Unified Process, on the other hand is a fully featured software development lifecycle framework that tries to encompass all aspects of software delivery. Unfortunately this leaves consumers of RUP with the impossible task of trying to get one's head around a complex, dense, and sometimes contradictory piece of work.

It's important to note that the creators of RUP refer to the unified process as a process framework, meaning that the process should be customized to the particular needs of a project, program or IT organization. RUP should not be taken out of the box and applied as is.

The RUP framework is a complex web of disciplines, (specific skill sets, i.e. requirements, design, development, etc.) phases, (product lifecycle stages i.e. inception, construction, etc.) process groupings, processes, artifacts, templates, gating criteria, and a bunch of other detailed process material, enough to make process geeks the whole world over cackle with glee, much to the detriment of those of us who are actually trying to get some work done. Here's a quick picture of all the pieces of RUP, written in UML, taken from my memory

So what's wrong with the above picture? For one, its complexity and sheer volume of information makes it incredibly difficult to actually find the valuable pieces of the process. Unfortunately, as of the last time I looked at the Rational Unified Process (which to be fair is ever evolving) it contained some incredibly good information (especially in the requirements and analysis and design of object-oriented systems) but there was a huge amount of process details that appear to be ad hoc, and added just to make the process complete. This seems to be a systemic problem of any detailed and "comprehensive" software methodology. The quality of different pieces of the process vary incredibly (the project management and testing is laughably bad), and it takes a very intimate knowledge of the process framework and some very real experience using it to determine what to use and what to completely ignore.

But even if all aspects of the rational unified process were of proper quality, RUP is just too complex.

But RUP Still Has Incredible Value, It Just Needs to Be Pared down a Little...

Since RUP is a process framework, it's okay to actually tailor it to make it compatible with an useful style of delivery, in fact many of the fundamental pieces of RUP are completely compatible with an agile way of doing things.

Listed below are the portions of the rational unified process that I consider actually valuable to a software development project...

• phases like it or not, the kind of work you do during specific iterations are going to change over the length of any project, lightly applying a notion of a lifecycle is a good idea

• disciplines, while agile projects categorized work into just a couple of roles, there really is more involved if you consider all aspects of delivery, RUP disciplines represent a reasonable way to organize the different types of work that need to be done, just be flexible in applying multiple roles to the same person

• principles and best practices are IMHO the most important part of the process. By tweaking the RUP best practices and principles to be a little more agile in nature, you're left with a solid foundation in which to base your development and delivery.

• Roles as stated previously, agile only has a couple of roles (e.g. Product Owner, Scrum Master, Developer) I personally like to see this list expanded, as long as everyone is clear that each single role does not equal one person, each person needs to fulfill many different roles.

• Discipline Oriented Guidelines where many software development processes fail is that they offer either too little advice, or offer to much and present this advice as standards that must be followed. Detailed process material should be represented as a suite of advice, patterns, and lessons learned based on real project experience. Above all this detailed process work needs to be presented as accelerators that can but don't have to be used. I think it's also okay to base guidance on the work of established authors, but make sure you test the waters on a real project before socializing the guidance across different project teams. So far, I have based my guidance, and implemented real projects, on work presented by the following authors, of course this is a continuing work in progress...

• Alastair Cockburn (Patterns for Effective Use Cases)

• Eric Evans (Domain Driven Design)

• Michael Cohn (Agile Estimating and Planning)

• Scott Ambler (Agile Modeling Method)

• Kent Beck (Implementation Patterns and Test Driven Development)

• and a whole bunch of other stuff...

Please refer to
Part 3 of this post for further details of other valuable components of RUP

Co-Presentation with Scott Ambler at Deloitte

It's been a while since I've had the opportunity to blog, but last week I had a opportunity to present on real-world agile development on large-scale projects with another firm employee, Sunny Ip, as well as the notable agile luminary Scott ambler . Our firm invited Scott to speak to some of our practitioners on why agile matters, and how it can be applied to environments that many critics state agile can not possibly thrive or succeed in.

As I have stated in previous posts, the firm that I work for it as much a management consulting firm as it is a applications development firm. Needless to say, many agile best practices and principles do not form the core tools that a lot of our practitioners use. While this is slowly changing, thanks to a core team is passionate and experience in using agile, having Scott present on the values of these approaches is essential to spreading this message within our firm.

I thought Scott did an amazing job in giving a no-nonsense and extremely frank presentation on why our firm and others like us need to start taking this seriously.

Scott managed to pack a lot of material in a relatively short period of time, topics included:

• agile overview


• surveys on the success of agile and traditional projects-a lot of great no-nonsense material was presented


• shortcomings of pure agile approaches (XP, scrum, etc.)


• extending agile with the rational unified process (RUP)


• applying agile in outsourcing staff situations


• applying agile to the database group and database platforms

I'm sure I did not list every topic covered here, but these were the major ones that resonated with me.

After Scott's presentation sunny and myself presented 4 separate case studies where our firm has taken advantage of agile best practices on large-scale delivery projects. Examples included:

• large-scale public sector transformation project with 20 plus developers in parallel


• assisting the enterprise architecture office in developing a reusable SOA platform


• using agile modeling techniques to properly analyze and design a domain driven solution for a provincial students assistance and loans application


• using agile delivery on a large-scale package implementation also dedicated to any students assistance and loan application

During our portion of the presentation, the audience began to ask questions which were mostly answered by Scott, with additional color/perspective from our firm added in by us.

For my part, my major take away from listening to Scott was that everyone in our firm needs to be much more forthright in pressing our clients on the need for change, and applying innovative/different delivery practices to their organizations. I think anybody who's done any solid delivery work for any length of time is more than aware of how dysfunctional many of our development processes and approaches are in a majority of organizations. But changing the situation requires courage, and the bravery to speak out to CXO level leadership. We really need to provide a frank assessment of the things we see on our day-to-day implementation projects. There's a huge amount of reward that can be gained from adopting agile in a pragmatic fashion. Costs can be lowered while simultaneously increasing quality and reducing risk. These things that business can buy into.

Again my thanks to Scott for a great presentation.

Great blog on "realistic" agile

Scott Ambler has recently started a blog Agile Software Development at Scale. Finally, an established authority on agile is providing advice on how could you to apply agile in a variety of real-world situations. Now this might seem like a funny statement from somebody whose blog is titled "Agile Consulting", but in my opinion a large portion of the agile community is too concerned with purity to be truly effective in a large number of real business world situations. This is not to say that agile is not full of incredibly useful material. The majority of development approaches and tools that I employ are almost always based on agile techniques. (CRC cards, agile modeling, velocity, release and iteration plans, test driven development, continuous integration, etc.)
What makes Scott Ambler's blog so refreshing (and much of his work on agile in general) is that it contains great advice on how to take all the great tools and techniques offered by agile and apply them to full program delivery, large-scale solutions, offshore, fixed-price engagements, and many other circumstances. Many agile purists simply throw up their hands in defeat when faced with one of these situations saying that "unless you follow practice number xx, you aren't agile". Which of course in my opinion is missing the point, which is how to apply agile techniques and common everyday situations that we find ourselves in.

SDLC-Guiding Principles

In my experiences, when a lot of resources speak about SDLC (short for software development lifecycle) in my organization they immediately focus on the concrete portions of the process. Tangible things such as roles and responsibilities, phases, activities, and artifacts all come in to mind. While these things are important, to a lesser and greater degree, depending on your point of view, they miss the most important point of a methodology, which is to describe the overall rationale and goals of the methodology itself.

The problem with using tangible, physical components to describe how an organization should develop software becomes immediately obvious to anybody who has tried to follow anybody else's SDLC. Primarily, a working environment, even one within an organization, fluctuates wildly from project to project. Resource experience is almost never consistent, team size can never be a given, nor is the scale of the solution. Other things such as technology, resource distribution, and project risks are also rarely a constant. What this means is that even a well thought out SDLC will never be more than partially applicable to any given project. I've seen organizations attempt to create a methodology that "scales" to various environmental factors. What this means is that a base SDLC is created, then offshoots are developed for projects that are smaller, larger, offshore, et cetera. Unfortunately, this has the effect of creating an incredibly dense and complex library of processes and approaches which become unwieldy in terms of communication and maintenance. Of course compounding this issue is actual process compliance, which is something that very few organization (I do not know of any) realistically achieve. In my personal experience, the majority of project members within organizations claiming to be CMI level XXX certified really just do the minimum they can to survive a project audit, and do not actually get a lot of value from their SDLC, often not following the "intent" of the SDLC.

So where do methodologies add value? IMHO, the most important part of any methodology is the section that describes the Best Practices and Guiding Principles of the SDLC. If your organization's SDLC does not frame its processes, artifacts and other tangible assets within a set of overriding goals and quantifiable values then resources will have no guidance around what to do when a process does not properly fit into a specific situation. (most situations in my experience) Best practices and guiding principles offer guidance on how software teams can/should strive towards success. They are goal/value oriented, and can thus be more applicable to many different situations than a set of concrete/tangible items such as processes, gating criteria etc. Principles and Practices also have the benefit of typically being shorter than a complex roles/responsibilities/process map which means that more people will actually read it, and may even actually follow it.

I recently was asked to put together a customization of the Rational Unified Process by one of my clients. While I believe the Rational Unified Process has a lot of value within it, much of this value is hidden away in detailed activities, artifact description, and low-level guidelines. I wanted to make sure that followers of the customization focused on the right values represented by the framework, which in my opinion was the emphasis on iterative work, modeling and extraction, and scenario driven requirements. I also wanted to put a clear emphasis on some of the more agile aspects of software delivery which are all compatible with RUP. The first two sections of the methodology customization dealt with these values. I have decided to post them here, as an example of what I believe IMHO is a good best practices and guiding principles section for an SDLC.

It should be noted, that I am about to share around three to four pages of advice on how to proceed for a number of different situations relating to development, requirements, testing etc. There's still a lot of leeway on how to interpret this advice, which I believe to be a good thing. Telling somebody that scenario driven requirements are a good thing and why, is much more valuable than telling somebody what the exact table of contents has to be for a requirements document. A particular artifact on a project might follow a specific template to the letter, but still miss the entire intent of the artifact. Better to focus on the values, some overall best practices and patterns, and allow the resources who are creating and/or using a particular artifact to determine what exactly needs to go in and go out.

Guiding principles

Uses an iterative approach:
· An iteration incorporates a set of activities from multiple disciplines (requirements, analysis and design, implementation, test, etc.) in various proportions depending on where in the development cycle the iteration is located.
· The schedule and scope of an iteration is fixed.
Risks are mitigated earlier, as most disciplines are engaged in the process from inception onward.

Follows a use case/scenario driven requirements approach:
· Requirements are organized in a way that tells a story of how someone may use the system, and how the system reacts to user behavior. Related scenarios are grouped into a specific use case or epic/theme.
· Nonfunctional requirements are captured in the context of one or more scenario driven requirements.
· Systems are developed by realizing requirements as part of the analysis and design discipline.
It is easier to develop requirements that are complete and consistent using stories.
A scenario driven approach provides a better understanding of a requirement from a user's perspective.

Is architecture centric:
· Design activities are centered around the notion of system architecture.
· The focus of early iterations of the process is to address architectural uncertainty
· Subsequent iterations may than reuse this architecture and focus more on completing business functionality
Technical risk is managed early on in the process
Greater control of system complexity and integrity

Is based on modeling visually:
· Visual representations of the system allow understanding at a higher level of abstraction than browsing through source code and other implementation artifacts
· Design models should focus on 3 fundamental aspects of the system
· Architecturally significant components, frameworks, and patterns
· A business domain model that clearly expresses the system in terms of business concepts understandable by business domain experts and system developers alike
· Common models/interfaces with other systems/programs that must be integrated
· While UML allows for the illustration of clear and unambiguous design decisions, it should not be the only modeling notation used (workflow, storyboards, ERD, etc.)
Modeling helps shorten the learning curve to understanding a complex system for both technical and business stakeholders

Is Agile, both in terms of development and documentation:
· All documents should have a specific audience, and the level of detail should be directed to that audience
· It is often not necessary to document all aspects of the system, focus on sections that are architecturally significant, difficult to understand, or have significant business logic
· Travel light when creating documentation. Every document that is created must be maintained over time. Some documents need only be temporary in nature
· All documentation should have a clear audience and purpose, and should assist in developing useful, high-quality software. Creating and maintaining documentation should not be a burden to the development team
· Use an evolutionary test driven approach to requirements and design
· Don't overcomplicate system design in an upfront manner in an effort to predict future requirements
· Use automated testing and build frameworks to enable the team to embrace change
· Architecture, requirements, and business concept should be prototyped with real code as early as possible, often in parallel with the documentation effort
· Continually validate with business stakeholders throughout the lifecycle of the project
Being Agile allows for greater emphasis on developing working software over comprehensive documentation and allows resources to “embrace change”, not resist it

Best Practices

Methodology Based Best Practices

Model and develop using an iterative, evolutionary approach
· Use more than one modeling artifact when determining requirements, design or architecture.
· Many modeling approaches, complement each other and may be conducted in parallel
o Use case, business domain, UI-prototyping
o Class modeling, database modeling, sequence modeling
· Architecture, requirement, and business concepts should be prototyped/validated with real code as early as possible, often in parallel with the modeling/documenting effort
· Within the same core team, design and implementation activities should be broken up into "miniature iterations" involving stakeholder review and clear testability

Use the simplest approach possible when modeling and developing the solution
· Use the simplest modeling tools necessary
o For smaller engagements involving co-located teams, whiteboards, sticky notes and other informal tools can be used
o As projects increase in scope, introduce case tools as necessary
· Avoid over designing solutions for requirements that have not yet been defined, as this can unnecessarily complicate the solution
· It is more important to have models that are correct than complete
o The purpose of models is to abstract knowledge and conveying meaning
o Extensive, overly detailed models typically obscure this meaning

Include testing activities within all phases of the lifecycle
· In order to support a culture that is able to embrace change, testing must be done early and often
· Modeling efforts should be validated with real code before committing to extensive documentation
· Automated testing harnesses should be used to validate all major system components
· When doing any work (documenting, planning, coding, etc.) always ask the question "how can I test/validate this?

Focus documentation efforts on creating essential artifacts necessary to develop and support the solution
· It is usually unnecessary to document all aspects of the system, focus on architectural significant pieces and business domain logic
· Some documentation and models, are used for understanding purposes only, and need only be temporary in nature
· Focus on creating detailed documentation (if necessary) after prototyping and other validation activities
· Clearly documented interface contracts are required between teams and systems
· Document with a purpose
o Focusing effort on what is required from the readers point of view
o Involve the "user" of the document to determine essential content

Take advantage of agile approaches , but structure them within more prescriptive processes as necessary
· Organize manageable units of work (app. 2 to 3 weeks) around moderately sized, co-located teams using XP, SCRUM, AM or other agile processes
· Document clearly defined interfaces between teams and major systems
· Organize these units of work using more prescriptive processes such as rup or sip

Organizational Best Practices

System stakeholders must be active participants in the development process
· Business representatives should own the requirements and be empowered to make decisions
· Organizational structure should support fluid lines of communication
· Requirements should be allowed to grow and evolve
Emphasize collaboration and communication
· Have a clear strategy that ensures that all documentation, modeling, and other system artifacts are clearly accessible to all project members and stakeholders
· Encourage collective ownership and development of system artifacts
· Ensure that the right tools are in place to support extensive collaboration and communication when using offshore/global teams

Favor staffing generalists, as opposed to specialists on projects
· Modern software development require a large number of skill sets to be successful (requirement gathering, interface development, object-oriented analysis and design, development, database, etc.)
· Employing "single artifact resources" will cause project resource counts to become unnecessarily large
· Specialists tend to be too narrowly focused on their own problem domain, and have difficulty seeing the larger picture (if the only tool you have is a hammer, all problems look like nails)
· Large amount of "document handoff required" resulting in information loss
· Hire/train resources to have 1 or 2 specialties, but have good general knowledge of the overall SDLC lifecycle
· Generalists require more experience than specialists and are not always readily available.
· However, working to maximize the number of generalists available for projects is essential to increasing delivery capability

Work to eliminate institutional barriers to change
· There are built in barriers to change specifically around infrastructure and databases in many IT organizations
· Processes and approaches can often be streamlined to be more responsive
Implement a governance model to support an enterprise system and process view
· Agile and iterative approaches are successful with small teams and 1 or 2 business experts
· Enterprise efforts require an enterprise view and a governance model to support a cohesive architecture and consistent approach

When taking advantage of offshore/global delivery capability use an agile approach to prepare key artifacts prior to hand off
· Functional prototypes, requirements, information architecture, and other artifacts can be prepared and validated using agile approaches
· Once validated, formally document these artifacts and transfer them to offshore/global resources

Technology-Based Best Practices
Focus development effort on mitigating areas of highest risk
· Elaboration and early construction iteration should be focused on addressing
o New/untested architecture
o Complex business logic
o Integration with external systems
· Risky areas of the system should be comprehensively covered by automated unit tests to ensure an early warning of possible faults
· Integration points
· Reusable assets
· Architectural risk
· Business process support

Make the right tools and resources available to support rapid application development
· Modeling/case tools are required for moderately sized and larger efforts
· Tools should support a superset of the uml notation (UML alone is not sufficient)
· Automated testing and build tools are the cornerstone to successfully implementing solutions using an iterative/ highly responsive approach
· Resources must have access to adequate development, staging, and testing environments

Develop using an appropriate platform supported by a robust technical architecture
· A solid enterprise architecture is a foundation that supports the ability to rapidly implement functional requirements
· Iterative and agile techniques along nicely to J2EE/.NET technologies
· Most web scripting and legacy technologies, languages lack appropriate tool support, and often are implemented using a more serial/waterfall approach

Implement a best in class Configuration Management Suite
· A comprehensive Configuration Management process is one of the most fundamental requirements of an iterative process:
o Automated unit testing support
o Automated build support
o Continuous integration several times a day
o Flexible versioning, release support
o Mock or stub object generation

Collaboration Responsibility Cards-domain modeling made easy

I recently just started work on what I consider to be a fairly large project with a large number of complex requirements. A massive amount of requirements had already been collected, (over a thousand pages worth). Unfortunately, a large majority of the requirements were not scenario driven, but of the "the system should make sure that..." variety. Extremely hard to follow especially giving this scale.
I was tasked with applying some structure and some modeling driven techniques to help frame the requirements and set the stage for downstream delivery. Needless to say, the environment for this project could be definitely considered less than ideal from an agile point of view. Project management did not want any development to start until requirements were better structured, and we had a better understanding of how different requirements mapped to different technology components.
One of the recommendations I quickly made was to develop a business domain model using package, class, and sequence diagrams. I'd read an excellent book by Eric Evans called Domain Driven Design, which nicely combined business centric modeling, UML, and a pragmatic, agile approach. I was hoping to use the functional resources that were responsible for putting together the business requirements in a business domain expert/reviewer capacity, as well as use a team of more intermediate resources to do the bulk of the modeling effort, with myself providing direction and guidance . We were constrained with using Rational Architect as our modeling tool, as that is what the client owned .
After about a month of modeling efforts, I had to admit I was not entirely pleased with our progress. Given a couple of days to reflect, I believe that the source of much of our inertia was due to the following
1) white board-based models are too hard to change once significant amount of effort has been put into creating the model. Whiteboards, while inclusive, need to be erased eventually, and require way too much redrawing.
2) most UML modeling tools might be a little bit easier to change, but are not collaborative in nature. There is way too much overhead in either printing off the models so that a large number of resources can collaboratively review and modify, or setting up a projector. Any changes require further work with the tool, and the turnaround time between making changes and putting them in the tool take way too long.
3) if business domain experts are not involved in directly creating the model, they do not focus enough on the model to see if they are correct. Our functional resources had several chances to take a look at many of our models before showing them to the client, but it was only until we did a live session with the client where the functional resources were actually asked to help the modeling team defend some of the modeling decisions did it become apparent how wrong our models were.
4) UML and the associated tools, just offer way too much detail. This detail actually can get in the way of initial modeling efforts. Despite my best intentions, I frequently found myself concerned with issues that were just not relevant at this point in the process. I can personally reflect on many situations where I spent too much time concerned with directionality of relationships, the use of inheritance over other more flexible typing mechanisms, and other issues.
CRC cards to the rescue
Now I admit to having heard of CRC cards before, but I've never paid much attention. I've always been happy to use TogetherJ or some other modeling tool for the majority of my diagrams, reviewing my work with other resources, or handing them off as necessary. Whoever, I believe that the the incredible business complexity of this project, the relatively large number of resources on the team, and scope of the proposed application dictates a different approach. I was doing some Internet research on various modeling tools and techniques and came upon CRC cards by accident. It didn't take very much reading for me to realize this approach had fascinating potential. I'm not going to go into too much detail around what CRC modeling is, however in brief CRC modeling involves placing responsibilities (methods and properties for classes) onto index cards (one index card for each class) as well as placing the collaborations necessary (with other classes) to fill these responsibilities. Index cards that represent classes with the lot of collaboration are placed close to each other, index cards that represent classes that implement or extend other classes/interfaces can be placed on top of each other. Potential models can be rapidly re-factored by rearranging these index cards. One can then trace through a set of classes and their responsibilities to see if a model will support specific requirements. One excellent resource concerning CRC modeling can be found here.
I quickly played with some CRC cards for about half a day, and was amazed at the speed I was able to model a large portion of requirements. I believe CRC cards will enable me to be effective on this project for the following reasons.
1) east of use-unlike UML, as well as most UML modeling tools, CRC cards are incredibly easy to learn, and do not require a lot of technical skill or object oriented knowledge to understand.
2) collaborative-CRC cards are incredibly collaborative, many resources (both technical and business oriented) can sit around a large table, and create CRC cards together. The result is a model that will have a very high alignment with true business concept situations and processes.
3) low fidelity-CRC cards natively support a lower precision fidelity then UML modeling notation. This lack of detail is actually a benefit when trying to focus on the nature of responsibility and collaboration, parameter ordering, datatypes, and other details found in UML just get in the way of this kind of modeling.
4) extremely easy to change-models made from CRC cards are very easy to reorganize, modify, and extend. Cards can be very easily ripped up, replaced, and changed.
Once I get back to work from the Christmas holidays I plan to introduce this technique to the modeling team, I would like to use CRC cards until the majority of the model has been completed, then translate the results to a more official modeling tool. I will certainly post the results here.

How to Make Agile Development Successful within a Fixed-Price Consulting Model

The purpose of this article is to discuss my experience in leveraging agile development approaches while delivering engagements working for a consulting firm. It should be noted that most of these engagements use fixed price contracts where scope, budget and schedule are defined well advanced of any detailed work. It is my opinion, that agile approaches and tools can still offer a lot of value on these kinds of projects.

What does it mean to develop using agile approaches?
While I assume that most readers are at least somewhat familiar with what it means to be agile I thought I would give a quick recap. The agile alliance defines the following manifesto:
Individuals and interactions over processes and tools

Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

The agile manifesto states that the items on the left of the statement are more important than the items on the right. However the manifesto clearly states that items on the right (including contract negotiation) still have value and are often a required part in any development project.
The agile manifesto further defines a couple of key principles that one should follow in order to truly claim that one is developing using agile methods. I've listed some of the most important ones below

1) early and continuous software delivery, with a preference towards small biweekly iterations.
2) acceptance and even welcoming of continually changing requirements
3) tight integration between the business and developers
4) giving motivated individuals the environment and support they need to be successful
5) a steady pace of software delivery that can be sustained indefinitely
6) face-to-face interaction is the best form of communication

While many of these tenants have become accepted by the larger developer community it's interesting to note that agile is often considered a "dirty" word among professional project managers and in many classical consulting organization in general.

fixed-price contracts: often a necessary evil when engaging with consultants
In order for any client to engage the development services of a consulting firm a contract must be created outlining the scope of the work. Often this is a fixed-price contract. Consultants typically work with client project managers to create a contract for an upcoming project, estimate the work, and determine the cost of the delivery in advance of any serious requirements or design work. Reasonably detailed contracts are required upfront to specify the nature of any engagement and the overall scope of any delivery. Needless to say, this environment can be perceived as not being conducive to an "agile approach" and as a result there's a fair amount of controversy around agile approaches at the firm that I work for.

Does an agile approach offer any benefit when following a contract model?
It is the opinion of many of the manager type resources within our firm that agile and fixed-price contracts are an absolute no-no. Typically our firm expresses a clear preference for structured, sequential, delivery methods. According to many in management, a highly structured approach supported by a robust and well defined change management process is the best way to manage any contractual risks to the engagement. Furthermore, management often equates agile development with "cowboy coding", and doesn't necessarily have the correct perception concerning the discipline required to make agile work. The problem with an overly structured approach, is typically things do change, scope does creep, and often an extensive, bureaucratic change management process only further complicates the situation. Far too often resources on the ground are asked to take up the slack. Often, the cost of change is not clearly perceived, quality begins to suffer, and the project is eventually impacted in some form or other.

My own experiences have led me to believe that an agile approach is crucial to the success of any development project that contains any technical or business ambiguity. I believe this represents the majority of most development projects, especially development projects that require outside consultant help. Typically, if the project was simple and risk-free, then clients would not be hiring consultants to manage the engagement in the first place.

The key to making agile work while also ensuring that the conditions of a particular contact between the consulting firm and the client are met is to make sure that agile approaches are supplemented with a suitable amount of structure that takes into account upfront pre-iterative activities such as initial planning, scoping, analysis and architecture.

What is interesting to note, is that agile and structured approaches do not need to be mutually exclusive. By being a little bit more liberal when interpreting explicit activities of some of the more structured approaches one is able to adapt and extend these approaches and execute them using an iterative and agile approach.

Listed below are set of best practices that have supported me well when applying agile approaches while still having to take into consideration factors such as budgeting, planning, and managing to explicit contract.

Attempt to structure contracts so that statements of work support an iterative approach
It is much easier to support an iterative approach when working against contracts that are smaller in duration. My experience has led me to believe that the ideal contract length is anywhere from two to four months. If contracts are any smaller you'll find that you're spending almost all of your time doing nothing but building contracts. You will also have very little room to make up for underestimated work within a particular contract. If contracts are any larger, you begin to open yourself to greater and greater risk of underestimating a particular piece of work.

Often master contracts are written for large engagements that can span several years. In this situation I found it helpful to negotiate with the client the ability to write separate requisitions of work against the master contract. These requisitions typically contain agreements for specific deliverables within a particular set of iterations. There will always be a situation where consulting firm will be forced to deliver work against a large multiyear contract. In these situations agile and iterative tools can still work to provide a internal baseline of progress and warning signals that the overall program might be in jeopardy well in advance of more typical waterfall like approaches.

Separate statements of work in sequence of perceived risk
While this might seem like an obvious statement, all too often I've been on contracts where statement of work have been written and executed in a typical waterfall like fashion as suggested by the order below.
1) requirements
2) design
3) implementation
4) testing

This approach is a completely naïve and does not take into account what risks should be mitigated in what order on a development project. While the specific risks on any particular project will vary depending on the situation, a more realistic structure for scheduling different statements of work is shown below.
1) Project Objectives - scope, business case, high-level cost, plan and schedule, conceptual architecture, POC
2) Project Definition - release planning, architecture, greater requirement stability, high risk prototyping and reference development
3) Project Construction-development in order of risk, code testing, detailed project documentation
4) Project transition-knowledge transfer, stakeholder product acceptance


I have frequently found it practical to include some degree of implementation work within each of these contracts. As an example, it is very difficult to know if the high-level conceptual architecture of the solution is feasible unless some portion of the architecture has already been implemented and proven. Frequently, it will be necessary to validate any conceptual architecture with a very simple strawman POC that ensures that all pieces of the solution are actually able to communicate with each other as intended.

Keep documentation (especially anything that could be considered "binding") at a high -level until it can be properly validated against an implementation reference
What this means is that requirements, domain models, architecture models, and any other project documentation that could be looked at as binding should be kept high-level and directional only until it can be verified with some kind of reference implementation. My experience has suggested that implementation has a massive effect on detailed design, almost a 100% delta on the documentation of the original solution. Where possible, I recommend keeping documentation at a high-level until a least some portion of it can be validated against a reference code base. Of course, this is not always possible and sometimes clients want documentation sooner, just be sure to communicate that all initial documentation needs to be validated with real implementation work, also make sure to be prepared to schedule the time to write the documentation at least twice.

Validate high-risk portions of the development effort prior to committing to engaging with the larger team
The majority of engagements that I involve myself in all contain some degree of technical risk. Examples of this type of risk are incorrect solution behavior, poor performance, poor reliability, or inaccurate estimation. Much of this risk can be effectively mitigated by initially deploying a small team of fairly senior resources in advance of the larger team. The sole purpose of this team is to address these technical risks. A senior team can use a combination of analysis and prototyping to resolve much of the technical and business related ambiguity prior to full-blown implementation. Once the mismatch between initial estimation, risk and actual solution is understood this information can be used to create a more finely tuned plan and accurate contract for later phases of engagement.

Use a pure agile development process to structure how work is conducted within each development team, but organize these teams and their efforts using more prescriptive milestone-based processes.
Agile development approaches are great for managing development teams on a micro/medium scale. They provide tools for estimating the real-time "velocity" of a development team, great techniques and tools for team collaboration, and place the proper emphasis on an automated test driven development method. What agile approaches don't typically do is describe how to use documentation, modeling, and planning to properly scope and coordinate multiple teams with each other. In my experience the following approaches supplement agile nicely to support larger engagements, especially with a bias towards a contracting model.
1) Explicit upfront documentation of the interfaces between teams and major systems
2) prototyping, testing, and documenting architecturally significant component in detail before a larger team begins development work using a untested platform or framework
3) documenting and testing nonfunctional requirements before architecture is considered to be properly validated
4) reasonably detailed functional requirements where continual business stakeholder involvement is not guaranteed
5) executing a explicit testing phase after the iterative development cycle

Long-term planning is still essential, even when developing using an agile approach, however planning detail should vary depending on the duration of the plan being used
In point of fact, agile development requires an awful lot of planning in order to be successful. Agile planning emphasizes the use of planning at different levels of details depending on how far out the plan goes. One approach, recommended by the excellent text "Agile Estimating and Planning" by Michael Cohn; is to create a set of product, release and iteration plans for a particular project, each at a different level of detail.

The product plan may span multiple years. The product plan describes the likely release schedule, as well as the contents and objectives of each release at a high level. Each release has a corresponding release plan. Typically, the release plan is only developed one or two months prior to the release actually been executed. The release plan defines how the release will be broken up into one or more iterations. Iteration deliverables (often called user stories, although I simply prefer the deliverables or features) are described at a high level, estimated, and allocated to a specific iteration. Iterations should be structured so that a small subset of system functionality is theoretically made available for the client to review. Each iteration is typically anywhere from two weeks to two months in duration, with a preference for the shorter side. Finally, each iteration is structured to support a more detailed activity-based iteration plan, however this planning is typically done at the beginning of the iteration or at most one iteration ahead of time.

Be flexible when deciding on how to adopt agile approaches
I found that one of the biggest inhibitors to adopting agile approaches is that agile evangelist can be somewhat overzealous/religious about agile in general. One of the interesting tenet of XP (a popular agile approach) is that "one must be extreme in applying all of XP practices". In my opinion, this is pure nonsense. There are many types of project environments that can benefit from many of the practices and approaches prescribed by the agile community without having to blindly follow every single principle to the letter. The question should not be "are you doing agile or not". Rather, the important question is "how can agile principles and practices benefit your current work environment?"

As an example, many agile approaches suggest that requirements be developed using "user stories", (an informal functional description of user and system behavior) and that all implementation work be based on developing code by realizing these user stories into code and allowing the common components, frameworks, and other architecture pieces to naturally evolve. This can be a very powerful approach, and I have always found that evolutionary forces have had a very beneficial impact on my architecture and design. However, larger scale systems developed in parallel by multiple teams typically require that some upfront architecture and design work be conducted. There is a middle ground where the right amount of modeling and/or documentation can be used to help direct and guide the development of one or more agile teams.

Furthermore, on a project that I was recently on, the team was directed to do nothing but develop a platform and framework for an SOA solution. This meant we had actually no capability to actually create "user stories" per se. Does this mean that we could not follow an agile approach? Of course not. Simply by being flexible we decided to designate the client architects as our "end users", and wrote user stories describing the features that are architects wanted the platform to do. We called our user stories "platform deliverables", assigned story points (calling them feature points), and developed and validated them with the client architects using an iterative approach. Examples of our platform deliverables were things like "Handle Service Request" or "Cache Service Request".

The point here is that one can adapt agile approaches to a surprising number of situations, one simply needs to be flexible, creative, and willing to treat the variety of approaches and methodologies out there, both non-agile and agile as tools, rather than as a prescriptive set of processes that should be followed "just because".