Agile over RUP, My Preferred Development Approach

In this post I thought I would describe my delivery approach of choice, in a brief, and (hopefully) understandable format.

As anybody who knows me can attest, I'm a big fan of agile development. That being said, I don't believe that agile alone is typically sufficient for me to successfully run all software delivery engagements. Although I always tweak the delivery approach I take to suit the conditions of a particular project, I frequently have relied on a combination of agile delivery combined with elements of the rational unified process (RUP). In a nutshell, the delivery approach I typically recommend for myself and my clients can be described as agile over RUP.


What does agile mean to me?
I have seen number different descriptions of what agile software delivery is and what it means to different people. The one that resonates the most with me describes agile development as:

1) a set of software development principles that emphasize:

• individuals over process and tools
• working software over documentation
• collaboration over contracts
• responding to change over following the plan
  

2) a set of complementary best practices that support these principles and are

• collaborative
• lightweight
• simple

Note the "collection of complementary best practices" part of this definition, this is important (IMHO). In my experience agile is not something that is "done" or "not done". I've never understood the comment "we are doing agile delivery" or alternatively that team "isn't doing agile". Rather agile is something that can be applied to any development engagement on a graduated, practice by practice basis. When taken in isolation an individual agile best practice can still add value to a software development project. However, when an agile best practice is combined with and complemented with other agile best practices, there is a potential to provide an exponential amounts of value to the project.

These agile best practices come from a number of different but complementary agile "methodologies". Examples of these methodologies include SCRUM, XP, Crystal-Clear and the Agile Modeling Method. Increasingly, there is growing industry evidence that many organizations are selecting these best practices à la cart, rather than choosing a specific methodology and following it in its pure form. This is an approach that I follow, and it's one that I recommend...

Here is a brief list of the agile best practices that have provided value to delivery engagements I have been a part of in the past...

Test Driven Development

Test driven development is a development technique where test cases are written first, then the code necessary to pass the tests is implemented, and finally the software is refactored to accommodate changes. The availability of tests before actual development ensures rapid feedback after any change. "Refactoring" source code means modifying it without changing its behavior, and is sometimes informally referred to as "cleaning it up". Refactoring improves the understandability of the code and changes its internal structure and design, and removes dead code, to make it easier to comprehend, more maintainable and amenable to change. Many of the other tenets of agile development (like evolutionary design, responding to change, etc.) start to fall apart if application code is not supported by a robust and well thought out automated testing framework and periodic refactoring cycles.

Iterative Development

Another core best practice essential to effective agile delivery is the delivery of working software using a just-in-time assembly model where potentially shippable software is created at the end of each iteration. The first step in iterative development is to create a upfront but simple plan for the implementation effort as a whole, estimating and scheduling work into iterations of equal length ranging from one week to one month. At the beginning of each iteration a more detailed version of the plan is developed, scheduled work is completed, working code is presented to the business, progress is measured, and the plan is adjusted as necessary. For iterative development to truly work, software must be thought of as being potentially shippable at the end of each iteration. Work must also be focused on delivering real business value to the business, rather than developer "gold plating". Finally, estimates and planning needs to be done by the resources were actually going to do the work.

Daily Standup Meetings
Daily Standup Meetings is a core agile practice that requires a very little effort to run successfully. Communication among the entire team is the purpose of the daily standup meeting. A stand up meeting every morning is used to communicate problems, solutions, and promote team focus. Everyone stands up in a circle to avoid long discussions. It is more efficient to have one short meeting that every one is required to attend than many meetings with a few developers each. Organization and involvement of all levels of team members is encouraged by requiring each member to specify what they plan to accomplish the day before, what they actually accomplish, what they plan to accomplish today, and what, if anything is preventing them from being successful. On large-scale projects, hold "super daily standups" once a day to coordinate work across different teams. This super daily standup should be attended by one representative from each separate team.

Continuous Integration

Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including tests) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.

Common Coding Standards And Self Describing Code

Coding standards tells developers how they must write their code. Instead of each developer coding in their own preferred style, they will write all code to a common standards. This makes sure that a large project is coded in a consistent style -- parts are not written differently by different programmers. Not only does this solution make the code easier to understand, it also ensures that any developer who looks at the code will know what to expect throughout the entire application. A typical supplementary best practice is to employ Self Describing Code, a set of coding conventions with the intent of making code a self-documenting artifact. Specific implementation patterns can be followed to ensure that code is developed with readability as a primary consideration, in effect reducing the need for code comments, supporting documentation, and other artifacts which may not always be kept up to date with production code.

Collective Code Ownership

Encourages every developer to contribute new ideas to all segments of the project. Any developer can change any line of code to add functionality, fix bugs, or refactor. Collective code ownership helps to mitigate the eventual transfer of resources from a particular project. This practice has been cited as being much more effective than conventional approaches such as code documentation, code walk-throughs and other knowledge transfer mechanisms.

Sustainable Pace

This best practice simply states that developers are much more effective at delivering high-quality code when they are put under the "right" amount of stress to perform. I.e. they are well rested, enthusiastic, and able to look at a problem with a fresh eye. This needs to be balanced with reasonably short term objectives, and a consistent and continual need to produce real results within a relatively short time. (I.e. a specific iteration)

Collaborative Modeling

Collaborative Modeling involves the use of simple and accessible tools to develop low fidelity, often throw away models of software that is to be built. The use of simple tools like index cards, whiteboards, and sticky notes allows for non-technical resources (i.e. the business) to actively partake in modeling HUP solution. Collaborative modeling calls for business users to actively engage and own any models that are developed to help guide the development.

An excellent example of collaborative modeling (one espoused by XP) are CRC cards (Class Responsibility Collaborator). CRC cards allow developers to create a low fidelity, object-oriented model. CRC models are created by laying out a set of index cards on a table or particle board, and labeling one or more of these cards with a major concept (a class) that needs to be represented in the software. Examples can include long-lived information such as a Student, an Account, or an Address. More temporary concepts can also be represented such as a ShoppingCart, or a RateCalculation. Using CRC modeling techniques facilitate a collaborative modeling atmosphere, one in which both business and technical resources can engage in.

Co-Located Teams

it is hard to overemphasize the amount of waste that is created by placing members of the same team in different locations. This should seem like a no-brainer, but the reality is many organizations really struggle with getting their team members into the same room for the duration of a project. Co-locating teams drastically reduces the need for official meetings, just go to the room where the work is being done, and resolve the issue. There is also a huge amount of information being simply from overhearing the conversation of others. On large-scale projects, organize teams to minimize the dependencies between teams and then make sure each team is in one room.

Retrospectives

While a set of best practices may look great on paper, the reality is that any best practice (even the ones described above) will require considerable refinement and adaption to the actual environment where the project is taking place. Retrospectives are short meetings held by the entire team at the end of every iteration. Retrospectives allow the team to add any best practices or processes that can help with the delivery of software, and remove one's that are getting in the way. Lessons learned are incorporated into the approach, and allow the team to continually improve over the duration of the project. Retrospectives are another essential agile best practice (IMHO) but one that is easy to implement.

Pair Programming

Pair Programming is a development technique involving the pairing of two developers to one computer, one mouse when keyboard. One developer enters in code (the driver) while the other developer (the observer) reviews, provide strategic direction, and refactoring suggestions. The two developers should switch roles for equally, a standard best practice is every 30 minutes. While seemingly simple, pair programming is (IMHO) probably one of the most difficult agile best practices to pull off, developers by their very nature like to work on their own computer on their own tasks. It is also counterintuitive to believe that their programming would not result in a drastic loss of productivity. There is actual empirical evidence that this is the case, with some studies showing overall effort reduced by 15% and code quality being increased by 50%. Pair programming also makes it incredibly easy to onboard and offboard resources, increases developer discipline (someone's always watching over somebody else) and lead to more effective problem-solving.

Active Business Stakeholder

Every form of agile methodology has some notion of this practice. XP calls the on-site customer, SCRUM calls it a product owner. In all instances, some form of user, or user proxy, plays an active part of the development team, providing advice, direction, and knowledge of what the product should do. The active business stakeholder is responsible for developing all requirements for the system, and has the final say on priority. The active business stakeholder is empowered to make decisions on the fly, and serves as a buffer between the development team and other business stakeholders.

Given the list above, the set of agile best practices could also be represented as:

Agile itself could be represented as a set of agile principles and agile best practices...

stay tuned for part 2 of this presentation where I described how RUP fit into the picture...

Co-Presentation with Scott Ambler at Deloitte

It's been a while since I've had the opportunity to blog, but last week I had a opportunity to present on real-world agile development on large-scale projects with another firm employee, Sunny Ip, as well as the notable agile luminary Scott ambler . Our firm invited Scott to speak to some of our practitioners on why agile matters, and how it can be applied to environments that many critics state agile can not possibly thrive or succeed in.

As I have stated in previous posts, the firm that I work for it as much a management consulting firm as it is a applications development firm. Needless to say, many agile best practices and principles do not form the core tools that a lot of our practitioners use. While this is slowly changing, thanks to a core team is passionate and experience in using agile, having Scott present on the values of these approaches is essential to spreading this message within our firm.

I thought Scott did an amazing job in giving a no-nonsense and extremely frank presentation on why our firm and others like us need to start taking this seriously.

Scott managed to pack a lot of material in a relatively short period of time, topics included:

• agile overview


• surveys on the success of agile and traditional projects-a lot of great no-nonsense material was presented


• shortcomings of pure agile approaches (XP, scrum, etc.)


• extending agile with the rational unified process (RUP)


• applying agile in outsourcing staff situations


• applying agile to the database group and database platforms

I'm sure I did not list every topic covered here, but these were the major ones that resonated with me.

After Scott's presentation sunny and myself presented 4 separate case studies where our firm has taken advantage of agile best practices on large-scale delivery projects. Examples included:

• large-scale public sector transformation project with 20 plus developers in parallel


• assisting the enterprise architecture office in developing a reusable SOA platform


• using agile modeling techniques to properly analyze and design a domain driven solution for a provincial students assistance and loans application


• using agile delivery on a large-scale package implementation also dedicated to any students assistance and loan application

During our portion of the presentation, the audience began to ask questions which were mostly answered by Scott, with additional color/perspective from our firm added in by us.

For my part, my major take away from listening to Scott was that everyone in our firm needs to be much more forthright in pressing our clients on the need for change, and applying innovative/different delivery practices to their organizations. I think anybody who's done any solid delivery work for any length of time is more than aware of how dysfunctional many of our development processes and approaches are in a majority of organizations. But changing the situation requires courage, and the bravery to speak out to CXO level leadership. We really need to provide a frank assessment of the things we see on our day-to-day implementation projects. There's a huge amount of reward that can be gained from adopting agile in a pragmatic fashion. Costs can be lowered while simultaneously increasing quality and reducing risk. These things that business can buy into.

Again my thanks to Scott for a great presentation.

Discussing government 2.0 with policy analysts

I just came back from presenting on the topic of Web 2.0 and government with a group of transportation policy analysts, of course the exact province shall remain nameless to protect the innocent as usual :-).
Paul Macmillan and myself were expecting a half an hour meeting with 4 or 5 folk, it ended up being a 1 1/2 hour meeting with around 40 people. I think my ability to improvise in getting a lot better, and of course Paul is a master.
This is about the sixth presentation that I have given within the last year on Web 2.0 and government and there are two things that I consistently come across.
1) excitement: I can say that with out almost any exception, members of government are incredibly eager to trying new Web 2.0 approaches to more effectively collaborate with citizens, concerned organizations, and each other. There is a palpable frustration around the churn involved using today's methods of providing information and getting feedback on matters of policy.
2) frustration: right now existing procedures and policies are making it very difficult for various members of government to engage using online forums, with user blogs. Facebook is banned for provincial employees within Ontario. So are many other social networking communities. Any time a group of government employees try to set up their own externally facing network they get hit with various roadblocks like the official languages act.
The issue here is that exchange is made on these types of forms and blogs, and message boards are much more like conversations than publicly broadcasted material. In fact the whole differentiation between a private conversation and a public conversation become a a lot murkier when using these kinds of tools. Clearly laws need to be updated, and policies need to be rewritten. It's a bit like the wild West out there currently. I find it difficult to advise on these issues except to say that being a risk taker by nature, I would recommend starting a social networking site anyway even if it violates a couple of policies. In this case it seems clearly easier to beg for forgiveness and ask for permission...

Government 2.0 And Municipal Government

Paul Macmillan and myself just had a great meeting with a municipal CIO, whom in order to protect the innocent will remain anonymous for the purpose of this post. :-)

We had a chance to briefly discuss Deloitte's view on the value of government 2.0, as well as some considerations around government 2.0 adoption. Our CIO had some great comments around some of the challenges in embracing of a Web 2.0 philosophy within the context of a municipal government. I thought that I would share these challenges with interested readers out there.

• Traditionally, municipal governments have a lot less IT budget than there provincial counterparts. When compared to provincial or federal, most municipal governments actually have an order of magnitude less IT staff per IT user. This is especially true for larger municipalities.
• This has required most municipal IT organizations to place greater emphasis on consistency, stability, and operations. This approach allows for better operational efficiency, and allows the municipal IT organization to stretch their dollar.
• Unfortunately, this also means a historical underinvestment in strategy, architecture, and it makes it much more difficult to innovate. There is very little money to support a heterogeneous environment.
• There are serious concerns around simple basic infrastructure which makes the idea of accessible Web 2.0 collaborative service challenging. As an example, many recreational centers in major cities do not have any Internet access whatsoever.
• Another concern has been raised around the need for cities that are multicultural and multilingual to support communities, collaborative environments, and other services in a fashion that is consumable by all of these varied citizens. Accessibility for all income levels and cultures is a priority.
• Finally, because of the emphasis on operations within the municipal government there is a fear that "opening up the floodgates" using Web 2.0 technologies to collaboration with citizens could make it very difficult for council members and other government employees to effectively sort through all the comments and feedback that could be provided by concerned citizens.

These are all valid concerns, and while I do have answers to almost all of them, I thought I would leave that for another post. If anybody out there has their own answers, or have some more challenges to add to a docking Web 2.0 tools and culture within municipal government and would welcome any feedback.

Changing the World, One Step at a Time

As some readers of this blog may be aware, I've had the opportunity to present on a number of occasions on the topic of government and Web 2.0. It's pretty obvious from where I'm standing that there is a lot of interest in pursuing Web 2.0 to enable the kind of cross organizational and government assistance collaboration required to make government really successful.


Paul MacMillan and I have tended to structure the presentation in two parts, where Paul gives a brief introduction of Web 2.0 and then proceeds to provide some great examples of where Web 2.0 is currently being used by governments around the world. I then tend to follow up with a discussion on what are some of the first things government organization should do in order to adopt not only Web 2.0 technology but a Web 2.0 "culture". I was genuinely surprised by the incredibly positive feedback and enthusiasm expressed by the audience concerning Web 2.0 adoption. Clearly the issue of how to get started is on the top of a lot of people's minds within government. The Canadian federal government is also starting to make some "Web 2.0" purchases, which while encouraging, might indicate an attempt to follow a traditional top-down mindset to a topic that clearly requires more collaborative, grassroots driven thinking to take close off.

Because of the perceived interest, we have started working on a
sequel to our original government 2.0 paper, Change the World or the World Will Change You that will be focused on advice, best practices, and
patterns that could make governments more successful when adopting Web 2.0. I plan to leverage a lot of the excellent patterns posted on wikipatterns , as well as continue to contribute patterns on an ongoing basis. Rather than developing the paper exclusively within our organization, I'm hoping to be able to share my thoughts and comments with the public as they evolve, and try to get feedback from anybody else out there who is equally passionate about the topic. So please feel free to suggest, flame, argue or otherwise criticize/contribute to any of the posts relating to Government 2.0 adoption.

To start, I've posted a overview of what I feel to be a successful government web 2.0 that emphasizes adoption in three stages. The first stage involves numerous incubation pilots, where the focus is on creativity, small targeted objectives, and a lot of learning by doing. In my opinion, members of government need to start getting more comfortable with experimentation and prototyping using a small team close rather than massive upfront design on massive scale. The second stage could still be thought of as prototyping, but now the effort focuses on how to better integrate Web 2.0 technology and culture within a government organization. Basically the focus here is creating a set of blueprints that can serve as the basis for larger adoption, again learning by doing and trial and error is way more important than philosophy and conjecture. The final stage, mainstream adoption is basically a self-fulfilling prophecy, providing leadership continues to focus on removing many of the roadblocks that will get in the way.

Feel free to post any initial comments here, or on the
overview itself.

Online Collaboration, Making the 1st Move

There is a clear and demonstrated interest in adopting Web 2.0 within the government, but what is the first step?

In the paper
Change Your World or the World Will Change You, Deloitte described the value that online collaboration could bring to governments and their citizens. The paper provided an overview of how Web 2.0 technologies can be leveraged to enable governments to engage in a highly collaborative, network based, communications model. The paper also described some of the top benefits of this fundamentally new culture of cooperation, referred to as Government 2.0. Some of the benefits listed were improving policy outcomes, streamlining internal operations, increasing effectiveness of government information, and attracting the next generation of tech savvy top talent.

In the course of presenting our paper, we have had a chance to have numerous conversations with various levels of government concerning Web 2.0 and how it applies to governments. It is clear that there is a lot of interest in adopting the technologies and culture represented by the Web 2.0 phenomenon within the public sector. A common thread that is often raised in these discussions revolve around how to get started. How does one apply a paradigm as revolutionary as Web 2.0 to an organization as large and complex as the government? A range of obstacles have been presented to us, ranging from issues of culture, restrictive web policies, to limitations of government IT.

Adopting an online collaborative approach supported by Web 2.0 will have an unprecedented impact on the way governments do their work, the scope of these changes threaten to stall any large-scale adoption

There is a significant amount of inertia to overcome when introducing any large-scale change within the government. Traditionally government work has taken place in a monolithic, highly structured, top-down environment. Adopting Web 2.0 technologies implies a significant change in working culture. Collaboration becomes much more important than control, workers are subjected to an unprecedented level of transparency, and the lines between different levels of government, supporting organizations, and citizens become much blurrier. IT will also need to rethink many of the ways that it has traditionally supported its stakeholders. IT will have the added responsibility of providing government and citizen end users with a constantly evolving "collaborative ecosystem". In order to properly fulfill this role, IT will have to adopt more lightweight, collaborative development approaches, becoming an enabler of change.

The scope of these changes can seem overwhelming and intimidating, to say the least. An enormous amount of analysis would be necessary before all of the possible issues and risks were covered to the extent necessary to satisfy all involved. Imposing a large upfront Web 2.0 analysis effort as a requirement prior to any government Web 2.0 adoption could effectively paralyze any meaningful progress within the space for many months and potentially even years. This would result in a significant loss of potential opportunity for the government, as it churns through the potential outcomes of any possible missteps relating to Web 2.0.

Successful Government 2.0 depends on demonstrating value immediately, starting with carefully chosen, tightly scoped 'incubation ' pilots -emphasizing innovation, trial and error, and learning by doing. Pilots will evolve over time to a more 'stabilized 'state, becoming more sustainable solutions-better integrating into existing infrastructure, considering usage policies and other standards, and becoming ingrained into the overall vision of government. Eventually solutions and platforms will be 'mainstreamed '-and considered an everyday part of government life

Complement Long-Term Strategy and Visioning with Incubation Pilots

Instead of dedicating initial Web 2.0 activity on a large, enterprise scale strategy, government organizations should also run a carefully selected set of Web 2.0 'incubation" pilots . Incubation pilots should be targeted at a set of internal and/or citizen end users that would most likely identify with the Web 2.0 approach. It is important to carefully limit the scope to a very specific set of objectives. This has the dual advantage of limiting the risks and exposure should things go wrong, while simultaneously allowing the piloting team the freedom to profit from the creativity, innovation, and trial and error learning necessary to determining the proper fit of Web 2.0 within a particular government environment.

Continue to Pilot, Stabilizing and Integrating with the Enterprise

As each incubation pilot comes to a close, experiences gained and lessons learned will be re-factored into the larger, enterprise-level Web 2.0 adoption strategy. As the strategy is refined, future initiatives will become more deeply integrated with existing government assets, examples include organizational standards, policies, and IT technology and procedures.

Once the value of Web 2.0 approaches have been clearly demonstrated through some of the initial "incubation" successes, the piloting effort shifts to realizing a fully integrated, controlled environment that is dedicated to developing and testing replicable Web 2.0 strategies. Pilots can now be used to determine how existing government assets need to evolve to support a Web 2.0 approach, including administration, policy, and other support structures that are essential to full government adoption.

Mainstream adoption is largely a self-fulfilling prophecy, providing the right components are in place

It will not be immediately obvious when a particular government or government organization has graduated from piloting to actual mainstreaming. One of the many appeals of the Web 2.0 approach is the viral pattern of adoption. The important point to take from this is that a Web 2.0 approach should be pursued in small increments, consistently revisiting the platform, policies, and any other components in an iterative fashion. As on the consumer Web, much of the work relating to mainstreaming the "Web 2.0" approach within the government will not be the result of any particular top-down decision.

That being said, leadership has a lot of power to make Web 2.0 a more effective force within the government. One of the most valuable things they can do first is give their members license to experiment, innovate, and try out these new technologies to better enhance collaboration. It is important that leadership foster a collaborative, transparent culture that rewards experimentation and places an equal value on lessons learned through failure as on a particular Web 2.0 success.

Further posts will further elaborates on how to execute this "trial and error" adoption approach.

The Highlights of SCA at JavaWorld 2008

Upon rereading my original post on SCA and Java world, I've realized that it's a little bit on the long winded side, so I thought I would summarize the key points in the following paragraphs.

What and why SCA?
· SCA is a set of specifications dedicated to developing a platform neutral service component model.
· SCA extends Web services to include multiple binding technologies (REST, RMI, JMS, and yes WSDL/SOAP) under a consistent model
· SCA makes SOA simple and possible by providing a clean POJO (or POCO, or POPO, etc.) development model, dirty nasty protocol details and service platform API calls are completely removed from service code,
· SCA does for service architecture what Spring did for the Java platform

Why is SCA better than JAX-WS, JAX-RS, etc.?
IMHO SCA is a better approach than comparable offerings from the JCP world (e.g. Jax-WS, Jax -RS, JBI, etc.). SCA is a true community process backed by both vendors, independence, and professional services organizations. We all heard complaints about Sun's dominance in the JavaEE world. The specifications have been submitted to Oasis for proper standardization, I have contributed to the process, although less than I like, but it truly is an open approach.
SCA offers a consistent programming model that is simpler, dependency injection-based, and consistent. We all know the realities of JavaEE, I know it's getting better, but in my opinion no comparison.

Tuscany is a great open source implementation of SCA, with real-world production implementations
Jean-Sebastien Delfino and Mario Antollini gave a incredible presentation on Tuscany my favorite open source implementation of SCA. The highlight of the presentation (IMHO) was when Jean-Sebastien showed had easily extend the SCA specification to include mashups/Web 2.0 component creation. My opinion this is one of the highest values of SCA, a truly comprehensive component model that spans technologies, from simple AJAX/ATOM components to more complex WSDL/SOAP style services. Brilliant.
This session really showed how easy it was to make services, or components of any nature using SCA

Sun has no intention of working with SCA, and there are some concerns about a fracture within the Java community
David Chapelle moderated a panel that included all the big SCA vendors. David made a big deal about the fact that SCA is causing a splinter in the Java community by offering another programming model. He also expressed big concerns around the fact that not everybody at the panel agreed on the level of support for the proposed programming model. I think David is being a little ridiculous, possibly intentionally so, he does come off as a really clever guy, and he is a "Microsoft" guy.

First of all, Java already has a multitude of programming models; Spring explicitly says that it has its own component model. Not to mention that working with components in a scripting language (like jRuby) is going to use a very different programming model than working with components using the core Java platform language. One of the core strengths of the Java platform is the diversity offers, Microsoft seems to like one approach, one platform, one model, sadly this approach suffers from getting rapidly date, and irrelevant to the people who actually need to use this stuff i.e. the developers.

Secondly, the SCA programming model is a POJO programming model, it's based on annotations, XML configurations, and dependency injection. So is uneven adoption somewhat of a concern? Sure because it's a great programming model, but the beauty of DI is that code can be easily plugged into different containers regardless of the model being supported, as long as it's a POJO one, (or POPO) with very little pain, so I think this big concern about disparate programming models is a much ado about nothing.

I did speak to Peter Walker from Sun after the panel was over, and certainly a good thing that Sun was actually at the panel this year, but according to Peter, Sun still sees no need to adopt SCA. As far as he is concerned, his "users" aren't asking for it. Ridiculous, there was a CapGemini employee as part of the panel, and I also know that Deloitte has huge interest in SCA. (I won't say why hint, hint) so the two major technology consulting firms don't count as users then who do? I think I smell a e-mail letter campaign coming...

JavaWorld 2008 part 2: Service Component Architecture

For the second part of my JavaOne 2008 coverage I'd like to talk about Service Component Architecture (SCA). I was very lucky in that I was able to go to three sessions talking about SCA pretty much in a row.

What is exactly SCA?
For those of you who are unaware of what SCA is, I'll give a brief introduction. SCA is a suite of specifications being considered on the the oasis mantle dedicated to defining a protocol and platform neutral service oriented component language. The primary purpose of this language is to allow developers or service assemblers to define a set of services, references, and bindings into a suite of composite structures.

SCA allows a developer to take code assets developed in a number of languages such as Java, Python, BPEL or C++ and wire them together with a minimum of protocol or SOA infrastructure knowledge. This wiring is typically done using the type of dependency injection techniques that many Java developers may have first come across when looking at the Spring framework.
In point of fact, when trying to describe SCA to people who have never heard of it before, I often compare frameworks using SCA to the Spring framework. But where Spring is focused on providing cross cutting concerns and dependency injection within a service platform, SCA focuses on how to use dependency injection to wire various components together regardless of the protocol of the wire, or the target platform/language of the service.

Services built using the SCA specifications have no inheritance or API dependencies like services often built using the set of JAX/Java EE approach. The approach is much more Spring like, with a focus on XML configuration and annotations where appropriate.
SCA is in my opinion, an extremely important specification, the "why" is hopefully self evident.

SCA: Flexible and Agile Composition of Distributed Service Oriented Architecture Applications

The first session I went to was titled SCA: Flexible and Agile Composition of Distributed Service Oriented Architecture Applications. The session was held by Mike Edwards, who's one of the main authors and contributors of the SCA specification, and is a SOA strategist for IBM.
The session was pretty good, basically a general overview of SCA, its value proposition, along with a couple of examples/usage scenarios. I had seen most of the information before, and much of it can be found at the OpenSOA website. The important point to note about this session was that it was full, also thousand people attended, which is a great indicator of the interest in SCA from members of the Java community.

Open-Source Service-Oriented Architecture with Service Component Architecture and Apache Tuscany

The following morning I went to the Open-Source Service-Oriented Architecture with Service Component Architecture and Apache Tuscany session. The session was presented by Mario Antollini, Jean-Sebastien Delfino and the topic was focusing on an open source implementation of the SCA specification.
I've had the opportunity to work with the Tuscany group developing an SOA solution so I have to admit that I am a little bit biased, but I thought that this session was fantastic. Mario gave an overview of SCA, and its importance as well as a general overview of Tuscany, talking about the number of downloads, and the size of the community. I'll admit to not be nailed to remember the exact numbers, but the growth looked impressive. Mario also mentioned that Tuscany was in production, and that the project had been implemented by Deloitte, which I thought was a great plug.

Jean-Sebastian then gave a great demo of Tuscany, using a fruit and vegetable store as an example. She sure how to do some examples using typical SCA functionality, i.e. how to create a component, expose some of its functionality as services and how to bind the component to other services using references and wires.
From a technical point of view, the demo is interesting because it showed two things. One, the service code was completely uncluttered with any service API or service protocol concerns. The only way one could differentiate the services in the examples from a regular POJO was the import statements necessary to bring in the @remottable and @reference annotations. I've yet to see any examples from the JAX/Java EE world that comes close to creating business services that are this clean.

Secondly, was the fact that the services were using two different protocols (JSON and ATOM believe) for each operation, not necessarily a realistic scenario, but as Jean-Sebastian put it, it was certainly "fun to do" and also more importantly it should have easily it was to switch binding protocols and have no impact on the service code.

Next, Jean-Sebastian showed how Tuscany was being used to prototype various bindings and implementation types that the formal always has SCA specification have not yet considered. One of the mission statement of Tuscany is to act as a live test bed for the specification work, providing very realistic feedback on what works, and what doesn't. Jean-Sebastian extended the demo to include "Web 2.0" style bindings. Showcased the
Jean-Sebastian demoed the use of a "implementation.widget" to create a JavaScript style component that would be hosted by the browser and was able to take advantage of the same annotations they were used in the Java examples. These JavaScript SCA Web 2.0 widgets were able to take advantage of asynchronous communications à la Ajax, while using the same dependency injected, nonintrusive approach showcased in the server-side components. I think the ramifications of this are incredible, a common component model that can describe how to wire components together, regardless of whether they are JavaScript/Ajax widgets, EJBs, REST services or full-blown WSDL/soap web services.

Open Standards for SOA and Java Technology

The final session they went to concerning SCA was titled Open Standards for SOA and Java Technology, and was basically a discussion panel hosted by David Chapelle, and attended by the following SCA experts.

• Mike Edwards of IBM
• Steve Jones of Capgemini UK
• Patrick Leonard of Rogue
• Mark Little of Red Hat
• Sanjay Patil of SAP AG
• Michael Rowley of BEA Systems, Inc.
• Scott Vorthmann of TIBCO Software Inc.
• Peter Walker of Sun Microsystems, Inc.
  

David started out with an extremely fast "SCA in less than five minutes" introduction. I must say that for the most part I found David to be quite entertaining, a great speaker, and he did a great job of introducing SCA.

The format of the question-and-answer portion was as follows. David asked a question to one of the members of the panel, going from left to right (top to bottom on this list) once the panel member had answered, other members of the panel were free to join in with their own opinions. Rather than try to repeat every question-and-answer code like to get to the heart of the controversy of the panel, namely that David seemed intent on representing SCA as a fracturing of the JavaEE community, and that it was mainly a competition to the JavaEE model. Although many different kinds of questions were asked of the panel, David kept circling back to this idea. David also seem to take a lot of pleasure in stating that SCA was an awful lot like the Microsoft method of composing components, known as WCF. I guess I could agree with the statement, if SCA was built by one vendor, had no grassroots community involvement, supported one platform, and one real language, (sorry Visual Basic is not a language)

One of the first questions asked by David was "is SCA reinventing the wheel?" Mike Edwards responded to this one, stating that SCA is being worked on by a diverse group of industry and non-industry members, under the Oasis umbrella. While currently SCA is only a specification, it has been submitted to oasis as an official standard. He then commented on the fact that while SCA does include a Java programming model that defines how to code Java in the SCA world, it does have a much larger scope than many of the JavaEE branded specifications. Mainly, SCA is concerned with how to compose services out of components using a variety of languages and platforms including but not limited to the Java platform. Eventually COBOL components could be wired together with SCA
Mike Edwards also discussed the fact that we are SCA did compete with Java (mainly the programming model) SCA offered a much more declarative, API free, dependency injection -based approach it was not limited to just the Java language, but also to other languages that could support it like Python for example.
Other members of the panel also jumped in to state that a new JavaEE SCA specification was also being built to allow components built in the JavaEE world to also be extended with SCA specific annotations and configurations.

Another contentious question asked by David was whether SCA was causing a serious fracture in the Java community, namely because Sun was not supporting it. David then stated that it looks like the Java community has lost "it's fear of Microsoft" because they are no longer united. It was entrusting that unlike last year, a member of Sun was present. Peter, that there has been a lot of distorted press around a comparison of SCA and the Sun backed JBI specification. Peter stated that the JBI specification is a bottom up infrastructure technology. JBI is meant to address concerns of the much lower-level than SCA.

Throughout the panel David repetitively came back to the issue of a fracture within the Java community because of the alternate programming model offered by SCA. Sanjay Patil of SAP had in my opinion a very eloquent response. According to Sanjay, while it is true that the originals of J2EE may have come out of a vendor unification is Microsoft, it is long since matured past those humble beginnings. Currently Java is now about bringing together a extremely diverse community that plays in a extremely diverse ecosystem. Java now caters to the "tattooed, and orange haired" scripters, as well as the business oriented application developers, as well as mobile software implementers.

This means that there are a large number of programming models out there already everything from the vanilla JavaEE, to the more scripting oriented ones offered by jRuby, Jython, to the spring programming model, to the one offered by SCA . This diversity is good for the platform, as it makes it stronger, and able to cater to different needs. An excellent comment, and one I'm not sure that an advocate of Microsoft would ever understand. That, or I believe David is just being extremely clever, and trying to add a little bit of confusion where possible.

Steve Jones, the only non-vendor at the table, (a member of Gemini) also made an excellent point around the value of SCA. Namely that SCA allowed one to declaratively describe the architecture of the system from the point of view of the various component technologies being used, and how these technologies relate to each other. This would allow very large programs to structure teams according to each component technology, with the programming provided a explicit description around how these terms should interface with each other. This sounds an awful lot like a bounded context map described by Eric Evans' Domain Driven Design book. In my opinion, this point of view alone was worth the price of admission for this panel. After the panel made sure to discuss this with him, and was very pleased to receive his opinions which were insightful.

Once the panel was over speak to Peter, as I want to get his opinion on whether Sun would ever think about formerly incorporating SCA into JavaEE. In my opinion piercing slightly defensive, and quickly tried to switch the discussion around the benefits of glass fish, the benefits of JBI, et cetera. What I certainly had no disagreements in anything that he was saying, I again tried to press him concerning the rationale of why Sun would not consider SCA, I certainly don't see anything being offered comparably by Sun. His response was that Sun would certainly be open to it, but that their users were simply not asking for it. So my final comment on this would be that if any of you out there see SCA as being something of value let Peter know, because he is currently unaware.

JavaOne 2008 conference San Francisco, Day 1 report

I've just finished the first day of attendance at the JavaOne 2008 conference at San Francisco.

The morning was kicked off with a general session where the general vision of what the objectives of the Java platform would be over the next year was presented. I have to say, after being a little bit skeptical lately about the relevance of Java, and specifically the Sun version of Java, I was pretty impressed. Maybe it was the 80s style breakdancing fighting, or presence of Neil Young on the stage, but I was impressed by Sun's vision of Java going forward.

Sun shared a very collaborative, Web 2.0, and dare I say agile vision on how technology was going to evolve over the next year. One of the key statements concerned the removal of various barriers. Barriers between corporate and personal workspaces, between business users, content owners, and designers and developers, and barriers between different kinds of information.

A key message was that organizations trying to hold onto information within the corporate firewall are going to be big losers going forward. Sharing and collaboration are vital ingredient for ongoing success. Another key message was that no technology or even business solution would be successful using big upfront design or planning. The emphasis needs to be on rapid turnaround time, short iterations, and small teams working in collaborative environments that can provide a large variety of technology, creative design, and business skills. Sounds a lot like agile to me.

So how does this all relate to the Sun vision for Java going forward?

The first thing is the official separation of the Java platform from the Java language. Going forward multiple languages (122 currently) will be supported. Many scripting languages are being targeted, after all, the open source community has long proven the developer productivity of things like Ruby, Python and others. The next version of Java will include an official scripting extension mechanism, to better support and integrate things like JRuby, JPython and Groovy into the platform. Multilanguage support especially scripting support is now integral to the Java stack.

Another very interesting and relevant announcement is support for Rich Internet Applications. A new scripting language called JavaFX, will enable Java developers to create unified workspaces taking advantage of rich multimedia capability. The platform supports deep integration with applications like Photoshop and illustrator to allow developers and graphic designers to work together to build the next generation of truly rich desktop applications. JavaFX also provides Web 2.0 style APIs to support the merging of information from different providers, instrumentation of user activity for powerful analytics, and aggregation of services. The goal of JavaFX is to support a mashup environment where developers can find, merge, deploy, share and monetize using a Web 2.0 model. JavaFX is being touted as one of the key components enabling to build applications "free from a hostile OS (think Windows), free from a hostile provider (Google), owned by the developer"pure

Personally, I think Java is found a relatively unique space offering some of the cloudlike platform capabilities of Google, but allowing people to retain more control than currently being offered by either Google or Microsoft. Of course the breakdancing was really impressive.

Finally, there's going to be emphasis over the next year to make Java more modular, extensible, and easier for developers to use.

JDK 9 is going to have support for in Module visibility, something I have personally been waiting seven years for. There's also going to be support for better packaging and bumbling including OSGI compatibility. Better packaging means glass fish is going to be offered in a extremely small (9 kB I believe) base package with a subsection of the time. Various containers can be redeployed, undeployed, and exist in parallel at run time. (Think multiple EJB containers being deployed as necessary)

It looks like Java annotations are going to become even more robust, with support for much stronger type checking, so things like no pointer exceptions, write once objects, and other neat tricks will be supported at compile time.

All exciting stuff, I'll try to comment in equal detail for every day of the conference.

Some of the interesting projects that I have been a part of...

After a flurry of initial activity, my blog has lain dormant for the last month or two. I thought I would kick off a new round of blogging by focusing topics on some of the more interesting projects that I've been a part of both before and during my blogging hiatus.
I believe the majority of the work I've been doing supports the notion of delivering software and agile way at least indirectly, and so should be of interest to anybody whois taking a look at my blog .
Think of this entry as a table of contents for my next numeral for to 5 blog entries. The next several posts will mostly be expansions of some of the areas I touch upon here.

1) Consulting 2.0
Outside of my regular job; consulting for specific clients, perhaps the most interesting thing that I've been a part of has been the effort to evangelize web 2.0 within the consulting firm that I am a member of. I've spent many an hour spearheading a small team focused on installing, implementing, and marketing a very extended version of mediawiki to serve as a collaboration platform, knowledge repository, as well as a centralized area were numerous groups can host various community portals.

The results have been pretty successful so far. I would have liked to see adoption and usage be a little bit more even across the firm, currently we are suffering from the typical 90-9-1 contribution ratio. Meaning that 1% of the people contribute a lot of content, 9% contribute occasionally, and 90% just look. Even so, I've recently become convinced that web 2.0 tools and technologies are an essential ingredient when attempting to deliver software using agile approaches. Software delivery, especially agile software delivery, is primarily a social exercise, and as such there is a potential to reap huge awards from using things such as wiki's and blogs as documentation and project management tools.

2) Accelerated Service Delivery
I have also been leading a small team to help focus our SOA related Eminence into a repeatable, reusable product offering. I spent the majority of the previous year, up until November 2007, working with a financial services client on integrating and validating an SOA framework that was going to serve as the common service platform for the entire organization. The team that I worked with to develop a platform felt so strongly about the technology that we decided to package it into a more marketable form.
The tie- to agile consulting and agile delivery in general is that the choice of technology is often one of the key factors in deciding how agile a team can be. First of all when I say that my team developed an SOA platform, I am perhaps using a too strong of a word. It would be more correct to say that we integrated a number of different technologies- almost all open source, developing reference implementations to validate the platform, and working closely with the open source communities (submitting JIRAs, contributing patches, etc.) to get everything working together.

The collaborative approach (both with our clients and with the open source community) was very agile and nature. More importantly, the open-source products we chose were primarily driven by our client's need to create an SOA platform that supported developer agility. The platform was based on Tuscany (an open-source version of the preview specification) and Spring. Using the two products together, we were able to completely decouple all infrastructure style coding logic from any of the business code. Developers were able to create services that were simple POJOs that could be dropped into our platform without any platform dependencies. I will go into more details in a later post, but suffice to say a product like Tuscany, that supports a specification like SCA, is a long time in coming. The Tuscany team provided fantastic support, better than I have ever received from any official vendor

3) Government 2.0
The firm I work for recently published an excellent paper around public sector and the use of Web 2.0 technologies. As I do not clear my blogs with anybody at legal from my work, I'm going to keep the name of the paper anonymous for now. (needless to say, our organization is still working through some of the details relating to Web 2.0) Regardless, the paper has received a very positive reaction, and many within both the federal and provincial public sector are asking how we can help to use Web 2.0 to assist in things like citizens collaboration, policy development, cross department interactions and a whole laundry list of other activity that could be helped by wiki's, blogs, mashups, as well as any of the other open, collaborative technologies that have been so successful on the web.

This is truly exciting from a number of perspectives, a lot of people talk about how web 2.0 is help to democratize the web, the same potential applies with government. Imagine a platform that would allow citizens to truly take a hand in forming policy, and providing real input into how we are governed. People in leadership positions in the government generally want better feedback, and Web 2.0 can provide that feedback. The mind boggles at the possibilities.

Again what is the tie back to agile consulting? Well again, a 2.0 project will not be successful without a lightweight development approach. Detailed requirements documents, massive upfront architecture and design, and extremely tight change control processes would be the death of any web 2.0 based project. The whole idea of Web 2.0 is that the platform is continually evolving, change is always happening, and responsiveness is key. In my opinion, at least from a development and delivery site, agile best practices are fundamental requirement of a successful Web 2.0 implementation.

Stay tuned for further announcements as I provide more information.

The Purpose of Modeling

My client wants a complete domain model, including every field and relationship...

I just finished a laborious, and somewhat exhaustive review of a relatively elaborate and extensive domain model with a set of business stakeholders last week. I know that a lot of upfront documentation is never considered a good thing on an agile project, but sometimes, when the scale of a project is incredibly large (think $20 million budget, and we inherited several thousand pages of requirements documentation before I came on board) it is necessary to capture business concepts in a form capable of providing effective guidance for future development.

The primary goal of this domain model was to take the vast ream of requirements given to us, and present them in a form that would be digestible by both business and developer/techie resources. In terms of total volume, the model consisted of approximately 16 diagrams, each describing a different domain concept, with approximately a page of text going along each diagram. In my opinion, this is not an awful lot of documentation compared to the source requirements we inherited which were thousands and thousands of pages long. When creating the domain model, we relied on many of the domain driven design approaches that can be found at http://www.domaindrivendesign.org/.


What exactly is the purpose of modeling?
While going over our models with business stakeholders, the biggest problem that I kept running into was that the client wished to see every single field and every single relationship on every class. They were under the impression that if the model wasn't perfect/complete it wasn't going to be useful. I did my best to explain that models did not need to be 100% perfect/complete in order to be useful, quite the contrary in fact. I'm not sure I did a perfect job in doing this, so I thought I would write up a quick blurb on what I believe are the primary purposes for modeling in the hopes that this will help me to defend my position better on in the future.

I should also point out that Scott ambler is an excellent job of describing modeling principles and best practices on his agile modeling method site

1. Modeling enables Understanding
   
   IMHO this is the most important part of modeling. Modeling enables people to understand what has gone on before their project, and also allows them to get a better understanding of what needs to happen next.
   
   As an example of the "gone on before", take the all to typical situation where a developer team inherits requirements; sometimes of good quality, sometimes of not so good quality. Developers are asked to design or develop based on these requirements, and frequently don't even know where to start. Modeling and diagramming is an excellent way to comb through these artifacts and apply some critical thinking around them so as to make some initial directional setting decisions.
   
   As an example of the "what needs to happen next", the same example also applies. The modeling done by developer type resources can also be used to apply critical thinking around what the data structures will look like, what the interface screens will look like, what services will be required, et cetera.
   
   This modeling exercise is also the best way for these developer resources to actually understand any legacy documentation. Simply asking someone to read a pound of documentation will rarely result in comprehension of that documentation. Giving someone some documentation and asking them to perform a discrete activity will result in exponentially more understanding. Examples include reading requirements to do initial analysis/domain modeling, initial vendor analysis, usability/interface modeling, actual development, et cetera.
   


2. Modeling Enables Communication
   
   Simple models that abstract and structure a large amount of requirements (or any other system information for that matter) are worth their weight in gold. The maxim "a picture is worth a thousand words" is no truer than in the system development space. Being able to illustrate a vast and incredibly complex system with 10 to 20 pictures (or 1 or 2 depending on the level of abstraction) goes a long way toward ensuring that a lot of distributed teams have the same understanding of what the system needs to do.
   

Complete models, be careful what you wish for...

So what does this have to do with my original clients desire to "have every field and every relationship described". You'll note that nowhere in these above two points did I mention "modeling to specify". Creating a specification means detailing every field, detailing every constraint, and every relationship, regardless of whether its fundamental to the point you are trying to make on a specific diagram or model. Creating a "complete" model actually makes it much more difficult to achieve the above two objectives. And here IMHO are the reasons why...

1. Important concepts are obscured: if every relationship, every field, and every business constraint is modeled and presented how can we tell the important ones from the unimportant ones? A model that is too detailed has the unintended consequence of intermingling important concepts with unimportant concepts. The art of actually hiding unimportant information is crucial to creating good models, it's probably harder to create models that only display relevant information than to create models that show everything. But the result is certainly a better model, one that enables understanding of the system across a variety of stakeholders.


2. The model is impossible to maintain: having been on several projects with a model oriented bent, I've had plenty of hands-on experience on the overhead required to maintaining readable and accurate models. In my experience, it is simply impossible to create maintainable models that are 100% precise down to the field level. At best, models are created that are "true when created" and get more and more irrelevant as time goes by. There are various experts out there who claim that modern tools make it easy to keep detailed diagrams and technical code in sync with each other. These experts need more actual hands-on practice with some of these tools, because this notion is still the stuff of ivory tower dreams, do not underestimate the effort involved in maintaining these types of models, it will equal, and often surpass your actual coding effort.


3. Complete models are really just "code-umation”: code-umation” is documentation that approximate the detail level of actual code itself. This type of documentation is no easier to navigate than code, it is certainly harder to change than code, and is not executable or testable. Run, run, run as far away from this approach as you can. It doesn't matter if you're in sourcing, outsourcing, working on a large or small projects, this type of detailed documentation does nothing to help developers do their work, try picking up the phone, having a face-to-face conversation, or any other method to communicate detail of this nature. Better yet, follow some of the excellent implementation patterns found in Kent Beck's excellent text here to ensure that your code is readable and self-documenting.


4. Drudgery versus innovation: creating detailed, field level, specification oriented models is mindnumbing, monotonous work. Good models are the result of critical thinking, innovative solutioning, and thoughtful analysis. These two styles of work are the opposite of each other. The more you focus on low-level details the less you'll focus on figuring out what the right things to do are, if you had the choice of receiving a high quality modeling backed by creative intelligent thinking or a complete model that was a result of monotonous repetitive effort what would you rather have. Don't ask for both, because it's simply not realistic.
   
   So in short, when modeling, focus on what's important, focus on what you need to communicate, and focus modeling on helping you understand what you need to do next. In most cases the simple advice will go a long way in streamlining your modeling efforts in ensuring that your modeling and models stay relevant.

Can Use Cases Be Used in an Agile Way?

Can one use use cases on a agile project? do use cases provide any value for agile teams?
I' m going to put out my opinion unequivocally Yes, Yes and Yes.

Most agile purists would disagree with me on all the three above points. Typically agile development approaches use user stories, backlog items, or other simpler mechanisms to describe requirements. They often have the format of "as a student i can purchase books". These requirements are typically put on an index card, and then not elaborated until it's time to code, and usually only by conversation, sometimes with a couple lines of text.

First of all let me state that I find user stories very useful for estimating , planning and tracking. I don't however find them very useful for providing context, structure, or abstraction. My experience has been that using user stories alone does not scale very well when working on large projects where multiple views of requirements are needed. For instance, a high-level view for project directors and other executives or a more detailed view for developers.

I can appreciate the bias that most agile practitioners have against use cases, because frankly, most use cases that I have come across in my experience are very poorly done. They are hard to read, they have way too much structure, rely way too much on UML syntax (extensions and usage relationships) and can often digress into what I call "functional decomposition". In order for use cases to provide value to any Project, agile or not, they need to be written properly. I specifically recommend two excellent books, Patterns for Effective Use Cases and Writing Effective Use Cases both of which Alastair Cockburn either authored or co-authored.

What I have euphemistically coined the "Cockburn Use Case Approach" describes how to write use cases with just enough formality and structure to allow proper abstracting, good context, multiple views, etc.; while still being readable by end-users, and not requiring a massive amount of effort to build and maintain. Most importantly, this technique focuses on writing scenarios using natural text, not on creating uber complex UML diagrams.

Alistair Cockburn has posted a pretty good piece justifying his use of use cases which I recommend taking a look at.

Some other comments on the topic can also be found here

A Brief Case Study

I have decided to share a brief case study regarding the own experience with user stories and use cases in general.

About five years ago I was asked to take a fairly large public sector program that was being previously conducted in a waterfall fashion and focus on getting some very quick functionality out the door. The client was previously frustrated by the millions of dollars spent with only paper documentation (and not very good documentation at that) to show for it. I lead a small task force of around five developers for about six months, our goal was to try to win back client confidence by delivering "something" quickly, and iteratively. We followed the XP process as it existed at the time, using user stories, velocity, etc.

While overall it was very successful, we quickly ran into many of the problems stated above. We had a lot of problems organizing our stories into a higher context, and we had no executive view which would allow client leadership to let us know if we were generally in the right direction. As a result, we wasted a lot of time developing functionality that was never actually used. (Often users on the ground working with agile teams do not have proper executive context, just a fact of life at least in all of my experiences)

As the program started to grow in size with multiple teams, I started looking into use cases, going through a variety of use case books until I came upon Writing Effective Use Cases and Patterns for Effective Use Cases. We started writing use cases according to the advice in these books, and saw great value immediately.

In all other respects, we still followed an iterative process, we developed use case briefs for a large portion of the use cases upfront. These then acted as our user stories in terms of planning, and estimating. The first thing we then did at the beginning of each iteration was to develop scenarios and extensions for use cases assigned to the iteration. Still agile, still iterative, but we used use cases.

Why Aren't These Cases Accepted within the Agile Community?

Throughout the last five years I have subsequently been a lead architect and process lead on no less than five separate projects (some very large and some very small) where I have used these techniques for use cases. I combined these use cases with other agile approaches (CRC, story points, test driven development, etc.) and have found them to be fantastically useful. This whole agile bias against use cases stems from the fact that use cases are probably one of the most abused SDLC related artifact. There are many different ways to do use cases, and from what I've read, most of them are just "wrong". But when done according to the advice found in these two books, they can really support an agile approach. I'm also confused by the agile maxim that you should "do what works", as long as you don't use use cases, never do any upfront modeling, don't use any case tools, etc., etc.

IMHO this sometimes religious bias against other tools can sometimes limit the appeal of agile by nonbelievers. I'm probably going little off-topic and will stop there.

Maybe the whole agile bias against use cases is a reaction against previous BUFD. But I still agree a little bit of upfront thinking is always going to benefit any development effort, as long as the emphasis is not on creating a requirements document, but doing the necessary thinking required to ensure that the right product is developed.

I think use cases can easily mapped to user stories, and can fit into almost any agile process
summary use cases > epics, themes
system use case > regular user stories
use case briefs >user story descriptions
use case details >part of iteration planning, determining the iteration objectives.

I will continue to use use cases on future projects, but combining them with more traditional agile techniques.

Why domain driven design is important

For several years I've been convinced that one of the core concepts of a system that should be modeled is the business domain. Being able to tell a common story using language, diagrams, and actual code that is focused on the business problem being addressed is one of the incredibly powerful capabilities of object-oriented technology.
This approach is being increasingly known as "Domain Driven Design". The term comes from an incredibly good book by Eric Evans. While browsing the Internet I came across a pretty good video that describes the benefits.

Interview with Rebecca

CRC cards-incredibly successful so far

During the last week of December I submitted a post discussing my intention to use CRC modeling techniques on what I would call a "massive" project. I needed a way to collaboratively build an overall domain model for an incredibly complex solution that was currently expressed in a thousand plus pages of linear, non-scenario-based business requirements. I said that I would report on my progress in a later post, and I'm happy to state that CRC modeling has been a fantastic success. While the overall constraints and structure of this project cannot in any way be called "Agile", applying this agile modeling approach across our internal team is providing a fantastic amount of value.

As I stated in my previous post, I initially followed a more traditional modeling approach, which involved a technical team:
1) Reviewing the requirements and creating models
2) Placing white board models into an official UML tool (Rational Software Architect)
3) Holding various workshops with business experts on our team to make sure that the models properly reflected the requirements

Again, as stated in my previous post, this approach had very limited success. The main problems being
1) Business resources were simply not a core part of creating the model. Reviews were simply not that successful/productive

2) UML repositories are really hard to share and maintain on a large team. Especially when rapid change is required.

Note: A lot of agile signatories (Scott Ambler especially) have strong cautions concerning using complex UML case tools; I never understood this before I tried using one with a team of 7 plus resources working on a shared model. It is difficult to overstate the aggravation involved in keeping this model in sync. An excellent resource on case modeling tools can be found on Scott Ambler's site here.

3) The case tool itself was causing the team (myself especially) to model with much more detail than required for this stage of the process. There's a lot of good advice out there on modeling that states that modeling should be at a high level until they can be validated through real implementation. UML diagrams by their very nature can sometimes seem too detailed for this initial analysis.

Within week one of the project restarting after Christmas break, (sometime around the second week of January) I had organized three parallel teams of two technical/modeler types and one business/functional resource. I myself played a floater role of mentor/facilitator/advisor across the three teams. Of course, I only had a week more experience than any of the other teams, but I'm used to doing what I call "just-in-time training". (meaning I train myself just in time to train somebody else)

The technical/modeling resources all had a pretty good understanding of OO and UML, while the business/functional resources had spent the last year and a half collecting 7000 plus business requirements, and thus had a pretty good understanding of what the solution needed to do in order to be successful.

We set up a process that involved spending three hours in the morning conducting modeling sessions using the separate teams mentioned above, then the modelers excluding the functional resources would spend the next half day consolidating the models, removing duplication, creating common structures and making sure that the separate work was combined into a cohesive whole.

Personally, I was personally concerned that CRC cards would not effectively scale, that it would be too hard to organize the cards//we organize them as necessary, that the model would be hard to maintain, and that we would be unable to get the right amount of detail. I was also concerned that the business/functional resources (who have no object oriented training whatsoever) would simply not be able to properly participate.

I have to say that my concerns had been completely unwarranted, every team appears to have had very good success in restating requirements into a CRC format that is now digestible in a matter of minutes, as opposed to hours. Even more important, functional/business resources have all played a vital role in deciphering/explaining business requirements and contributing to the quality/accuracy of our CRC models. I was personally satisfied we are on the right track when I saw specific CRC cards being built by these functional resources. Even more important, the proposed system appears to be a lot simpler than perceived by simply looking at the original requirements. Certainly there still is a lot of complexity, and this will by no means be an easy system to build, but it has become apparent that there is a lot of shared pieces that apply to 70 plus percent of the requirements.
Our client has also been very firm in wanting a Rational Software Architect version of the model. What that means is that we've spent the last week or so entering our CRC cards into this tool. The work involved in ensuring the integrity of the model within the tool, and the level of effort required to make simple changes more than validates the incredible flexibility provided by using simple index cards and particleboards. Changing the contents of a CRC-based model is incredibly easy, incredibly quick, and painless. I'll probably never attempt any serious modeling exercise without CRC cards again. And actually probably little embarrassed that I've never used a CRC before. So two thumbs up IMHO for CRC cards.

Great blog on "realistic" agile

Scott Ambler has recently started a blog Agile Software Development at Scale. Finally, an established authority on agile is providing advice on how could you to apply agile in a variety of real-world situations. Now this might seem like a funny statement from somebody whose blog is titled "Agile Consulting", but in my opinion a large portion of the agile community is too concerned with purity to be truly effective in a large number of real business world situations. This is not to say that agile is not full of incredibly useful material. The majority of development approaches and tools that I employ are almost always based on agile techniques. (CRC cards, agile modeling, velocity, release and iteration plans, test driven development, continuous integration, etc.)
What makes Scott Ambler's blog so refreshing (and much of his work on agile in general) is that it contains great advice on how to take all the great tools and techniques offered by agile and apply them to full program delivery, large-scale solutions, offshore, fixed-price engagements, and many other circumstances. Many agile purists simply throw up their hands in defeat when faced with one of these situations saying that "unless you follow practice number xx, you aren't agile". Which of course in my opinion is missing the point, which is how to apply agile techniques and common everyday situations that we find ourselves in.

SDLC-Guiding Principles

In my experiences, when a lot of resources speak about SDLC (short for software development lifecycle) in my organization they immediately focus on the concrete portions of the process. Tangible things such as roles and responsibilities, phases, activities, and artifacts all come in to mind. While these things are important, to a lesser and greater degree, depending on your point of view, they miss the most important point of a methodology, which is to describe the overall rationale and goals of the methodology itself.

The problem with using tangible, physical components to describe how an organization should develop software becomes immediately obvious to anybody who has tried to follow anybody else's SDLC. Primarily, a working environment, even one within an organization, fluctuates wildly from project to project. Resource experience is almost never consistent, team size can never be a given, nor is the scale of the solution. Other things such as technology, resource distribution, and project risks are also rarely a constant. What this means is that even a well thought out SDLC will never be more than partially applicable to any given project. I've seen organizations attempt to create a methodology that "scales" to various environmental factors. What this means is that a base SDLC is created, then offshoots are developed for projects that are smaller, larger, offshore, et cetera. Unfortunately, this has the effect of creating an incredibly dense and complex library of processes and approaches which become unwieldy in terms of communication and maintenance. Of course compounding this issue is actual process compliance, which is something that very few organization (I do not know of any) realistically achieve. In my personal experience, the majority of project members within organizations claiming to be CMI level XXX certified really just do the minimum they can to survive a project audit, and do not actually get a lot of value from their SDLC, often not following the "intent" of the SDLC.

So where do methodologies add value? IMHO, the most important part of any methodology is the section that describes the Best Practices and Guiding Principles of the SDLC. If your organization's SDLC does not frame its processes, artifacts and other tangible assets within a set of overriding goals and quantifiable values then resources will have no guidance around what to do when a process does not properly fit into a specific situation. (most situations in my experience) Best practices and guiding principles offer guidance on how software teams can/should strive towards success. They are goal/value oriented, and can thus be more applicable to many different situations than a set of concrete/tangible items such as processes, gating criteria etc. Principles and Practices also have the benefit of typically being shorter than a complex roles/responsibilities/process map which means that more people will actually read it, and may even actually follow it.

I recently was asked to put together a customization of the Rational Unified Process by one of my clients. While I believe the Rational Unified Process has a lot of value within it, much of this value is hidden away in detailed activities, artifact description, and low-level guidelines. I wanted to make sure that followers of the customization focused on the right values represented by the framework, which in my opinion was the emphasis on iterative work, modeling and extraction, and scenario driven requirements. I also wanted to put a clear emphasis on some of the more agile aspects of software delivery which are all compatible with RUP. The first two sections of the methodology customization dealt with these values. I have decided to post them here, as an example of what I believe IMHO is a good best practices and guiding principles section for an SDLC.

It should be noted, that I am about to share around three to four pages of advice on how to proceed for a number of different situations relating to development, requirements, testing etc. There's still a lot of leeway on how to interpret this advice, which I believe to be a good thing. Telling somebody that scenario driven requirements are a good thing and why, is much more valuable than telling somebody what the exact table of contents has to be for a requirements document. A particular artifact on a project might follow a specific template to the letter, but still miss the entire intent of the artifact. Better to focus on the values, some overall best practices and patterns, and allow the resources who are creating and/or using a particular artifact to determine what exactly needs to go in and go out.

Guiding principles

Uses an iterative approach:
· An iteration incorporates a set of activities from multiple disciplines (requirements, analysis and design, implementation, test, etc.) in various proportions depending on where in the development cycle the iteration is located.
· The schedule and scope of an iteration is fixed.
Risks are mitigated earlier, as most disciplines are engaged in the process from inception onward.

Follows a use case/scenario driven requirements approach:
· Requirements are organized in a way that tells a story of how someone may use the system, and how the system reacts to user behavior. Related scenarios are grouped into a specific use case or epic/theme.
· Nonfunctional requirements are captured in the context of one or more scenario driven requirements.
· Systems are developed by realizing requirements as part of the analysis and design discipline.
It is easier to develop requirements that are complete and consistent using stories.
A scenario driven approach provides a better understanding of a requirement from a user's perspective.

Is architecture centric:
· Design activities are centered around the notion of system architecture.
· The focus of early iterations of the process is to address architectural uncertainty
· Subsequent iterations may than reuse this architecture and focus more on completing business functionality
Technical risk is managed early on in the process
Greater control of system complexity and integrity

Is based on modeling visually:
· Visual representations of the system allow understanding at a higher level of abstraction than browsing through source code and other implementation artifacts
· Design models should focus on 3 fundamental aspects of the system
· Architecturally significant components, frameworks, and patterns
· A business domain model that clearly expresses the system in terms of business concepts understandable by business domain experts and system developers alike
· Common models/interfaces with other systems/programs that must be integrated
· While UML allows for the illustration of clear and unambiguous design decisions, it should not be the only modeling notation used (workflow, storyboards, ERD, etc.)
Modeling helps shorten the learning curve to understanding a complex system for both technical and business stakeholders

Is Agile, both in terms of development and documentation:
· All documents should have a specific audience, and the level of detail should be directed to that audience
· It is often not necessary to document all aspects of the system, focus on sections that are architecturally significant, difficult to understand, or have significant business logic
· Travel light when creating documentation. Every document that is created must be maintained over time. Some documents need only be temporary in nature
· All documentation should have a clear audience and purpose, and should assist in developing useful, high-quality software. Creating and maintaining documentation should not be a burden to the development team
· Use an evolutionary test driven approach to requirements and design
· Don't overcomplicate system design in an upfront manner in an effort to predict future requirements
· Use automated testing and build frameworks to enable the team to embrace change
· Architecture, requirements, and business concept should be prototyped with real code as early as possible, often in parallel with the documentation effort
· Continually validate with business stakeholders throughout the lifecycle of the project
Being Agile allows for greater emphasis on developing working software over comprehensive documentation and allows resources to “embrace change”, not resist it

Best Practices

Methodology Based Best Practices

Model and develop using an iterative, evolutionary approach
· Use more than one modeling artifact when determining requirements, design or architecture.
· Many modeling approaches, complement each other and may be conducted in parallel
o Use case, business domain, UI-prototyping
o Class modeling, database modeling, sequence modeling
· Architecture, requirement, and business concepts should be prototyped/validated with real code as early as possible, often in parallel with the modeling/documenting effort
· Within the same core team, design and implementation activities should be broken up into "miniature iterations" involving stakeholder review and clear testability

Use the simplest approach possible when modeling and developing the solution
· Use the simplest modeling tools necessary
o For smaller engagements involving co-located teams, whiteboards, sticky notes and other informal tools can be used
o As projects increase in scope, introduce case tools as necessary
· Avoid over designing solutions for requirements that have not yet been defined, as this can unnecessarily complicate the solution
· It is more important to have models that are correct than complete
o The purpose of models is to abstract knowledge and conveying meaning
o Extensive, overly detailed models typically obscure this meaning

Include testing activities within all phases of the lifecycle
· In order to support a culture that is able to embrace change, testing must be done early and often
· Modeling efforts should be validated with real code before committing to extensive documentation
· Automated testing harnesses should be used to validate all major system components
· When doing any work (documenting, planning, coding, etc.) always ask the question "how can I test/validate this?

Focus documentation efforts on creating essential artifacts necessary to develop and support the solution
· It is usually unnecessary to document all aspects of the system, focus on architectural significant pieces and business domain logic
· Some documentation and models, are used for understanding purposes only, and need only be temporary in nature
· Focus on creating detailed documentation (if necessary) after prototyping and other validation activities
· Clearly documented interface contracts are required between teams and systems
· Document with a purpose
o Focusing effort on what is required from the readers point of view
o Involve the "user" of the document to determine essential content

Take advantage of agile approaches , but structure them within more prescriptive processes as necessary
· Organize manageable units of work (app. 2 to 3 weeks) around moderately sized, co-located teams using XP, SCRUM, AM or other agile processes
· Document clearly defined interfaces between teams and major systems
· Organize these units of work using more prescriptive processes such as rup or sip

Organizational Best Practices

System stakeholders must be active participants in the development process
· Business representatives should own the requirements and be empowered to make decisions
· Organizational structure should support fluid lines of communication
· Requirements should be allowed to grow and evolve
Emphasize collaboration and communication
· Have a clear strategy that ensures that all documentation, modeling, and other system artifacts are clearly accessible to all project members and stakeholders
· Encourage collective ownership and development of system artifacts
· Ensure that the right tools are in place to support extensive collaboration and communication when using offshore/global teams

Favor staffing generalists, as opposed to specialists on projects
· Modern software development require a large number of skill sets to be successful (requirement gathering, interface development, object-oriented analysis and design, development, database, etc.)
· Employing "single artifact resources" will cause project resource counts to become unnecessarily large
· Specialists tend to be too narrowly focused on their own problem domain, and have difficulty seeing the larger picture (if the only tool you have is a hammer, all problems look like nails)
· Large amount of "document handoff required" resulting in information loss
· Hire/train resources to have 1 or 2 specialties, but have good general knowledge of the overall SDLC lifecycle
· Generalists require more experience than specialists and are not always readily available.
· However, working to maximize the number of generalists available for projects is essential to increasing delivery capability

Work to eliminate institutional barriers to change
· There are built in barriers to change specifically around infrastructure and databases in many IT organizations
· Processes and approaches can often be streamlined to be more responsive
Implement a governance model to support an enterprise system and process view
· Agile and iterative approaches are successful with small teams and 1 or 2 business experts
· Enterprise efforts require an enterprise view and a governance model to support a cohesive architecture and consistent approach

When taking advantage of offshore/global delivery capability use an agile approach to prepare key artifacts prior to hand off
· Functional prototypes, requirements, information architecture, and other artifacts can be prepared and validated using agile approaches
· Once validated, formally document these artifacts and transfer them to offshore/global resources

Technology-Based Best Practices
Focus development effort on mitigating areas of highest risk
· Elaboration and early construction iteration should be focused on addressing
o New/untested architecture
o Complex business logic
o Integration with external systems
· Risky areas of the system should be comprehensively covered by automated unit tests to ensure an early warning of possible faults
· Integration points
· Reusable assets
· Architectural risk
· Business process support

Make the right tools and resources available to support rapid application development
· Modeling/case tools are required for moderately sized and larger efforts
· Tools should support a superset of the uml notation (UML alone is not sufficient)
· Automated testing and build tools are the cornerstone to successfully implementing solutions using an iterative/ highly responsive approach
· Resources must have access to adequate development, staging, and testing environments

Develop using an appropriate platform supported by a robust technical architecture
· A solid enterprise architecture is a foundation that supports the ability to rapidly implement functional requirements
· Iterative and agile techniques along nicely to J2EE/.NET technologies
· Most web scripting and legacy technologies, languages lack appropriate tool support, and often are implemented using a more serial/waterfall approach

Implement a best in class Configuration Management Suite
· A comprehensive Configuration Management process is one of the most fundamental requirements of an iterative process:
o Automated unit testing support
o Automated build support
o Continuous integration several times a day
o Flexible versioning, release support
o Mock or stub object generation

Collaboration Responsibility Cards-domain modeling made easy

I recently just started work on what I consider to be a fairly large project with a large number of complex requirements. A massive amount of requirements had already been collected, (over a thousand pages worth). Unfortunately, a large majority of the requirements were not scenario driven, but of the "the system should make sure that..." variety. Extremely hard to follow especially giving this scale.
I was tasked with applying some structure and some modeling driven techniques to help frame the requirements and set the stage for downstream delivery. Needless to say, the environment for this project could be definitely considered less than ideal from an agile point of view. Project management did not want any development to start until requirements were better structured, and we had a better understanding of how different requirements mapped to different technology components.
One of the recommendations I quickly made was to develop a business domain model using package, class, and sequence diagrams. I'd read an excellent book by Eric Evans called Domain Driven Design, which nicely combined business centric modeling, UML, and a pragmatic, agile approach. I was hoping to use the functional resources that were responsible for putting together the business requirements in a business domain expert/reviewer capacity, as well as use a team of more intermediate resources to do the bulk of the modeling effort, with myself providing direction and guidance . We were constrained with using Rational Architect as our modeling tool, as that is what the client owned .
After about a month of modeling efforts, I had to admit I was not entirely pleased with our progress. Given a couple of days to reflect, I believe that the source of much of our inertia was due to the following
1) white board-based models are too hard to change once significant amount of effort has been put into creating the model. Whiteboards, while inclusive, need to be erased eventually, and require way too much redrawing.
2) most UML modeling tools might be a little bit easier to change, but are not collaborative in nature. There is way too much overhead in either printing off the models so that a large number of resources can collaboratively review and modify, or setting up a projector. Any changes require further work with the tool, and the turnaround time between making changes and putting them in the tool take way too long.
3) if business domain experts are not involved in directly creating the model, they do not focus enough on the model to see if they are correct. Our functional resources had several chances to take a look at many of our models before showing them to the client, but it was only until we did a live session with the client where the functional resources were actually asked to help the modeling team defend some of the modeling decisions did it become apparent how wrong our models were.
4) UML and the associated tools, just offer way too much detail. This detail actually can get in the way of initial modeling efforts. Despite my best intentions, I frequently found myself concerned with issues that were just not relevant at this point in the process. I can personally reflect on many situations where I spent too much time concerned with directionality of relationships, the use of inheritance over other more flexible typing mechanisms, and other issues.
CRC cards to the rescue
Now I admit to having heard of CRC cards before, but I've never paid much attention. I've always been happy to use TogetherJ or some other modeling tool for the majority of my diagrams, reviewing my work with other resources, or handing them off as necessary. Whoever, I believe that the the incredible business complexity of this project, the relatively large number of resources on the team, and scope of the proposed application dictates a different approach. I was doing some Internet research on various modeling tools and techniques and came upon CRC cards by accident. It didn't take very much reading for me to realize this approach had fascinating potential. I'm not going to go into too much detail around what CRC modeling is, however in brief CRC modeling involves placing responsibilities (methods and properties for classes) onto index cards (one index card for each class) as well as placing the collaborations necessary (with other classes) to fill these responsibilities. Index cards that represent classes with the lot of collaboration are placed close to each other, index cards that represent classes that implement or extend other classes/interfaces can be placed on top of each other. Potential models can be rapidly re-factored by rearranging these index cards. One can then trace through a set of classes and their responsibilities to see if a model will support specific requirements. One excellent resource concerning CRC modeling can be found here.
I quickly played with some CRC cards for about half a day, and was amazed at the speed I was able to model a large portion of requirements. I believe CRC cards will enable me to be effective on this project for the following reasons.
1) east of use-unlike UML, as well as most UML modeling tools, CRC cards are incredibly easy to learn, and do not require a lot of technical skill or object oriented knowledge to understand.
2) collaborative-CRC cards are incredibly collaborative, many resources (both technical and business oriented) can sit around a large table, and create CRC cards together. The result is a model that will have a very high alignment with true business concept situations and processes.
3) low fidelity-CRC cards natively support a lower precision fidelity then UML modeling notation. This lack of detail is actually a benefit when trying to focus on the nature of responsibility and collaboration, parameter ordering, datatypes, and other details found in UML just get in the way of this kind of modeling.
4) extremely easy to change-models made from CRC cards are very easy to reorganize, modify, and extend. Cards can be very easily ripped up, replaced, and changed.
Once I get back to work from the Christmas holidays I plan to introduce this technique to the modeling team, I would like to use CRC cards until the majority of the model has been completed, then translate the results to a more official modeling tool. I will certainly post the results here.