Tuesday, September 3, 2013

Agile Transformation Is Extremely Hard, and the Challenges Are Not Addressed by Today's Change Management Techniques

Executing on Lean and Agile Requires a Fundamental Shift In the Way Most Organizations Work

Organizations using traditional management methods rely on detailed planning, command-and-control, and a hierarchical structure. Work is processed by departments that employ specialists of a specific skill set, and typically passed across specialist departments in large batches.
By contrast, organizations using lean and agile methods place a much bigger emphasis on learning rather than planning, and on self organizing teams who rely on a network structure to deliver business value. Work is consumed in frequent iterations, or just in time by teams that are cross functional and contain the majority of skills required to satisfy a particular unit of customer demand.
Organizations using traditional methods can be thought of as an industrial cargo boat or cruise ship. This design is incredibly efficient, and a lot of cargo or passengers can be carried across vast distances for a relatively low cost.

Organizations using agile and lean methods are a lot more like a speed boat. The cost per passenger seems to be a lot higher, but this is made up for in speed and maneuverability.

Metaphors aside, the real point here is that an organization using traditional methods does not look a lot like an organization using agile or lean methods. Just like a cruise liner and a speed boat may both be boats, they really have very little in common when it comes to operation, objectives, and capability. Agile and traditional organizations are really not the same thing. And the change required to transition from one to the other is huge.

Big C-Change with Big-C consulting firms - Fundamentally Wrong for Agile Change

There is a huge industry in helping organizations make dramatic changes necessary to helping them survive. Most major consultancies have an army of practitioners and partners who have dedicated their careers to helping organizations rewrite the way they think and operate.

While exact consulting methods vary depending on the firm in question, our experience is that organizational change methods follow a typical pattern. Consultants work with organizational executives, managers, and occasionally key team members to articulate a desired target organizational state based on a set of key business drivers.

Consultants and then work with the client to examine the current state of the organization, compare it to the target, and come up with a set of gaps between the two.
A roadmap is then developed based on prioritizing business drivers to come up with an implementation plan around how the organization will effectively transform to the desired state. This roadmap frequently comes in the form of a detailed plan that outlines key milestones, required activities, and effort required by both employees and external consultants.
This approach comes with a number of risks, especially when trying to affect dramatic change in an organization, such as transforming from traditional methods to one leveraging lean and agile thinking.
This change management approach typically results in the implementation of big C change. Dramatic changes are rolled out to the organization, sometimes on a department by department basis, resulting in wholesale shifts in job titles, processes, and technology. When any change is introduced into an organization, even a change that is good for that organization, the drop in performance will result. New methods need to be learned, new responsibilities take time to master, and new skills take time to acquire.

If the organization is able to successfully stick with the change, then performance will bottom out at the new, deteriorated level of performance, eventually the desired changes will have the effect of improving performance allowing the organization to reap the benefits of the target state.
This however is an almost naïve prediction of how big change actually occurs. In reality many big changes stall well before organizational benefits achieves the promises of the target state.

Organizations naturally resist change, and professionals within an organization will resent any force that asks them to change the way they are working. Fear is a major cause of change resistance, fear of losing one's job, fear of no longer being relevant, as well as fear of being asked to work in a way that
one may not be used to.

Organizations are susceptible to abandoning the change project when organizational performance drops to its lowest, it's at this point that the change agent might find himself fired. Paradoxically, organizations that tend to operate at lower levels of maturity will drop more in performance when asked to make a large change, but will also have a lower tolerance for this drop, and be more likely to abandon the change as panic sets in.

Even if change agents and change champions can effectively keep the organization on track, and get the organization to the target state, the promise of improved performance may not materialize. The reality is that the suggested change may be wrong for the specific context of the organization in question.

As stated before, the agile and lean body of knowledge contains a diverse set of methods, practices, and principles, not all of these are equally suitable to every organization's context.
Every organization has a different set of business drivers, level of maturity, willingness to change, and an endless host of other factors that will ensure that no two agile transformations are alike.

If we remember our original discussion around planning driven methods is that they leave us open to the risk of building the wrong thing. This is true of products, software, as well as a change management target state. In an environment of high variation and complexity, faithfully completing a plan that was built the beginning of the engagement leaves us open to creating something that has no value.

Using Scrum to Inspect and Adapt - A Better Approach to Agility, but Often Results in Too Large a Change That Does Not Fit All Contexts
Many organizations attempting to increase their agility and transform their technology delivery capability to take advantage of lean and agile thinking have elected to use Scrum. Scrum is a deliberately minimalistic agile method that is meant to help teams get started with agile, as well as provide a framework to help technology knowledge workers adopt and improve.
Customer demand is placed into a product backlog, the content of this backlog is primarily the responsibility of a product owner, who coordinates with other business stakeholders to prioritize, refine, and otherwise groom this backlog. The product owner is considered to be the "one throat to choke" when it comes to making sure that the right product features are being delivered according to the desire of business. Work is typically delivered to the end-user in iterations of between 5 and 30 days. This iteration is a heartbeat for delivery, and is known as a sprint.
Software, (and other business valued work) is completed by a cross functional, self empowered team that is typically co-located. In order to encourage this cross functional nature, scrum specifies a very limited set of roles, a team member, a Scrum Master, and optionally a coach. Team members may have specific specializations, but are not constrained to only doing any one role. (E.g. a developer may test if it is helpful), Scrum Masters are expected to be servant leaders who help the team by removing impediments, protecting them from organizational politics, and providing other advice and guidance.

A particularly important part of Scrum is that teams only commit to delivering what they feel they can possibly accomplish as part of a specific sprint. Commitment is done at the team, rather than individual level.

Sprint cycles are typically accompany by a very lightweight set of artifacts and meetings. Scrum provides advice on how to conduct Sprint planning sessions, daily scrums, and product demos, as well as guidance in how to track velocity using burn down charts.

Scrum helps teams and the organizations they work for to become more agile in a number of ways. First of all, sprint cycles are designed to maximize customer feedback allowing teams to course correct and maximize customer value. Sprint cycles are also accompanied with what is known as a retrospective, a regularly recurring meeting where team members focus on continuous improvement and addressing existing issues and problems.
The iterative, customer facing nature of Scrum is also designed to make organizational dysfunction visible, encouraging the organization to make more systemic changes that are required to ensure that individual teams can be successful. This focus on organizational impediments provides the data required for change agents to transform to be more agile
Using scrum to help organizations transform share some, but not all of the risks found within a big C change approach. Because scrum is a minimalistic framework, the performance impact of change from initial adoption is not as adverse as found in larger structurally-based transformations. Subsequent waves of change can also be smaller, as the pace of change can be graduated based on the impediments found by different teams.
That being said, different organizations have faced a number of challenges while trying to adopt scrum, and a significant portion of technology knowledge workers have "failed" to successfully adopt.

While scrum is a lightweight method, it asks executives, managers, and staff to work in a fundamentally different paradigm. The whole notion of cross functional teams, self organization, and delivering in small iterations cause major conflict with organizations that have grown up using traditional methods and traditional thinking. Many people resist the alien terms, and are threatened by this unfamiliar way of working.

What many folks adopting scrum fail to realize is that implementing scrum must be quickly followed by implementing other practices from the lean and agile world. The hyper agile model of scrum breaks down if organizations are not willing to invest in significant, further changes.

Scrum also faces challenges due to the fact that a pure agile model is not always the right recipe for all delivery work. Not all domains lend themselves to permanent, generalist, self empowered team members. Not all work neatly fits into the notion of short, time boxed intervals. In many organizations are simply not ready for the large and sudden shift required to make scrum successful.

Kanban - a Viral, Evolutionary Approach to Change Management, Many Contexts Require Significant Investment and Expertise to Be Truly Successful

Kanban has been quoted as being a "viral, evolutionary" change management approach designed to help organizations gradually become more agile over time.

Kanban is designed to provide an even more gradual way for technology knowledge workers to improve than the minimalist scrum process framework. Knowledge workers start by mapping out their existing delivery process and visualizing this process using a Kanban system, typically manifested as a Kanban card wall.

The Kanban system is then directly connected to business stakeholders responsible for prioritizing and validating business valued work. Each customer is given a specific queue and allowed to place work on that queue based on the amount of capacity dedicated to that customer.

Delivery agility is introduced in the system by putting explicit limits on how much work can be consumed at a time. Very low working process limits will force teams to adopt a much more agile and lean working style, collaborating frequently, resolving impediments quickly, and keeping quality high through best in class techniques. High work in process limits will allow teams to operate in a much more familiar way to traditional methods. Multitasking will be allowed, and there will be less pressure to collaborate and continuously improve.

Current work in process is then visualized on the Kanban system based on where it is currently in the process. Working process as well as work that is in the various business stakeholder teams are expressed as work tickets, which may represent enhancements, user stories, or other business valued work.

Simple visual indicators such as color are used to annotate work so that technology knowledge workers and their customers can immediately recognize important attributes such as risk, priority and capabilities required to complete a specific unit of work. This allows knowledge workers to assign unique processes two different work types, for example emergencies could be completed as soon as they are received regardless of what other work is in progress.

Hot color tickets can also be used to annotate business valued work with, blockers, and other impediments. This visualization serves the purpose of connecting with knowledge workers and their managers on an emotional level, being able to see how much work is in process, as well as the healthiness of that work encourages people to make better decisions and improve immaturity.

Knowledge workers will work through the process using a pull-based approach. Using this method, work is only moved into particular state when doing so will not violate the inventory limits for that particular state. This ensures quick turnaround of delivery work, or what is known as delivery flow. Delivery flow creates a high feedback system similar to that found in scrum or other approaches that rely on iterations.

Kanban helps organizations adopt a lean mindset, based on collaboration, continuous improvement and self empowerment. Kanban places a heavy emphasis on improvement followed by experimentation, borrowing lean measurement techniques such as the use of statistical process control and cumulative flow diagrams to measure performance, flow and lead time.

The combination of visualization, empirical-based improvement, and focus on flow allows knowledge workers to design customized process solutions that match their own context. Different Kanban systems are allowed to evolve at their own pace within the same organization based on the unique needs of the workers and customers involved in delivering business value. This process and method diversity is enabled through a common process improvement methods.
Kanban has been labeled as a meta-method, not a software delivery method in its own right, technology knowledge workers and their managers are encouraged to design the right method for their own needs, using an incremental evolutionary approach. Kanban has also been described as a viral improvement method. Kanban can be adopted in a small part of the organization, which eventually encourages other workers to connect to the Kanban system, and adopt their own Kanban implementation.
When Kanban works as designed it offers the least disruptive path to organizational change. Because getting started does not require anybody to change roles, changed job titles or change the way they are organized, more conservative and traditional organizations can get started with less impact than adopting scrum.

Kanban contains many design elements such as process policies, work in process limits, and work types that are customizable to the context of the business value being delivered, this means the Kanban can be matched to the unique constraints of the organization.

By design, a viral and evolutionary method can take a long time to provide desired performance improvement results. When asking an expert how long a Kanban improvement method will take to achieve high maturity, the answer is often as long as it needs to. While this approach may arguably be the one that makes the most sense, it is often an answer that most organizations desiring some type of transformation are not willing to or able to accept.

Kanban is also not designed to provide specific process answers for particular problems. Knowledge workers are expected to use Kanban to make problems visible, and engage in continuous improvement to make the system work better. Our experience is that this can actually cause churn and confusion in organizations with little practical agile experience. The wealth of options and the customizable nature of Kanban can actually make change harder for some organizations.

Finally, Kanban adoption can stutter just like scrum due to the lack of existing maturity within the organization. Kanban systems that do not process fine grained units of work, similar to user stories found in scrum and other agile methods, may exhibit a lack of feedback as well as performance and stability that interfere with continuous improvement. This means that the initial jump to successfully adopting Kanban can be larger than it is first thought to be.

The Case for Lean Change
My team and I designed Lean Change to help overcome the challenges inherent in some of these approaches. Lean Change helps organizations navigate through agile change by providing a set of tools that support deliberate change planning without falling into the trap of a fixed upfront design. It appeals to organizational leaders who want to execute a deliberate change, without locking them into a particular approach up front. When using the Lean Change method, the entire change model is represented by assumptions that are validated through the change process. Lean Change is designed to require an agile improvement method such as Kanban and Scrum, as these provide teams with insight on what to improve next.

Check out the Rest of Lean Change - Chapter 1
  1. Why Today's Technology Organizations Need to Change
  2. Challenges with Current Organizational Change Methods
  3. Presenting the Lean Change Method

No comments:

Post a Comment