In-Depth
"Being" Agile vs. "Doing" Agile
Why don't agile development methods when applied to BI always yield success? It comes down to understanding that agile is really about behaviors, attitudes, and values.
by Ken Collier
It's beginning to look like 2010 may be the year of agile business intelligence. I've been preaching about agile BI since I first started doing it in 2004. Although there are a few converts, the BI and data warehousing community hasn't been fast to adopt. I'm suddenly hearing a lot more buzz in BI/DW circles about agile this year, including the agile theme of the upcoming TDWI World Conference in San Diego. I'm excited about this because I've personally experienced the joy of iteratively building BI systems in small increments with tons of collaboration with business experts and end users, and I want you to experience the same joy.
Since 2004 I've had one foot firmly planted in the agile software development community and the other in the data community. Agile seems to have "crossed the chasm" in the software community, and there is an interesting (and undesirable) tendency emerging that the BI/DW community can learn from. Some software organizations are falling into the trap of looking to agile as a prescriptive methodology that can be imposed on software engineering teams to make them faster and better. Guess what -- it isn't working!
It isn't working for these organizations because they have fundamentally misunderstood agile. Agile is a style of working, not a methodology or process. Agile is founded on a shared set of core values including:
- Individuals and interactions
- Working software
- Customer collaboration
- Responding to change
(For more, see agilemanifesto.org.) Agile espouses a set of guiding principles including:
- The early and continuous delivery of value
- Welcoming changing requirements
- Frequent delivery of a working system
- Daily collaboration between developers and business experts
- Self-organizing teams of experts
among others. (To learn more, visit manifesto.org/principles.html).
Sure, there are methodologies such as Scrum, eXtreme Programming, and Lean that introduce practices that are consistent with these values and principles, but agile is inherently a way of "being" that involves the entire organization, not just development teams. The specific practices and processes are simply manifestations of agile behaviors. Agile's core values and guiding principles give rise to good process, not the other way around. The great agile success stories (such as Google and Salesforce.com) reflect organizations that embrace this notion and have found agility to be very powerful. Organizations that adopt agile practices without embracing the values and principles often struggle.
As an agile consultant I get to work with many companies in various stages of agile transformation with varying corporate cultures and experiencing different challenges and successes. Through these experiences, success patterns and failure patterns emerge. I've begun referring to success patterns as being agile and failure patterns as doing agile. Doing agile refers to organizations that fail to move beyond the simpler trappings of agility: iterations, daily stand up meetings, and so on. Being agile refers to organizations with inherent values, behaviors, and guiding principles that exhibit the essence and spirit of agility: adaptive, evolutionary, value-driven, and quality-driven development. Organizations that are agile also do agile, but the inverse is not necessarily true. Many organizations are decidedly non-agile while still utilizing many agile practices.
Symptoms of Doing, Not Being
I was recently asked to help a struggling agile organization figure out why they weren't seeing the benefits of their investment in agile adoption. Senior management in this company had expectations of improvements in quality, increased productivity, better customer satisfaction, and happier developers. Unfortunately, their progress was going backwards in every area. After some digging, I began to see some familiar anti-patterns emerging. These are the symptoms of an organization that is doing, but not being, agile.
Iron Triangle Planning: Agile BI projects deserve sufficient planning, but agile plans are projections, not promises. Experienced managers sometimes have difficulty breaking the habit of expecting a fixed price, fixed scope, and fixed schedule project plan. An agile plan reflects the teams' best projections based only on the information that is currently available. As requirements change and uncertainty is uncovered, those projections are likely to become obsolete. Agile managers anticipate and adapt to this.
Management Styles Don't Change: The best agile BI teams are self-organizing, self-managing, and self-responsible. This doesn't mean that the role of management is subverted -- instead, it changes. Managers are enablers, decision makers, and facilitators. They work with development teams to remove barriers and protect the team from unwanted outside pressures. Agile managers are effective at replacing command and control leadership styles with more collaborative ones.
Emphasis on Productivity: The promise of agile BI is the delivery of a high-value, high-quality working system, not increased productivity. Leaders who emphasize productivity are surprised when quality suffers and end users are unsatisfied with results. Yet, when developers are pressured to be more productive, they take shortcuts such as reduced testing and hurried workmanship. Conversely, when the emphasis is on high-value and high-quality BI features, user's needs are often met after only 60 to 70 percent of the planned features are complete, effectively shortening the project cycle by shrinking the scope. Agile managers emphasize quality and value; they trust their teams to be productive.
Adapting to Change is Only Lip Service: Change is inevitable, especially in today's climate. Unfortunately, embracing and adapting to change is not a normal part of human nature. We go to great lengths to control and limit changes. We establish change control boards and change management processes. Instead of controlling changes, these processes only made them more disruptive. Embracing change means seeking it out, inviting it in, and encouraging more of it to assure that we build the right solution for our customers. Agile managers are eager to add new requirements, eliminate unnecessary requirements, overhaul project plans, rearrange priorities, and even discard working BI features in order to respond to -- and embrace -- change.
Customer Collaboration is Short-Circuited: One of the four agile core values is customer collaboration. Unfortunately, effective customer collaboration is hard. Our customers are busy and hard to pin down. They sometimes tell us things that don't make sense or won't work. Customer collaboration cuts into development time, so we create roles such as product owner or business analyst to be the "voice of the customer." It's easier to do this, but it's almost never as effective as real customer collaboration. BI customers have interesting stories to tell, and when developers get to hear these stories first-hand, they get a more holistic understanding of the BI system they are building – and they build it better. Agile leaders insist on frequent collaboration between BI development teams and end users, and they enable effective collaboration between these groups.
The Last Word
What is the lesson for us in the BI/DW community? Although agile BI developers are responsible for many agile engineering practices, the ultimate success of agility cuts across the entire organization. The management community, the customer community, the stakeholder community, and the development community are equally responsible for adopting agile attitudes and behaviors. Only then will an organization be agile rather than simply do agile.
Ken Collier, Ph.D. is the president of KWC Technologies, Inc., a consultancy devoted to agile business intelligence. As an agile trainer, coach, mentor, and project team lead, Ken has worked with organizations across industry sectors to promote agility. He is also a senior consultant in the Cutter Consortium Agile in Development and Business Intelligence practices. You can contact the author at [email protected].