In-Depth
Commentary: Survey of Agile in the Real World Confounds
A pair of analysts surveyed how enterprises are using an agile methodology. The facts are interesting; their analysis is downright foolish.
A new report from voke Research sheds light on how the agile methodology is being used by 200 survey participants. The firm's Market Snapshot Report: Agile Realities aims to provide insight into how enterprises are working with (and benefiting from) agile in the real world between June 2011 and February 2012; it also examines the pluses and minuses of the methodology. The full report is available to voke's clients only, but the firm provided a full copy to Enterprise Strategies.
Let’s start with background from the report that sheds light on the participants and their projects. Most surveryed project's' total team size is usually small. Twelve percent used 1-5 people, 22 percent employed 6-10, 20 percent had 11-15, and 18 percent had 16-20, with an overall average of 30.
Teams typically had 1-5 (44 percent) or 6-10 (28 percent) developers; the average was 16 (the average is probably skewed because 8 percent of organizations said they had up to 175 developers on a single project). Sprints, on the other hand, tend to have 6-10 developers (40 percent) or 5 or fewer (38 percent).
Agile is also being used for geographically distributed teams: just over a third (35 percent) say their whole development team works at the same location; 32 percent have distributed teams.
Project duration was most typically 6 months (20 percent), followed by 7-11 months (16 percent) or 12-17 months (15 percent). Total project budget was most often between $100,000 and $999,999 (20 percent), followed by $1,000,000 to $1,500,000 (12.5 percent). Sprint durations typically lasted 1-2 weeks (43 percent) or 3 weeks (25 percent).
Is Agile Keeping the Customer Satisfied?
More than half (55 percent) of tech companies have used "Agile" for more than three years; the figure was 27.5 percent for "enterprise organizations." The move to the methodology doesn’t appear to be smooth; according to the report, only 7 percent called the transition easy, and 64 percent said it was "confusing, hard, or slow." That's not unreasonable; in IT shops where I've worked, there's always an initial stressful period when new methodologies are adopted.
Respondents noted that adoption could be "politically stressful," "a work in progress" (with the methodology's goals elusive), and "initial[ly] confusing, [but] now the 'quest for Agile' is embraced with ongoing help of coaches."
Are customers satisfied with, or reaping the benefits of, the agile methodology? The researchers point out that survey participants report that "Agile" is best used for smaller projects, custom development, and Web apps, and should be used by "mature teams with [a] solid understanding of objectives and requirements." (The last point should apply to any development project.)
Survey participants say the methodology isn't a good fit for large projects, commercial off-the-shelf implementations, projects involving legislation or third-party supplies, or for software designed for hardware. When asked about how customer satisfaction has changed, 42 percent of tech companies and 28 percent of enterprise companies said it has increased; it stayed the same or decreased at 18 percent and 9 percent of those companies, respectively.
From the report’s survey results, I conclude that the number of changes users experience could have a negative impact on satisfaction. For example, 71 percent of participants report 10 sprints per release; 29 percent have 20 to 30 (or more) sprints per release.
Agile's de-emphasis on documentation and the acceptance of frequent requirements changes (something traditional methodologies control tightly) may also account for the amount of rework required on such projects. Although 44 percent can't say how much rework is required, 28 percent put the figure at up to 10 percent, and another 34 percent put rework at between 10 and 20 percent. That's change on top of change, for customers. Requirements gathering may also be affected when considering that only 41 percent of "Agile" projects "include a proactive customer representative." Another possible downside: "[Survey participants] noted that features took precedence over quality."
What Is Agile, Anyway?
The authors are so busy looking at the trees that they miss the forest, so to speak. They spend considerable time trying to figure out just what agile is and fail to grasp its key attribute: business agility.
The researchers point out that of the 100 terms used by survey participants to define agile development (iterative, Scrum, shorter releases, fast/less time to market, and customer-driven, among them), only one (changing requirements) comes from the original "Manifesto for Agile Software Development" and its 12 principles. So? That does not justify their conclusion that there's massive confusion about the methodology.
In focusing on the "disconnect" between the terms people associate with "Agile" and the Manifesto, the analysts miss the bigger picture: a key benefit of the "Agile" methodology is shorter, more frequent releases, which is why such terms appear in their survey. Indeed, the most frequently cited benefit of the methodology in the report was "faster releases" (14 percent).
This seems to miss the whole point of "Agile." In fact, the authors try to spin their own results that distinguish the methodology's very reason for being.
"In terms of release frequency, we see that both Agile and non-Agile projects have similar rates of maintenance releases occurring in quarterly intervals or less." They point to the 22 percent of "major Agile projects span durations of 6 to 18 months, while 36 percent of major non-Agile projects span the same duration." That's a difference of 14 percentage points, which is significant.
However, they're looking at the wrong numbers. Of all "Agile" projects in the survey, 52 percent of major projects and 82 percent of maintenance projects have a release scope of every two months or less, while for non-Agile projects the figures were 35 and 71 percent, respectively. That's a substantial difference, and equates to shorter, more frequent releases to me -- a central tenant of "Agile."
Furthermore, their focus on the Manifesto also leads them to draw some dicey conclusions.
"Based on the first statement of the Agile Manifesto, 'We are uncovering better ways of developing software by doing it and helping others do it,' we assume that the purpose of the Agile movement is to sell services." Sell services?
Later in the report, they claim that authors trying to sell books or vendors hawking training classes somehow reinforce this idea. This appears to contradict the fact that the Agile movement was designed to get applications implemented and updates installed more quickly so as to be more responsive (whether to customers or to its own users).
What IT shop will take a set of 12 principles and simply change directions without training or conferences to learn what they're getting into? It is, indeed, rare when a new development methodology didn't spawn books that explained the principles (in theory and in practice) and give enterprises deeper guidance. Conferences run sessions on best practices for a reason. Having worked with a number of fads, approaches, and products, from rapid application development (RAD) to a product/methodology called SDM/70, they all came with books and consultants. To say that that's why the methodology was developed is a major jump in logic.
The authors have anticipated criticism, noting:
Some people might not agree with our approach to analyzing the definition in terms of the Agile Manifesto. In our discussions about Agile, we have encountered people who indicate that Agile is more than the Manifesto and offer alternative definitions. We believe this point of view is exactly what is fueling the confusion over the words agile and Agile. What is Agile if not derived from the Manifesto? We believe this point of view will ultimately lead to an infinite number of Agile definitions that will eventually render the word agile meaningless in terms of technology.
The authors claim that confusion over terminology (that is, "agile" meaning nimble and "Agile" meaning the development methodology) has created an "Agile Dilemma" -- that IT believes that when management says it wants the enterprise to be "agile" that they're being given the green light to use an "Agile" methodology. Have these people ever worked in IT?
The report is valuable for its review of the real-world use of "Agile" Just skip the out-of-touch analysis.