In-Depth
Q&A: Agile Software Development Maximizes Project Rewards
Using existing development methods, companies end up with software that doesn’t do what they need it to do. Agile software development offers a new mindset and approach to building applications that succeed.
Independent consultant Roy Miller is a veteran of the former Andersen Consulting—in its pre-Enron heyday, of course—and RoleModel Software, a custom development house that applies the principles of so-called eXtreme Programming (XP) to real-world application requirements.
Miller has worked extensively with IBM DeveloperWorks, among other clients. He’s also written, or co-written, a pair of books—including one on XP—as well as a new book, "Managing Software for Growth: Without Fear, Control, or the Manufacturing Mindset."
Don’t let the latter book’s title fool you, however. It’s anything but a snoozer. Miller pulls from a variety of sources (including chaos theory) to argue that too many companies are approaching software development from an outdated, manufacturing-oriented perspective. What’s needed, he argues, is an emphasis on lean, flexible, and adaptive—agile—programming methods, which produce software that maximizes rewards. Using existing methods, Miller says, companies defer, but don’t minimize, risks—and frequently end up with software that doesn’t even do what they need it to do.
You argue that manufacturing principals developed more than 100 years ago today predominate in many aspects of software development. What specific methods do you identify here, and why do you feel they’re inappropriate for software development at the enterprise level?
A lot of [the way software development is done today is] based on the work of Frederick Taylor, who lived in the late 1800’s, the early 1900’s, and he got his start with steel manufacturing and machine tools in the steel industry. [W]hat he saw was just rife inefficiency all over the place. He came up with something he called “scientific management” to try to fix these problems, and what he meant, essentially, is that managers manage, workers work, and never the twain shall meet.
So he did time-and-motion studies, where you really would watch somebody work and you’d hold a stopwatch and time them and then try and figure out the most efficient way for them to use a shovel, for example. There wasn’t really anything wrong with this approach for something like bricklaying. It actually revolutionized the brick masonry industry. The problem was it was dehumanizing. It treated humans as cogs in a machine.
This is the same approach that companies are still bringing to software development?
This idea of scientific management was just the way things got done in the industry by the 1950s. And when did software development start in businesses? The early 50s, of course. So when companies started doing software development, they used this approach. It was a silly notion, but at the time, it seemed reasonable. That was back in an era when a computer fit in a gymnasium. It didn’t really fit then, and it really just doesn’t fit the modern software development world.
I’ll play devil’s advocate here. Even though I agree that what you’re saying intuitively makes a lot of sense, why doesn’t what worked for manufacturing map to the production of software?
Knowledge work is not physical labor. It’s all between your ears. So when management tries to translate a physical labor approach to knowledge work, they really screw things up, and that can become very dehumanizing. If you’re expected to be the guy who screws on the third nut in the left fender, when in fact what you’re trying to do is figure out why there’s this last-minute bug in eight million lines of code—it’s not exactly the same thing.
But surely organizations aren’t taking a manufacturing mindset and applying it in toto to software development. What’s the most common approach to software development in enterprise or large-IT organizations today?
In software development, most of the time, you start with planning. And if you look at the typical software development plan, it has heft. It took months to figure it out.
The way it used to work when I was at Andersen Consulting [was that] every plan I ever did was of the same variety. First you analyze, then you design, then you build, then you test, then you deploy. Your plan is based on a detailed, bottom-up analysis. You talk to lots of people, you spend some time prototyping some things, and you come up with a list of assumptions and heuristics that you can use to create this bottom-up estimate. Things like, “We assume it will take one-half day to do each screen, we’ve got 36 screens, so it will take 18 days to do our screens.” Most of the time, these plans aren’t worth the paper they’re printed on.
Why?
Because you’re guessing. You did your best. You weren’t trying to sandbag or stonewall. You can have an educated guess or a silly guess, but most of them are guesses in one way or another. When you put in all that effort in creating those plans, you have assumed a particular kind of problem. It’s a problem where you assume that all you have to do is follow a plan—all you need to do in order to get to the end result you want is control things and not deviate from the plan. But the problem you’re trying to solve isn’t that kind of problem at all.
What kind of problem is it?
It’s a “messy problem,” where the act of solving it changes the problem you’re trying to solve, such that your solution and understanding of the problem evolve at the same time, and one day you just decide to stop and you call it done. But that’s not how people want to think about software development.
The way most software projects run, the only tool they give management is control. We’re going to start with this plan. We’re going to control things to stick to the plan, but that’s like trying to put your software development on rails, like a train. You’re going to get from Point A to Point B, but what happens if—when you’re halfway to point B—you see that where you really want to go is 100 yards over there to the right?
You have two choices: stop, or go all the way to Point B. Because of the risks involved, most managers push home to Point B, because they have a plan, someone signed off on it, and this gives them plausible deniability. But they end up with software that doesn’t do what they need it to do.
You champion an approach that you call “agile development,” which is related, in many ways, to what is popularly called eXtreme Programming (XP). What does agile development imply, other than increased flexibility or responsiveness?
"Agile methods" is an umbrella term for various methods. One kind of agile methods is extreme programming, which is the most commonly known and used. I believe that XP is the right way to develop software. Bar none, it’s the best approach I’ve ever seen in balancing all of the forces on the development team.
What agile development methods say is you want to do just enough stuff other than writing code to let you write the right code. You don’t want to clutter up the process with lots of formality and documents and meetings and such. You want to focus on creating the software that people want.
[Agile methods] focus on things like conversation and planning very early and very often. You plan once a week. You don’t have a three-month planning session. That’s the only way to do it. You plan for a little bit. You write code for a week, you show the code to real users. They say, "That’s not what we meant at all." Since you’ve only worked on it for a week, you can still change direction without losing too much.
It sounds like a no-brainer in terms of a method, but you said yourself it’s not the way things are most often done. Why?
It’s a no-brainer in terms of how to solve the problem. It’s not a no-brainer in terms of psychology and interpersonal relationships. Most organizations are not set up to be as informal as XP demands they be. [XP] really [does] assume an organization that is almost agile itself. I coined a term called “extreme business” to go along with extreme programming, and that extreme business notion goes along with companies that want to develop software. They really have to change the way they do business—they’ve got to overcome a lot of the internal resistance and inflexibility.
What are the biggest pockets of resistance?
The resistance comes from people. If you have a manager and he’s been taught his whole career that he’s got to make plans the way I’ve described and he’s got to keep things under control—if you tell him he’s got to be a little bit less formal, and actually give up control in order to gain it … that’s anathema to him.
The primary reason people do things the way they do in the software development world is to reduce anxiety and increase risk. The irony is that the approaches they’re using do not decrease anxiety and do not decrease risk. As a matter of fact, they store up anxiety for the future, and dramatically magnify your risk—but it’s deferred. Almost three-quarters of software development projects fail, meaning they’re late, they’re over budget, or they don’t deliver what they said they’re going to deliver, or some combination of all three. And by far, the most important is that they really don’t deliver the software people want.
But agile or extreme programming isn’t for everyone, is it? If managers have been managing the same way for decades, programmers have probably become pretty set in their programming ways as well. How receptive are enterprise developers to a paradigm that turns most of what they know on its head?
Some software developers don’t like it because it’s very different from what they’re used to. For example, on an XP team, your requirements are all written on note cards. Some people think that’s just the height of idiocy. First of all, it doesn’t feel all that professional. You’ll get objections like “What happens if you lose one? We don’t have the document that lets Bob Smith over there in accounting sign off on it and say, 'It’s his butt on the line.'”
You have to get rid of these notions of antagonism—where you’re managing to minimize your risks rather than maximize your rewards. You have to come together as one bunch of people and say, “We’re trying to create software here. We’re all part of the same bunch, we’re going to get together and we’re going to periodically talk about it, and we’re going to make small steering adjustments.” So the real resistance, I would say, is more psychological than methodological. It’s just because people are not used to thinking this way, or they’re biased against … thinking this way.
Doesn’t it often come down to differences of personality, where certain kinds of people are going to be better suited for an agile approach than others? Given this, how can organizations—which are bound to have both types in house—effectively implement an agile approach?
I think there is a way. One way is for the people who are going to solve the production-level bugs where software’s been in use and something goes wrong, it’s probably a good idea for them to be the super-duper analytical types. For people to create new software that blazes new trails, however, you don’t need those people there. You need people who are a little bit different there. You need people who are more agile.
If the people who are more Taylorist thinkers, if they don’t want to be on the production side and want to be making new software, you should try to isolate them so they can’t hurt anybody else. But force-fitting software to fit their mindset produces crummy software. It really does.