In-Depth

Q&A: Agile BI Basics

The what, who, and why behind the move to agile BI.

• By James E. Powell
• 06/09/2010

What is agile BI, who's using it, and what's behind the rising interest in the approach? To learn more, we turned to Cindi Howson, the founder of BI Scorecard and a distinguished faculty member of TDWI who teachers at TDWI World Conferences. She's an experienced BI practitioner, having used, implemented, and evaluated business intelligence tools for more than 15 years.

BI This Week: What is Agile BI?

Cindi Howson: The Agile Manifesto was first published in 2001 by a group of software engineers (see agilemanifesto.org) trying to improve the software development process and customer satisfaction. There are 12 principles, but the six that most apply to BI are:

• Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
• Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
• Business people and developers must work together daily throughout the project.
• The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
• Simplicity -- the art of maximizing the amount of work not done -- is essential.
• The best architectures, requirements, and designs emerge from self-organizing teams.

If it’s been around for so long, why all the hype around agile now?

With any new development approach, it takes a while for companies and developers to get comfortable with new methodologies. As people use them and find success, adoption grows. Some of the most successful BI deployments rely on agile development techniques, so as we hear more about their approaches and successes, it breeds interest.

The other reason why I think agile is getting a lot of attention now is the rapid pace of business change. BI has to adapt at the speed of business. A recent example is the attempted terrorist attack in New York. Shahzad could board a plane because the Do Not Fly lists from the TSA are not synched with airlines’ databases. That it has taken two years to get the lists synched even on a daily basis is just one example where long development times have negative consequences.

Who is using agile development and how important is it?

I do get the sense that more innovative companies are using agile development, but I have also seen it in established manufacturing companies. It is less well suited to companies that have outsourced BI because it makes it harder to build things to a specification. Then again, I’m not a supporter of outsourcing for BI.

Agile BI emerged as a common theme among successful BI case studies when I began researching my book Successful Business Intelligence in 2007, so last year, we included this data point in the survey. Overall, agile development was identified as being not that important.

What do the survey results mean -- that this is all hype?

No! I think it means there is still a lot of education we need to do around agile BI. Survey respondents might have rated it as not that important because many don’t know what it means, and yet, some of those principles in the agile manifesto -- IT and business partnership -- make the Top 3 list. Probably, if we phrased it as “collaborative development with fast deployment and release schedules,” it would have been rated as more important.

Agile sounds like the Wild West of BI with no requirements, no documentation.

If you are used to having everything highly documented with requirements precisely defined, then less formal requirements definition can seem like the Wild West. The difference is in the how and degrees. Requirements are still gathered, but perhaps through rapid prototyping, collaboratively, rather than the business writing out their specifications before they can look at any results.

You tend to focus on the BI tools portion in the industry. Is there anything companies can do right now to apply agile concepts in that portion of the BI lifecycle?

I have long advocated that the business needs to have greater control over the delivery of BI content, whether the metadata layer, dashboards, or reports. Using collaborative development here, or even letting the business do more themselves, is a step toward agile development for the BI applications. IT should take the role of applying best practices and putting checks in place to ensure scalability and re-usability.