In-Depth

The New Breed of BI Analyst

A new type of business analyst has emerged, and though many have filled that role, little hard research has been done. Recently LyzaSoft and Microsoft looked and came back with similar conclusions.

• By Ted Cuzzillo
• 04/08/2009

Two makers of data-analysis tools have taken a hard look at a kind of analyst that many know of but few have measured. Conclusions of the two generally agree.

Recent research by LyzaSoft, maker of the new data analysis tool it calls Lyza, and Microsoft's Gemini program agree on basic characteristics: this group is determined, resourceful, and distrustful of data managers who presume to think for them. They are not technical in the way IT people understand it -- for example, most don't know SQL -- but they do know business processes intimately.

They crave data. Scott Davis, CEO of LyzaSoft, recalls one focus group member's statement. She had said, "I want to be close to the data." He didn't understand at first. The data was right in front of her, neatly summarized. He then saw she meant all of the data, every little bit of it. She wanted to snap open a long window and scroll down through the figures, millions at a time. She wouldn't try to read them, only see their shapes. She could say, for example, "Hmm, about two-thirds are under 1000." Davis calls that "visualization with browse" -- as legitimate a use of the term visualization as any I've heard.

Such intimacy with the data must include access to formulas. They want to see what's in each number, which helps explain Excel's popularity, Davis says. If a number shows up that doesn't look right, the formula is always easy to check. "You realize, 'Oh, that's the annual figure,'" he says. "'I forgot to divide by twelve.'"

Gemini researchers came to similar conclusions. "They build their analysis interactively, trying things out, turning back from dead ends," Microsoft's principal program manager of SQL Server Analysis Services, Donald Farmer, wrote in an e-mail message. They start in Excel, typically, and they build their analysis interactively, trying things out, turning back from dead ends." They're like "a painter working on the canvas."

"Users always find a way," Farmer wrote. "If they need to get data, manipulate data, analyze, and report, then they will do so. Sometimes they do it with the blessing of their IT and governance teams, sometimes they fly under the radar, sometimes they are in open rebellion."

A few other vendors have also had their eye on this group. Many Tableau Software employees came from just such jobs, writes Tableau Software vice president of marketing Elissa Fink. "[These analysts] are often self taught, and they apply brute force to tools. They are the massively under-served of the business intelligence world." The company estimates that the number of such analysts has grown ten-fold since 2005.

Three forces drive this trend, says LyzaSoft's Davis. First, data volume is growing like mad, and much of the data has to be analyzed. Second, hardware capacity is now better able to handle the work, even while software capability has lagged. Third, a new generation of users has arrived for whom computers are second nature. He tells about one particularly difficult problem he and several engineers had worked on. His team simply couldn't find a solution. One morning, an engineer walked in with news: a solution had come from an unexpected source. A young woman in the office, about 24 years old -- not a coder, not an engineer -- had announced it casually. When Scott asked her how she did it, she shrugged, "I just went home and wrote a Perl script."

Wayne Eckerson, director of TDWI Research, says that the LyzaSoft and Microsoft research has confirmed his notion of types of analysts and activities they perform.

Former TDWI education director Dave Wells recalls once being one of the distrusted data managers and later one of the scrappy analysts. He sees parallels with the mid-'80s emergence of PCs. The same will likely happen as happened back then, he says. "The IT departments had to learn how to federate them instead of fight them."