In-Depth

Q&A: Social Media and BI Overlap Ahead

Using BI and data warehousing tools on data collected from social media applications could allow companies to better store, search, and analyze that data. The founder of a firm that makes a capture and storage platform for social media sees synergies just ahead.

• By Linda L. Briggs
• 09/22/2010

Given the huge growth of data from social media applications, as well as instant messaging and SMS, an overlap with business intelligence tools seems logical. Using appropriate BI technologies, companies might begin to capture, organize, and analyze the comments and chatter about their products posted on applications such as Facebook and Twitter as well instant messages.

In that context, BI This Week spoke with Derek Lyman, the founder and CEO of Dexrex Gear, a company that makes an enterprise capture and storage platform for social media, instant messaging, and SMS. Given his focus on social media and real-time communications in the enterprise, we asked Lyman about the overlap between social media in the enterprise and BI, and what might be ahead.

Regarding the integration of BI and social media, Lyman sees trends in which communication and collaboration platforms increasingly resemble BI tools. “We are approaching a very interesting point,” he says, “where we’ll be leaping two steps forward” in the synergies between the two technologies.

BI This Week: First of all, how well-developed is the overlap between social media and business intelligence right now?

Derek Lyman: Although there are some examples of integration and overlap between these two technology ecosystems, I think it’s fair to observe that they are currently two distinct and different worlds.

Exceptions aside, the professionals and companies that develop and implement communication technologies and platforms are not the same as those who do supply chain or processes management, and BI technologies. (There is even a clear divide here in the analyst and media journalist beats.)

However, there are trends on both sides that are causing communication and collaboration platforms to resemble BI tools in some ways. I also think that as BI tools (and process management tools) seek to inform and structure collaboration, they, too, are increasingly integrating with and replicating the communication environments.

So there’s movement happening on both sides?

Yes. As various platforms rise and fall, there will be some rough points. The overall trend may involve one step backward for every two steps forward. That said, I think we’re approaching a very interesting point where we’ll be leaping two steps forward.

In terms of applying BI tools and concepts to social networking applications, what’s out there right now?

I’m not aware of any well-adopted tools yet for running analytics in social networks, particularly for analyzing proprietary enterprise-oriented platforms. Although I know of some companies that operate online communities and have developed their own in-house BI capabilities, these systems generally rely on a sort of custom cobbling-together of systems, and more often than not include the extensive use of Microsoft Excel.

On the other side of things, companies that offer corporate social networks, such as Jive Software, SocialText, or Yammer, don’t yet have much of a focus on analytical capabilities, but that may change in the future.

What about BI tools that can be used on data from a more traditional medium -- e-mail?

In terms of analytics capabilities, that’s the area where a lot of work has been done. Interestingly, the vast majority of commercial initiatives to analyze e-mail can trace back to the Enron case. When Enron’s e-mail records were made public, it exposed an obvious soft spot in corporations, and in turn kick-started work on communication and network forensics. Enron and related cases of the time also set the stage for Sarbanes-Oxley and a new regulatory environment. All of this acted to drive the development of electronic communication retention products to fulfill new regulatory compliance objectives.

Once these data-retention products were in the market and were aggregating and storing vast amounts of e-mail messages, the records were increasingly used to support litigation and investigation efforts. The stockpiles of data grew, and resulting e-discovery costs skyrocketed. As a result, a number of analytics and search tools were developed to facilitate and reduce the cost of e-discovery. One example of such a tool is Orchestria, which was acquired by CA in 2009 for its information-security technology. There are many similar in-house and proprietary tools that have been developed and are quietly clicking away.

In the middle to latter part of the last decade, in an effort to improve corporate communications and corporate community awareness, a number of companies attempted to develop next-generation analytics tools. Unfortunately, just as many of these products were reaching maturity, we entered the 2008 downturn, and spending on new and innovative productivity tools dropped off. Many of these communications-oriented BI tools, therefore, died on the shelves.

For example, a product called Visible Path was acquired by Hoovers for $20 million during the peak of activity and renamed Hoovers Connect, but it was recently discontinued as a service. That’s unfortunate, since many great products were cut short when the market turned against them at the wrong time. As a result, it’s primarily in the regulatory compliance and e-discovery product spaces that BI products focused on in-house communication were developed and have been able to survive over the past few years.

Where do you see these tools and technologies heading?

Community analytics products have had a hard time selling as standalone products. However, I think they make great features and packages on top of a unified communication or corporate social networking platform. They demonstrate well and they can add genuine insight and value once users understand how to use them. I’m currently seeing a trend where these tools are being developed in line with new social networking platforms.

An example of this is that both Cisco and Novell have products coming out of their emerging technology units, tools that run on top of their communication platforms. Cisco’s Pulse product seeks to identify experts in various subject matters within a corporate community by analyzing and indexing keyword patterns within the communications of its users.

A smaller but still very interesting company is Saba. They have a software-as-a-service (SaaS) communication platform called Saba Live that has an interesting angle. Saba has a background of working in the world of human resources and organizational development. Customers in that space were early adopters of tools for corporate social network analysis to inform things such as succession planning and work force integration through organization transitions like mergers. Saba has built tools into its communication platform for doing corporate network relationship mapping and influence mapping.

Is there a natural overlap or some clear connections between those sorts of tools and existing, more traditional BI tools?

There’s lots of overlap and you can see it coming from both sides. As the traditional providers of communication and collaboration platforms are developing more BI capabilities, companies with strong backgrounds in business intelligence are developing stronger communications offerings, while keeping corporate knowledge management and BI objectives in mind. An example of this can be seen in SAP’s Streamworks.

In my opinion, with few exceptions the recent trends in corporate communication and social analytics and intelligence are driven by the implementation of new SaaS offerings on the communications side, and centralized data management on the BI side. These both result in the aggregation of social interaction data, which is being opportunistically cultivated for new BI feature sets. Both these trends are riding on the trend of unified platforms.

Unified platforms are limited in their reach, however, by the limits of their deployment and adoption. A task that is much more common in the BI product space is that of data integration, including the ability to import data across systems and from outside, third-party systems. As the value of new BI tools becomes more apparent, I suspect there will be a recognition that more complete data will result in better intelligence. Then we’ll see a push for social and communication data aggregation and integration capabilities. If we now circle back to the compliance and e-discovery product space, it’s there that tools for communications data capture and aggregation have been being developed and deployed all along.

As they did following the Enron collapse, compliance data sets make for good research and great feeder systems for communications BI.

Regardless of how all this plays out, I’m confident in saying that there’s an increasingly apparent convergence in the capabilities of traditional BI offerings and communication and collaboration offerings as they seek to harness the value of communication data to power new features and create new insight. Ultimately, there will be a push for better communication and social data capture and integration toolsets.

How does your company’s product, ChatSync, fit into what we’ve talked about?

The ChatSync platform powers the capture and integration of instant messaging, SMS, and social media data and provides those capabilities for leading communications data management solutions.

Although the ChatSync platform is primarily being used in partnership with compliance and e-discovery solutions, its general mission is to make all communications and social data easy to retain and manage. It’s uncertain now how industry will choose to handle communications data -- as part of master data management, in a subsystem that is specific to communications data only, or both. Regardless, ChatSync is a capable universal adaptor for message data, and Dexrex Gear is committed to working with industry leaders to advance the next generation of communications data management technologies. Our goal is to make all communications data more accessible, portable, and useful.