Data Logs: For IT, More is Better

When it comes to keeping and managing system logs, a new SANS Institute report makes it clear the relevant word is “more.” IT is collecting more information from more sources and want to use the data for more far-reaching purposes.

“When this survey started seven years ago, log collection was only being done by 43 percent of respondents, compared with 89 percent who indicated they collected logs this year,” notes the report’s author, Jerry Shenk, who is a senior analyst for the SANS Institute and a senior security analyst at Windstream Communications. However, rather than just using logs to detect suspicious behavior or troubleshoot problems, enterprises now are collecting log data for forensic analysis and correlation “to meet/prove regulatory compliance” (PCI DSS leads the pack of regulations in this regard). Respondents said they wanted to make better use of the data in cost management, for example.

More data is coming from physical plant and operations sources such as HVAC systems according to 14 percent of respondents, and 59 percent say they are collecting log data from their line-of-business applications. These sources didn’t even register as major sources in last year’s survey. Other new sources this year: mobile devices (15 percent of respondents) and cloud services (14 percent). In addition to sources, more devices are being tapped: most organizations collect information from more than 50 devices.

As more enterprises are using log data, the challenges have changed. “The mechanics of collecting, storing, and archiving the log data are no longer the challenge in today’s world of almost unlimited data storage,” the report explains. The biggest problem: the industry still hasn’t devised standard formats. Enterprises must still deal with inconsistent date formats between log sources, for example.

I asked Bill Roth, executive vice president of marketing at LogLogic (one of the report’s sponsors) what’s lacking. “The state of standardization is poor,” he explained. “There are fake standards like CEF and then there are underdeveloped CEE. My rant against CEF is that it is not open. Arcsight lists it on their Web site but does not provide the specification. There is no way any other vendor would implement it.”

When asked about log management tools, real-time alerts remain the most important feature for survey respondents, and their displeasure with log management system’s interfacing capabilities with third-party tools is their biggest gripe. Windows systems were cited as particularly unfriendly for log analysis; it is difficult to draw out and normalize log data from these systems.

The survey points to respondents’ dissatisfaction with searching and reporting. According to Roth, it’s a two-fold problem. “First, everybody hates search. You hate Google Search and Yahoo Search. Search is hard. The issue is not to make people like search, but to make them hate it less, so search will always be a problem. Second, challenges from reporting happen because everyone expects their reports in exactly the way they want them, so LogLogic realizes this and that's why we're spending a huge amount of investment on reporting capabilities.

That could be a big challenge, given that respondents weren’t happy about the analytic capability of log solutions. Roth gets it. “People want the analytics for their business. Every business is unique. The typical problems with analytics tend to be that they are too slow and that they don't have access to enough data. That will be a perennial problem and again, we are investing to make it better.”

I asked Roth if there were any surprises in the survey results. “Yes, that so many people are doing some form of logging -- 89% is really high and indicates that the market has moved to a new phase where it means that nearly everyone is doing log management so that vendors have to get more innovative in order to survive.”

I asked Roth about what sources IT will add to its log collection. “Rail systems, satellites, and geo-spatial data,” he said. We’ll see a year from now.

The report is available at http://www.loglogic.com/sans. Access is free but registration is required.

-- James E. Powell
Editorial Director, ESJ

Posted by Jim Powell on 05/04/2011