In-Depth

Study Targets Windows "Hooking" in Antivirus Software

Microsoft has confirmed that it has been working with a security firm investigating a fundamental flaw in antivirus (AV) software for Windows.

• By Jabulani Leffall
• 05/25/2010

Microsoft has confirmed that it has been working with a security firm investigating a fundamental flaw in antivirus (AV) software for Windows.

Microsoft said it worked directly with security research firm Matousec. Earlier this month, Matousec published a paper describing a Windows "hooking" technique practiced by some AV software makers that may make those products open to attack by hackers. Specifically, the research examined a technique it calls "the argument-switch attack or KHOBE [Kernel HOok Bypassing Engine] attack."

AV software makers may use the Windows hooking technique to modify the Windows kernel and run their security software, but it could also be exploited as an avenue of attack by hackers, according to Matousec. Last week, Matousec listed software security vendors subject to the hooking vulnerability. Those security vendors include Norton (Internet Security 2010), McAfee (Total Protection 2010), Sophos, Trend Micro (Internet Security Pro), Symantec and BitDefender.

Off the list were Microsoft's software security products, which do not use the hooking technique. "[Microsoft] has worked with Matousec to confirm that Microsoft Security Essentials and Forefront Client Security products are not affected by their KHOBE research due to the design of our real-time protection," said a Microsoft spokesperson in an e-mail statement.

According to Matousec, an argument-switch attack can patch the Windows kernel to "enable it to intercept certain operations like opening files or killing processes." In such an attack, a hacker switches out benign code on an affected system with malware before third-party security software can detect anything.

The Microsoft spokesperson said that such an attack method requires the hacker to have the ability to execute programs on the client machine as a prerequisite."In other words, the client machine is already running undetected programs -- and in some cases drivers -- making the practical impact of this technique very limited," the spokesperson said.

Microsoft's real-time protection is built using the file system's mini-filter driver model, which properly validates user mode parameters, synchronizes scanning, and "allows us to ensure we are examining the actual content that is being loaded for execution," the Microsoft spokesperson explained.

McAfee and Kaspersky Labs, to name a few software security vendors, are on the defensive this week. Spokespersons from those companies have suggested that administrative access is also needed for any kernel hooking to work. McAfee downplayed the Matousec research, saying that "several mitigating factors" make the research unlikely to lead to a viable, real-world, widespread attack scenario.

Security experts say it's too early to tell what actions software security firms will take in light of the Matousec research findings.

Joe Nardone, president of Expert Data Labs, said that security research can only be truthfully examined and applied when something actually happens and an exploit actually destroys a system or bricks an application. "This, like all other security issues, is a cause-and-effect relationship," he said. "This is unfortunately the case when you're talking about what malicious software could do and examining risks -- most of it is theoretical. When something actually happens, then the market dictates what the standards will be and these firms will adjust their functions accordingly."

Another research analyst predicts that the third-party firms mentioned in the research will look to make adjustments and will likely roll out fail-safe measures in the weeks and months to come. "Now that Microsoft has said its programs aren't affected, what you're going to hear from other vendors is that they have rules characteristics and parameters that stop such attacks even when disconnected from the kernel," said Jon Oltsik, a principal analyst at Enterprise Strategy Group.