In-Depth
Buyer Beware: Putting Intrusion Protection to the Test
A new report examines IPS products in rigid performance, security, and usability tests.
If you're considering purchasing an intrusion prevention system (IPS), what should you look for?
“Performance and throughput are key criteria when selecting an IPS,” notes Greg Young, an analyst at Gartner Inc. “IPS as a proactive defense can be a powerful part of an organization’s enterprise protection model. However, there is no ‘one size fits all’ IPS. The decision drivers involve differences in throughput, features, memory, and cost.”
So far so good. But when it comes to testing and adopting an IPS, how accurate are vendors’ performance claims, especially in real-world environments? Those claims are one method for vetting IPSs. Typically, security managers will then test a shortlist of choices in a production version of their corporate network.
Few organizations have the equipment or software necessary to accurately simulate their network under heavily loaded conditions. A new, free report from the NSS Group should help. It details the real-world performance of a number of IPS.
IPS Options Increasing
IPS is alive and kicking today, despite some analysts’ predictions of IDS or IPS death, says Bob Walder, head of testing at the NSS Group Security Testing Laboratories in Sumème, France, and author of the second edition of the “Intrusion Prevention Systems” report.
The test lab checked the performance, reliability, security, and usability of all of the devices through a battery of tests. Last year, average speeds were about 1-2 Gbps (gigabytes/second). This year, however, speeds were much higher—topping out at 8 Gbps. NSS will release a report on multi-gigabit IPS later this year. One surprise for the second edition of the report was that the lab “had a number of newer and unknown companies” sign up, says Walder, demonstrating the IPS market is still growing. Even so, “I got the impression that some are just rushing product to market, and just aren’t ready to be tested.”
Buyers should still beware. This year, nine vendors signed up for the second version of the report, each submitting an IPS, but only the IPS from BroadWeb, Fortinet, SecureSoft, Top Layer Networks, and V-Secure passed.
Vendors pay for their products to be tested. “Our tests are very rigid and very thorough,” notes Walder. “It’s not about paying a test fee and getting a rubber stamp.” Some vendors actually use the report and testing as part of the quality-assurance development phase for the submitted product, since NSS uses testing equipment some don’t have access to. “To replicate the type of test we do would probably cost a minimum of a million or two million dollars,” given the test equipment and expertise employed.
Putting IPS to the Test
In general, NSS categorizes IPS into two categories: rate-based and content-based scanning. Rate-based scanning looks “for anomalies in traffic patterns, a high rate of packets from a particular port, for example,” says Walder. It’s not looking for known bad worms, but just anomalous activity—resembling an attack, or worm activity—which it then blocks.
By contrast, “if a product looks at the contents of a packet or the contents of a stream, and applies it to a signature or some kind of analysis, then it’s content-based scanning,” he says. Signatures might include Code Red and Slammer.
While most products use either one or the other approach, one used both content and rate scanning. “The only one that worked on both fronts this time was Top Layer, because they’ve come from the attack-mitigation front and are trying to get to content front,” he says, though mentions “they’re still lacking functionality in the content” area. Still, “they’re the only product we know at the moment able to pass both tests.” Most of the content vendors are likewise working toward rate-based scanning but “are not quite there yet.”
Defending Against DDOS
In general, devices can’t handle attacks from many different IP addresses yet. Still, over the next year, Walder expects to see all of the IPS devices he tested handle both denial-of-service (DOS) and distributed denial-of-service attacks (DDOS). “DOS is something like a SYN flood coming from one source IP address, whereas a DDOS is the same coming from multiple source IP addresses,” he notes. “The devices behave very, very differently when you throw a DOS attack at it versus a DDOS attack.”
With a DDOS attack—attacks originating at hundreds or thousands of IP addresses—the devices’ ability to mitigate the attacks suffers, and the devices can become a network choke point. “If you could manage to send a gigabyte at these devices via a DDOS attack, you could bring them to their knees,” Walder says. Even so, “it’s a bit churlish to point it out, because if you didn’t have it there, it would bring the network down anyway.”
As Walder’s comment illustrates, he tested devices to their breaking point. Sometimes that point was far beyond where vendors rated their devices. Sometimes it was less. (See the report for product-specific details.)
Even so, when it comes to the gigabit-per-second throughput many devices can handle today, “how many people are really deploying these at a gig, and at a true gig? Most people never see more than 200 or 300 megs on the networks,” he notes. Customers are “basically going for this over-engineering thing,” choosing a device able to handle more traffic than their network carries. Still, “that’s a pretty sensible way to go about it.”
Walder doesn’t recommend gigabit IPS for the edge of the network. “If it’s at the perimeter, you can easily put up with milliseconds of latency,” using a product rated to handle a smaller load.
On the other hand, some networks really do have multiple gigabits flowing through the internal network. Here, the critical factor is latency. Internally, “you don’t want to be looking at latencies of more than 300 microseconds,” he notes, “because you can look at simple tasks, like doing an NFS file copy,” and see them get out of control. For example, a file copy that normally takes 40 seconds might require four minutes with too much latency. “That’s because NFS uses lots of small packets, and lots are obviously going through the device twice—once there and once back—so when you start to add up all these latencies, it can move it into hundreds of milliseconds.”
In other words, “latency is very important. Some vendors are trying to gloss over that at the moment. We think it’s more important than they’re letting on.”
Remember, he says, IPS are “switches with security in them, so the network guys who’ve spent many hours trying to architect their networks to get the minimum latency are not going to be happy if you put a security device in-line that doubles or triples their latency.” The only two devices able to minimize latency in really high-traffic situations, he notes, are the Top Layer IPS 5500 and SecureSoft Absolute IPS NP5G devices. One device tested in last year’s report, from TippingPoint, also had very low latency, he notes.
Software Consoles a Weak Spot
Beyond the number of machines that failed some part of the test this year, there was another major surprise. “Of the ones that passed, we were quite shocked at the quality of the software. I still think the management software on these things has a long way to go,” Walder observes. “To me, if you can’t mange these things efficiently, I don’t care how fast they go.” He sees the state of the software as evidence many products are being rushed to market.
Another interesting finding is the disagreement vendors have over which kind of IPS is best. “That’s one thing that makes it so fascinating from our point of view. This is such a new market that it’s one of the few areas of IT now where there’s a genuine disagreement between the vendors on how you should do this.” Some advocate protocol analysis, others go for pattern matching. Then vendors will fragment and reassemble streams in different ways too, with corresponding performance wrinkles.
Hold Off on Multifunction Devices in Enterprise
What about the emergence of multi-function devices, containing firewall, IPS, and antivirus? For now, Walder says they’re the provenance of small and home offices. “Layered defenses are much better,” he explains. “For now, the dedicated devices do a better job.” In other words, enterprises should adopt IPS and firewalls as separate devices. “I don’t care what the firewall vendors tell you about deep-inspection firewalls, these devices are not powerful enough to do firewall and IPS-type inspections in one box.”
The chief problem is throughput. “[With] multifunction devices, the more functions you turn on, the slower they get.” Hence IPS devices in general need to have increased throughout before having multiple functions becomes useful.
Small offices, on the other hand, may benefit from multifunction devices. For example, the Fortinet Fortigate-800, which NSS tested, maintained throughput of 400 megabits per second (Mbps)—a conservative figure, notes Walder—for IPS. With antivirus and anti-spam features activated, throughput drops to about 100 Mbps. Yet that’s “probably sufficient for your average branch office.”
The NSS Group study link can be found at http://www.nss.co.uk/ips
Related Article:
Product Shootout: Intrusion Prevention
http://www.esj.com/news/article.aspx?EditorialsID=836