Surviving Network Blivets
From its World War II military definition -- 10 pounds of stuff in a five-pound bag -- the word "blivet" has become a synonym for all kinds of intractable problems. One common blivet is the networking problem of sudden WAN saturation.
This problem recently crept up on a friend of mine. Having successfully installed fractional T1 lines connecting his company's remote offices several years ago, my friend all but forgot they were there. Until last week, that is, when they suddenly stopped working properly.
The symptoms were subtle at first -- a few sporadic client/server application failures, which he chalked up to recently applied service packs and software upgrades. But within a few days, the frequency of the outages rose dramatically, until several of his offices went almost completely offline. This couldn't have been software, he reasoned. He dusted off his WAN analyzer and started sampling packet traffic.
To his surprise, he found the WAN links to the offline offices were routinely operating at more than 95 percent of capacity, and the T1 lines to the offices with no problems were running at over 80 percent. He recalled that while he was taking his WAN infrastructure for granted, his company rolled out several hundred Internet-browsing workstations and dozens of new client/server applications -- all of which pushed increasing traffic through the existing wires. My friend suddenly had himself a blivet -- 10 pounds of data stuffed into five-pound bags -- with catastrophic operational results.
What lessons can be learned from my friend's misfortune?
An ounce of prevention can prevent a 10-pound blivet. Pay attention to traffic levels on WAN links, and don't wait until they're pegged at 100 percent utilization. Remember, when it comes to networking, 100 percent isn't just 20 percent more than 80 percent -- it may as well be infinitely more. An 80 percent-used line won't lose any packets, but a line running at 100 percent will lose a lot of traffic. Also, there is no telling how far over 100 percent a line is being pushed once it reaches the top mark. The consequences of running WAN lines on the edge of capacity are unpleasant and precipitous. Applications running through an 80 percent-utilized line one day might not be running the next day when demand grows and the line hits 100 percent. Start monitoring WAN lines now -- when you find sustained traffic rates of 70 percent, intervene before they hit 100 percent.
How should you deal with them? If you have time, upgrade them. If you haven't considered your WAN options in years, start from scratch. Invite several vendors to bid on faster interoffice links, and encourage a price war. You might be surprised how much more bandwidth you can get for your money now.
In a critical situation, you may be able to buy some time by enabling data compression in your routers, a feature offered by most major router vendors. Remember, however, that enabling compression will burden the routers' CPUs could degrade performance elsewhere in the network. Also, while most vendors offer a compression feature, routers aren't always interoperable: If you're using different vendors' routers at either end of your WAN links, router-based compression may not be an option. A good reference about data compression can be found at Cisco Systems' Web site (www.cisco.com/warp/public/cc/cisco/mkt/ios/tech1/compr_wp.htm).
Finally, check your Windows NT network infrastructure. Have you deployed backup domain controllers in your remote offices, or is everyone authenticating through servers in a central location? Are you unnecessarily accessing databases through your WAN links, and would replicating databases to local servers help? Are all your users browsing the Internet through one pipe, and would implementing additional Internet connections in your largest remote offices help? Have you configured chatty BackOffice products correctly, given your WAN layout? A few simple client/server configuration changes can make a big difference in your WAN traffic profile. --Al Cini is a senior consultant with Computer Methods Corp. (Marlton, N.J.) specializing in systems and network integration. Contact him at al.cini@computermethods.com.