In-Depth

Bandwidth Bloodhounds

Hold Brothers, a trading company, tracks down bandwidth usage to contain skyrocketing costs.

• By Steve Zurier
• 04/01/2002

Hold Brothers' exponential growth rates had an unfortunate side effect: Bandwidth costs grew even faster. But as top IT brass scurried to contain costs, they discovered a classic corporate mystery: No one really understood where all that bandwidth was going.

Revenues at the Jersey City, N.J. direct-access trader doubled to $134 million in 2000 from $61 million in 1999. The company currently employs 300 traders at 15 offices nationwide and is holding its own in a very tough economy, reporting steady revenues of $135.6 million in 2001 and a No. 150 ranking on the Inc. 500 list.

But as Hold Brothers grew, bandwidth costs were going through the roof. In a single year, the telecom budget went up by a factor of 10. And despite the additional bandwidth, users still complained of slow network performance.

"I wanted to drill down and find out how much bandwidth each tier used,” says Chris Lukas, CTO of emerging technologies at Hold Brothers. "The Cisco tool we were using would just say our WAN link was 90 percent utilized, so we kept on adding T1 [lines]," an approach that Lukas says the company couldn't continue to afford.

The network delivers market data to each trader, allowing them to execute trades on seven major Electronic Communications Networks (ECN) including Instinet and Archipelago, as well as on the New York Stock Exchange. The company has five different network configurations, spread across 15 branches.

At the branch level, the network includes a mix of client PC machines, an order server, a custom order routing application written in Visual Basic and C++, and servers with links to seven ECNs. Lukas says the company's headquarters in Jersey City, for example, has client machines, an order server, plus a link to each ECN.

Although each branch has access to all seven ECNs through the Hold Brothers network, only the larger branches have local access. Typically, the larger the branch's dollar and trading volume, the more local ECN connections Hold Brothers supports for that branch.

To put a stop to the rising bandwidth budget, the company needed to determine the bandwidth consumption of each trader, each branch and the entire network infrastructure. To do that, the company turned to probes from NetScout Systems Inc. to get a clearer picture of its bandwidth situation.

Lukas' team installed the NetScout equipment in late 2000 between its routers and the WAN connections it uses from AT&T, Verizon, Broadwing and WorldCom. The probes capture and decode the headers on every packet as they cross the WAN link. The packets are then categorized according to user-specific business rules; the Web server then graphically displays the usage information via a browser.

A tool like NetScout, Lukas says, offered the granular information the IT department needed to get bandwidth costs under control. The NetScout's probes let Hold Brothers find out how much bandwidth each trader uses, plus much more detailed information, such as how much bandwidth any one office uses for market data feeds and how much bandwidth each network tier used per branch and per user. Using the new tools, Lukas' team was able to get a handle on its network costs in several areas:

Reduced bandwidth costs. Hold Brothers has been able to reduce the number of T1s to each office by one or two lines while improving performance and efficiency. That alone translates into more than $1 million in savings in the year since NetScout was installed.

Streamlined business processes. The company used to send all available trading information on more than 6,000 stocks to each trader. NetScout reports showed that sending all this data was one of the most serious drains on network resources. Seeing this, business managers could make the strategic decision to sends traders only the specific stock information they request. That move reduced quote traffic on the WAN by more than 90 percent.

More efficient use of ECN servers. Hold Brothers had been sending several copies of the same real-time data to each trader, tying up precious network resources. Lukas and his group found that installing ECN servers in the remote offices allowed the company to send only a single copy of a data feed to the office. The local server then duplicates the feed to each remote trader, a move that puts much less stress on the network.

Enhanced bandwidth awareness company-wide. Having an accurate picture of individuals' bandwidth usage let Hold Brothers develop conservation training. That, in turn has prompted traders to pay more attention to the way they use the network. For example, the traders don't use tick charts to analyze stocks because they take up too much bandwidth. Each branch manager has been given a NetScout tool for a more detailed read on how the branch uses bandwidth.

"Installing the probes allowed us to see how the network was being used and by whom," Lukas says. "We were able to change how we distributed quotes and we rewrote our trading application to use less bandwidth on the WAN. Before the probes, we had no way of telling how much bandwidth the apps or the traders were using."