In-Depth
Accelerating File Transfers in the Mid-Size Enterprise
The mid-size enterprise needs simple, fast, reliable, and secure ways to transfer large files. PDP and UDP technologies may provide the answer.
Mid-size enterprises in a broad range of industries are transferring ever-increasing amounts of data, both within the business and with external business partners. While e-mail has been the primary medium of data exchange, these businesses are now producing much larger data sets. Rich media files in the entertainment industry, seismic data produced by oil and gas exploration, and airplane reservation information in the transportation sector are just three examples that are anywhere between 10 MB to 1 TB in size.
Most companies have relied on the File Transfer Protocol (FTP) over Transmission Control Protocol (TCP) to move files between computers, but only recently has the transfer of such large file sizes become a common occurrence.
Because TCP is a “reliable” protocol (received packets are checked for errors or missing bits and retransmitted if such a problem is detected), the receiving computer must respond with an acknowledgement after it receives a group of packets, and this acknowledgement must return to the transmitting computer before further packets can be sent. If the acknowledgement is too slow in arriving, TCP concludes that there is congestion on the network, re-transmits the packet(s), and throttles back the rate at which it sends further packets.
An unintended consequence of this behavior is that latency in the network will trigger such a slowdown, even if the latency is simply caused by a long round-trip time (RTT) for the packet and its acknowledgement.
Medium-sized business typically react to their large data transfer needs by ordering larger bandwidth connections from their Internet service providers, often under the mistaken impression that bandwidth is necessarily the limiting factor in transmission throughput.
Figure 1: TCP throughput as a function of latency. These data were
taken using a WAN emulator with no packet loss between two
conventional PCs using FTP. Sample city pairs shown for reference.
However, Figure 1 shows how throughput falls off as a function of network latency, regardless of the available bandwidth. For example, the throughput of an FTP file transfer between a PC in San Francisco and a PC in London (separated by a ping time of around 200 ms) will never be greater than 1.3 Mb/s, even if a direct Gigabit Ethernet link is available.
UDP and PDP: Compelling Alternatives to TCP
Recently, commercial products have been released that use alternatives to TCP to overcome this latency-induced bandwidth limitation. One such protocol is the User Datagram Protocol (UDP).
Unlike TCP, UDP is not inherently a reliable protocol. However, reliable variations of the original UDP now exist, and these have proved to have far better characteristics for high-performance file transfer characteristics.
For global exchange of very large files over large pipes—now a common occurrence for large businesses—the new, reliable UDP implementations are far superior to FTP-over-TCP. However, for medium-sized businesses, pipe sizes can often be more modest, and firewall issues sometimes prevent easy implementation of UDP. For this market, an additional protocol has recently been developed.
This new Parallel Delivery Protocol (PDP) consists of several simultaneous TCP streams. While parallel TCP implementations have been in research labs for several years, it is only recently that PDP has been commercially available. PDP has the inherent reliability and firewall-friendliness of TCP while providing acceleration over a TCP-like connection in the presence of network latency, making it a very good fit for mid-size enterprises. In addition, PDP is available with built-in security features that can be extremely important when the files in question are to be transported over the public Internet.
Given the new commercial availability of UDP and PDP, there are very few times when FTP remains the optimal file transfer protocol for a file transfer. The choice now is more often between UDP and PDP, as shown in Figure 2.
Figure 2: Determining whether UDP or PDP is ideal for an
application depends on both the available pipe size and the
RTT. For medium-sized businesses, the combination of UDP
and PDP in one product is a very compelling solution.
Mid-size businesses often have external WAN connections ranging from 10 Mb/s (Ethernet) to 45 Mb/s (DS-3), and may need to transfer data over a broad range of distances (and hence a broad range of RTTs). In these cases, as shown in Figure 2, a simple-to-use, reliable, and secure combination of UDP and PDP is optimal to address this market.
Applications for Mid-size Businesses in Many Industries
Any business that sends large amounts of data—either in single, large files or in large collections of smaller files—is a good candidate for UDP and PDP solutions. Since these companies do not typically have large IT groups, any solution needs to be simple enough that the end user, not the IT professional, can implement it. Most solutions come in client/server configurations, with particular emphasis on ease-of-use for the client.
Despite the need for simplicity and intuitive interfaces, powerful automation features such as hot folders, job transfer scheduling, and synchronization are also required for these products to become widely adopted. Such automation tools are particularly important when collaborating across time zones. For example, an office in Paris can receive new files whenever a development team in Los Angeles posts them without the need for a manual transfer, and the sales teams in Frankfurt and Singapore have access to those files whenever they need them.
A further consequence of the need for simplicity is the desire for an all-software solution. Hardware-based WAN accelerators have been commercially available for several years, but the complexity involved in installing and testing a new piece of network hardware, coupled with the high cost of these devices, has primarily limited their adoption to large enterprises. The new breed of software-based solutions is approximately 10 percent of the cost of the hardware appliances.
Summary
For mid-size companies working in distributed geographies or with vendors, partners, and customer around the globe, FTP-over-TCP is no longer the tool of choice for exchanging and sharing large files. For shorter distances or for small-bandwidth connections, PDP provides the ideal protocol for file transfers; for large pipes and large distances, UDP provides the maximum throughput.
The mid-size enterprise can now solve its need with simple, fast, reliable, and secure all-software solutions that are commercially available and affordable.
- - -
Steffen Koehler is chief marketing officer, Scott Fairbairn is CEO, and Anthony Ogden is an intern at RocketStream, Inc. You can reach the authors by writing to Mr. Koehler at [email protected].