Could It Get Any Easier?
Talk.com Customers Subscribe and Pay Online for Long-Distance Phone Service
For consumers, the new appeal of competitive telecommunication and long-distance service providers is the pursuit of greater conveniences, even a means for cutting time spent on such ordinary "nuisances" as bill paying – especially when the convenience of online payment at Internet retail services already provides a highly successful comparison.
Talk.com is one of the largest long-distance carriers in the United States, yet what is newsworthy about this provider in quiet Reston, Va., compared to dozens and dozens of noisy competitors, is an unconventional technology application for customer billing and payment. Talk.com is the only company of its kind to have tackled back-office processes that contribute to stacks of paper bills arriving at our homes every month and provide, instead, online billing services via credit card to approximately 1.6 million customers.
Think about how this works for everyday customers – the public. One of the company’s largest partners to date is AOL, whose members need only to click and register online for long-distance service, and who, subsequently, receive bills on the Web and pay automatically and electronically through their credit card records already stored in the AOL database. Customers from other non-AOL distribution channels also benefit from online rating, billing and customer service.
The good news about this is a back-office billing revolution for Talk.com, one that slashed overhead costs for billing, collections and other customer interfaces an estimated 1.5 cents per customer, involving the processing of over two million call detail records per day.
Talk.com is a rapidly growing integrated communications provider marketing long-distance and, recently, local services to residential and small business customers. In the last two years, it began utilizing a proprietary "realtime" online billing and customer service platform, building heavily upon an HP infrastructure of application engines and database servers, as well as critical system support. Talk.com’s services are marketed directly, through various business arrangements in addition to AOL, which include such prominent entities as Prodigy, E*trade, First USA, DSI and Quintel marketing. In addition, it has gained accreditation to become a registrar of domain names on the Web.
What’s So Different About Online Billing?
Several years ago, corporate executives at Talk.com looked at the tremendous costs the company incurred associated with acquiring and billing customers. They decided there had to be better ways of, one, attracting customers to the service, and two, avoiding the costly, labor-intensive process of billing by mail.
"Here’s a long distance provider that recognized standard billing arrangements commonly used throughout the industry were long overdue for change," says Al Schraeder, HP’s Account Manager for Talk.com. "They were the first to look closely at e-commerce and the Internet as supporting technologies for feasible adaptations that provided the right foundation to appeal to Internet users and transform them into customers."
Using Web-enabled technologies and proprietary applications, the company saw ways to offer its customers online conveniences including:
• Detailed rate schedules and product and service-related information are easily accessible for customers online.
• Online sign-up for the company’s telecommunications services.
• Credit card billing that frees customers from having to write another check each month.
• Realtime and 24x7 billing services, customer service and online information.
• Up to the hour billing information. The cost for most calls is posted within minutes after completion of the call.
Another key difference in all this is that users have instant access to their call records and other call-related information, instead of waiting for a monthly phone bills to arrive with the regular mail. Instead of making calls oblivious to the costing structure, online rating schedules enable at-a-glance decisions about call initiation times. Round-the-clock customer service makes sense for many people nowadays who do their personal business online during peak service-call hours between 5:00 p.m. and midnight. Talk.com’s system saves lengthy phone waits for signup procedures and billing questions – customers just input their own signup information or research up to a year of call record history on their own time.
Horsepower and Architecture Behind the Transformation
As discussions of reinventing rating and billing went through the hallways at Talk.com, the challenge was how to deal with accumulating millions of process transactions per day. Talk.com reengineered its hardware and software infrastructure in response to higher levels of computer demand and availability. In order to store and process what now amounts to two terabytes of history contained for one billion call data records (CDRs), Talk.com has invested in hefty hardware engine and storage technology, and has tuned its software applications to exploit all that power.
Talk.com’s new computer infrastructure revolves around several hubs of superservers in the eastern U.S. – one in New Hope, Pa., and the other in Reston, Va. About 1,000 geographically distributed client systems located in such venues as Florida, home base for Talk.com’s customer service staff, round out the company’s IT hardware.
Executives’ view of hardware and storage costs is that these are relatively inexpensive assets, and as a result, the IT team has followed a strategy of installing sufficient power for the volume of data being transacted, and then maximizing that equipment to what managers there call "the nth level." In fact, a hallmark of Talk.com’s infrastructure is the high degree of process automation and parallelism that reduces operating costs.
For example, the IT group recognized that application databases of the scale used at Talk.com could potentially become a bottleneck. As such, the applications and databases were architected specifically to operate in parallel and significantly improve the overall performance of rating and billing processes. Fully leveraging hardware throughput is one of the most interesting aspects of how the infrastructure was tuned to Talk.com’s business. Because the company utilizes High End Parallel Databases (Informix), the company developers also needed to write software to operate in a parallel mode. Looking at this another way, if the application is single-threaded, then one processor provides maximum performance since one thread of a process runs on one CPU at a time. In this case, the database might be capable of more, but its capacity cannot be utilized and the application has become a bottleneck.
In contrast, most of the applications at Talk.com create from 20 to 40 child threads or processes upon invocation and attack the data at the same time. By creating applications from the ground up, the company is able to maximize the processing power of the HP multiprocessor machines.
It’s this very capability that enables Talk.com to process millions of call records daily, on a near realtime basis. As the company has had large growth in almost every year since inception, it was necessary to not only look at immediate needs, but at other forward-looking requirements on an ongoing basis. The general power and scalability of HP’s platform was instrumental in this respect, as was software compatibility across all of HP’s processors in the same family series.
This compatibility allowed the company to write software for requirements at the time and have assurance it would all be supported with tomorrow’s hardware. It was important for company executives to know that when HP releases newer members of its family of computers, the company IT team can immediately begin running software without investing time and effort into porting to a new processor. Moreover, it allows the company to take advantage of a hardware upgrade almost immediately, even as the IT group begins tweaking the solutions in order to get as much out of the new hardware as possible.
The heart of Talk.com’s computer infrastructure runs off an HP V2600 with 32 CPUs and 16 GB of RAM. It is based on 18 fiber-channel controllers running three each to six EMC units. Talk.com’s 500 to 700 customer service staff is also using the system to support its 24x7 operations. Instead of hosting them on an application server, they run client/server with an application the company developed that connects their Windows box to the database on the V-class, and does the queries against the V. The V2600 is backed up by a V2250 in a service guard configuration to ensure 24x7 operations 365 days a year. This system runs a 1.5 Terabyte Informix database.
The billing system was envisioned to support heavy transaction flows. In actual use, this is proven to be greater than two million call detail records per day and necessitates industrial-strength processing hardware. A pair of HP 9000 K570 servers support realtime call detail through the Talk.com Web application. Several billing cycles accommodate 400,000 to 450,000 site hits per day. The HP K570s store all the billing CDRs – an average record approaches 1KB in size – as well as unbilled call records, which appear within 30 minutes of a given call.
It’s no surprise, then, that Talk.com’s storage systems must transact, process and make available two Tbytes of data without the applications themselves becoming bottlenecks. The more that companies buy into massive parallel systems like the V-class, with 32 CPUs, the greater the need for a database that can parallelize queries and take advantage of the parallel architecture in the SMP world. The IT group believes it is the only one of its kind that also parallelizes their applications so that the database can get maximum throughput. It doesn’t make sense for databases to be able to return 10,000 records/second if the application can only take and handle two.
Talk.com’s parallelized applications includes three components:
1. The rater, that rates at least 2 million calls/day and could rate more.
2. The billing applications, which host multiple billing cycles within a day, along with all the permutations of monthly calculations per customer to be posted online. That application also needs to distinguish which records shouldn’t be billed again. That means the company must be handling roughly 6 million call records a day in a transaction.
3. The sending of e-mails to communicate with customers, and directly with the RBOCs. Talk.com is connected by direct circuits, including T1 lines, to every RBOC in the country and communicates with them each day. Additionally, the company will send as many as one million e-mails in a day, demonstrating the system’s large volume capacity.
HP servers are also supporting associated processes within Talk.com’s IT infrastructure. Talk.com’s network administrators use an HP 9000 N4000 server with four CPUs and four GB of memory to generate network statistics and analyze network traffic patterns and performance. For example, administrators can see how many people are calling from Los Angeles to New York City at a given time, and use that information to decide if they should put in more circuits.
Talk.com has another HP N-class server, in the same configuration, running billing statistics, such as the average number of months paid on the service, a customer’s average bill, or the ratio of international vs. interstate vs. intrastate calls. This type of data helps Talk.com answer key questions such as, "What’s the impact going to be if we lower our rate five cents/minute to try and get more customers? What would we lose, and what would we gain?"
With all these innovative services available today, where is Talk.com headed? On the business side, Talk.com’s long-distance will stay entrenched in the consumer market, eyeing what it believes to be 100 million potential subscribers. Company planners think the natural bundle going forward is long distance, local and dial-up ISP for the residential customer.
The impact on technology is for continually expanding power to support a similar trend in CDR volume. Company executives expect to increase the number of EMC storage solutions being used from six to eighteen, as terabytes of data resources grow from two to eight. This gives Talk.com even more reason to continue stretching and exploiting its HP hardware and storage systems to the max.
– Greg Wood is CIO of Talk.com. He has been with the company for five years.