In-Depth

Case Study: Software Quality Controls Keep Game Site Purring

Online gaming company brings software development lifecycle under control

Online game companies might look like fun from the outside, but the software development lifecycle must be highly businesslike. To stay in business, gaming companies need to maintain close to 100 percent uptime, produce new games and features at a rapid rate, and meet some of the same compliance requirements as banks.

WorldWinner is an online gaming provider that hosts more than 10 million online skill games, awarding millions of dollars in prizes every month. WorldWinner vice president and CIO Joe Bai summarizes his challenge this way: “It’s like banking, but with a [different] user interface. … Money changes hands, but you have to play a game to do it. A lot of the constraints and business objectives associated with a banking-like service exist here. … Our users think of us as purely entertainment, but we can’t act like fun and games.”

In March 2006, WorldWinner was acquired by competitor FUN Technologies in a $23 million deal. The combined company is one of the world's leading online casual gaming providers. It partners with some of the largest game sites on the Internet, including Yahoo!, Lycos, Excite, Shockwave.com, and iWon, which support more than 30 million visitors combined who compete daily for prizes.

At a gaming site, any downtime can result in significant lost revenue and damaged reputations. Keeping applications up and running over multiple platforms and infrastructures, with hundreds of thousands of variables, is a complex job. Even the smallest software change that isn’t thoroughly tracked and tested before release can bring the entire system down. Poor application performance can result in slow page loads that frustrate users and drive them to other sites.

Several years before its acquisition by FUN Technologies, WorldWinner faced a significant business problem, one that Bai bluntly summarizes as “We had unsuccessful product launches [and] pushed things out that didn’t work in production.”

Bai joined the company as CIO in early 2003 to address that very problem, by making what he calls “a cultural shift in the development environment” from the WorldWinner’s start-up attitude. For example, he found the gaming firm’s application development infrastructure laden with problems. A configuration error in one part of the lifecycle would ripple throughout the process, affecting service levels throughout the system.

WorldWinner was using a method familiar to other startup software development shops—a senior systems administrator mentally kept track of what was happening throughout the software lifecycle. As the company grew, that method became increasingly untenable. Bai recalls an instance shortly after he joined WorldWinner when the system went down for three hours. “Three hours is a very appreciable amount of revenue,” he says. The downtime resulted, he says, because WorldWinner’s small development staff “pushed out a release that was developed and QA’d on a system configuration that did not map to production.”

To solve its software quality issues, Bai wanted better control of WorldWinner’s applications, Web infrastructure, and overall IT environment. He needed to redesign the deployment process to support parallel development in both pre-production and production environments. He also wanted to manage configuration files without adding IT staff, to improve application uptime, and to boost speed to market with new games and features—key to maintaining and adding customers in the gaming world.

WorldWinner chose a configuration, change, and release management solution from mValent called Integrity 4.0 that provides an automated, repeatable, process-driven approach to application infrastructure configuration management. Integrity includes features for configuration, change, and release management, and audit and compliance initiatives.

When Bai first suggested purchasing a software configuration management tool, he met predictable resistance. “I had a couple of senior sys admins who said they could write something like that,” he relates. “But it’s not the core competency of the company. We do games; we don’t write infrastructure management tools.”

Based partly on the fact that Bai knew several executives at mValent, and on his quick survey of the change and configuration management market six months earlier, he made his decision, partly driven by time. “When it came time to do what I needed to do at WorldWinner,” he says frankly, “I didn’t have time to do many analytics of feature sets [and so forth]. … I knew what I knew, and I knew that I had already taken a three-hour outage because of this.”

One immediately measurable benefit from the move to the mValent solution was improved uptime. “Before I started with [WorldWinner], they struggled to make 98 percent availability, partly because of long service outages for monthly maintenance.” Now, Bai says, actual uptime is close to 100 percent, and scaled against feature availability, and the company routinely hits “99.8-plus” in uptime.

“Just having all the pages up isn’t enough,” he notes. “It’s having every single feature of every product line up at the same time that gets me to 99.99 percent availability.” That continues to be his goal—uptime plus the constant addition of new features that keep gaming aficionados coming back.

WorldWinner was already using version-control and requirements-tracking systems, and the impact of mValent on developers has been minimal, Bai observes. “The software is very intelligent and non-intrusive from their point of view.”

Now, when a WorldWinner or FUN partner asks for a feature, “the whole process is incredibly predictable,” Bai says. “At no point can we drop or lose someone’s deliverable.” Once a feature is in the system, it takes on a life of its own. It’s tied to a release, tied to a date—it all just happens now.”

The mValent system gives management lots of control as well. “It’s so much easier from my point of view. … I just jump into the requirements system.” From there, Bai can view dates, release contents, and related work estimates. “I can tell which people are spending what amounts of time on which features associated with which release.”

By bringing software development under control, the product has also helped with compliance issues. Bai’s first external audit with WorldWinner took seven and a half months on the systems side, he says. The most recent audit before the acquisition, with mValent in place, took just under 20 business days.

Although mValent was partly responsible for that, the shift is also indicative of a change in philosophy in dealing with systems infrastructure, Bai says. “I have to, at all points in time, know what I’m running in every environment, how it got there, who wrote it, why they wrote it, and who authorized it. … That whole logical string has to make sense all the way into production, and mValent is a very important piece of that.”

Just as the product release cycle gradually changed at WorldWinner with Bai on board, he is now several months into the same process with the new, merged company. “With the acquisition, we’re making plans to start rolling [Integrity] out to different infrastructures and platforms.”

It will be a challenge, since the merged company is split between the east and west coasts in terms of development environments and languages. The east coast division runs Linux, Pearl, and Apache, among other things; the west coast is primarily a Windows .ASP, and SQL Server shop.

For now, only the east coast is running Integrity, Bai says, and not yet at the level he would like. With the acquisition, he plans to extend Integrity to the entire company.

Must Read Articles