In-Depth
Case Study: Patching the SAFE Federal Credit Union
The need to automate time-consuming, manual patch processes drove one financial institution to adopt patch management software.
Some organizational names carry inherent assumptions. Just ask the Sacramento Air Force Employees (SAFE) Federal Credit Union in California, which maintains assets of $1.2 billion for its roughly 120,000 members.
“With a name like SAFE, it implies certain things to our members, and we pride ourselves on protecting our member data and keeping our customers’ data safe,” says Marc Buzard, the network services manager for SAFE. “So we’re always looking at what we can do better. We’re on top of the patching, monitoring for various vulnerabilities, looking at things internally, seeing what it makes sense to do. It’s just part of our culture.”
Part of doing things better means finding ways to improve existing processes, and at SAFE that often that means finding ways to transition from time-intensive, manual processes to automated ones.
The credit union isn’t alone. According to “Best Practices in Security: Network and Infrastructure,” a report written by Jim Hurley, vice president of research for security, compliance, and risk management for Boston-based AberdeenGroup Inc., leading companies are evaluating which security processes most merit automation. The goal is to get the most bang for security-improvement buck, plus make better use of an always-scarce resource: security staff time.
Yet not everything can, or should, be automated. “Business pressures dictate where automation throw-weight is most useful,” notes Hurley. “For example, automation is frequently cited as the key contributor for business procedures that involve large numbers of people, systems, and segmented networks,” among other things.
Automating Patch Management
Such concerns drove SAFE to find an alternative to manually patching its 450 Windows and Macintosh computers and 110 Windows and Unix servers located at 15 different branches. Simply put, managing operating-system and application upgrades, plus the installation of security patches, took an inordinate amount of time.
Furthermore, the company faced the prospect of somehow scaling these time-consuming, manual-patch processes. “We were starting to grow, and realized it just wasn’t cost-effective to have one or two people running around to all the branches at the time and manually updating them with all the latest patches,” says Buzard. Also, insuring the credit union just kept up with new patches drove automated patching, since at the time, “Microsoft was putting them out left and right,” and SAFE is a Microsoft shop.
About two years ago, SAFE investigated products from Ecora Software Corp. and PatchLink Corp., plus Systems Management Server (SMS) from Microsoft, and the Windows Update functionality built into Windows. SAFE dinged SMS for being too expensive, and Windows Update over of its lack of reporting and inability to schedule patch installations.
Ultimately, the credit union selected PatchLink’s Update product, especially for its particular Web-based interface and reporting capabilities. “One of the key things we liked about PatchLink was its centralized reporting: the ability to run a report and see which machines had been patched,” notes Buzard.
After a straightforward implementation, including installing the requisite agents on all computers to be patched by PatchLink, the product has functioned as expected. When needed, he says PatchLink’s technical support staff has fielded questions quickly. “With any kind of product you always run into issues, but most of the issues we’ve run into are learning curves.”
Simplifying Patch Processes
With automated patch management now deployed, SAFE quickly realized the “feature” it most desired: “We didn’t have to run out to each branch to patch,” notes Buzard. This, of course, saves valuable IT time. Indeed, while two people previously handled patching, now it’s just one person’s job, and “it’s not even a full-time job for him.”
The credit union patches on a fixed schedule (barring any unexpected, critical updates) based largely on Microsoft’s patch-release schedule—new patches get released on the second Tuesday of every month. The next day, a SAFE IT team meets to plan what needs patching and in what order. (Internet-facing servers typically go first.)
The credit union uses PatchLink to update all of its Windows, Macintosh, and Linux computers when possible. “Some Linux boxes are running a stripped-down version of Linux provided by one of our vendors, so there are some ‘support issues,’” notes Buzard, meaning patching the box might produce a version of Linux the vendor doesn’t support. In such cases, “we do follow up with the vendor to make sure they’re up to date, as far as deploying patches.”
The Patch-Testing Imperative
Before distributing any patch, however, SAFE first thoroughly vets it. “Because we have a test bed we can deploy on and test on, it saves us a lot of downtime,” Buzard says. All patches are tested against the array of front- and back-office applications typical to the credit union. “You walk through the basics, just make sure everything seems to work correctly, there’s still Internet connectivity, applications launch and behave as they’re supposed to, and merely applying the patch doesn’t crash the machine.”
Then the credit union rolls the patch out to just a branch or two at a time—just in case. One particular worry: even with extensive testing, on a certain make and model of machine the credit union uses, applying a patch can sometimes result, after rebooting, in a blue screen. “We’ve gotten to the point where it doesn’t happen often, and when it does we know how to get around it,” notes Buzard. For example, “this last go-round, we had two machines out of 450 that exhibited a blue screen.” Still, it’s one more reason to play it safe and deploy patches gradually.
In the near future, Buzard might need to extend patching capabilities to devices that remotely connect to the LAN, and so may tap a new PatchLink product able to update remote or wireless devices. “We’re just now staring to deploy some wireless solutions, so we may be wanting to look at that in the future—to patch those devices remotely as well.”
Related Articles:
About the Author
Mathew Schwartz is a Contributing Editor for Enterprise Systems and is its Security Strategies column, as well as being a long-time contributor to the company's print publications. Mr. Schwartz is also a security and technology freelance writer.