In-Depth

Q&A: Virtualization and Rigorous Testing

We examine how virtualization can benefit developers, the drawbacks of the technology when it comes to testing, and best practices developers can use to exploit the capabilities of virtualization.

• By James E. Powell
• 10/28/2008

There is more to virtualization that just saving floor space or power use in the data center. IT developers can profit from virtualization in many phases of a project’s lifecycle. Dennis Powell, Senior Product Manager, StackSafe Inc. shares his perspective on virtualization’s benefits when it comes to testing, the technology’s drawbacks, and best practices IT should follow to exploit virtualization’s capabilities.

Enterprise Systems: Virtualization has been around for years. How is its use different today than, say, 10 years ago?

The past decade has seen a lot of different concepts, ideas, and tools put forth to support this technology we call virtualization. In fairness, I should acknowledge IBM as a forerunner in the area of virtual machine development well before the past decade -- think Big Iron logical partitions (LPARs).

If I were to put my finger on the primary changes to virtualization over the past decade, I'd point to three things:

First, virtualization as a technology has advanced from primarily open source projects to a highly commercialized technology. From the late 90s forward, we've seen the introduction of a variety of virtualization technologies such as the FreeMWare project which yielded today's Plex86 (an open source Linux VM), Wine, a non-Microsoft alternative Windows API for running Windows programs on Linux, Solaris, Mac and FreeBSD, and DOSEMU to run DOS on Linux. I guess you could also point to the plethora of commercial mergers and acquisitions, for example EMC, Citrix, and Novell respectively acquiring VMware, Xensource, and Platespin as proof that virtualization is viewed as a strategic play and a foundation technology.

Second, the hype cycle for virtualization is peaking. Much of the current escalation in the industry to adopt virtualization is, in my opinion, being driven by society and in turn IT industry's focus on green computing and the need to save energy in the data center. Virtualization is a proven value in terms of server consolidation, energy footprint reduction, and reduced floor space. The commercial success of VMware, Xensource, and Parallels (MAC virtualization) is undisputable. Couple this with a pending hypervisor-based competition between Microsoft and these vendors, particularly VMware, and what you have is a considerable number of IT organizations needing to get smart on virtualization quickly.

Third, while many virtualization users still see the value of consolidation (and resultant cost savings) as the primary driver to deploy virtualization, we're seeing a “second generation” of virtualization use. This includes a growing focus on virtualization as a production platform for mission-critical systems, the growth in tools to help manage the virtual production environment, and the use of virtualization as an enabling technology for change management staging and testing platforms.

How does virtualization impact the way IT tests, particularly at each stage of testing (e.g., unit testing, system testing, etc.)?

The impact of virtualization on IT testing depends on the source of, and deployment target for, tested components, and the development and testing methodology and approach that an IT organization deploys.

Virtualization's strongest benefit to IT testers occurs when it is used to establish staging environments for testing. Virtualization is of more benefit to IT during the development, unit testing, and QA phases of the component lifecycle than during the systems testing phase. Virtualization’s benefits also vary according to the type of application for which IT is responsible. On the other hand, virtualization increases IT's testing workload when it is the target of the test, meaning when IT needs to test virtualized production systems.

Most organizations establish separate environments for development, unit tests, QA, and pre-production (systems and user acceptance) testing. During the development and unit test phase, testing focuses on features and functions of the individual component being developed or updated. During the QA cycle, the component is often tested against a subset of infrastructure components to confirm stress, load, and performance. As you can imagine, the challenge to understand the impact of a systems change increases as the number of pieces, parts, and inter-dependencies increase. With each successive testing phase, it becomes more difficult to maintain configuration compatibility between the pre-production system and the production system. The intent of systems testing and user acceptance testing (UAT) is to understand the impact of new components, upgrades, and changes to a full pre-production environment.

Creating staging environments for these phases requires significantly more time, effort, and budget to provision, build, and maintain a representation of production. It is simply easier and more accurate for an IT group to import software images of physical servers into a virtual staging environment than it is to manually provision, modify, and synchronize a pre-production staging environment to ”simulate” the physical production infrastructure. Importing physical server images into the virtual sandbox also provides certainty that the imported software image represents what is running in production, and gives IT a greater level of confidence in the results of tests.

If mistakes are made during testing, if a pre-production configuration is corrupted, or if IT needs to recreate a baseline configuration to complete a more complex test, it saves time and improves IT confidence in results to re-import a production image, rather than to separately configure a pre-production environment to match production.

Certain application types lend themselves to virtualized staging and testing, too. Large, diverse systems such as ERPs, transaction processing systems, and custom (home grown) systems run on non-x86 platforms, require large disk-space requirements (terabyte databases, for example), and introduce significant dependencies with other components and platforms (zSeries, iSeries…) that aren't suitable for a virtualized testing environment. Web-site (hosted) applications, CRMs, and Linux and Windows infrastructures, can be completely configured for virtual imports.

Virtual staging platforms are better suited to support testing during development (TDD), unit testing, and QA testing because the interdependencies are fewer, but user acceptance testing (UAT) and systems testing are more difficult to stage in their entirety, creating the potential for gaps in testing. Of course, if you are developing and testing systems in a virtual sandbox for the purpose of deploying the developed components into a virtual production environment, there are fewer external interdependencies to take into consideration, so the test coverage should be more complete. The difficulty with testing virtual production systems is caused by the lack of maturity with processes and testing tools for the virtual environment.

What are the benefits of virtualization during testing?

Testing organizations have traditionally struggled with finding the time and effort needed to perform effective pre-production staging and testing. That statement is backed up by research showing that nearly 25 percent of all changes put into production cause unexpected impact, and approximately 10 percent of all changes to production must be rolled back because they cause problems that can't be resolved. Here are some of the benefits of virtualized testing that I see:

• Helping IT quickly build and maintain representative staging environments is a significant timesaver. IT can spend time managing the test process rather than creating pre-production components.
• Helping IT consolidate its staging environment to multiple virtual machines (VMs) running on a single or small group of physical servers is a significant cost-savings benefit both in terms of reduced hardware costs and fewer resources needed to set up, manage, and tear down the testing infrastructure.
• Being able to run automated tests against VMs on a virtual network improves end-to-end testing and leverages existing test investments.
• Using virtualization to capture a representative image of a production server that is imported into the virtual staging sandbox gives IT confidence that they are testing against the software image that actually exists in production.
• Re-importing a physical server software image or a virtual workload to ensure configuration synchronization between production and pre-production environments reduces the time to maintain the staging environment, and also increases confidence that pre-production represents production.
• Establishing a staging lifecycle in which VM workloads can be promoted (imported) between virtual stages is less time-consuming than promoting system components between physical staging environments.
• Being able to virtualize infrastructure middleware, such as an application server farm, increases the flexibility and dynamic allocation of this component of the infrastructure when conducting systems testing and UAT.
• Virtualization supports different departments, applications, users, and teams from a single staging environment.
• Especially when testing virtual components for virtual production environments, being able to certify component readiness because the virtual staging and testing platform is similar to the virtual product infrastructure

What are the challenges of virtualization during testing?

When testing applications and systems with inter-dependencies to components sitting outside of the virtual staging platform, it can be difficult to configure a representative “hybrid” staging and testing environment, including virtual-to-physical network communication. It can increase the overall testing workload when IT needs to validate test scripts that must run across the virtual and physical network before they are complete. It adds complexity when invoking one system layer in the virtual sandbox that must verify a process at a different layer of technology outside of the sandbox. Other testing roadblocks must be overcome:

• Simulating network device behavior
• Simulating SOA service interfaces to external services
• Simulating load and volume that must be tested between virtual and non-virtual components
• Accurately predicting roundtrip transaction performance
• Performing hardware tests (driver behavior, for example)
• Simulating a service to which an IT organization may have limited access (such as a for-fee service)
• Controlling user (tester) access to only certain systems and data sources within a virtual pre-production environment
• Validating a virtualized system service against remote system services not owned by IT

Virtualization in many cases has been introduced into an organization by the business unit without providing IT the opportunity to evaluate, learn, and establish effective controls over such areas as system configuration, security, and change management.

When it comes to testing, for example, many traditional testing tools used by QA have not yet been optimized to run in a virtual staging environment.

Virtual production system security compliance and threat prevention is immature. Virtual production servers that deploy a virtual production environment need to be tested for vulnerabilities outside of the individual VM operating system layer and at the individual VM operating system level. This ensures that intruders to the virtual production server cannot access all VMs on the server if they gain access to the server itself.

How has IT been addressing these challenges?

IT today is limiting virtual testing to proven benefits: IT can quickly build testing and staging environments for x86-based environments; consolidate physical test lab servers to a series of VMs on a single physical server; or quickly clone physical and virtual machines. Most organizations license commercial solutions. Nearly all Fortune 1000 organizations own VMware to construct virtual development and QA testing labs. IT groups are licensing virtual lab management products to assist them with the development-through-QA phases of the system lifecycle.

The challenges noted above are currently preventing IT organizations from implementing fully virtualized testing and staging platforms and unless a solution is developed to virtualize all platforms and operating systems, there will always be a reliance on physical as well as virtual testing and staging platforms. Note that a number of second-generation independent software vendors are hard at work developing solutions for a number of challenges listed above.

What best practices can IT follow when testing in a virtualized environment?

IT should exploit the capabilities that virtualization provides to support best-practice guidelines for maximum testing rigor and maturity. I recommend IT:

• Use virtualization to establish development, unit test, QA, and pre-production sandboxes when developing systems for virtual production environments
• Use virtualization to establish x86-based staging environments and use existing import technologies to clone representative images of the physical production software infrastructure stack within the virtual staging environment
• Test all changes in the virtual staging and testing sandbox that are to be deployed to production
• Test the change across as much of the end-to-end infrastructure as can be established within the virtual sandbox and connected to across the physical test lab
• Schedule changes on a regular basis rather than making changes on an ad-hoc basis