In-Depth
Q&A: Change Management Challenges
Keeping changes under control is vital to keeping business services available.
The one constant in an IT environment is change -- new features, new technology, updates to applications, you name it -- but keeping those changes from disrupting business operations is a major challenge.
In this question-and-answer exchange with Harish Rao, CEO of nSolutions, we explore the issues affecting IT and what you do to manage changes effectively.
Enterprise Systems: What are the issues facing change management?
Harish Rao: The challenge that enterprises face is to ensure continuous availability of business services across the entire physical and virtual IT environments consisting of network, server, storage, and applications. In many enterprises, any interruption of IT services, even for a fraction of a minute, puts the business at risk as the cost of unavailability is unacceptably high. The leading cause of IT service interruption can be attributed to a change in the system. Industry analysts have identified that 70 to 80 percent of service unavailability is due to problems resulting from infrastructure changes. IT must have complete visibility and control of change in the infrastructure, including change impact assessment, for planned changes and for changes resulting from incidents (forced outages).
Managing IT change has become quite a challenge. This challenge arises from an increase in infrastructure complexity in security requirements, VoIP, virtualization, outsourcing, consolidation, etc. Not only is the number of IT changes on the increase, but the frequency of change is rising, too: 250 access control list changes a week, 1500 changes a quarter, are not uncommon.
Most IT organizations are required to “do more with less,” resulting in a situation where managing change can only be achieved through automation and control in order to meet the requirements of business service delivery.
What are the factors IT has to deal with when making changes?
There are processes as well as technical and organizational factors that IT must contend with when properly managing change:
- From a technology point of view, IT does not have adequate visibility into the infrastructure. Typically, the knowledge of the infrastructure is poorly documented, incomplete and generally in the minds of a few “experts.” A logical first step is to have IT staff establish a configuration baseline of all the infrastructure resources -- physical, logical, or virtual. Once the baseline is established, change has to be detected and correlated by extracting the information from the deluge of data..
- From a process point of view, managing change combines the operational aspect of effecting change and the workflow aspect of change management. There are several excellent tools available for change management that handle the workflow and trouble- ticket process very well. However the data they rely on for effecting the operational changes (such as configuration changes) is often out of date or incorrect.
- From an organization point of view there are “IT silos” of operational control to contend with. These silos -- Windows group, UNIX administrators, network administrators, etc. -- create barriers for effective communication and data exchange that are vital for managing change. Security and compliance issues that cross these silos require correlation of data across all silo domains of control. The consequence is that organizations responsible for compliance are often unable to get the right information in time for assuring their objectives.
If changes are so complex, why are they still being done manually, as you mention? Aren’t there automated tools to help IT?
There are many point tools that are being used in individual silos. Many of these tools are based on writing scripts which are easy to deploy, but have inherent limitations in terms. Many products that have been introduced in the last couple of years have automated the scripting approach. They are mostly silo oriented and where there is some coordination across silos, it tends to be loosely implemented.
These tools do not address the fundamental problems of correlating data across domains from a business services perspective and their ability to assess the impact of a change -- before the change is effected -- is limited.
What is your view of the current solutions and their limitations? How have tool features and tool market changed?
Consider the network infrastructure as an example. Current tools are based on building a library of drivers for each class and type of network device. These drivers are based on scripts written in PERL, PYTHON, etc. When the network equipment vendors introduce new devices, there is a significant effort in writing new drivers for these devices. Technology is moving fast and several advances change the very behavior of the devices. A router today could incorporate the behaviors of a switch, firewall or even a server. This complicates the problem of adding drivers.
Another example is the challenge of managing change in a virtual server environment. Lack of visibility into the virtual structure is a limitation in many of today’s tools for managing change.
You’ve mentioned how visibility is a key to knowing how a proposed change will impact the organization. What new or emerging technologies make gaining this visibility more complicated?
A good example to illustrate this would be the manufacturing or retail environments. Global manufacturing organizations and multinational retail organizations have many locations and must be able to set operational and security policies globally and — manage them locally — for effective IT operations. Going across the WAN optimization devices or multiple firewalls also poses significant challenges to the visibility needed to mange change.
Another element of complication is the virtual environment. Virtualization of servers clearly has escalated the manageability problems and once again, if IT cannot ‘observe’ or reconstruct the system, they cannot manage it effectively.
Additionally, the emerging developments in PDAs will also create visibility challenges. These PDAs will undoubtedly have behaviors that might include additional networking functionality and introduce new change requirements and processes.
What should a good change management process include?
Managing change in real time requires three key components:
First, you need visibility into the infrastructure. This means that IT must be provided with deep configurational visibility into the physical, logical, and virtual resources of the infrastructure at the right level of granularity. The discovered data must be normalized and correlated and the resulting information should be stored in a real-time repository for operations.
Second, it needs to manage change by exception. Because of the sheer size of the infrastructure components and the associated change data, an important consideration is to manage the change by exception. This means that IT must set operational policies for managing change and violations to these policies are detected and managed in real time.
Finally, the process must be automated. Continuous compliance with the operational policies allows IT to operate in an ‘auto-pilot’ mode. This means that IT can manage change proactively and not have to react to incidents.
The net result is higher availability, lower error rate and lower operational costs which together contribute to better delivery of business services.
What products or services does nSolutions offer to help IT manage change?
nSolutions offers a product line of appliances called NOVA (Network Ontology and Virtualization Appliance). NOVA is a line of secure appliances that manages change in global heterogeneous infrastructures. NOVA appliances can scale globally because the appliances can be networked in a peer-to-peer or a hierarchical architecture. NOVA provides global visibility through a browser based application.
NOVA provides a single, unified, up-to-date view of the configuration state of the infrastructure.
NOVA processes auto-discovered information from physical, logical, and virtual resources to create a NOVA Repository with a virtual construct of configuration interdependencies among the components of core (routers, switches, firewalls, etc.) and the applications infrastructures (servers, databases, applications etc.). In addition, NOVA detects change in the infrastructure in real time and generates an alert if operational policies are violated; provides continuous compliance with operational and regulatory policies; and provides a proactive solution for managing change by provide Change Impact Assessment (CIA). I should point out that NOVA CIA provides a CIO with a tool that accounts for the service level agreements (SLA) goals for the business.