Capacity Planning in Distributed Environments, Part 2

In the conclusion of his two-part article, third-party, capacity planning offerings, the cost-effectiveness of distributed computing and controlling "service levels" of projected workloads are explored.

• 02/01/1999

Capacity Planning efforts today must focus on the multi-system aspect of distributed systems. Figure 1 above is meant to illustrate how mainframe based systems evolved to client/server architectures. The diagonal line depicts the network; functions below the network line are functions that are performed on a client; functions above the line are performed on a server. The evolution shows how distributed systems have emerged by placing more emphasis on the client as it became more intelligent. The term Remote Management refers to managing the data and processing resources that exist at a remote server. The term "distributed" is used to indicate that a key function (e.g., "presentation" or "processing" or "data management") is performed on several machines. Thus Distributed Presentation refers to systems where the client has enough intelligence to offload some presentation functions; Distributed Processing implies that the client can offload presentation functionality and some of the processing chores; and Distributed Management indicates that the actual management of data occurs on both clients and servers.

Today, we have the intelligent client that can perform some significant functions on its' own; we have the network, whose bandwidth can define the system-wide performance; and we have the server (or servers) that often contain the data being sought for analysis and presentation on the client. This is a multi-tiered architecture, and is fundamentally different from old mainframe systems in that (1) there were no intelligent clients, only dumb terminals, (2) the network only connected machines of the same type, all using the same communication protocol, and (3) the one single server contained all of the data necessary on it and it alone.

Today's systems have a great challenge to conquer: they often are asked to pass information along in a three-tiered computing architecture: data often resides on a large mainframe; midsize machines often house smaller, but key, databases; and even smaller "client/server" machines often characterize a LAN. Capacity planners must face this reality of multiple systems connected via one or more networks, where the systems are of different sizes, running different operating systems and containing different database management systems. Again, the key to capacity planning must be to deliver useful information in a cost-effective way. Thus, the capacity planner must examine how applications will be using these different tiers. Common questions would include (a) getting many small servers vs. few large servers, and (b) where should specific processing functionality reside: on the client or on a server?

And let's not lose sight of networks! The sensitivity of the backbone network of multi-system configurations could easily be the single resource that dominates poor service by applications. The network bandwidth and speed is sensitive to choice of protocols. If an organization is considering using a Wide-Area Network, this often will require some study - perhaps involving a network simulation! The available/useful bandwidth of the public network, too, is extremely sensitive - and is less controllable! Router capacities must be carefully considered as well. A common performance/validation technique is to place a sniffer at some point on the network to collect network traffic statistics from which response times can be calculated.

Presumably, a great advantage of distributed systems is being able to buy needed capacity in small increments. Small systems have exhibited this characteristic; this allows a more accurate sizing of necessary equipment to business needs. All of this translates into less excess capacity and therefore lower overall cost - or so one might think.

As we've said before, scalability is key for capacity planning for distributed systems. Economically, scalability is a primary contributor to reduced cost. The theory is you buy enough capacity to do your processing now, and if additional capacity is required in the future, it is acquired at a reduced unit cost because of the constant improvement in price/performance ratios.

Is there a fallacy in this thinking? Consider what happens during the entire life cycle of equipment - especially client/server equipment. You buy what you need today - and incur both acquisition and installation costs. Over time, you also incur an operational costs (licensing fees, support personnel and maintenance). But is that all of the costs? What happens when additional capacity is needed (i.e., one server needs more capacity). Yes, you acquire a bigger server, but what do you do with that old server? Do you throw it out? Most companies would rollover the server to a new place - that is, the old server is likely to replace an even smaller machine somewhere else in the organization. And as this may cause a cascading effect, consider, too, that there are costs that have to be incurred when installing each old machine in a new place (e.g., installing new software, testing, support personnel costs, etc).

In the mainframe environment, these rollover costs were seldom encountered. Processors were generally exchanged or added. But in the distributed environment, a processor swap can cause multiple rollovers. If the rollover costs become significant, it may become uneconomic at some point to do the rollover! Leilani Allen at a meeting of the Financial Management for Data Processing Association said "By the year 2000, it will cost more to keep old technology than to upgrade." (Allen, Leilani E., Ph.D., I/T Investment in the Year 2000, FMDPA Annual Conference, May 1994)

While this may be surprising, the situation does point out that we should understand the actual magnitude of rollover cost in building a financial model over the life span of a distributed system. James Cook (Cook, James R., Should You Drive a Used Car on the Information Superhighway?, Proceedings of the Computer Measurement Group, 1994, Orlando, Fla.) proposes such a financial model; namely that the life cycle cost equals the initial acquisition cost, plus the operational cost, plus N times the installation cost, where N is the total number of swaps in the rollover series. The net impact of a first processor swap may cause an increase of nearly 50 percent to the original acquisition cost, 100 percent for the second swap, and 150 percent for the third swap! Note, too, that at some point, rollover costs will consume any savings that may be gained on cheaper MIPS being available in the future. Thus, spending a little more initially on capacity may actually avoid a processor swap (and its costs) in the future.

The bottom line: The focus of financial management strategies for IT has long been on acquisition. But the realities of the life cycle of equipment in distributed systems dictate that ongoing operational costs (that address rollover) demand more attention. Old, conventional wisdom just doesn't apply to distributed systems.

The capacity planner should attempt to base a capacity plan on projected workloads that each receive an acceptable degree of service - service levels. Workloads should, ideally, be specified in business terms - sometimes referred to as natural forecast units or business transactions. One advantage of basing the capacity plan on business transactions is that a clear correlation exists between the capacity plan and the business plan, and should the business plan change, the effect on the capacity plan will be immediately obvious. Remember, the focus of the capacity planner should always be on business needs.

But before service levels can be managed, they have to be specified and agreed on. Application designers, capacity planners and performance analysts need to agree on the answers to questions like:

• What is the relative importance of each application to the business?
• What is the required availability of each application?
• What is the required performance (i.e., response time) of each application?
• What are the current workload volumes for each application?
• Are the workload volumes or relative importance expected to change over time? If so, how?
• Are new applications planned? What are their relative importance and workload volumes?
• What financial constraints exist on the acquisition of computer resources?
• Can IT costs be charged back to the business units that use the applications?

Service level measures (specifically availability), response times and workload volumes, should be reported by business unit and application. When compared to service level requirements, we can see whether goals are being met, and, based on priority, determining that impact on the business. Re-planning capacity requirements may be necessary if the stated service level requirements are adversely affecting the business.

But there are several obstacles to hurdle before managing service levels across distributed environments. The nature of client/server applications is such that the application architecture distributes some portion of its processing to the client. This introduces additional points of potential failure or bottleneck, and also complicates the definition of a transaction. In the network, heterogeneous systems coexist using different communication protocols and application-to-application session protocols (e.g., SNA APPC, OSF's DCE, etc.). Thus, managing end-to-end service levels becomes very complex because of the different computing and communications device in a path. And there is a relative lack of maturity, as compared with the mainframe, of measurement, monitoring and applications management tools - but the technology problems in building similar tools are different for distributed systems. There is hope - base measurement data, like utilization and traffic, is available from most intelligent networked devices. Standards have been proposed for system performance data across platforms. Agents have been and still are being developed for database instrumentation.

The problem remains one of intelligently correlating the various component measurements into an end-to-end picture of the service provided to application users. To obtain performance at an individual transaction level will require code instrumentation within the application. Application-specific measurements would be collected and fed to an enterprise manager, which would allow the end-to-end correlation of performance.

HP's MeasureWare Agent Transaction Tracker is an example of a tool that can be used to manage IT service levels. User-defined applications enable you to group processes together by user or application name for better tracking of an individual's or application's impact on your resources. MeasureWare Agent is consistent across multiple platforms - HP-UX, Sun SPARC, IBM AIX Bull and NCR WorldMark - using a common interface and standard metrics.

To create application specific measurements, the joint efforts of HP and Tivoli have produced ARM - Application Response Measurement. ARM is an API/software developer's kit that allows application developers to add appropriate performance instrumentation to their code easily. Applications deployed using C/C++, Microsoft Visual Basic, MicroFocus COBOL and Delphi are currently supported under all of the popular PC platforms.

Summarizing, we note that as distributed applications are deployed, and as management responsibilities for these distributed applications shift to the central IT organization, service level management becomes increasingly important, complex and difficult. Many of the management tools required to effectively manage service levels across the enterprise are now first coming into their own. The first step toward implementing enterprise service level management is to approach systems management from the end-to-end application workload perspective, rather than viewing the environment as a collection of physical components.

Capacity Planners have some very critical questions to consider, including:

• What type of servers should be deployed to support specific applications?
• How many servers are needed? How much bandwidth does the network need to provide acceptable service levels?
• How big should the servers be to handle the application volume?
• Can we define service levels for different distributed applications?
• How can we tell if the servers are optimized for the network operating system and communications software that is in place?
• Are applications properly configured for a mixed-platform environment?
• Are appropriate measurement sources available?
• What are the true costs of incremental additions to capacity?
• Is effective service level management possible for distributed systems?

Capacity management for distributed systems faces many challenges. The one single driving challenge is to constantly ask whether Capacity Planning is helping IT deliver the best service possible to its customers, now and in the future. IT managers must always address building scaleable architectures that have market-driven compatibility. Evaluating a "checkoff" item capacity planning philosophy for your environment may prove to be a real cost saver - especially when taking into account the cost of maintaining a full-time capacity planning staff. And we shouldn't forget to include the true costs associated with adding incremental capacity.

Instrumentation across platforms is key - especially for new applications. Applications should be designed from the outset with the goal of being able to provide application-specific performance measurements so that we understand the quality of service delivered. Without application-level instrumentation, service-level management across the enterprise may not be possible.

Data reduction/summarization/reporting software will be required to manage the volume of customer data across platforms, and must address the network, as well as the client and the server. Modeling tools must become graphical to allow network topologies to be easily defined, and must then address modeling heterogeneous combinations of hardware and system software platforms. Modeling should be able to provide predictions of IT usage and service from a global perspective, as well as a detailed focused perspective.

It is our hope and desire to see these challenges addressed in the near future with the development and application of new techniques.