In-Depth
Job Scheduling: The Three Generations
In the first of a three-part series, we outline the evolution of workload automation technology.
by Mike Gilbert
We inhabit a world in which everything we need seems to be available on the Internet, so we might suppose that no one uses batch scheduling anymore, right?
Wrong. The market for batch job scheduling software is growing at about 6 percent compound annual growth rate. This growth is driven by specialist systems management vendors who, alongside BMC Software, CA, and IBM, are competing in a vibrant market for workload automation, the latest generation of job scheduling software. Job scheduling and workload automation are at the very heart of today’s real-time enterprises.
Technology advances in Microsoft Windows Server and in the underlying multicore, multiprocessor architectures being developed by Intel and Advanced Micro Devices have combined to provide a Windows Server platform that is capable of handling workloads equivalent to a 4000-MIPS mainframe computer (see note 1). Enterprise-level organizations that install the Windows Server platform need job scheduling tools to manage such large processing capacity effectively.
This is the first instalment of a three-part series based on an original report (see note 2) that compared mainframe-based and distributed-based job schedulers and to assess the readiness of Microsoft Windows-based job scheduling technology to manage workloads at the enterprise level. This article can also help IT organizations that plan to migrate mainframe workloads to the Windows Server platform as part of a legacy modernization initiative.
In this instalment we outline the evolution of workload automation technology. Part 2 will provide an analysis of job scheduling capabilities on both mainframes and distributed platforms based on detailed comparisons of a selection of products targeting both platforms. Part 3 concludes the series with a look at JCL emulation products and a summary of the status of job scheduling products for the Windows platform.
Job Scheduling Overview
We begin our discussion by charting the progress of job scheduling through three generations: batch processing, workload management, and workload automation. (Note that different vendors use different terms to describe their products and features.)
First-Generation Job Scheduling: Batch Processing
Scheduling computing tasks by means of batch processing remains the backbone of most mainframe-based IT operations. In batch processing, jobs are scheduled to optimize use of costly computing resources. One of the primary goals of early batch job scheduling was to keep CPU usage as close to 100 percent as possible, night and day. Today, however, operations managers prefer to keep processing capacity in reserve to handle peak demands.
The feature baseline for job scheduling products includes automatic restart and recovery, file management, integration with security systems, operator alerts, service classes, spooling devices, and high throughput and availability.
Second-Generation Job Scheduling: Workload Management
The business demands for information processing continually increase in volume and complexity. These needs have placed great burdens on operations staff. Organizations have diverse application workloads to process, which may include packaged applications, Web applications, several different platforms, and significant integration of operations across business functions.
This change has driven the second generation of job scheduling features, called workload management. Workload management provides the capability to manage and analyze workloads spread across diverse platforms from a central point of control. It provides functions that define processing priorities by business deadlines (calendar scheduling) and by cross-functional dependencies.
Workload management, which builds on basic batch processing features, is achieved through richer scripting features, more sophisticated scheduling engines, and multi-system, cross-platform workload-balancing capabilities.
Third-Generation Job Scheduling: Workload Automation
The most recent advances in job scheduling have been driven by the broader integration needs that have arisen from Internet-based business activities. To compete in their own markets, organizations have been forced to rethink the way they do business. Customers want self-service applications and expect such applications to provide real-time, integrated access to personalized information. For these scenarios, any product and departmental silos must be hidden behind Web applications that present the company, products, and services to the outside world.
To address these needs, IT organizations are building complex, real-time, automated business processes using a patchwork of existing and packaged applications. A Web-based product order may trigger several dependent applications to complete the ordering process, such as order tracking, billing applications, product assembly and packaging, shipping notifications, and inventory management. A workflow of jobs that use existing applications is often built with batch integration techniques using job scheduling software.
Another innovation in job scheduling products is event-driven schedules. This represents a clear move by batch tool vendors into the online camp. Jobs may be scheduled to respond in real time to business events such as the web-based product order described above. A process percolates through a chain of activities—a workflow—using events such as file creation, e-mail arrival, new log entries, or console messages to automatically trigger the next step in a sequence.
Dynamic business models are driving the need for more dynamic workload scheduling. Job scheduling products are evolving to provide "live" process management features to meet online response deadlines using features such as critical path monitoring and dynamic resource adjustment. Workload planning and forecasting are essential to ensure that the business processes will perform during times of heavy customer demand, such as the Christmas shopping season.
This third generation of job scheduling software, called workload automation, builds further on the workload management and batch processing features described earlier. Workload automation permits IT departments to define service level agreements for critical business services and to monitor performance against them. Additional features include conditional dependencies, graphical workflow tools, mobile access, scheduler APIs, and support for virtualization.
Application integration software is moving to address the same set of automation needs, but from an online perspective. TIBCO and webMethods products, and the Microsoft Biztalk server, are examples. The line between batch and online processing is blurring as technology converges on solutions to address the current thirst for process automation.
These technologies come from different starting points and are typically used by different members of the IT department. Application integration software is used by developers to construct composite hard-wired applications spanning multiple platforms, organizations, and businesses. Job scheduling products are used by operations staff to maximize operational efficiency to meet service deadlines. There are indications that job scheduler vendors are continuing this trend toward convergence as they find new customers: application development teams and departmental users looking for ways to put together new business processes quickly. This trend is, in turn, driving new feature requirements such as the ability to run multiple instances of a schedule side by side with no resource or data-flow contention.
Next week, in part 2 of this series, we’ll take you through a detailed analysis of product capabilities and market trends based on a survey of the leading vendors of mainframe- and distributed-based job scheduling products.
- - -
Notes:
- Independent research based on data published by the Transaction Processing Performance Council (http://www.tpc.org) assumes a 16-processor server and SAN configuration
- See the white paper “Job Scheduling on Windows”
- - -
Mike Gilbert is an independent consultant and owner of Legacy Directions where he is responsible for providing advisory services to the IT industry focusing on the issues around legacy technology. You can reach Mike at mike.gilbert@legacydirections.co.uk.