In-Depth

Back-End Processing Support for E-business Applications

  • By Yossi Aloni, Einat Weiss
  • 12/01/2000

Technology has changed the way companies conduct business. Today, the Internet is leading this change as e-business transforms industries. Accordingly, analysts predict that, by 2003, the Internet will become the predominant mechanism for conducting business. Most of the growth will occur through electronic markets. These markets will serve as nodes of commercial activity, linking buyers and suppliers.

Catalyst for Change

Although the Internet has existed since the 1970s, 99 percent of the Internet user population became familiar with it only as recently as the past six years. It was the World Wide Web that opened the Internet, paving the way for e-business. What was previously a public Web site used for marketing presence and lead generation, has now become the primary tool utilized by enterprises to leverage their retail sales, product support, customer service, and as a channel for the delivery of electronic services.

Suddenly, organizations have a way to propagate information and to use the large information pool in a new and effective communication method. A recent survey of top CIOs found that companies are able to reduce the time and costs of delivering products and services by giving staff, clients, suppliers and distributors direct access to information via the Internet. In other words, the Internet has shortened the time-to-market for companies. However, the opportunities opened by the Internet have also created new challenges mainly due to the large volumes of available data and massive access by consumers.

Although the traditional brick and mortar companies are quickly joining the "dot.com" revolution, it is important to remember that e-business is not expected to be the ONLY business. Amazingly, even in the beginning of the 21st century, only 94 percent of Americans have telephones. Furthermore, out of that population in 1999, only 1/3 had access to the Internet. Although estimates indicate that total consumer spending through the Web will reach $350 million in 2002, this amount will account for only five percent of total spending.

The challenge, therefore, is to develop the new e-business applications in an evolutionary (versus revolutionary) way. The key is to integrate these e-business applications into the existing infrastructure of the applications that are generating the rest of the 95 percent of revenues.

Today’s enterprises are rapidly adopting Web-based technologies and processes that dramatically change the way their organizations do business. The new Internet-enabled technologies allow enterprises to be more productive. An organization’s data center has become crucial in gaining the competitive advantage. As a result, Web applications are being implemented at a faster rate, and, in many cases, they are treated as separate and experimental.

Applications are implemented fast in order to gain advantage over competitors. However, to implement Web applications within an existing organization requires the ability to integrate the Web front-end to new and existing back-end applications, creating one enterprise production environment. Create a consolidated enterprise production environment is therefore one of the greatest challenges that enterprises face.

The Production System and Challenges of the Internet

Analysts predict that by 2002 75 percent of e-business projects will fail to meet their objectives due to fundamental flaws in project planning. One of the key factors in a successful implementation is providing the proper e-nfrastructure and the integration of the new applications, especially when a production environment already exists. Here are some of the implementation challenges:

Adopting new technologies in the fastest way possible. Many of today’s leading enterprises have already established e-business channels. Pressured by Internet only companies, there is a sense of urgency among these established businesses to compete for online market share. In order to use the rapidly developing Internet to gain competitive advantage, the enterprises within these companies have to be able to handle new technologies at a very fast pace.

Creating one common methodology for ALL applications. The new applications are viewed as either extensions or a new generation of the traditional applications by the IT department. The successful convergence of legacy and Web applications is therefore crucial. The banking industry, for example, illustrates the need for an efficient way to converge these technologies.

For years, banks have provided tools for internal end-users to issue requests from a remote location and process them. A clerk in a remote branch can issue transactions from his terminal to the bank’s main computer and receive feedback for this action. Most banks will soon—if they are not already there—allow external consumers to perform the same transaction from any location in the world, 24 hours a day, using the Internet. A new methodology of work will have to be created to handle the convergence between the new and existing applications.

Addressing the characteristics of the Internet – volume and unpredictability. In the traditional IT world, the number of processes submitted every day is known in advance, allowing for efficient planning of the workload amongst the various systems. Advanced knowledge of the workload on the systems helped IT departments determine the load on the machines during different periods of the day. For example, the batch jobs usually run during the night; during the day, most transactions are online. The Internet changes this level of predictability. Users worldwide can perform all kinds of transactions when and where they want, and often. Opening applications to the Internet has created an added challenge of volume and unpredictability for IT personnel.

Controlling all the enterprise mission-critical applications in a focal way. Some organizations still treat e-business applications as experimental. They are rapidly implemented so that organizations can gain advantage over competitors. However, these e-business applications should be treated in the same way as the other applications. Some applications that are now open to the Internet are actually becoming part of the organization’s mission-critical applications and should be treated accordingly. As such, another challenge is to find the way to handle and manage mission critical applications that are open to the Internet.

E-business Impact on Batch Scheduling

The e-business production environment is not totally different than the traditional production environment. However, it is stretched to the limit. Transactions once used only by internal employees, can now be activated by consumers directly into the production environment through the Web. Some of these transactions will perform real-time operations; others will accumulate data to be processed by a back-end processing engine. As more and more data becomes available through the web, analysts see more usage of back-end processing, providing the middleware layer for the applications. Batch processes will be directly or indirectly triggered to handle the massive accesses from the web. Most of the new e-commerce sites already defer some of the transaction processing to batch to assure efficiency, scalability, high availability and better control.

Back-end processing is not used only as part of the applications processing. Many critical processes are done in batch, like EDI, backups, data recovery and file transfers. These processes play an important role and must be treated as an integral part of the datacenter’s processing.

What are the main factors that will impact the back-end production environment?

  • Volume.
    •  Since the number of transactions and users rapidly increases, there are many more jobs, and in many case each job handles a huge number of transactios. Furthermore, many of these new jobs will be unpredictable in time or capacity.
  • Criticality.
    •  Each production job is now more critical. These jobs process more transactions, some of them accumulated from the web front end applications. Analysts see EFT (Electronic Funds Transfer) jobs to be one the most critical processes of the new e-commerce applications. Any halt in the production flow may affect the availability of the system. With most of the users being external consumers, the service level is much more critical. These external users are far less forgiving if their transactions are not processed properly.
  • Availability.
    •  The production system must be continuously available. This shrinks the production batch window to a minimum. Furthermore, the system cannot be totally suspended for the duration of that batch window. External customers worldwide are now impacting the systems as they are utilizing the Web 24X7 for their business transactions. The reliability and availability of the production management system should support the highest possible service level, making sure that any job failure is handled in the most efficient way.
  • Efficiency.
    •  Data centers need to operate more efficiently due to the high volume of transactions and the difficulty in predicting when the jobs will be submitted and how. While the competitive market forces the best use of all available resources, the service level still needs to be high.
  • Scalability.
    •  Data center processing will grow exponentially and this will put the infrastructure in a position that it has to expand rapidly and constantly. Systems that used to handle 5,000 jobs per day may find themselves dealing with hundreds of thowsands of jobs as they open to the web.

 

The New Production Management System

Some of the old batch-scheduling systems do not have the robust capabilities to meet the challenges that today’s enterprises face. Today’s production management systems need to be able to support the new applications requirements in conjunction with the existing traditional applications.

What are some of the characteristics inherent in a job scheduling and production management system with the robust capabilities to handle the back-end processing of the new applications? These characteristics include the following items:

(image not provided)A Solid Architecture
A production management system should be based on a three-tier architecture that will address the challenges of huge volumes of information generated by e-business. It also addresses the availability and scalability challenges. A true, three-tier architecture can support configurations driven by business requirements, irrespective of the different platforms or specific applications.


The enterprise tier consists of a Graphical User Interface (GUI) that provides a common look and feel to all platforms and manages cross-platform dependencies. This consolidated view of

scheduling information, task statuses and exceptions allows staff members to manage and control any environment, eliminating the need for platform-specific experts, cross training, and the associated costs.


The server tier consists of a full-function scheduler available on all popular platforms. Each server component maintains its own database which consists of scheduling information for that platform and the agents. Real-time scheduling and execution information on each platform environment is communicated to the enterprise level for inclusion in the consolidated view.


The agent tier consists of multiple agents that are managed by a server. These agents, which provide the same functionality as the server, should include task submission, tracking and results analysis capabilities. The agents eliminate the need for a local database and routine administration since scheduling decisions are made by the server and communicated to the agent for actual execution.

Reliability and Availability
The following availability and recoverability characteristics should be built into a production management system.

  • The production management system should be built to overcome any communication failures with automatic recovery and synchronization processes in place.
  • Each server component should function independently and continuously regardless of a communication failure with the enterprise level.
  • An administrative user interface on both enterprise and server levels help ensure that the system can be monitored from either level in case of a failure.
  • The system should support automatic fail-over of agents from one server to the other.

Open Dynamic Workload Balancing
The production environment must be managed in a way that fully and efficiently utilizes all resources. A truly open and dynamic Workload Balancing component should provide the necessary automation tools. This Workload Balancing component will enable the production management system to send the jobs only to those machines that have enough available resources for their execution. The jobs will be sent only when these resources are available.

The "ADOBE" characteristics are important for the Workload Balancing component:

  • A
    • utomatic – A true production management system should make regular scheduling decisions based on resource availability. That same mechanism should be used when implementing the workload balancing. Based on pre-defined rules and parameters, the system should optimize the workload balancing process without any need for manual intervention.
  • D
    • ynamic – The workload of the different systems and the utilization of the existing resources constantly change. Making accurate scheduling decisions depends on using the most up-to-date data. Therefore, any workload balancing solution must be dynamic and have the capability to decide where to send the process for execution, based on real-time and constantly updated data.
  • O
    • pen – The production management solution should be open and capable of being integrated with any external component that can provide information about the resource utilization in the system. Resources can vary from CPU utilization to software license count to the size of temporary disk space. As such, the workload balancing solution should be open to retrieve the information from these different sources using the same mechanism and infrastructure.
  • B
    • uilt-in – Workload balancing should be an integral part of the solution. It should be built into the product and based on its standard infrastructure. A production system should be able to manage different resources, and using that same mechanism, to balance the workload. Implementing a workload balancing solution is preferable to buying another component.
  • E
    • asy to Implement – It should be easy to define, maintain and monitor.

    This dynamic Workload Balancing component is available today from vendors such as BMC Software through an integrated solution that combines CONTROL-M’s Dynamic Workload Balancing with PATROL’s sophisticated agent technology.

    Simulation
    In an environment with numerous variables, it is important to have good enterprise level simulation tools. "What if" scenarios should be used to analyze and predict. The simulation tool helps the production manager to ensure that the service level to Web users is not adversely affected by any planned changes. Furthermore, the simulation tool should be enterprise wide to cover the whole production environment, regardless of the platform that is running the jobs.

    End User Triggering Facility
    Not all e-business processing can be done in real-time; some require back-end processing due to system load or synchronization issues with other processes in the system. In such cases, external users can trigger some of the jobs/processes that were once initiated by internal users/administrators. The production management system should provide a tool that will enable job triggering by consumers through the Web. This tool should act as a "screen" between the consumer and the production system. It should also be a "screen" from the security perspective. The end user should not have to worry about jobs and dependencies; they should just be able to provide input to and initiate the various processes.

    Deadline Scheduling
    The production management system should issue an alert when exceptions occur. In an e-business environment, the systems should also be able to issue alerts when a production flow is not going to end successfully. This is critical to provide the service level needed in such an environment. If the administrator knows that a problem is likely to happen, preventive action can be taken to keep users from being affected. To minimize any negative impact on the end-user, the system must provide a built-in deadline scheduling component that will alert and also initiate automatic recovery procedures in case a critical production flow is predicted to be delayed.

    Enhanced Online Administrative Tools
    A production management system in an environment hosting e-business applications should provide online tools with the following capabilities:

    • Enhanced viewing, navigation, and drill-down capabilities to efficiently manage the huge systems.
    • Focal point of control for the entire enterprise, including all processes, on all platforms, with all applications.
    • Windows and Web-based tools.
    • Intelligent alert capabilities that are based on management by exception, including alerts to emails, paging systems and cellular phones.

    Summary

    Industry experts recommend that organizations approach e-business holistically. Rather than create ancillary or parallel businesses, the integration of technology into the core of the business is a key challenge that must be met for organizations to be successful. Enterprises that fail to achieve this integration might waste valuable time and resources. As a result, the integration of e-business applications into the rest of the production environment will be critical. As the number of new e-business applications continue to increase, organizations will have to re-evaluate their existing batch scheduling systems to verify that they can meet the availability challenge of the e-business world.

    The new production management systems, replacing the old schedulers, will help enterprises to achieve their major goal: opening the traditional system to the Internet. Solutions, like
    CONTROL-M from BMC Software, enable enterprises to address the challenge of implementing the e-business applications.

    Yossi Aloni has a degree in Computer Science and Business and has served as a Systems Programmer, a Development Manager, a Systems Engineer, and an International Technical Director. Contact her at yossi_aloni@bmc.com

    Einat Weiss has a degree in Computer Science, and has worked with the INCONTROL products for six years and has experience as a software developer and a senior software consultant. Contact her at einat_weiss@bmc.com.

    Must Read Articles