Net Management: Less Can be More
How many network management products do you own? How many are enough?
On a recent tour through the information systems department of a large organization, I was proudly shown an extensive library of network management products. Optivity, OpenView, Ciscoworks, Manageworks, plus several different kinds of LAN sniffers and traffic analyzers -- these guys had it all. They spared no expense when it came to equipping themselves with network management products, so you would think they were ready for anything, right?
Later that afternoon, as I attended one of their departmental planning meetings to discuss NT network architecture, I noticed one of the network support people grab his pager from his belt, glance at its display, then run urgently from the room. When he was followed soon by two colleagues it became obvious to everyone that something was wrong. The meeting broke up, and we all headed off to find out what happened and to see if we could help.
We found the network guys huddled in a large wiring closet, puzzling over the maze of patch cables. When we asked what was wrong, we were told that remote offices were reporting that an important application was failing when trying to access a central database in headquarters.
Was it the database server? We followed the network support team down the hallway to the server room. They spent 20 minutes trying to figure out how to use the shared monitor/keyboard/mouse switching equipment to connect to and log onto the database server, only to announce that it looked okay. They rebooted it anyway, just to make sure. Predictably, 10 minutes later, after the server had come back up, the application still wasn't working.
Maybe it was the local router behind the database server. We watched the guys ping it, listened to them talk about it, watched them ping it again. Fifteen minutes later, they confirmed the router was working properly.
Was it the WAN line? We followed the guys back into the wiring closet where they fiddled with termination gear for 10 minutes until they were convinced the WAN line was working.
This painful fault-isolation process continued for a while, until the organization's network crew finally identified and fixed a disconnected serial cable on the router in front of the WAN line. The problem was solved, but only after a critical application was knocked offline for more than an hour, frustrating hundreds of users. Later that afternoon, I heard the IS director remark sarcastically, "And these guys wonder why we want to outsource our network support."
What really went wrong that day was the organization's network support procedures. The company made little or no use of its investment in those fine network management products. If the system had been installed and configured properly, any one of the tools on hand could have told them pretty much where to find the problem, maybe even before users called to complain that the application was down.
We later learned that many of the organization's network management products were still in shrink-wrap. The few products they had taken the time to install were loaded onto workstations that, covered with dust, had been down for weeks. When we asked why they didn't use these tools to find the problem, we learned the software was set up by consultants and former employees who never transferred their knowledge of these tools to their successors. When the company's network went down that day, the support guys resorted in panic to using what they knew, which turned out to be little more than how to ping an interface and reboot a server.
In fairness to the technicians, management was equally at fault. They pointlessly bought many fine network management products without bothering to plan for their deployment. If they had devised a plan, just one product would have been enough. Another technical lesson learned the hard way: Network management isn't something you buy, its something you do. --Al Cini is a senior consultant with Computer Methods Corp. (Marlton, N.J.) specializing in systems and network integration. Contact him at al.cini@computermethods.com.