Tough Lessons
One of the worst mistakes I ever made occurred around 1980. I was the assistant director of computing at the most prestigious engineering college in Terre Haute, Ind. In truth I was the No. 2 guy in a two-person IT department, but it was a nice title.
We had a DEC PDP-11/40 to handle the administrative departments and a PDP-11/70 for the academic workload. I was in charge of administrative computing, which meant I took care of the registrar’s office, the alumni department, student affairs, and all other administrative functions. I still have samples of tuition bills in a cardboard box from a massive tuition bill program I wrote.
Backups were a hassle; the 11/40 did not have a tape drive. It had four or five DEC RK05 disk drives that housed removable 2.5 MB -- yes, that’s megabytes -- platters about the size of a pizza. We had one disk for each department. For backups, every week for each department, we mounted an RK05 in an unused drive and did a disk-to-disk copy. Since it took about half an hour to copy each disk, backups would easily consume two to three hours each week. Every four weeks we would remove each department’s disk from the 11/40, mount on the 11/70, and copy to a nine-track tape.
We were supposed to keep a rotation with a couple tapes for each department and, as I recall, a couple disks. But the disks went bad so fast and the tapes were so unreliable that we were lucky to keep any kind of rotation.
One month got particularly busy, and I went four weeks without doing a backup. Finally, I could no longer put it off. I dismounted the registrar’s disk from the 11/40, mounted it in the 11/70, and started my backup. I vaguely remember thinking that I really should have used an older tape, this tape was four weeks old and was the most recent copy of everything on that disk. I figured, what were the odds of something going wrong? I could afford to be a little careless, just this once, couldn’t I?
You know the rest of the story. A little way into the tape, I heard a grinding noise coming from the disk drive. It sounded like a lathe, with the chisel digging deeper and deeper, grinding away precious data with each rotation. The disk was ruined, and since the tape copy had started, the tape also was worthless. In one incredibly dumb move, I blew away all the data from the registrar’s office, along with the most recent backup, which was already four weeks old. I single-handedly destroyed over a month of work. It took them weeks of data entry to catch up. I feel bad about it to this day, almost 20 years later.
I share this embarrassing story because of an e-mail that a reader sent me about a column I wrote for the Sept. 8 issue, called "A Taste of My Own Medicine." In that column I shared a few other stories about some of my, shall we say, less-than-perfect moments. The reader wanted to know how some skinny bald guy like me, with a 20-year track record of mistakes, could possibly be competent to deliver consulting services to customers.
After I got over being mad at the insult to my already deflated ego, it occurred to me that this is a thought-provoking question. If I were a customer, would I trust my computer systems to some frozen nut from Minnesota who openly admits he made serious mistakes?
My weasel answer is, it depends. It depends on how the mistakes are presented and the point in sharing them. You see, every mistake we make in life should be a learning experience; each real-world mistake could end up being worth more than months or years of theoretical training. So a long history of making mistakes could be a valuable asset.
Nearly 20 years ago I learned an unforgettable lesson about the value of proper backups. That embarrassing incident saved a major electronics plant from disaster four years later, saved a major software development project seven years later, helped more customers than I can remember set up proper backup rotations and probably averted more disasters than I will ever know about.
That’s why I always tell people new to this business that you aren’t a professional until you’ve blown away somebody’s critical data and you have no way to recover it.
I learned lessons from every mistake that I’ve shared over the years in this column and, frankly, those mistakes make me a better consultant. That’s why I share them. Learn tough lessons from your mistakes, recover from them and they will pay dividends over and over for years to come. --Greg Scott, Microsoft Certified Systems Engineer (MCSE), is Chief Technology Officer of Cross Consulting Group (Eagan, Minn.). Contact him at gregscott@scottconsulting.com.