Mark Kraemer works as a CITI Network Administrator, and shares his experiences in his column Tech Diaries: Musings of a Network Administrator.
Dear tech aficionados and non-techies,
I am very excited to announce the launch of my column, Tech Diaries: Musings of a Network Administrator. During the day I am busy troubleshooting all kinds of IT issues, but I also enjoy writing and sharing stories. In this column, I will share with you interesting stories that I hope will make you think and spark your interest in a subject considered dry by many.
Today I want to share a story with you that may get you thinking about backup solutions.
Jimmy works at a branch office of my client. It is his sole responsibility to change the nightly backup tapes and to take those tapes off site. Each night data for the entire office is backed up. Each night he takes the previous day’s tapes home. Once a month he takes one tape to the safe deposit box. Most weeks things go as planned. But one recent week, Jimmy was on vacation. The task he handled for the last several months went unnoticed in his absence. No one even knew that the backups were at risk. On Saturday evening, a good eight days after Jimmy’s last trip to the office, the lone server at this branch was taken out by a fire; an obstacle, but a surmountable one under ordinary circumstances.
But these were not ordinary times. Jimmy was gone. The last real back up happened on Friday, a week before the fire. An entire week’s worth of data would have been lost. But it was worse than that. Jimmy is a fallible person and he had bad habits. He hadn’t taken any of the tapes home the previous week. These tapes were sitting on top of the server, and were burned as well.
This story doesn’t have a happy ending. The branch office lost roughly two weeks of data for five people. Despite the heroic efforts of the IT staff and the manual labor of the admin staff, the damages were severe. Countless hours were lost, thousands of dollars spent, and unknown amount of lost revenue because of missed deadlines, and unhappy donors.
The reverse of this story is how a good backup saved a less than ideal situation. At one of my client’s other field offices, all of the important shared files lived on one person’s computer. One fateful day, through no fault of the user, the machine was taken out by a very nasty virus. After several hours, I declared the machine a loss, and began prepping a new hard drive. But what about the data?
For this location, we used an automatic, online backup solution. While not applicable in all situations, it fit our needs. Because a snapshot was taken nightly, we only lost about 6 hours of data. Restoring it took some effort because of the specifics, and the office didn’t have access to many of their files for nearly a week, but it could have been far worse. The former situation was a disaster, the latter an inconvenience.
Thankfully most of first story is not true. While the fire didn’t happen, the details regarding “jimmy” certainly did. I spun the very real details with a little bit of “what-if” as a lesson for my client. The second story IS true. And just as good a lesson. Both are worth sharing. The message is simple but very important: there needs to be accountability when it comes to backing up mission critical data. If you are a manager, do you know how your data is backed up and who is responsible for it? If you don’t have IT people in day-to-day charge of your backup solution, then you need to think long and hard about who is, and make sure there are checks in place. No system is fool proof, but good planning can minimize your risk. The day of an emergency is not the time to find out about your backup solution. Ask yourself these questions. If you don’t have immediate and satisfactory answers, call your IT pro sooner than later.
- If we had to rebuild our office from scratch tomorrow, what would we have lost?
- Is our data backed up off-site?
- Can it be retrieved within an acceptable time frame?
- Who is in charge of making sure our data is backed up? IT people? Office staff? Others?
- If we have more than one location, who is in charge of backups at those other sites?
- Who is in charge of restoring lost data from the backups?
- Has our backup solution been tested and verified recently?
I welcome your comments, questions and suggestions. I look forward to hearing from you!