Comments on: GitLab’s Database Outage Postmortem

By: Tim Hall

Tim Hall — Sun, 05 Mar 2017 13:59:57 +0000

You do have to wonder why the engineer never asked what the other nine tapes were for. If it never occurred to him to ask it inplies a worrying level on incuriosity. If he was afraid to ask it doesn’t cast the management culture in a very good light.

By: Michael

Michael — Mon, 27 Feb 2017 22:59:11 +0000

Remember the engineer in question did not act against his training.

The training given and the standing instructions made an incorrect assumption: that a different tape would be used for each backup without a specific instruction to do so being given.

By: Tim Hall

Tim Hall — Fri, 17 Feb 2017 21:00:17 +0000

Ouch. Sounds like a serious training issue there.

By: Michael

Michael — Fri, 17 Feb 2017 20:43:52 +0000

The computer room where this happened has been decommissioned for some years now, but once upon a time there was a remote site with a lone support engineer who had to busk his way though all the procedures and problems using written instructions supplied from the main data center.

One of these scripts was the weekly backup to tape. A stack of 10 tapes were provided.

The time came for a restore and the tape was found to be corrupt. The other nine tapes were blank.

The written instructions did not say put the backup tape at the bottom of the stack when you take it out of the drive. The same tape had been used since the computer room was commissioned and it had worn out.