Note: this is an unfinished slush-pile article.
A pilot friend of mine told me that flying is hours of boredom punctuated by moments of sheer terror, which could also describe the life of a SysAmin or DBA with a little bit left over to cover the unique despair of finding yourself at 41,000 feet with no fuel left in either tank. IT, like the airline industry, has evolved a tapestry of emergency plans and engineering safeguards--codified in more manuals than we can read in a lifetime--to provide two important things: multi-failure tolerance, and options for the poor bastard who has to deal with everything else. Looking at the history of industrial disasters we can conclude that while procedure can postpone catastrophe by millions of hours, the gaps it leaves can only be bridged by smart, well trained human beings.
Let's take the poor fellow who ran out of gas. His name was Bob Pearson, captain of Air Canada Flight 143 on July 23rd, 1983. His Boeing 767 had 61 passengers on board, five crew, state-of-the-art avionics and an infrastructure that spanned world-class engineers, veteran mechanics and ground crew, certified air traffic controllers and a civil aviation authority. But in the hours before their flight they discovered a minor mechanical failure in the fuel gauges.