Unexpected problems with your Data center

I was out Saturday night for a friends birthday in Glasgow. We went to Jaunglers, one of those branded comedy clubs which are around the country. It was good night so the inevitable hangover arrived Sunday morning. I got up at 10am and went to the gym for a few hours which squared me up. Still, I was certainly not going to be very productive so a few of us went to my friends house. 2 games of football and the Ryder Cup ensured that we would not be bored.

Just before the Ryder cup started my friend James told me a funny story which I thought I would share with you guys. James used to work with a large telecommuications company, one of the biggest in the UK. They got a contract with a huge bank and James was part of the team which looked after it. Being such a large company, it was important for telephones and other communications to be on 24 hours a day, 365 days a year. He told me the contract was worth around 900 million pounds so it's safe to say they were an important customer.

Soon after the contract started the bank started complaining. Everything was going down in the middle of the night – all the phones etc were just offline. Apparently this was easy to fix and just needed a reset but they couldn't figure out what the hell was going wrong. This went on for a week or so and the bank were understandably pissed off and were threatening to take their business elsewhere.

As a last resort they sent 2 of the technicians down to the datacenter to see what was wrong. So the 2 guys had to do a nightshift and sit all night in the freezing cold datacenter to see what, if anything, was going wrong.

Nothing was happening but then, at around 4.30am, an old cleaning lady came into the room. She promptly took out one of the plugs which powered one of the servers, plugged in her vaccuum and starting cleaning up. Shellshocked is probably the best way to descrive the reaction of the 2 tech guys. ‘What the hell are you doing!!!!' or something along those lines was shouted in a high pitched voice. Turns out the lady had been lifting up the small piece of carpet and innocently pulling out a random plug so that she could do her job. All the other power areas had been locked but for some reason this one didn't have a lock which is why she used that one.

So to summarise, this company nearly lost a contract worth close to a billion UK pounds because of a hard working old cleaning lady who was clearly obvlious to the carnage she was causing. I would love to have been a fly on the wall when senior management were told why all the problems were ocurring.