The domino effect with incompetent clients is alive and well in I.T. For as long as there have been computers and clients, situations like the one in this story will persist.

See how this guy had to handle a client make things worse and worse with his servers.

Bricking ten servers

This is from the old days when I was working for the on-site service of a big PC/Server Company. I was responsible for the on-site service in my region.

It was a dark Friday night in September and I had just lit a nice fire in my fireplace, had a nice hot chocolate and a book when my phone rang.

There was a big problem.

I needed to head to a client NOW as ALL of his ten servers were out and the hotline could not find out why and what to do.

As I arrived I could confirm that indeed all ten servers where dead. Like no light no nothing. The “I.T. guy” was a middle aged electrical engineer who was was very upset and quite angry and so it took me a little time to find out what happened.

The guy thought it was a good idea to do some firmware updates via the iDRAC while no one was there that could complain about the servers rebooting. That is indeed a valid reason to do this on all servers at once on a Friday evening.

So he clicked on “Update all” and went to do other stuff.

Then he did a little more. And then he did something else. (He told me all he did in excruciating detail – nothing he did had anything to do with the servers but he could not be stopped.)

But this guy was blissfully unaware of the crisis brewing.

As the servers where still updating he then went out to have a smoke.

When he returned, the servers were offline and he was not able to connect to the devices. So he obviously did what any responsible USER would do: he /tried/ to power cycle the devices.

Each and every one of the poor things.

He did it the hard way by cutting the power to the enclosure.

Then he basically killed the heart of the system.

This was the exact moment he learned that power supplies have a BIOS too.

He also learned that this BIOS can be updated.  He learned that when this happens, everything else shuts down. He learned that an update on a PSU is a very slow thing.

And he learned that cutting the power to a PSU that is updating instantly kills the poor little thing.

Well, I ordered 20 new PSUs. Installing them revived all servers.

Here is what people are saying.

Lazy, incompetent gamblers. That’s who.

I bet!

I wonder if this is a controversial take.

Suspenseful indeed!

Well, let’s hope neither worker has a bad heart.

I hope he doesn’t have to deal with another client like that.

If you liked that post, check out this story about a customer who insists that their credit card works, and finds out that isn’t the case.