When you work in IT, you need to be very careful not to cause more damage than you are fixing when working on an issue.

What would you do if an employee got a ticket to take an action, but he misread it to request that he delete all the users out of email, so that is what he did?

That is what happened to the senior IT guy in this story, and he couldn’t get the technician to take accountability for his actions, and eventually had to fire him.

How helpdesk and HR managed to ruin my day.

I work as an Senior Engineer for a managed solutions IT firm.

I hope it is a good one.

This isn’t a stupid user story this is a stupid employee story (and stupid recruitment manager).

We had just hired 2 new helpdesk guys. One from interstate, one from another rival firm.

That is an impressive credential.

I hadn’t been on the interview, but one was apparent an MCSA (who we will call Dave) which makes me wonder why he was applying for a level one help desk position, but we give them level 2 tasks from time to time and me being the sole exchange guy here wanted at least to make people have exchange knowledge so I’m not fixing every job that comes through and this was to help with the noise.

I run mainly via PowerShell for the cloud based clients, so it helps to be a bit knowledgeable.

Upgrading servers is apparently quite profitable.

I’m in the middle of upgrading servers for one of our clients who pay quite a lot of money, our biggest client, and at the moment their backup solution isn’t currently working thanks to issues with the vendor sorting out some bugs.

So, I’ve left a big note on the company file asking if anything needs to be changed to see me first, this flashes up on your screen if you even click the company because god forbid anything bad happens the latest backup we have is a month old and a mess of issues will happen even if we restore from that with the database store.

Well, this sounds fun.

I had some personal leave off one day to see my girlfriends violin performance I had this booked 2 months in advance and god forbid anything interrupt me during that, but left my mobile on just incase.

• 45mins in: I feel my phone vibrate, I ignore this hoping that they see my calendar that says “do not call today”
• 50mins in: I have 6 missed calls.
• 55mins in: I have 12 missed calls.
• 60mins in: Message “please contact the office something is wrong”

He should have had his phone silenced so they didn’t hear it.

I get up during the performance and see my girlfriends face look at me like “what are you doing” point to my phone and I’m in a suit and tie in a classical performance going through a bunch of grumpy people in a quiet part with my phone still ringing.

Ring the helpdesk and ask the other helpdesk guy what’s wrong:

Is he really the only one who can help?

Me: “What’s wrong that you cant escalate to another senior tech”

H2: “Well, its x company, nobody has mail”

This should be simple.

Me: “So the server is down? can you remote in”

H2: “Yeah”

Me: “So, when did this happen?”

H2: “This morning”

He is giving them plenty of things to check.

Me: ~usual are the drives full, is the database mounted, are the services started etc etc~

Me: “Have you worked on it at all in the last 24 because nobody can at the moment until I get the backup sorted.”

They just gave up?

H2: “No David was working on it yesterday”

I end up remoting my work pc on my phone on a toilet in the building. I look in the ticketing software and see an exchange ticket made 46 hours ago titled “All users removal”.

The ticket asks if people can be removed from All Users, pretty simple.

Oh my gosh, who could be this dumb?

The notes say: “Removed all users.”

David is not that great at English, so I just thought it was a blunder, I mean it cant be that bad so I phone back H2 and (given I’m working on an iPhone 5 and i cant see anything when i RDP through that) ask him to check the exchange server for users as a joke.

I can’t believe this happened.

Me: “Okay so this is a far stretch but is there any users in exchange console at all”

H2: “No”

Me: “So they’re all gone”

H2: “Yep”

And a restore isn’t going to be possible. Ouch.

Me: “You’re kidding me”

H2: “Nope”

Me: “So there’s no users in exchange what so ever? What about AD?”

H2: “Nope”

Someone needs to figure this out.

Me: “Can you put me through to the TL”

H2: “Okay”

Me: “Hey TL can you check if there any users on the X company SBS”

TL: “Sure, nope, this is bad”

Me: “Yes it is…”

The clock is ticking.

Given they have a 48 hour retention on mailboxes if you delete via SBS i was realizing if i miss the 48 hour mark i am completely out of luck. I end up speeding down the street to my work (on my pushbike in a suit because no time to change) and end up baseball sliding to my desk and restore all the users back, this takes me 45mins with about 15mins to spare before it ticks over and end up having to confirm with about 30 people they can get their mail, a lot of which hasn’t been delivered over the weekend.

He has every reason to be upset.

Given my rage at this point, and for some reason cant get the image of my girlfriends angry face looking at me and her families for missing half of her performance, I pull David into a room and ask him what the problem was.

Me: “When did you think removing people from a distribution group meant deleting the whole entire company, did the entire company call you and say ‘yeah we give up and we don’t want mail anymore’?”

Surely he can’t be that stupid.

David: “I was following the ticket, they said remove all users”

Me: “No it says remove from all users i.e the mail group, you deleted them all from SBS. Do you not think of the repercussions of that? Do you not read company notes that flash in big letters when you open their company profile up, I’m fine to teach you whatever but that’s just the simplest thing in the book not to do”

He is still not taking real responsibility.

David: “well you should have trained me on it”

I took a deep sigh at this point because I didn’t want to be rude by insulting his intelligence but I was pretty much close to yelling.

Some people need to be put in their place so they will actually learn.

I am not really an authoritarian either, this guy was in his 30’s and i am 23 and obviously just your lanky long haired IT nerd with a tux doll at his desk.

Despite the guy almost costing us a $200k a year contract,-I am sitting there as he grumps out on me and gives me attitude and can’t fathom how someone comes across educated and nice in an interview but now shows about much life as a potato in a mud puddle.

It’s called accountability.

I do my best to try teach him to say its better to say you messed up because all I need to do is reproduce logs to say otherwise, but he keeps trying to say he followed the ticket rather than just admitting he messed up.

At this point, the big boss a big Italian bloke, walks in and asks what the problem and I have about 2-3 HD workers and a TL all watching now.

Finally, some consequences.

I go into the other room and explain what’s happened and a phone goes hurdling across the room and before I can even say “think about it” the guy is asked to leave the building within 5mins and tries putting up a fight about it.

This ultimately ends up with someone calling the cops because he refuses to leave and a fist fight almost occurred.

Hiring is so important.

We have now since reorganized our hiring process and I am now sitting in on all interviews.

My girlfriend did end up speaking to me eventually, but I will have my phone off at the next performance.

Wow, it is amazing how much of a problem someone can cause when they aren’t careful, but it is even worse when they won’t take accountability for their actions.

Read on to see what the people in the comments have to say about it.

This is a pretty basic thing.

Yup, don’t let work ruin personal time.

This is absolutely true.

Sadly, there are lots of people like this.

Being good at passing an exam is different than being good at your job.

If you mess up, just admit it and move on.

If you liked this post, check out this story about an employee who got revenge on a co-worker who kept grading their work suspiciously low.