Sometimes, apologies can save a lot.

Even your entire job!

But this boss refused and made an irreversible mistake that got him fired!

Let’s check out what happened.

It is our job to know!

This starts as malicious compliance and ends with Petty Revenge.

A place I worked at in 2004 had three DBA teams (Data Base Admin). The team I had been on since I joined was tight and ON-call rotation was pretty easy.

If anyone got paged for a database that you were the primary on, then you had to buy the team drinks.

You rarely got paged. 🙂

He has some flashbacks..

So End of an era. Do you remember “Al Bundy” doing commercials to save big money on long-distance phone calls? That was us! Cell phones and free long-distance put us under.

During the death throes, they had massive layoffs and combined 30 DBAs on three teams into one team of 7. I made the cut.

They had no mercy on them..

HOWEVER, they refused to offline any systems so the workload of 30 remained!

My old boss did not make the cut and my new boss was a toxic butt-kisser. I don’t even care anymore– his name was Steve. Worst boss ever! He would defend ANYONE but someone on his team.

I believe he legit thought throwing his people under every bus made him a “team player” Other departments quickly figured out, “Blame them and he will blame his people” so they did!

He hated how things were!

My first on-call rotation on this new combined team was utter hell! The pager went off every 10 to 15 minutes. Messages like:

Job 104 completed step 5.

Job 110 starting step 12

Job 110 completed step 12.

In case you did not know-

THESE ARE NOT THINGS YOU PAGE OUT FOR!!

He was done with it!

I have no idea what any of those jobs are, where they are kept or what do about them if they should fail.

No idea!

On and on all night long – You can’t ignore them as you might miss a REAL message about an actual database problem!

So you get zero sleep and with the diminished staff they expect you there the next day.

When I started ranting the next day the folks left from that particular team said it’s always been that way.

That was nuts and we are short staffed so I set out to fix the problem.

He wanted to fix the issue!

I looked at every message that got sent to the DBA pager queue and added the prefixes “INFO” or “WARN” to every message (kept it short for the pager)

Then I changed the pager daemon to route anything starting with “INFO” to e-mail and ONLY route “WARN” to pagers.

Worked like a champ! We went from 50 pages a night to maybe one per night and often none.

One day a team meeting was called and Boss Steve was upset that a print job had failed and our on-call person (me) did not respond.

But uh oh…

“Why would I respond to a print job failure? We (database admins) have nothing to with that.”

“When you get a page you respond. Period”

“Why would I get a page that a print job failed?”

“Because we always have, didn’t you get the page?”

“Ohhhh, that, no I fixed that a while back. DBA’s only get paged for DBA stuff now.”

“WHAT? You need to put that back the way it was!”

Ok, I am cutting a bunch out here for brevity – I said no!

He had a valid point!

The people who were actually responsible should get paged not us– we ended up in a shouting match!

He was yelling he would take me to HR and I told him that worked both ways as he was flat-out abusing us not letting us sleep AND expecting us in at 8:00 AM sharp.

That slowed him down a little and then he calmly said, “Our department must know everything that happens overnight so we can answer for it in the morning, make sure we get the pages!

Oh the evil light bulb went off. I smiled and said, “Yes, sir!” Ding! Ding! Ding! “Our department” eh? “We” eh? Okie Dokey then!

He did what was necessary..

I reworked the pager daemon so that all pages went to HIS pager but left the code that only routed WARN to the on-call pager in place.

Now WE and OUR DEPARTMENT know everything that happens overnight!

Yes, I am so very proud of that one! I slept soundly that night!

The next morning I kept waiting for him to come over, and either tear into me again OR admit it was a horrible night, that he did not grasp how bad it was but now let’s get a better fix in place.

The boss knew he had messed up.

I was disappointed– He said NOTHING to me at all. Did he get all those pages? I double-checked that everything was routed to his pager and it was.

He just would not admit he was wrong! I was impressed!

I thought he was tough (but stupid) getting all those pages every night and never saying a word!

END OF MALICIOUS COMPLIANCE

And now the fun part.

Start petty revenge. 🙂

A few weeks later I was doing an upgrade outage overnight.

We were getting to a critical point and needed a decision to roll back or continue and exceed the outage window.

I paged him several times. No answer!

The boss was going to get himself in trouble.

We all sleep through pages from time to time so I rolled back and e-mailed all concerned. No big.

But he NEVER got back to me. That was odd.

The next Monday I mentioned it to the team and several folks said yeah, he does not answer pages after hours anymore!

AAAAAA-HA! He was not tough! He just started turning his pager off before bed! Still too chicken to admit he was wrong.

A few weeks later I was on call again and late Sunday night I got paged to come in immediately!

To keep it brief the system that went down was a federally mandated system with severe financial penalties for downtime beyond a limit.

Things were chaotic!

We were reaching that limit! It was a huge amount of money! All hands on deck! Even the CEO and CIO were there.

Developers, sys admin, network, security and DBA with their bosses (with one exception) were all on laptops in the conference room.

This was BIG! Thank god it was not a DB problem – It turned out to be network.

He didn’t mind the work.

The CIO said he appreciated us and wanted us all there until it was resolved… just in case. No problem… sometimes that is the life we choose! 🙂

At about 5 am Monday after a long night, the CIO says to me, “Hey, I have paged your boss like a dozen times — he is not answering”

OH GLORIOUS DAY! With a shrug, I very casually say, “Oh yeah, yeah, Steve *never* answers pages at night.”

He said nothing out loud but I saw his face cloud up.

This is where it picks up pace!

About 7:30 issues are resolved and we are watching transactions flow across to the feds.

I am watching the DB to make sure the huge backlog is not gumming it up but I am just watching as all is well.

At about 7:45 Ole Steve comes strutting in the door, fresh-faced with a full night’s sleep! As he walks past the full conference room he does a double take and stops.

When he sees all the heavy hitters (and us plebes) in the room, his face falls, he knew right then he screwed up!

Everyone was pissed at the boss!

The CIO Says, loud and mean, “Where have you been?!” Then the CIO and CEO take him out of the room.

I have no idea what was said but I bet it was not pretty! lol

The CIO and CEO come back in about 10 minutes. They heartily thanked us all for the dedication then sent us home and said don’t come back until tomorrow! Good leaders!

Poor Steve!

I came in the next day and no Steve. I never saw him again!

I sure am glad I kept that short! 🙂

Yikes! That must’ve stung for Steve. But everyone in his team probably got a sigh of relief.

Let’s check out what the Reddit community thinks of this story.

This person has a sarcastic comment for the boss.

This person is delighted to hear what happened to Steve.

This person loves the way the story goes.

This person is done with mean bosses.

This person doesn’t get why people can’t simply make an apology.

Steve should have being fired. Period.

Thought that was satisfying? Check out what this employee did when their manager refused to pay for their time while they were traveling for business.