Some tech issues can really become technical if not fixed on time!

This guy shares how a power outage caused a lot of trouble at his workplace!

Check out how things got worse.

Sometimes, you have to be firm with them

One of our stores had a power outage. Not a huge deal, it happens, we can deal.

Due to some recent work on the network, however, we didn’t have everything in the network closet running on the UPS (yet), and when the power came back on, the SDWAN box – the one single point of failure – didn’t boot up correctly.

This is where it gets interesting…

It would route WAN traffic, but not LAN.

Note: When we upgrade our phones to VOIP, we pull cable for each phone, separate from the computer network, so the phones are entirely outside of our network. More expensive, but there are . . . reasons (and good ones).

The SDWAN box thus has (or, rather, uses) two LAN ports, one for us, one for the phones.

Our telecom provider had, of course, detected the outage and automatically opened a ticket.

They detected that the SDWAN was back up, and closed the ticket.

UH OH…

The store was still offline, and was less than 100% efficient at letting me know, but when they did, they told me the phones were still down as well.

This was the second store this exact thing had happened at in two weeks, and the first one took an on-site visit by a tech to reconfigure the SDWAN box. This was late on a Friday.

Once I found out, I reopened the ticket, and explained that with the phones down as well, it has to be the SDWAN box that’s acting up.

He wanted them to take some action!

They looked at it, saw the SDWAN was showing as online, couldn’t see our firewall behind it, and closed it again. “We show you’re online in the orchestrator.” Sigh.

So I reopened it a second time, made it clear we were still offline, and so are the phones, which are outside of our network, same result, same message, word for word, closed again.

This is Saturday afternoon.

We’ve been offline for two days at this point, which is painful, but not a huge deal, but the phones have been down as long, and that is a big deal.

That’s INSANE!

So, OK, we’re a six figure a year account (with about two dozen locations now) with a platinum support team, so I emailed our account rep and asked her to escalate the ticket.

It was Saturday afternoon by this point, so I didn’t actually expect a quick response. I was wrong.

Less than five minutes, I get cc’d on the message to the escalation team. And less than two minutes after that, I get an email from the escalation team. Woo-hoo! Things are happening!

The SDWAN is up. I do not see your internal network equipment switches/routers/firewalls).

Yeah, things are happening, but all progress is backwards.

Finally things got a bit better!

“For the fourth time, the phones are not working. The phones are not on our network. They go through a separate switch that plugs directly into the SDWAN. There is no part of our network that can affect phone service. The SDWAN is not communicating with its own LAN side.

“For the fourth time, this is the exact same thing that happened after a power outage at another location last week that required an on-site visit to reconfigure the SDWAN because the configuration got corrupted by the power outage.”

I did get an apology after that, and he was able to get into the box remotely and get it working in less than half a hour (unlike the previous incident).

He knows how things work!

I guess there are advantages to being a big fish in a medium sized pond, and they’re better than any other phone or internet company I’ve dealt with, but there are times . . .

GEEZ! That’s a bit messy!

Why would everyone be so ignorant towards the issue?

Let’s find out what folks on Reddit think about this one.

This user thanks this guy for not being lazy about the situation!

This user is happy they have retired from the field!

This user has had almost the same experience once!

This user shares another similar experience!

Trying hard enough always pays out and this guy knows it!

If you liked that post, check out this story about a customer who insists that their credit card works, and finds out that isn’t the case.