A New IT Guy Was Deleting An Unused Vlan From A Switch Configuration, But It Took Down The Entire Network Until His Boss Could Show Him How To Fix It

Michael Levanduski

3 weeks ago

Reddit, Unsplash

Keeping a network working smoothly is important, especially for businesses that rely on communication between locations or even devices.

While managing switches is normally not that difficult for an experienced IT professional, sometimes they are set up in unusual ways that can cause some issues.

That was the case for the IT guy in this story, and when he tried to clear off an unneeded vlan from a switch, it took down the entire network until his boss could show him how to fix it.

How I nuked the network at a small gaming facility with one line.

There was a general data segment for most of the computers at a small gaming facility I worked for before we granulized our segmentation.

Simple networks aren’t always the most secure.

On this data segment you could find the computers for all of the departments and the POS up front. Printers, servers, switches, ATMs, gaming machines, phones, cameras and a few other devices were excluded from this segment and had their own.

The departments affected were generally security, surveillance, cashier cage service counter, player club service counter, food services, counting room, gaming inspection, slot mgmt, tables mgmt, operations mgmt, facilities mgmt, custodial services, receiving and IT helpdesk.

Outside IT consultants can be very good. Or very bad.

Some context, the previous IT administrators were actually an outside consulting firm that came out and did IT work for both sites.

Needless to say, they were great at talking up large goals for infrastructure change and development, and had absolutely zero follow through, ending up in a spaghettified network full of crap configurations, SPOFs, and general lack of foresight and ability.

Having one guy manage the network is always a mistake.

Only the main-site gaming facility a few cities away had a de facto network administrator, an overworked sysadmin who managed basically every application and server and the network configuration cleanup after that firm was terminated.

The company would not approve a network technician for the off-site smaller gaming facility only a couple years after parting with that disaster.

This sounds like an interesting job.

I was working on helpdesk and was a fairly new unofficial off-site network technician working with approval and under the discretion of the main-site IT director.

I was working on organizing and relabeling the IDF cables with verbally approved minimal downtimes for each endpoint, manually clearing out bad switch configuration lines and replacing them with our preferred agreed upon configurations, and in general documenting the wild frontier we were stuck with.

Switches can work for a long time without updates.

These were the first major change these switches had seen in years, and it was clear that they had been manually configured at different times with different intents.

Many also had common bad practices security holes that are easily fixed with a line or two. At this point the IT budget was abysmal so there was no good remote management solution aside from the singular SecureCRT license afforded to the department, or custom PuTTY configs shared amongst us.

What is going to happen?

Well, one unlucky day on the gaming floor working on one unlucky access switch in particular, I was clearing the vlan database of unused entries.

At this point, I was new and self-taught mostly alone, and I was unaware of a certain unpopular protocol that would be my ultimate doom. Did i mention our enterprise was Cisco?

Ok, clearing unneeded vlans is good.

Well, I was just getting started and picked the first vlan to clear – the data vlan. On this access switch, for its purposes of connecting slot machines back to the distribution layer, it did not need this one.

So, I simply did my thing as I had on a few other switches beforehand, getting the hang of it, and entered the command “no vlan ” and saved. I didn’t notice any immediate change. I didn’t even notice my Wi-fi went.

That’s never a good sign.

Away from me all around the gaming facility, departments erupted into chaos. Although the slot machines kept going so the patrons were mostly unphased, all the customer-facing service counters, the point of sales, the back of house, security and surveillance, gaming operations, even our helpdesk lost network connectivity.

The phones worked. And I soon found out so did everyone’s legs and voices, as the IT office was swarmed a few moments after my return.

He better get it fixed fast.

I assured everyone I would look into the issue and get it resolved immediately, and I called up the IT director, who at this time was the best network engineer I knew with 20 years of experience, and I explained what happened and what I had been doing.

He instructed me to go to core switch at our site and manually connect to it, and check the VLAN database.

At least this guy knows what he is doing.

Checking, I found that the entry for data vlan was missing from the core switch. He instructed me to put it back and once I did and saved the config, everything came back up.

He informed me that I had fallen prey to the aforementioned consulting firm’s sloppy management practices. They had VTP still on site-wide, and even worse was that some of the access-layer switches were in server mode.

This should not even be possible.

What I had so innocuously done from the access switch on the gaming floor brought down pretty much the whole site in a moment. Luckily the core switch was also in server mode, so once I put it back the change was basically undone.

At that point we made it a policy to never allow VTP on the network.

Wow, that must have been a scary moment for this new employee. Fortunately, it worked out.

Let’s see what the people in the comments think of this story.

Yup, he handled himself well.