Imagine working in tech support and volunteering to take on the job of figuring out why there is a lack of security on some workstations. Do you think the problem would be easy to solve?

In this story, it sure wasn’t!

The person who took on the task found it extremely interesting, but as he does some digging, it sounds more and more frustrating as he uncovers what’s going on. He keeps working towards a solution, but he still doesn’t sound confident that it’s going to work.

Let’s read all about it.

160 Manhours (so far)

The main endpoint security system for the MSP I work for is basically a host-IDS powered by machine learning. Let’s call this system “MIDS”.

In October someone suspects that there are workstations without MIDS and set up SNMP alerts.

Over the course of the day we find probably 300 machines (out of 2k) that don’t have MIDS.

Since my long term interests are in security, I volunteer to fix this.

He gets to work.

Our RMM lets me upload files (such as the MIDS installer) and run shellcode (such as the install command) without bothering the user.

Keep that workflow in mind, it will be important.

After about a week I’ve gotten most of them taken care of, but there are some that the install process fails on.

Through some digging I realize that these have some of the services and some of the registry keys, but not all of them.

This sounds frustrating.

So I email the vendor.

They explain that it looks like these are failed installs or failed updates.

Why would this happen? I will ask months later and the answer is basically 🤷

There are three options.

MIDS isn’t in the control panel, but the vendor shows me how to uninstall it:

Option 1: The server’s web console. But, if the install is broken, it probably isn’t talking with the server, right?

Option 2: A shell command that requires an uninstall password. The command may not work, possibly because the password hash on the endpoint is corrupted

Option 3: Go into the advanced boot menu, delete the services, delete some stuff from C:\Program Files, delete some stuff from C:\ProgramData, reboot, delete some registry keys (huge pain to delete that many keys from the command line), reboot, and now you can install it. Except, sometimes there’s no command line option in the advanced boot menu, and sometimes when you navigate to the C: drive it just doesn’t believe that there are any directories. Then you… have to reimage? I haven’t tried to figure an alternative yet.

There’s also the forced update tool, but I’ve never gotten it to work, so I’m not going to count it.

OP starts to figure out what’s happening.

Earlier this year I was doing something on a server and realized it had a failed MIDS install.

In a rage I spend a day going through the list of computers in the MIDS web console and the RMM and find another TWO HUNDRED devices that have some kind of problem.

Turns out, the monitoring we setup was only on workstations, not servers. And was based on the presence of a particular file that is deleted then remade during the update process. So if there’s a failed upgrade, the alert is triggered. But if the upgrade just never starts (or fails really early or really late in the process), no alert trigger.

Also, the name in the MIDS server is the name the machine had when MIDS was installed, not its current name. And machines that have had MIDS uninstalled are still in the server. This is a big part of why it took me 6 hours to audit 2k machines.

But a customer has a problem.

When I’m down to under a hundred problem machines, one of our customer’s starts having work-stoppage system slowness.

Yes, it was because of having MIDS and Defender running at the same time.

No, we didn’t disable Defender with GPO. Yes, ownership was mad at me. No, it could not possibly be all my fault.

Recently, now down to ~50 issue machines, the owner realizes a VM host is like three years out of date. He asks me if there’s a way to get alerts about this so we don’t go three freaking years with a vm host having basically no security. Well, actually, he sent a furious message in Teams about why this happened in the first place, then asked about monitoring after I explained it.

Never Miss a Story

Sign up to get our BEST stories of the week straight to your inbox.

This sounds frustrating.

The vendor’s answer seems to be “lol, no, we don’t have monitoring for that.”

But, I happen to know already that there’s a log file that updates every five minutes when it checks in with the server. And it includes the current version. Which means we all get to hope I can figure out enough about SNMP to query this file on at least our servers, because if not, I think my boss is going to have a stroke.

Also there are 12 VM’s that need to have uninstall option 3 done. Can you go into advanced boot menu in a Hyper-V VM?

Not sure. Hope so.

I don’t think I’m techy enough to catch all that, but it sounds really frustrating.

Let’s see how Reddit responded to this story.

This person shares their experience.

Another person offers a suggestion.

I agree that this sounds like a mess.

I know he took on this issue willingly, but I’d regret that if I were him. It sounds like a very frustrating problem to solve.