A Triumvirate of Stupidity
Whew. I have to consider myself pretty lucky. I haven’t encountered enough stupidity to be worth ranting about in almost a year, but as guaranteed by interacting with humanity, that time has come to an end.
See, I’ve been decently happy at my current job. I’ve got a cool NOC role and spend a lot of time automating all the things, and I’m learning to wear that coveted badge called DevOps. I’ve been able to avoid most Layer 8 issues and idiot vendors, insulated by my lair of monitors and Python code. But that all came to a crashing halt several weeks ago, when a Level 2 technician (who I would consider Level 1.2 because lol he had like 10 Microsoft certs but didn’t know how to allow a port in a Sonicwall) departed for, oddly, another MSP.
Why would he do that? I’ll never know. But I feel for the poor MSP who had no idea what they were hiring (a paper tiger cert hoarder).
Anyway, the Higher Ups decided they didn’t need to replace him. Why not let the NOC engineer take some tickets even though he’s already pretty busy? He’s the only person besides the CTO who can write Powershell and Python scripts and automate, but, eh, lol why not.
So fine, I’ll take care of some stupid users.
But it all came to a head today because I can tolerate stupid users, for the most part. I cannot tolerate stupid vendors and people who think they’re good at IT.
Let us begin:
I. Stupid Web Host/Designer Trying to Blame Us for Issues
I fucking hate web hosts. I have yet to encounter one that is competent at administration tasks, like setting DNS. These fuckers, whose name ends in Site Solutions, initially whined about not having control of our client’s DNS. They relented and gave us the records they wanted to change. Great, I changed the records for them, and then the client’s website loaded, but it was giving an SSL certificate error. I let the client know to contact their web host since that’s their issue, BUT the unfortunate fact is that I trusted the web host to be competent when they were not. They were indeed, shit.
Our client tries to contact the web host. For FOUR days. They were given the runaround bullshit explanation that they needed to clear their browser cache. Repeatedly. Then they sent a message that DNS wasn’t changed. So our client came back to us and thought we were in the wrong.
No. Fuck that shit. I took screenshots of the lookups from mxtoolbox and sent them back to Shite Solution’s support along with their new message which wanted a different A record IP, and I made sure to attach a screenshot of the original e-mail requesting a different IP. Those fuckers weren’t about to get away with boldface lying to our client, because they made it personal and tried to pin it on me. No, Tech-Dragon was now on the case, and they were gonna fix their shit or get flamed.
Funky thing was the original IP was actually loading the website. I hounded their asses on the phone and verified the correct IP address. They claimed the original server I pointed DNS to was theirs, explaining why the website loaded, but it had a broken certificate so changing it would fix it. LIES. Changing the A record to the new IP did nothing.
But our end was done. Fixing the SSL certificate was beyond my power. DNS was pointing to where Site Solutions wanted it. I sent more screenshots and let our client know they should contact the web host about the SSL errors. They were not fully managed anyway so it didn’t make sense to spend money every minute to have me contact them when the conversation should have been this easy:
Client: “Hello, our website is showing SSL errors.”
Site Solutions: “Yes, we see and shall fix it immediately.”
Instead, Shite Solutions support kept telling our client to clear their browser cache again. What the fuck? I look at the damn SSL certificate and it’s broken because it’s self-signed and there’s no issuer. HELLO?
Our client let us know they were being given the runaround. Tech-Dragon’s on the case.
I call support and MAKE them load the website. “Oh, it does say it’s insecure.” NO SHIT YOU FUCKWADS. ARE YOU TELLING ME YOU NEVER ONCE ACTUALLY CHECKED THE WEBSITE WHEN OUR CLIENT CALLED IN?
Oh, and it gets even better! The reason the SSL certificate was self-signed? “Our records indicate <Client Name> never purchased an SSL Certificate.”
What.
So let me get this straight: you fuckers told our client to keep clearing their browser cache and never actually looked at the website’s certificate, and you were telling them this when the whole time they hadn’t even purchased one from you?
Great. Fine. They told me to tell our client to call their sales rep to buy an SSL certificate.
The cherry on top: our client says they tried to buy one, but their sales rep had gone home for the day and he’s the only person authorized to make this purchase. My. Fucking. God.
And you wanted DNS control? You dipshits are never going to get that shit now, ever. Fuck y’all.
Moving on to the next hour of my day…
II. Stupid Idiot Pretending to be IT at Another Client Because He’s the Owner’s Son
Ticket comes in. IT person at one of our managed clients has a question about our antivirus. Okay fine.
I call.
Ten minutes later I already want to strangle this “IT person.”
He asks me some wonderful nuggets like: “How are IPs assigned, are they like random or what?”; “What’s a public IP?”; “How do ports work?”; “I found a Sonicwall in the garbage, it looks like it’s a newer model than ours, how do I replace ours with it?”
I hope that last question raised an eyebrow, because the questions before it pretty much should flash a giant warning, “I AM IN IT BECAUSE I BUILT A GAMING COMPOOTER ONCE AND MY DAD THINKS I SMART BUT I REALLY HAVE NO BUSINESS TRYING TO INSTALL A SONICWALL IN A PRODUCTION ENVIRONMENT.”
But of course I can’t just tell him he’s retarded so I answer his questions while thinking it. Loudly. In my head, of course. An hour of idiot IT questions seriously threatened to bring those thoughts to my vocal cords though.
Cue e-mail to account manager warning him their client’s internet is probably going to go down sometime next week.
III. Stupid Solarwinds Support’s Silly Response
I modified an n-Central custom monitor that runs on an AMP that has a Powershell script embedded into it two days ago. I updated the AMP, and the service shits itself and won’t read information anymore. The AMP runs in Automation Manager. The Powershell script gives no errors. I triple check the damn script for any typos, and actually find one that wouldn’t break the damn thing but make an output null, which I know automation policies can handle. I fix that. All my devices with this monitor show No Data or Stale status, except for strangely a handful so I know it has to be working, but have no idea how. I peruse the n-Central logs and they’re useless. They only seem to tell me if an AMP works and doesn’t give me any information on timeouts.
I open a ticket to Solarwinds support.
I expect very little because of the dark and terrible words, “Powershell” and “Automation.” I get a little worse than what I expected.
Yesterday, they told me to try recreating the policy in the new version of Automation Manager (version 12). Fine, I try that. Surprisingly, it works on a test device. I try propagating it to all devices. It doesn’t work. The next day, they’re all stale.
I respond with this information. They surprisingly reply the same day (except almost at the end after 5pm), something they haven’t been able to do in ages.
Solarwinds support tells me this requires restarting the n-Central agent service on the devices to update the modified AMP into their cache. Oh, that curse word again today. Cache. Motherfucker. And it doesn’t make sense either. I’ve never had to restart the agent service to get an updated AMP monitor working before. This monitor is deployed on servers, meaning I have to restart the agent service on all of them and risk the agent not coming back online. Fabulous. This clearly can’t be as intended. Unless they’re telling me the version 12 update made it so, but that would be retarded.
I’ve had enough stupidity for one Friday evening. I forward this shit to my boss who will message his higher up n-Central contact about this garbage.
It’s Friday night. Fuck it all.
