A few days ago, a client’s data center (well, actually a server room) "vanished" overnight. My monitoring showed that all devices were unreachable. Not even the ISP routers responded, so I assumed a sudden connectivity drop. The strange part? Not even via 4G.
I then suspected a power failure, but the UPS should have sent an alert.
The office was closed for the holidays, but I contacted the IT manager anyway. He was home sick with a serious family issue, but he got moving.
To make a long story short: the company deals in gold and precious metals. They have an underground bunker with two-meter thick walls. They were targeted by a professional gang. They used a tactic seen in similar hits: they identify the main power line, tamper with it at night, and send a massive voltage spike through it.
The goal is to fry all alarm and surveillance systems. Even if battery-backed, they rarely survive a surge like that. Thieves count on the fact that during holidays, owners are away and fried systems can't send alerts. Monitoring companies often have reduced staff and might not notice the "silence" immediately.
That is exactly what happened here. But there is a "but": they didn't account for my Uptime Kuma instance monitoring their MikroTik router, installed just weeks ago. Since it is an external check, it flagged the lack of response from all IPs without needing an internal alert to be triggered from the inside.
The team rushed to the site and found the mess. Luckily, they found an emergency electrical crew to bypass the damage and restore the cameras and alarms. They swapped the fried server UPS with a spare and everything came back up.
The police warned that the chances of the crew returning the next night to "finish" the job were high, though seeing the systems back online would likely make them move on. They also warned that thieves sometimes break in just to destroy servers to wipe any video evidence.
Nothing happened in the end. But in the meantime, I had to sync all their data off-site (thankfully they have dual 1Gbps FTTH), set up an emergency cluster, and ensure everything was redundant.
Never rely only on internal monitoring. Never.
#IT #SysAdmin #HorrorStories #ITHorrorStories #Monitoring
Schmaker
Unknown parent • •Neni to ještě v pohodě, uráží to mý cítění pro design.
Všimni si - bez reklam krásně klidnej blankytnej den, do kterýho rád vstaneš. S reklamama? "STO TABLETŮ TÝDNĚ, NAKUPTE KRETÉNI!!!! .. jo a taky se můžeš podívat, jaký bude počasí"
Schmaker
Unknown parent • •Schmaker
in reply to Schmaker • •Přišla odpověď .. tato stránka není přehlcena reklamou 😁
like this
muž s klapkami na očích likes this.
FoxVK
in reply to Schmaker • • •Schmaker
in reply to FoxVK • •Jan Antoš
in reply to Schmaker • • •Schmaker likes this.
Jan
in reply to Schmaker • • •Schmaker
in reply to Jan • •Mě vlastně celkem baví ty zpětný vazby dávat. Člověk si udělá představu, co tam sedí za člověka.
Musí se ale nechat, že odpovídat v neděli ráno je víc než slušný
Jan
in reply to Schmaker • • •Schmaker likes this.
Schmaker
Unknown parent • •Piggo
Unknown parent • • •Schmaker likes this.
Piggo
in reply to Piggo • • •Schmaker likes this.
Jan Antoš
Unknown parent • • •Jan Antoš
in reply to Jan Antoš • • •FoxVK
Unknown parent • • •Schmaker
Unknown parent • •V pořádku. Já mám adblock a když ho začnou přečůrávat, prostě tam nebudu chodit.
Dělám to tak i s neodmítnutelnýma cookies
Koubik
in reply to Schmaker • • •Schmaker
in reply to Koubik • •Já bych ho zkousnul i kdyby se hejbal, ale jak říkáš - pak přibyde Alzák a najednou koukáš na informace v reklamě 😁
Mně v principu nevadí in-line reklama, klidně video, ale nesmí zvučit. Pokud by to tak bylo, adblock bych jim vypl a měli by ze mě aspoň něco. Ale takhle?
Josef Zettl alias Princezna
in reply to Schmaker • • •Daniel Tóth 🌿
in reply to Schmaker • • •Schmaker likes this.
Schmaker
in reply to Daniel Tóth 🌿 • •@Daniel Tóth⚡
Vidím, že užívatel už je dneska reklamama celkem otupělej. Já pamatuju začátky internetu a - světě div se - uživili se i s reklamou neintruzivní.
Snižovat standarty nebudu 😀
Unreed
in reply to Schmaker • • •Schmaker likes this.