A few days ago, a client’s data center (well, actually a server room) "vanished" overnight. My monitoring showed that all devices were unreachable. Not even the ISP routers responded, so I assumed a sudden connectivity drop. The strange part? Not even via 4G.
I then suspected a power failure, but the UPS should have sent an alert.
The office was closed for the holidays, but I contacted the IT manager anyway. He was home sick with a serious family issue, but he got moving.
To make a long story short: the company deals in gold and precious metals. They have an underground bunker with two-meter thick walls. They were targeted by a professional gang. They used a tactic seen in similar hits: they identify the main power line, tamper with it at night, and send a massive voltage spike through it.
The goal is to fry all alarm and surveillance systems. Even if battery-backed, they rarely survive a surge like that. Thieves count on the fact that during holidays, owners are away and fried systems can't send alerts. Monitoring companies often have reduced staff and might not notice the "silence" immediately.
That is exactly what happened here. But there is a "but": they didn't account for my Uptime Kuma instance monitoring their MikroTik router, installed just weeks ago. Since it is an external check, it flagged the lack of response from all IPs without needing an internal alert to be triggered from the inside.
The team rushed to the site and found the mess. Luckily, they found an emergency electrical crew to bypass the damage and restore the cameras and alarms. They swapped the fried server UPS with a spare and everything came back up.
The police warned that the chances of the crew returning the next night to "finish" the job were high, though seeing the systems back online would likely make them move on. They also warned that thieves sometimes break in just to destroy servers to wipe any video evidence.
Nothing happened in the end. But in the meantime, I had to sync all their data off-site (thankfully they have dual 1Gbps FTTH), set up an emergency cluster, and ensure everything was redundant.
Never rely only on internal monitoring. Never.
#IT #SysAdmin #HorrorStories #ITHorrorStories #Monitoring
:
Schmaker
in reply to Schmaker • •Dear #PeerTube admins - guy from #Mozilla team kindly asked me to stream "Browser and web platform" room as well.
As resources are limited, we are only going to stream 3 rooms. Is there any instance admin able to stream this room so we can federate together?
Please boost, time is running out slowly 😀
reshared this
Archos, Milan Hubáček, jacqueline, Nico Rikken, Huubje, Valentijn Sessink a Mirek Svoboda reshared this.
MakerTube
in reply to Schmaker • • •Schmaker
in reply to MakerTube • •Thank you kindly for your offer, that's amazing! 😀
I can't run it on my own as it requires ffmpeg (preferably on stream host machine) as well. I'm coordinating this room with peertube.uno admins now.
If you would like to join us to serve more rooms or there will be any more rooms requested, can I count on you? Are you able to run ffmpeg on your server for copying the stream from FOSDEM?
We can create federated "mesh" with #FOSDEM streams, there are 28 more rooms to go 😁
jacqueline reshared this.
MakerTube
in reply to Schmaker • • •Schmaker likes this.
Schmaker
in reply to MakerTube • •Sure 😀
1. Create separated FOSDEM channel on your instance
2. Federate with fosdem@VHSky.cz (and tell me your channel link so I can federate back)
3. Choose room(s) from FOSDEM you would like to stream (so far we have both Mains, Social networks and Web already settled, others should be free to go 😀)
fosdem.org/2026/schedule/rooms…
4. Create recurring livestream with 60s cache for most usage of P2P
5a. Feel free to test if it's working, the command is:
ffmpeg -re -i https://stream.fosdem.org/aw1120-1080p.m3u8 -c copy -f flv rtmp://<instance>/live/<KEY>5b. Set up ffmpeg for the actual stream and don't forget to run it when FOSDEM starts
ffmpeg -re -i https://stream.fosdem.org/<ROOM>-1080p.m3u8 -c copy -f flv rtmp://<instance>/live/<KEY>6. Profit 😀
I think it would be handy to give the channel also a playlist with all live rooms for switching, but not sure about that
Also feel free to contact me on Matrix @Schmaker:schmaker.eu
Edit: Fixed the rooms link
Edit2: Fixed the handle to federate - it has to be handle, not URL!
FOSDEM 2026 - Rooms
fosdem.orgMakerTube
in reply to Schmaker • • •I created a new channel makertube.net/c/fosdem and sent a federation request to your account handle. If have somehow trouble replicating the test stream. I get the following output:
[hls @ 0x55bb3cda0200] Skip ('#EXT-X-VERSION:3')
[https @ 0x55bb3d175240] Opening 'stream.fosdem.org/aw1120-1080p…' for reading
[https @ 0x7fde94007540] Opening 'stream.fosdem.org/aw1120-1080p…' for reading 1x
Any ideas?
fosdem
makertube.netSchmaker
in reply to MakerTube • •MakerTube
in reply to Schmaker • • •Untitled - Pastebin
PastebinSchmaker
in reply to MakerTube • •I wonder what could possibly go wrong. It seems like it opens the stream just fine. I wonder, that PeerTube might block incoming packets from local address.
Can you try the public url of stream instead?
We tried only with public adress and the testing stream worked
Schmaker
in reply to MakerTube • •MakerTube
in reply to Schmaker • • •Schmaker
in reply to MakerTube • •That's wierd, there is clearly something wrong on PeerTube "input side" - I just double checked and we only copy the stream from one part of the internet to our PeerTube instance just fine.
Maybe some missing lib?
Encoding should happen on PeerTube's side...
MakerTube
in reply to Schmaker • • •Schmaker
in reply to MakerTube • •MakerTube
in reply to Schmaker • • •MakerTube
in reply to Schmaker • • •Schmaker
in reply to MakerTube • •MakerTube
in reply to Schmaker • • •Schmaker
in reply to MakerTube • •Thank you!
Can I ask you for reserving one for the Web browsers room in case PeerTube.uno for some reason would fail? Just in case?
MakerTube
in reply to Schmaker • • •Schmaker likes this.
Schmaker
in reply to MakerTube • •Not really .. I only was asked to make the browser room happen.
Choose your bests 😀
Schmaker
in reply to MakerTube • •So, it's second day peertube.uno is quiet about livestreams, so I'm giving up on them and want to confirm that you gonna stream the Web browsers room. Thank you.
Also - can you set up these streams already so I can propagate them to unified playlist?
MakerTube
in reply to Schmaker • • •Schmaker likes this.
Schmaker
in reply to MakerTube • •MakerTube
in reply to Schmaker • • •tH91FM3JdSe7b3cnqsfhmW
makertube.netSchmaker likes this.
Schmaker
in reply to MakerTube • •Just added them to our unified playlist .. feel free to add ours to yours as well. Thanks for the help once again!
Schmaker
in reply to MakerTube • •Just created thumbnails for you in case you're interested
next.oscloud.cz/s/DR2jmcJsgHT6…
MakerTube
in reply to Schmaker • • •Schmaker likes this.
Nico Rikken
in reply to Schmaker • • •Schmaker likes this.
Schmaker
in reply to Nico Rikken • •Schmaker
in reply to Schmaker • •@Fedi.Video and #PeerTube admins:
If you want these streams to show up on your instances, you have to follow our fosdem channels using the handle. If you follow them using URL, they acts like they are successfully connected, but are not.
MakerTube reshared this.