Follow

So you have a server at home, and you're on the road. You find that your server is offline, but nobody is at home to troubleshoot. Your modem, router, or server probably just need to be rebooted, but because your server is down, you can't remote in to trigger a reboot.

How do people solve this problem?

@pcrock I haven’t tried to do this before, but maybe a RaspberryPi based KVM?

@joseph I'm leaning in the same direction. Though a KVM solution only works if the router / modem is still functioning properly and it's just your server that needs to be rebooted. I'm thinking the RPi would still need to be smart enough to know when to turn things off and back on again without my intervention...

@pcrock maybe a smart outlet combined with a custom script that stops / starts power to the modem if it can’t reach google or some other IP with expected 100% uptime.

@pcrock it’s very difficult. There’s no simple way to get around it, and part of the reason I don’t host anything that requires uptime (like email) at home.

@kev Agreed. I have stuff like a VPN and Jitsi server in the cloud, but some things like a personal workstation are best kept at home. I am thinking about becoming more dependent on remotely accessing a home workstation instead of requiring an expensive powerful laptop. But Internet connection reliability becomes an issue at that point...

@hund If only I could replace a faulty ISP.... Sadly, "turn the modem off and back on again" is too often the solution, and I _suspect_ those issues have more to do with the ISP. I suppose I could try to find a smarter modem that could reset itself...

@pcrock TP-Link HS110 controlled through Home Assistant, with the server set to auto power-on after a power outage. Normally it’s on a sealed-off iOT VLAN, but I open a connection in my firewall to the plug if I’m going to be away for weeks at a time. I think Linus tech tips did a video a while back looking at different solutions to this exact problem.

@pcrock best solution would be to set up a cronjob to cURL reset the modem every x minutes you can't ping any of a list of ip addresses

@pcrock
I don't 😁
"Sensitive" services are not at home, and that's it.
I know, this does not help a lot. 😉

@pcrock If you've had a failure, and that includes a software one, that renders the system inoperable the only solution I've found is manual intervention.

I had a situation where I was away from holiday and an OS bug caused my server to freeze. A helpful neighbour (with keys) and instructions on how to restart the system was the only solution.

@neildarlow @pcrock I was gonna say this. Sometimes the best solution is analog. Although I also like the cron job that reboots the router when internet connection is lost.

@pcrock rpi with failover gsm modem. Or.. if just an issue with server.. rpi to serial console of server maybe of course hooked into switch so can perform power reset of server.

One of the people I follow on twitch actually started doing something like this

@geekgonecrazy Ohhh... yeah GSM failover is totally an option for me! Now to find a smart plug (or better yet, UPS) that I can control without a LAN.

@pcrock I know you can monitor a lot of UPS with serial.. but not sure if I’ve ever seen a reboot from that serial connection.. 🤔

@pcrock what if the server has an encrypted boot. :/ Any way of solving that issue?

@jordan31 There's a good argument for disabling drive encryption for servers. That said, I'd probably configure my server so that it does nothing more than host VMs and containers, and I'd use drive encryption inside those VMs / containers.

@jordan31 @pcrock IPMI so you can enter the boot password. That's how I keep all my servers encrypted
@pcrock reliable firewall that doesn't crash/go down, always able to VPN to that. Get a network controlled power outlet on your cable modem and have the firewall power cycle it if the internet is unreachable for 10 mins. IPMI on the server.

@pcrock
Back in the old paging days, I used a pager. I opened up a pager, hijacked the output of the relay that drove the vibrate motor and ran that to a relay on the power input of a computer. RadioShack had a battery replacer that plugged in with a wallwart to keep the pager powered. Worked like a champ. Of course if someone accidently paged that number, I had an unintended reboot to deal with.

@pcrock
I suppose you could do the same with an older or cheap cellphone, plugged in all the time using a Ting SIM to keep costs down.

@oranje That is _awesome_! Makes me want to whip out the old electronics toolbox again...

@pcrock A :raspberrypi: with RealVNC and alternative network connection.

@pcrock for server restarts... mtlynch.io/tinypilot/ comes to mind (RaspberryPi + a $30 HDMI grabber)... as for modem restarts, most smart plugs seem to be cloud-only, but e.g. etekcity ones seem to be using plain HTTP so you could probably switch them from the LAN with some DNS hackery?

Sign in to participate in the conversation
Fosstodon

Fosstodon is an English speaking Mastodon instance that is open to anyone who is interested in technology; particularly free & open source software.