mastodon.sdf.org is part of the decentralized social network powered by Mastodon.
"I appreciate SDF but it's a general-purpose server and the name doesn't make it obvious that it's about art." - Eugen Rochko

Administered by:

Server stats:

2.3K
active users

Learn more

One of my Raspberry Pis is freezing every night, because the amount of I/O and CPU to run incremental backups is too high, and is causing other processes to misbehave.

Trying to decide whether I want to run backups from a different system (all data is in NFS), or run a different backup software (currently using Deja Dup). 🤔

#HomeLab

For now, since rebooting every morning is getting tiresome, I just created a script that will shutdown all the services before the backup starts, and bring everything up again after it completes. Should survive a couple of nights without intervention. #HomeLab.

Stopping all the services while running the backup not only stabilized the system, but it also cut the backup to 1/4 of the time. There's definitely some improvement to be made here. #HomeLab

Uh-oh. Spoke too soon. The Raspberry Pi just froze 8 hours after the backup has completed (with all services down). Poking around the logs, it seems something related to InfluxDB trying to compact old files, which overloads the CPU and saturates the I/O on the SD card (temp files?).

Suggestions? Upgrade to InfluxDB 1.11 or 2.7 (1.8 now)? Maybe convert everything to Victoria Metrics?

I would prefer to retain the data, if possible (historical knowledge).

#HomeLab #InfluxDB #VictoriaMetrics

offdutypirate

@badnetmask I've had OK luck using USB disks instead of SD cards for more I/O intensive stuff on the Pi. Maybe that's an option?

@offdutypirate
Not really. The apps are running out of an NFS mount, so in this case it's just a matter of redirecting the temp dir out of the SD card.