Deployed a backup script on a new server, tested it manually — worked fine. Set up crontab, came back next morning, no backup. Turns out /usr/local/bin wasn’t in cron’s PATH so pg_dump just silently didn’t exist.
Switched to absolute path and it worked immediately. Fifteen minutes of debugging for a one-character fix. I keep making this mistake every time I set up a new box. At this point I should just have a checklist taped to my monitor.
I’d recommend self-hosting healthchecks and using runitor for your cronjobs. Runitor pings healthchecks when the cronjob starts, then pings it again on completion with the status (success or fail) along with the stdout and stderr.
Healthchecks can be configured to expect a ping periodically (once per day, once per hour, whatever) and alert you if it doesn’t receive one.
For backups, Borgmatic has a healthchecks integration.