Deployed a backup script on a new server, tested it manually — worked fine. Set up crontab, came back next morning, no backup. Turns out /usr/local/bin wasn’t in cron’s PATH so pg_dump just silently didn’t exist.
Switched to absolute path and it worked immediately. Fifteen minutes of debugging for a one-character fix. I keep making this mistake every time I set up a new box. At this point I should just have a checklist taped to my monitor.
hey, at least you caught it within a day.
EDIT : Next time make the cron job run when you’re finished deploying it.
That’s one reason why I like using systemd timers instead of cronjobs. You can easily test it by running the unit instead of setting the job to “in a minute”.
Downside is that you don’t easily get notification mails for any output.
I’d recommend self-hosting healthchecks and using runitor for your cronjobs. Runitor pings healthchecks when the cronjob starts, then pings it again on completion with the status (success or fail) along with the stdout and stderr.
Healthchecks can be configured to expect a ping periodically (once per day, once per hour, whatever) and alert you if it doesn’t receive one.
For backups, Borgmatic has a healthchecks integration.
Looks like you need to collect your crontab logs somewhere you can read.
Or maybe even set up a valid MAILTO in the crontab so that failures are emailed to them.
My “pro” tips:
- set up an email alias from root to an email you actually read
- always use absolute paths
- anything complex, put it in a shell script
You can append
2>&1 | logger -t my-cronjobto any command, and it will write logs to system journal which you can view with
journalctl
