Close-up of a dark server board with blinking lights

My Cron Job Ran Fine by Hand and Failed Every Night for a Month

Posted on by Navid

A month. That’s how long our nightly report job ran at 1 AM, exited politely, and produced absolutely nothing. I found out because our biggest client asked why the Monday report was three weeks old.

The Scene

The job was embarrassingly simple. A bash script that generated a CSV export and dropped it in S3, every night at 01:00. I wrote it in one sitting, tested it by hand twice, considered it done. Cron was supposed to be the boring part. (You see where this is going.)

The Spiral

First move: SSH in, run it by hand. Works perfectly. CSV generates, file lands in S3. Shrug, must have been a one-off. Bad call.

Day two I got properly paranoid: script permissions, S3 credentials, disk space. All fine. Then /var/log/syslog showed this, every night at 01:00: CRON[4123]: (navid) CMD (/opt/scripts/nightly-report.sh). So cron was running it. The file still didn’t exist. By then it’s 1 AM (fitting, I know) and I’m reading forum threads where every answer is “check your permissions”, which I had. Four times.

Still with me? Good, because the next part is the humiliating one.

The Discovery

The script ended with node /opt/app/export.js. Node, installed the way I always do, via nvm, in ~/.nvm/versions/node/v20.11.1/bin. When I SSH in, my shell loads nvm and everyone is happy. Cron does none of that. Cron runs your command with a bare PATH=/usr/bin:/bin, no dotfiles, no nvm. I proved it in one line:

Ran the script the way cron sees it: env -i /bin/sh -c '/opt/scripts/nightly-report.sh'. Output: node: command not found. Every night, for a month. And the crontab line had > /dev/null 2>&1 on the end, added because a blog post said cron output gets noisy. The job was trying to tell me. I’d taped its mouth shut.

The Fix

One line at the top of the crontab:

PATH=/home/navid/.nvm/versions/node/v20.11.1/bin:/usr/bin:/bin

That’s it. The next night the report generated at 01:00 like nothing had happened. I’m not proud of the two days it took to check everything except the one difference between my terminal and cron’s world.

The Lesson

  • Cron is not your shell. It starts nearly empty: no nvm, no rbenv, none of your dotfiles. Every binary needs an absolute path, or a PATH line at the top of the crontab.
  • Silent failure is usually self-inflicted. I added > /dev/null 2>&1 for tidiness. Shorthand for “never tell me about your problems”. So it didn’t.
  • syslog proves the job started, not that it worked. That CMD line is a timestamp with a command attached, nothing more.
  • Testing a cron script from your own terminal tests nothing about cron. One env -i run ends this on day one.

What I’d Do Different

  1. Absolute paths for every binary in anything cron touches. which node once, paste, done.
  2. PATH set explicitly at the top of every crontab, even when I’m sure it won’t matter. Especially then.
  3. stderr to a log file I actually open, not /dev/null.
  4. A heartbeat ping on important jobs, so a missing run alerts somebody before a client does, three weeks too late.

So yeah. A month of a job doing nothing, fixed by one line at the top of a crontab. I check /var/mail/navid every morning now, like a responsible adult. Small wins.