I think job scheduling capability (not to be confused with process scheduling) is a very important, yet somewhat neglected aspect of an operating system. I have worked on big and complete production systems that actually do nothing but run scheduled jobs at their predefined intervals. And Cron is probably the most common job scheduling system on Unix-based systems. As a good Unix citizen, cron conforms to the “do one thing and do it well” mantra, which means it leaves out many features to other tools which are purposefully built for them. These features are critical when devising fairly demanding scheduling schemes. Curiously I have not seen people talk about the intersection of cron with the rest of a Unix system as much as they should. As a result, compared to other job scheduling systems like the one on recent versions of MS Windows Server operating system, cron might seem very lacking.

I’m gonna try to change this perspective by introducing some tips and tricks. This collection of random points are what I have found to be some best practices on job scheduling via cron. A basic knowledge about cron is assumed.

• Users should use crontab command to create, edit, list or delete their very own crontab file and manage their set of cron entry lines. Although the set of crontab files for all users are stored in /var/spool/cron/crontabs directory, they are not intended to be edited directly. Users can run crontab -e to edit the file or crontab -l to list all their scheduled tasks. Software packages, on the other hand, should drop a separate crontab file in /etc/cron.d directory to setup their scheduled jobs. The format of these files is a little different from user crontabs. Debian has mandated a user name field before the program name, using which the program will be run. Cron package lets the system administrator create allow and deny lists to choose which users can access the service. On Debian, these files are /etc/cron.allow and /etc/cron.deny respectively. Generally cron package in Debian has been heavily patched with many customizations and deviations from upstream defaults. This is something to consider when using multiple distros.

• Use flock program in a crontab entry to ensure only one instance of command runs at a time. If the program creates side effects like changing the state of a file or database, and it has not been designed to be graceful about multiple instances of itself competing to change the state at the same time, using this little utility is a must.

  0 * * * * flock --verbose --exclusive --wait 60 $HOME/my_program.lock -c "/path/to/my/program"
  

• Use timeout command to put an upper limit on how long the job can stay running. Combine this command with flock to make sure multiple instances of a job do not run at the same time, and a single instance does not take forever to run either. For example:

  0 * * * * timeout --signal=TERM --kill-after=60s 15m flock --verbose --exclusive --wait 60 $HOME/my_program.lock -c "/path/to/my/program"
  

  Play with the command line parameters as necessary. This has been a life-saver for me, because most scheduled jobs are untested scripts created by a single developer with no review that are demanding on system resources and run unattended! It’s great that we can leave out monitoring and locking from our programs and rely on well-tested Unix utilities for that.

• cron cannot be relied on to keep a history of when it runs the scheduled programs, when they finish executing and whether they finished successfully or not, including their exit code. To be fair, the cron daemon can be invoked like cron -L 15 to produce the most verbose logs, including some of the information above. But the truth is in reality most distros do not run the cron daemon like this and as a result only the start of cron jobs are logged. On Debian, /etc/default/cron can be edited like this to enable the highest amount of logging:

  EXTRA_OPTS='-L 15'
  

  After which cron messages are logged via the ‘cron’ facility to syslog. Also to debug cron issues, it is possible to run it in the foreground with highest log level like this: cron -f -L 15 and then watch for cron messages.

  So, lack of proper event logging by cron makes it crucial for the program itself to log necessary events. Shell redirection of the program output to a file is handy but it does not really fit the bill in production systems, because it cannot connect to syslog for serious log management. Syslog is a great Unix service which provides centralized logging facility to other running programs. Fortunately syslog, provides a handy command-line tool to equip existing programs with simple logging without the need to fiddle with the code of programs which do not support syslog internally. Use logger in combination with shell output piping to log any terminal output that the command could produce for later inspection. Output lines are automatically timestamped and tagged.

  * 2 * * * * /path/to/my/program |& logger --tag my_program --priority user.notice
  

  Optionally an rsyslog configuration file can be specifically written for this program to customize how rsyslog treats them. For example to forward all messages tagged as above into a separate log file, create /etc/rsyslog.d/50-my_program.conf to look like this and then restart the rsyslog service:

  :syslogtag,isequal,"my_program:" /var/log/my_program.log
  &~
  

  Logging via syslog offers much more flexibility about what to do with log lines compared to the email functionality in cron. rsyslog is able to format, buffer, store and forward log messages into various sinks according to a filter criteria. Output methods like a file, a database table, a network socket, or to another rsyslog instance over the network are supported and pluggable modules can extend its functionality even further.

  For the example above, don’t forget to set up an automatic log file rotation scheme like logrotate if the program spews so much output that it might fill the entire disk. An example logrotate config file could look like below and should be placed in /etc/logrotate.d/my_program:

  /var/log/my_program.log {
      daily
      rotate 5
      compress
      notifempty
      missingok
  }
  

  For simple scripts, shell redirection could also work, otherwise all output printed by the program to stdout and stderr during cron execution is lost.

• Other Unix utilities that might be beneficial to incorporate into the command line are: sudo, nice, time, taskset, cpulimit, chrt, parallel, pidstat, chronic, etc. All these programs support command line parameters to customize their default behavior. cgroups can be used to further confine a process to a control group which can then be used to limit its visibility to the rest of the system and manage its resources. See man cgroups for more information.

• If all we need is to run a job at a specific time and not periodically at specific intervals, at program, not cron should be used. I say this because I’ve seen people adding a cron entry for midnight tonight before leaving work and then commenting it out the next day. at is the safer way to do this.

  at midnight <<EOF
    reboot
  EOF
  

  In all honesty, for such a use case, a crontab entry like below that specifies all 5 criteria probably will run the job this midnight and hopefully not match again for a year from now, but still it visually pollutes the crontab file.

  # m h dom mon dow usercommand
  45 2 22 feb wed /path/to/program
  

  There is also a noteworthy gotcha in cron that could result in a bug in the example above. Unlike the minute, hour, and month selectors which are anded together to find a match, ‘day of month’ and ‘day of week’ selectors, if both present, are ored together and then anded to the rest of the selectors. For any single crontab entry, I try to skip one of them (always set it to *) to avoid this source of confusion.

  For simple one-off job scheduling at command and its job management associates atq and atrm commands have worked flawlessly for me. These commands depend on a different daemon than cron called atd which pops the jobs from the queue and runs them at the specified time. Command composition with utilities such as the ones mentioned above works for at too, thanks to Unix philosophy!

• Use bash as the SHELL to avoid surprises when writing complex command lines that take bash facilities for granted. Put this at the top of the crontab.

  SHELL=/bin/bash
  

  The default shell is /bin/sh.

  While talking about shell, its worth mentioning that all job workflow mechanisms supported by shell are readily available to a cron job because the program section of the line is simply passed to the shell to be executed. Specifically ;, &&, || and ! operators in a bash command list allow easy definition of dependency among programs in a job based on the success or failure of each other.

  15 2 * * * filename=backup_$(date +"%Y_%m_%d_%H_%M_%S").sql; pg_dump -f ${filename} postgres && gzip ${filename}
  

  bash conditionals allow checking for further criteria before running the job, like checking for a flag indicating of the job being disabled or being already run. bash loops allow job retires in case of failure and running the job more than once a minute. If the command becomes too long, it maybe the time to move it into a separate shell script because crontab does not support multiline entries.

• Default value assigned by cron to PATH environment variable in crontab files could also be a source of surprise and frustration, because it is set to `/usr/bin:/bin. To set a less restrictive value for this variable, try adding this line at the top of crontab file:

  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  

• Disable the email notification in crontab by adding the following directive at the top. This effectively empties this reserved environment variable. Logging combined with log monitoring is how service notification should be done. I don’t like to get automated and unfiltered emails about every single run of the job, especially when no mail server is installed on the local system. Because these emails are perpetually accumulated on the local disk and are in fact the cause of a service outage waiting to happen.

  MAILTO=""
  

• Assigning to reserved variables like SHELL, PATH and MAILTO at the top of the crontab changes the default behavior of cron. But cron also supports assigning to arbitrary variables, which then can be referred to in cron entries below them by their name. This simple variable replacement is often times invaluable in writing cron entries, since they make lines more readable and maintainable. Note that these have nothing to do with shell variable assignment and cannot refer to shell environment variables. They cannot even refer to each other. Examples include setting a base path for the project directory, settings a common file name for the lock file or the log file and setting common command line options.

  PRJ_BIN_PATH=/opt/awesome/project/bin
  PRJ_TMP_PATH=/opt/awesome/project/tmp
  LOCKER=timeout --signal=TERM --kill-after=60s 15m flock --verbose --exclusive --wait 60
  
  0 * * * * $LOCKER $PRJ_TMP_PATH/program1.lock -c "$PRJ_BIN_PATH/program1"
  5 * * * * $LOCKER $PRJ_TMP_PATH/program2.lock -c "$PRJ_BIN_PATH/program2"
  

• While cron only focuses on running jobs based on time interval events, it supports one other event: system bootup.

  @reboot /path/to/program
  

  Debian makes sure that this event does not match if only the cron daemon and not the whole system had been restarted. See man 5 crontab for some other shorthands for specifying the 5 time and date fields, like @hourly and @midnight. Note that this and some other features are only available in “Vixie cron” which is the cron daemon written by Paul Vixie in 1987 and nowadays has completely replaced the original cron program for Unix written by no other than Ken Thompson. But still there are cron versions that lack such “advanced” features, like the cron binary in busybox.

• cron daemon does not read the symlinks in /etc/cron.d directory. If like me, you have committed the crontab for the project into the version controlled project directory, make sure to copy the actual crontab file to this directory instead of symlinking it. crond service does not need to be restarted to read newly added crontab files.

• Earlier I mentioned cron does not keep a history of jobs that it runs. As a result cron cannot try to re-run a job if the cron process itself has not been running when it should have run the job. This happens for example when the machine is turned off when it should have run a job. If this is not an acceptable behavior for a job, anacron program should be installed. This program tries to run missed jobs at earliest time that the system boots up.

• cron does not support:

  • time granularity less than one minute
  • job inter-dependencies
  • one-shot jobs
  • triggering based on time in different time zones
  • triggering based on events other than time. Events like when system is idle or when user logs off are not supported.
  • running the jobs at random time intervals or with random delays
  • schedules based on relation of weekdays to the month in a calendar

  That being said, a programmer can implement most of these features to some extent by creating simple shell scripts that call onto other external programs for deciding whether the job should run, thus creating a wrapper that allows the job to be scheduled via cron. For example while crontab scheduling syntax is fairly powerful, it cannot express schedules like “last day of month”, “first weekday of month” or “third Saturday” because all of those require building a calendar representation of days in a month. But a simple shell snippet invoking ncal command can add the appropriate check before running the actual job via cron.

  As another example, to set up a scheduled task to be run in a random time between midnight and four in the morning:

  (crontab -l 2>/dev/null; echo "$(( RANDOM % 60 )) $(( RANDOM % 4 )) * * * /path/to/program") | crontab -
  

  Note that after this task is setup, the schedule is set and no longer random, but it is easy to see how the command can be changed with a search and replace and executed at the end of the task to set new random values for the task itself.

  Otherwise if first class support for these features is needed, alternative job schedulers should be explored. Some other dedicated job schedulers or software with such feature are at, mcron, systemd, snooze, Celery, jobber, Rundeck, Jenkins and Apache Airflow scheduler.

  There are also task scheduling libraries which can be used inside programs to run processes, threads or functions at defined intervals. For example for Python programs, I’ve had good luck with apscheduler. This library actually supports the cron notation for defining task intervals. Quartz is another notable such library written for the Java programming language.

• For better or worse, on most of the newer Linux systems, systemd has re-implemented scheduled job execution functionality which can be configured via its timer unit files. This might have its benefits compared to cron. For example, systemd allows much finer control over system resources that are available to the process and it offers better logging which is very helpful when debugging issues with the command that is being executed. See man systemd.timer for a reference and to explore the possibilities.

• The syntax for setting the time schedule in crontab could look peculiar at first and lead to mistakes by new users. To help compose a particular cron schedule expression according to some criteria, you can depend on crontab.guru website until you can get comfortable with the syntax.