I think job scheduling capability (not to be confused with process scheduling)
is a very important, yet somewhat neglected aspect of an operating system.
I have worked on big and complete production systems that actually
do nothing but run scheduled jobs at their predefined intervals.
And Cron is probably the most common job scheduling system on Unix-based
systems. As a good Unix citizen, cron conforms to the
“do one thing and do it well” mantra, which means it leaves out many
features to other tools which are purposefully built for them.
These features are critical when devising fairly demanding scheduling
schemes.
Curiously I have not seen people talk about the intersection of cron with
the rest of a Unix system as much as they should.
As a result, compared to other job scheduling systems like the one
on recent versions of MS Windows Server operating system, cron might
seem very lacking.
I’m gonna try to change this perspective by introducing some tips
and tricks. This collection of random points are what I have found
to be some best practices on job scheduling via cron.
A basic knowledge about cron is assumed.
-
Users should use crontab
command to create, edit, list or delete their
very own crontab file and manage their set of cron entry lines.
Although the set of crontab files for all users are stored in
/var/spool/cron/crontabs
directory, they are not intended to be
edited directly. Users can run crontab -e
to edit the file or crontab -l
to list all their scheduled tasks.
Software packages, on the other hand, should drop a separate crontab
file in /etc/cron.d
directory to setup their scheduled jobs.
The format of these files is a little different from user crontabs.
Debian has mandated a user name field before the program name, using
which the program will be run.
Cron package lets the system administrator create allow and deny
lists to choose which users can access the service. On Debian, these
files are /etc/cron.allow
and /etc/cron.deny
respectively.
Generally cron package in Debian has been heavily patched with many
customizations and deviations from upstream defaults. This is something
to consider when using multiple distros.
-
Use flock
program in a crontab entry to ensure only one instance
of command runs at a time. If the program creates side effects like
changing the state of a file or database, and it has not been designed
to be graceful about multiple instances of itself competing to change
the state at the same time, using this little utility is a must.
0 * * * * flock --verbose --exclusive --wait 60 $HOME/my_program.lock -c "/path/to/my/program"
-
Use timeout
command to put an upper limit on how long the job can stay
running. Combine this command with flock
to make sure multiple
instances of a job do not run at the same time, and a single
instance does not take forever to run either. For example:
0 * * * * timeout --signal=TERM --kill-after=60s 15m flock --verbose --exclusive --wait 60 $HOME/my_program.lock -c "/path/to/my/program"
Play with the command line parameters as necessary. This has been
a life-saver for me, because most scheduled jobs are untested
scripts created by a single developer with no review that are
demanding on system resources and run unattended!
It’s great that we can leave out monitoring and locking from
our programs and rely on well-tested Unix utilities for that.
-
cron cannot be relied on to keep a history of when it runs the
scheduled programs, when they finish executing and whether they
finished successfully or not, including their exit code.
To be fair, the cron daemon can be invoked like cron -L 15
to produce
the most verbose logs, including some of the information above.
But the truth is in reality most distros do not run the cron daemon like
this and as a result only the start of cron jobs are logged.
On Debian, /etc/default/cron
can be edited like this to enable
the highest amount of logging:
After which cron messages are logged via the ‘cron’ facility to syslog.
Also to debug cron issues, it is possible to run it in the foreground
with highest log level like this: cron -f -L 15
and then watch
for cron messages.
So, lack of proper event logging by cron makes it crucial for the
program itself to log necessary events.
Shell redirection of the program output to a file is handy but it
does not really fit the bill in production systems, because it cannot
connect to syslog for serious log management.
Syslog is a great Unix service which provides centralized logging
facility to other running programs.
Fortunately syslog
, provides a handy command-line tool to equip existing
programs with simple logging without the need to fiddle with the
code of programs which do not support syslog internally.
Use logger
in combination with shell output piping to log
any terminal output that the command could produce for later inspection.
Output lines are automatically timestamped and tagged.
* 2 * * * * /path/to/my/program |& logger --tag my_program --priority user.notice
Optionally an rsyslog
configuration file can be specifically written
for this program to customize how rsyslog treats them. For example to
forward all messages tagged as above into a separate log file, create
/etc/rsyslog.d/50-my_program.conf
to look like this and then restart
the rsyslog service:
:syslogtag,isequal,"my_program:" /var/log/my_program.log
&~
Logging via syslog offers much more flexibility about what to do
with log lines compared to the email functionality in cron.
rsyslog is able to format, buffer, store and forward log messages
into various sinks according to a filter criteria.
Output methods like a file, a database table, a network socket,
or to another rsyslog instance over the network are supported and
pluggable modules can extend its functionality even further.
For the example above, don’t forget to set up an automatic log file
rotation scheme like logrotate
if the program spews so much output
that it might fill the entire disk. An example logrotate config file
could look like below and should be placed in /etc/logrotate.d/my_program
:
/var/log/my_program.log {
daily
rotate 5
compress
notifempty
missingok
}
For simple scripts, shell redirection could also work, otherwise
all output printed by the program to stdout
and stderr
during
cron execution is lost.
-
Other Unix utilities that might be beneficial to incorporate into the
command line are: sudo
, nice
, time
, taskset
, cpulimit
, chrt
,
parallel
, pidstat
, chronic
, etc. All these programs support
command line parameters to customize their default behavior.
cgroups can be used to further confine a process to a control group
which can then be used to limit its visibility to the rest of the system
and manage its resources. See man cgroups
for more information.
-
If all we need is to run a job at a specific time and not periodically
at specific intervals, at
program, not cron
should be used.
I say this because I’ve seen people adding a cron entry for midnight
tonight before leaving work and then commenting it out the next day.
at
is the safer way to do this.
at midnight <<EOF
reboot
EOF
In all honesty, for such a use case, a crontab entry like
below that specifies all 5 criteria probably will run the job this
midnight and hopefully not match again for a year from now,
but still it visually pollutes the crontab file.
# m h dom mon dow usercommand
45 2 22 feb wed /path/to/program
There is also a noteworthy gotcha in cron that could result in a bug
in the example above.
Unlike the minute, hour, and month selectors which are anded together
to find a match, ‘day of month’ and ‘day of week’ selectors, if both
present, are ored together and then anded to the rest of the selectors.
For any single crontab entry, I try to skip one of them
(always set it to *
) to avoid this source of confusion.
For simple one-off job scheduling at
command and its job management
associates atq
and atrm
commands have worked flawlessly for me.
These commands depend on a different daemon than cron called atd
which pops the jobs from the queue and runs them at the specified time.
Command composition with utilities such as the ones mentioned above
works for at
too, thanks to Unix philosophy!
-
Use bash
as the SHELL
to avoid surprises when writing complex
command lines that take bash
facilities for granted.
Put this at the top of the crontab.
The default shell is /bin/sh
.
While talking about shell, its worth mentioning that all job workflow
mechanisms supported by shell are readily available to a cron job because
the program section of the line is simply passed to the shell to
be executed.
Specifically ;
, &&
, ||
and !
operators in a bash command list
allow easy definition of dependency among programs in a job based on
the success or failure of each other.
15 2 * * * filename=backup_$(date +"%Y_%m_%d_%H_%M_%S").sql; pg_dump -f ${filename} postgres && gzip ${filename}
bash conditionals allow checking for further criteria before running
the job, like checking for a flag indicating of the job being disabled or
being already run. bash loops allow job retires in case of failure and
running the job more than once a minute.
If the command becomes too long, it maybe the time to move it into
a separate shell script because crontab does not support multiline entries.
-
Default value assigned by cron to PATH
environment variable in crontab
files could also be a source of surprise and frustration, because it
is set to `/usr/bin:/bin
. To set a less restrictive value for this
variable, try adding this line at the top of crontab file:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
-
Disable the email notification in crontab by adding the following directive
at the top. This effectively empties this reserved environment variable.
Logging combined with log monitoring is how service notification
should be done.
I don’t like to get automated and unfiltered emails about every single
run of the job, especially when no mail server is installed on the
local system. Because these emails are perpetually accumulated on
the local disk and are in fact the cause of a service outage
waiting to happen.
-
Assigning to reserved variables like SHELL
, PATH
and MAILTO
at the
top of the crontab changes the default behavior of cron.
But cron also supports assigning to arbitrary variables, which then
can be referred to in cron entries below them by their name.
This simple variable replacement is often times invaluable in
writing cron entries, since they make lines more readable and maintainable.
Note that these have nothing to do with shell variable assignment and
cannot refer to shell environment variables. They cannot even refer
to each other.
Examples include setting a base path for the project directory,
settings a common file name for the lock file or the log file and
setting common command line options.
PRJ_BIN_PATH=/opt/awesome/project/bin
PRJ_TMP_PATH=/opt/awesome/project/tmp
LOCKER=timeout --signal=TERM --kill-after=60s 15m flock --verbose --exclusive --wait 60
0 * * * * $LOCKER $PRJ_TMP_PATH/program1.lock -c "$PRJ_BIN_PATH/program1"
5 * * * * $LOCKER $PRJ_TMP_PATH/program2.lock -c "$PRJ_BIN_PATH/program2"
-
While cron only focuses on running jobs based on time interval events,
it supports one other event: system bootup.
Debian makes sure that this event does not match if only the cron daemon
and not the whole system had been restarted.
See man 5 crontab
for some other shorthands for specifying the 5 time
and date fields, like @hourly
and @midnight
.
Note that this and some other features are only available in “Vixie cron”
which is the cron daemon written by Paul Vixie in 1987 and nowadays
has completely replaced the original cron program for Unix written by
no other than Ken Thompson.
But still there are cron versions that lack such “advanced” features,
like the cron binary in busybox.
-
cron
daemon does not read the symlinks in /etc/cron.d
directory. If like me,
you have committed the crontab for the project into the version controlled
project directory, make sure to copy the actual crontab file to this
directory instead of symlinking it. crond
service does not need to be
restarted to read newly added crontab files.
-
Earlier I mentioned cron does not keep a history of jobs that it runs.
As a result cron cannot try to re-run a job if the cron process itself
has not been running when it should have run the job. This happens
for example when the machine is turned off when it should have run
a job.
If this is not an acceptable behavior for a job, anacron program
should be installed. This program tries to run missed jobs at earliest
time that the system boots up.
-
cron
does not support:
- time granularity less than one minute
- job inter-dependencies
- one-shot jobs
- triggering based on time in different time zones
- triggering based on events other than time. Events like when system
is idle or when user logs off are not supported.
- running the jobs at random time intervals or with random delays
- schedules based on relation of weekdays to the month in a calendar
That being said, a programmer can implement most of these features
to some extent by creating simple shell scripts that call onto
other external programs for deciding whether the job should run, thus
creating a wrapper that allows the job to be scheduled via cron.
For example while crontab scheduling syntax is fairly powerful, it
cannot express schedules like “last day of month”,
“first weekday of month” or “third Saturday” because all of those
require building a calendar representation of days in a month.
But a simple shell snippet invoking ncal
command can add the
appropriate check before running the actual job via cron.
As another example, to set up a scheduled task to be run in a random
time between midnight and four in the morning:
(crontab -l 2>/dev/null; echo "$(( RANDOM % 60 )) $(( RANDOM % 4 )) * * * /path/to/program") | crontab -
Note that after this task is setup, the schedule is set and no longer
random, but it is easy to see how the command can be changed with a search
and replace and executed at the end of the task to set new random
values for the task itself.
Otherwise if first class support for these features is needed,
alternative job schedulers should be explored.
Some other dedicated job schedulers or software with such feature are
at, mcron, systemd, snooze,
Celery,
jobber,
Rundeck,
Jenkins and
Apache Airflow scheduler.
There are also task scheduling libraries which can be used inside
programs to run processes, threads or functions at defined intervals.
For example for Python programs, I’ve had good luck with
apscheduler. This library
actually supports the cron notation for defining task intervals.
Quartz is another notable such
library written for the Java programming language.
-
For better or worse, on most of the newer Linux systems, systemd
has
re-implemented scheduled job execution functionality which can
be configured via its timer unit files.
This might have its benefits compared to cron
. For example,
systemd allows much finer control over system resources that are
available to the process and it offers better logging which is very
helpful when debugging issues with the command that is being executed.
See man systemd.timer
for a reference and to explore the possibilities.
-
The syntax for setting the time schedule in crontab could look peculiar
at first and lead to mistakes by new users. To help compose a particular
cron schedule expression according to some criteria, you can depend on
crontab.guru website until you can get
comfortable with the syntax.