Linux Log Rotation

Automatic Log File Rotation

The automatic housekeeping of log files is handled by a routine known as "logrotate". This process significantly reduces the risk of log files growing to excessive sizes. The logfile rotation process is generally handled by a cron entry which calls upon the logrotate utility.

Log files that need to be managed are generally added into a configuration file called "/etc/logrotate.conf". This configuration file also specifies that individual files can be added into another directory called "/etc/logrotate.d. Within these files we can specify retention periods,size as well as specifying compression to be used.

By default most Linux distributions will have the "logrotate" package installed. If you do need to install the package, it can be installed with the following commands:

Ubuntu/Mint/Debian systems: sudo apt install logrotate

RHEL/CentOS systems: dnf install logrotate

Rotates files in /var/log

The main purpose of the "logrotate" package is to rotate system files or any other files that utilise the storage area "/var/log". It is important that this area has sufficient free space available at all times as many applications and system utilities write their logs to this area.

Below is a view of files that have been rotated within the "/var/log" area on a Linux Mint Desktop.


-rw-r-----  1 syslog            adm               390921 May 31 19:34 syslog
-rw-r-----  1 syslog            adm               984656 May 31 14:20 syslog.1
-rw-r-----  1 syslog            adm               116900 May 30 11:53 syslog.2.gz
-rw-r-----  1 syslog            adm               163746 May 29 13:30 syslog.3.gz
-rw-r-----  1 syslog            adm                59625 May 28 07:41 syslog.4.gz
-rw-r-----  1 syslog            adm                52118 May 27 19:13 syslog.5.gz
-rw-r-----  1 syslog            adm                55723 May 26 17:08 syslog.6.gz
-rw-r-----  1 syslog            adm               220812 May 25 19:35 syslog.7.gz
drwxr-xr-x  2 root              root                4096 Oct  6  2020 timeshift
-rw-------  1 root              root                6006 May 31 19:22 ubuntu-advantage.log
-rw-------  1 root              root                 182 May 14 22:55 ubuntu-advantage.log.1
-rw-r--r--  1 root              root                 113 May 31 19:12 ubuntu-system-adjustments-adjust-grub-title.log
-rw-r--r--  1 root              root                  26 May 31 19:12 ubuntu-system-adjustments-start.log
-rw-r--r--  1 root              root                  26 May 31 15:20 ubuntu-system-adjustments-stop.log
-rw-r-----  1 syslog            adm                27544 May 31 19:14 ufw.log
-rw-r-----  1 syslog            adm                61053 May 29 20:51 ufw.log.1
-rw-r-----  1 syslog            adm                 3242 May 22 20:19 ufw.log.2.gz
-rw-r-----  1 syslog            adm                 3849 May 15 22:28 ufw.log.3.gz
-rw-r-----  1 syslog            adm                 5473 May  8 22:19 ufw.log.4.gz

From the above you will notice that many of the files such as "syslog" and "ufw.log" all have older versions. The most recent version is the current version, the next most recent version has a ".1" appended to the name and older versions have all been compressed as we can see that they end with ".gz".

The above log files are all managed by rules that specify how many versions need to be kept, and whether the files should be compressed.

/etc/logrotate.conf

The file "logrotate.conf" is the default configuration area for log rotation. Below is an example taken from a Linux Mint System.


# see "man logrotate" for details
# rotate log files weekly
weekly

# use the adm group by default, since this is the owning group
# of /var/log/syslog.
su root adm

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
#dateext

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# system-specific logs may be also be configured here.

By default "logrotate.conf" will configure weekly log rotations (weekly) for files owned by the "root" user and the "adm" group (su root adm). Four log file copies are kept (rotate 4), and new empty log files being created after the current one is rotated (create).

/etc/logrotate.d

As mentioned previously, the location "/etc/logrotate.d" is where additional or custom log rotation entries can be added. This area is specified as "include /etc/logrotate.d" within the logrotate.conf file.

As an example, we can look at one of the entries within this location called "apt". You can view the contents by issuing the following command:

"cat /etc/logrotate.d/apt"


root@mint01a:/etc/logrotate.d# cat apt
/var/log/apt/term.log {
  rotate 12
  monthly
  compress
  missingok
  notifempty
}

/var/log/apt/history.log {
  rotate 12
  monthly
  compress
  missingok
  notifempty
}

The above configuration contains two separate blocks of code. These sections are for managing the areas "/var/log/apt/term.log" and "/var/log/apt/history.log". Both sections are using the exact same options. Any options that have not been set here will pick up the default settings from the "logrotate.conf" file.

Below is a listing of the above files as seen on a Linux Mint System.


$ pwd
/var/log/apt

$ ls -l
total 352
-rw-r--r-- 1 root root 84396 May 29 14:31 eipp.log.xz
-rw-r--r-- 1 root root 15482 May 29 14:32 history.log
-rw-r--r-- 1 root root  4868 Apr 29 21:34 history.log.1.gz
-rw-r--r-- 1 root root  1839 Mar 31 18:49 history.log.2.gz
-rw-r--r-- 1 root root  2168 Feb 28 20:45 history.log.3.gz
-rw-r--r-- 1 root root  4597 Jan 29 21:01 history.log.4.gz
-rw-r--r-- 1 root root  1990 Dec 31 00:16 history.log.5.gz
-rw-r--r-- 1 root root  2752 Nov 30 12:35 history.log.6.gz
-rw-r--r-- 1 root root 33302 Oct 31  2020 history.log.7.gz
-rw-r----- 1 root adm  77907 May 29 14:32 term.log
-rw-r----- 1 root adm  16391 Apr 29 21:34 term.log.1.gz
-rw-r----- 1 root adm   4969 Mar 31 18:49 term.log.2.gz
-rw-r----- 1 root adm   5929 Feb 28 20:45 term.log.3.gz
-rw-r----- 1 root adm  15169 Jan 29 21:01 term.log.4.gz
-rw-r----- 1 root adm   5251 Dec 31 00:16 term.log.5.gz
-rw-r----- 1 root adm   7402 Nov 30 12:35 term.log.6.gz
-rw-r----- 1 root adm  31076 Oct 31  2020 term.log.7.gz

The options specified within the "apt" configuration are as follows:

rotate 12 - Retention period set to 12 weeks for old log files.
monthly - Rotate one a month.
compress - Compress the rotated files. Files compressed will be indicated by the ".gz" extension.
missingok - Don't write an error if the log file is missing.
notifempty - Don't rotate the log file if it is empty.

Manually rotating Log files

If you create a new logrotate configuration, it is always a good idea to test that the rotation is correctly configured. An easy way to do this is to issue the following command:

logrotate -d /etc/logrotate.d/custom

The above will carry out a "dry run" of the log rotation process. Any errors will be reported. The "-d" specifies a dry run. You will need to change the name of the configuration to match the one you have created.

To carry out the log rotation for your new configuration, you would remove the "-d" option and issue the command below (amend name to match your config).

logrotate /etc/logrotate.d/custom

Directives

The "logrotate" tool has many parameters that can allow you to automate many actions with your log files. These can range from running a custom script and to sending an email indicating that a rotation has taken place. Below is a list of the directives that can be configured.


DIRECTIVES
       These directives may be included in a logrotate configuration file:

       compress
              Old versions of log files are compressed with gzip(1) by default. See also nocompress.

       compresscmd
              Specifies which command to use to compress log files.  The default  is  gzip(1).   See  also
              compress.

       uncompresscmd
              Specifies which command to use to uncompress log files.  The default is gunzip(1).

       compressext
              Specifies which extension to use on compressed logfiles, if compression is enabled.  The de‐
              fault follows that of the configured compression command.

       compressoptions
              Command line options may be passed to the compression program, if one is in  use.   The  de‐
              fault,  for  gzip(1), is "-6" (biased towards high compression at the expense of speed).  If
              you use a different compression command, you may  need  to  change  the  compressoptions  to
              match.

       copy   Make a copy of the log file, but don't change the original at all.  This option can be used,
              for instance, to make a snapshot of the current log file, or when some other  utility  needs
              to truncate or parse the file.  When this option is used, the create option will have no ef‐
              fect, as the old log file stays in place.

       copytruncate
              Truncate the original log file to zero size in place after creating a copy, instead of  mov‐
              ing  the  old  log file and optionally creating a new one.  It can be used when some program
              cannot be told to close its logfile and thus might continue writing (appending) to the  pre‐
              vious log file forever.  Note that there is a very small time slice between copying the file
              and truncating it, so some logging data might be lost.  When this option is used, the create
              option will have no effect, as the old log file stays in place.

       create mode owner group, create owner group
              Immediately  after  rotation  (before  the postrotate script is run) the log file is created
              (with the same name as the log file just rotated).  mode specifies the mode for the log file
              in  octal  (the  same as chmod(2)), owner specifies the user name who will own the log file,
              and group specifies the group the log file will belong to. Any of the  log  file  attributes
              may  be omitted, in which case those attributes for the new file will use the same values as
              the original log file for the omitted attributes. This option  can  be  disabled  using  the
              nocreate option.

       createolddir mode owner group
              If the directory specified by olddir directive does not exist, it is created. mode specifies
              the mode for the olddir directory in octal (the same as chmod(2)), owner specifies the  user
              name  who  will own the olddir directory, and group specifies the group the olddir directory
              will belong to. This option can be disabled using the nocreateolddir option.

       daily  Log files are rotated every day.

       dateext
              Archive old versions of log files adding a date extension like YYYYMMDD  instead  of  simply
              adding  a number. The extension may be configured using the dateformat and dateyesterday op‐
              tions.

       dateformat format_string
              Specify the extension for dateext using the notation similar to strftime(3)  function.  Only
              %Y  %m  %d  %H  %M %S %V and %s specifiers are allowed.  The default value is -%Y%m%d except
              hourly, which uses -%Y%m%d%H as default value.  Note that also the character separating  log
              name  from the extension is part of the dateformat string. The system clock must be set past
              Sep 9th 2001 for %s to work correctly.  Note that the datestamps generated  by  this  format
              must be lexically sortable (that is first the year, then the month then the day. For example
              2001/12/01 is ok, but 01/12/2001 is not, since 01/11/2002  would  sort  lower  while  it  is
              later).  This is because when using the rotate option, logrotate sorts all rotated filenames
              to find out which logfiles are older and should be removed.

       dateyesterday
              Use yesterday's instead of today's date to create the dateext extension, so that the rotated
              log file has a date in its name that is the same as the timestamps within it.

       datehourago
              Use  hour  ago  instead of current date to create the dateext extension, so that the rotated
              log file has a hour in its name that is the same as the timestamps within it.   Useful  with
              rotate hourly.

       delaycompress
              Postpone compression of the previous log file to the next rotation cycle.  This only has ef‐
              fect when used in combination with compress.  It can be used when  some  program  cannot  be
              told  to close its logfile and thus might continue writing to the previous log file for some
              time.

       extension ext
              Log files with ext extension can keep it after the rotation.  If compression  is  used,  the
              compression extension (normally .gz) appears after ext. For example you have a logfile named
              mylog.foo and want to rotate it to mylog.1.foo.gz instead of mylog.foo.1.gz.

       hourly Log files are rotated every hour. Note that usually logrotate is configured  to  be  run  by
              cron daily. You have to change this configuration and run logrotate hourly to be able to re‐
              ally rotate logs hourly.

       addextension ext
              Log files are given the final extension ext after rotation. If  the  original  file  already
              ends  with  ext,  the extension is not duplicated, but merely moved to the end, that is both
              filename and filenameext would get rotated to filename.1ext. If  compression  is  used,  the
              compression extension (normally .gz) appears after ext.

       ifempty
              Rotate  the  log  file even if it is empty, overriding the notifempty option (ifempty is the
              default).

       include file_or_directory
              Reads the file given as an argument as if it was included inline where the include directive
              appears. If a directory is given, most of the files in that directory are read in alphabetic
              order before processing of the including file continues. The only files  which  are  ignored
              are  files which are not regular files (such as directories and named pipes) and files whose
              names end with one of the taboo extensions or patterns, as  specified  by  the  tabooext  or
              taboopat directives, respectively.

       mail address
              When a log is rotated out of existence, it is mailed to address. If no mail should be gener‐
              ated by a particular log, the nomail directive may be used.

       mailfirst
              When using the mail command, mail the just-rotated  file,  instead  of  the  about-to-expire
              file.

       maillast
              When using the mail command, mail the about-to-expire file, instead of the just-rotated file
              (this is the default).

       minage count
              Do not rotate logs which are less than <count> days old.

       maxage count
              Remove rotated logs older than <count> days. The age is only checked if the logfile is to be
              rotated. The files are mailed to the configured address if maillast and mail are configured.

       maxsize size
              Log  files  are  rotated  when they grow bigger than size bytes even before the additionally
              specified time interval (daily, weekly, monthly, or yearly).  The  related  size  option  is
              similar  except  that it is mutually exclusive with the time interval options, and it causes
              log files to be rotated without regard for the last rotation time.  When  maxsize  is  used,
              both the size and timestamp of a log file are considered.

       minsize  size
              Log files are rotated when they grow bigger than size bytes, but not before the additionally
              specified time interval (daily, weekly, monthly, or yearly).  The  related  size  option  is
              similar  except  that it is mutually exclusive with the time interval options, and it causes
              log files to be rotated without regard for the last rotation time.  When  minsize  is  used,
              both the size and timestamp of a log file are considered.

       missingok
              If the log file is missing, go on to the next one without issuing an error message. See also
              nomissingok.

       monthly
              Log files are rotated the first time logrotate is run in a month (this is  normally  on  the
              first day of the month).

       nocompress
              Old versions of log files are not compressed. See also compress.

       nocopy Do not copy the original log file and leave it in place.  (this overrides the copy option).

       nocopytruncate
              Do  not  truncate  the  original log file in place after creating a copy (this overrides the
              copytruncate option).

       nocreate
              New log files are not created (this overrides the create option).

       nocreateolddir
              olddir directory is not created by logrotate when it does not exist.

       nodelaycompress
              Do not postpone compression of the previous log file to the next rotation cycle (this  over‐
              rides the delaycompress option).

       nodateext
              Do  not  archive   old versions of log files with date extension (this overrides the dateext
              option).

       nomail Do not mail old log files to any address.

       nomissingok
              If a log file does not exist, issue an error. This is the default.

       noolddir
              Logs are rotated in the directory they normally reside in (this  overrides  the  olddir  op‐
              tion).

       nosharedscripts
              Run  prerotate  and  postrotate scripts for every log file which is rotated (this is the de‐
              fault, and overrides the sharedscripts option). The absolute path to the log file is  passed
              as  first argument to the script. If the scripts exit with error, the remaining actions will
              not be executed for the affected log only.

       noshred
              Do not use shred when deleting old log files. See also shred.

       notifempty
              Do not rotate the log if it is empty (this overrides the ifempty option).

       olddir directory
              Logs are moved into directory for rotation. The directory must be on the same  physical  de‐
              vice  as the log file being rotated, unless copy, copytruncate or renamecopy option is used.
              The directory is assumed to be relative to the directory holding the log file unless an  ab‐
              solute  path  name is specified. When this option is used all old versions of the log end up
              in directory.  This option may be overridden by the noolddir option.

       postrotate/endscript
              The lines between postrotate and endscript (both of which must  appear  on  lines  by  them‐
              selves)  are  executed  (using  /bin/sh) after the log file is rotated. These directives may
              only appear inside a log file definition. Normally, the absolute path to  the  log  file  is
              passed  as  first  argument  to  the script. If sharedscripts is specified, whole pattern is
              passed to the script.  See also prerotate. See sharedscripts and nosharedscripts  for  error
              handling.

       prerotate/endscript
              The lines between prerotate and endscript (both of which must appear on lines by themselves)
              are executed (using /bin/sh) before the log file is rotated and only if the log  will  actu‐
              ally  be  rotated.  These directives may only appear inside a log file definition. Normally,
              the absolute path to the log file is passed as first argument to the  script.   If   shared‐
              scripts  is  specified,  whole  pattern  is passed to the script.  See also postrotate.  See
              sharedscripts and nosharedscripts for error handling.

       firstaction/endscript
              The lines between firstaction and endscript (both of which must appear  on  lines  by  them‐
              selves)  are  executed  (using  /bin/sh) once before all log files that match the wildcarded
              pattern are rotated, before prerotate script is run and only if at least one log will  actu‐
              ally  be rotated.  These directives may only appear inside a log file definition. Whole pat‐
              tern is passed to the script as first argument. If the script exits with error,  no  further
              processing is done. See also lastaction.

       lastaction/endscript
              The  lines  between  lastaction  and  endscript (both of which must appear on lines by them‐
              selves) are executed (using /bin/sh) once after all log files that match the wildcarded pat‐
              tern  are  rotated,  after postrotate script is run and only if at least one log is rotated.
              These directives may only appear inside a log file definition. Whole pattern  is  passed  to
              the script as first argument. If the script exits with error, just an error message is shown
              (as this is the last action). See also firstaction.

       preremove/endscript
              The lines between preremove and endscript (both of which must appear on lines by themselves)
              are  executed  (using  /bin/sh)  once just before removal of a log file.  The logrotate will
              pass the name of file which is soon to be removed. See also firstaction.

       rotate count
              Log files are rotated count times before being removed or mailed to the address specified in
              a mail directive. If count is 0, old versions are removed rather than rotated. Default is 0.

       renamecopy
              Log  file  is renamed to temporary filename in the same directory by adding ".tmp" extension
              to it. After that, postrotate script is run and log file is copied from  temporary  filename
              to final filename. This allows storing rotated log files on the different devices using old‐
              dir directive. In the end, temporary filename is removed.

       size size
              Log files are rotated only if they grow bigger than size bytes. If size is  followed  by  k,
              the  size is assumed to be in kilobytes.  If the M is used, the size is in megabytes, and if
              G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G are all
              valid.

       sharedscripts
              Normally, prerotate and postrotate scripts are run for each log which is rotated and the ab‐
              solute path to the log file is passed as first argument to the script. That means  a  single
              script  may  be  run multiple times for log file entries which match multiple files (such as
              the /var/log/news/* example). If sharedscripts is specified, the scripts are only run  once,
              no  matter  how many logs match the wildcarded pattern, and whole pattern is passed to them.
              However, if none of the logs in the pattern require rotating, the scripts will not be run at
              all.  If  the  scripts  exit  with error, the remaining actions will not be executed for any
              logs. This option overrides the nosharedscripts option and implies create option.

       shred  Delete log files using shred -u instead of unlink().  This should ensure that logs  are  not
              readable after their scheduled deletion; this is off by default.  See also noshred.

       shredcycles count
              Asks  GNU shred(1) to overwrite log files count times before deletion.  Without this option,
              shred's default will be used.

       start count
              This is the number to use as the base for rotation. For example, if you specify 0, the  logs
              will be created with a .0 extension as they are rotated from the original log files.  If you
              specify 9, log files will be created with a .9, skipping 0-8.  Files will still  be  rotated
              the number of times specified with the rotate directive.

       su user group
              Rotate  log files set under this user and group instead of using default user/group (usually
              root). user specifies the user name used for rotation and group specifies the group used for
              rotation.  If  the  user/group  you  specify here does not have sufficient privilege to make
              files with the ownership you've specified in a create instruction, it will cause an error.

       tabooext [+] list
              The current taboo extension list is changed (see the include directive  for  information  on
              the  taboo  extensions). If a + precedes the list of extensions, the current taboo extension
              list is augmented, otherwise it is replaced. At startup, the taboo extension list  ,v,  .cf‐
              saved,  .disabled,  .dpkg-bak,  .dpkg-del, .dpkg-dist, .dpkg-new, .dpkg-old, .rhn-cfg-tmp-*,
              .rpmnew, .rpmorig, .rpmsave, .swp, .ucf-dist, .ucf-new, .ucf-old, ~

       taboopat [+] list
              The current taboo glob pattern list is changed (see the include directive for information on
              the  taboo extensions and patterns). If a + precedes the list of patterns, the current taboo
              pattern list is augmented, otherwise it is replaced. At startup, the taboo pattern  list  is
              empty.

       weekly [weekday]
              Log files are rotated once each weekday, or if the date is advanced by at least 7 days since
              the last rotation (while ignoring the exact time).  The weekday interpretation is following:
              0  means  Sunday,  1  means  Monday, ..., 6 means Saturday; the special value 7 means each 7
              days, irrespectively of weekday.  Defaults to 0 if the weekday argument is omitted.

       yearly Log files are rotated if the current year is not the same as the last rotation.

Additional help

For additional help using logrotate. You can issue the command "man logrotate". There you will find more examples and all of the available parameters that can be used within your configurations.