Displaying system log files with journalctl
What is journalctl?
From as early as 2012, many Linux distributions moved away from the traditional "System V" init system to a new system called "systemd". Systemd is used for the booting of a system and managing of processes. Included within systemd is an ancillary component called "journalctl" whose prime purpose is to manage the event logging of a system. journalctl is a system utility that gives you the ability to query the contents of the systemd journal. journalctl is used to view your systems log files. By default, the command when used with no other parameters will display the full contents of your journal, starting with the oldest entry collected. All users by default are granted access to their own journals, however, access to the main system log files and other users log areas are restricted to privileged users ("wheel" group and "systemd-journal" groups) and the root user. Because the system logs can be quite verbose and contain a lot of information, journalctl allows you to pass parameters that can limit the output to specific entries or time frames.
When the journalctl command is issued and executed with no additional parameters, the full journal is displayed. By default, the oldest entries are displayed first. The output is displayed a screen at a time, however, you may scroll through the output using the cursor keys (arrow keys) or by pressing the "space" bar to scroll a whole page at a time. By default, lines are truncated, however, you may use the arrow keys to scroll "left" and "right".
Below are some examples of the journalctl command in use. The commands below were executed on a "CentOS" system and an "Ubuntu" system, however, the majority of Linux systems now have adopted "systemd" and this command may be executed.
journalctl
When passed with no additional parameters, all output is displayed with the oldest entries first.
Command Issued: journalctl
-- Logs begin at Thu 2016-08-04 19:04:49 BST, end at Thu 2016-08-04 19:19:52 BST
Aug 04 19:04:49 centos72m systemd-journal[90]: Runtime journal is using 8.0M (ma
Aug 04 19:04:49 centos72m systemd-journal[90]: Runtime journal is using 8.0M (ma
Aug 04 19:04:49 centos72m kernel: Initializing cgroup subsys cpuset
Aug 04 19:04:49 centos72m kernel: Initializing cgroup subsys cpu
Aug 04 19:04:49 centos72m kernel: Initializing cgroup subsys cpuacct
Aug 04 19:04:49 centos72m kernel: Linux version 3.10.0-327.28.2.el7.x86_64 (buil
Aug 04 19:04:49 centos72m kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-327.2
Aug 04 19:04:49 centos72m kernel: e820: BIOS-provided physical RAM map:
The above output is displayed with a header indicating the start of the journal entry. Where the lines have been truncated, you may use the arrow keys to scroll to the right to display the missing information.
journalctl -r
When the "journalctl" command is passed with the "-r" parameter. This instructs the system to display the output in reverse order. This means the latest output will be displayed first.
Command Issued: journalctl -r
-- Logs begin at Fri 2016-08-05 09:07:14 BST, end at Fri 2016-08-05 09:22:14 BST
Aug 05 09:22:14 centos72m systemd[1]: Started Cleanup of Temporary Directories.
Aug 05 09:22:14 centos72m systemd[1]: Starting Cleanup of Temporary Directories.
Aug 05 09:22:10 centos72m systemd[1]: Configuration file /usr/lib/systemd/system
Aug 05 09:22:10 centos72m systemd[1]: Configuration file /usr/lib/systemd/system
Aug 05 09:17:07 centos72m systemd[1]: Starting Session 2 of user root.
Aug 05 09:17:07 centos72m sshd[1447]: pam_unix(sshd:session): session opened for
Aug 05 09:17:07 centos72m systemd[1]: Started Session 2 of user root.
Aug 05 09:17:07 centos72m systemd-logind[667]: New session 2 of user root.
The above journal entries are displayed with the newest entries first.
How to tail a log file
To tail a log file (read the latest entries) using journalctl, you simply pass the parameter "-f" after the command. By default, this will start displaying the most entries that are written to the main journal area.
Command Issued: journalctl -f
# journalctl -f
-- Logs begin at Thu 2016-08-04 19:04:49 BST. --
Aug 04 20:01:01 centos72m CROND[1539]: (root) CMD (run-parts /etc/cron.hourly)
Aug 04 20:01:01 centos72m run-parts(/etc/cron.hourly)[1542]: starting 0anacron
Aug 04 20:01:02 centos72m anacron[1548]: Anacron started on 2016-08-04
Aug 04 20:01:02 centos72m run-parts(/etc/cron.hourly)[1550]: finished 0anacron
Aug 04 20:01:02 centos72m run-parts(/etc/cron.hourly)[1552]: starting 0yum-hourly.cron
Aug 04 20:01:02 centos72m run-parts(/etc/cron.hourly)[1556]: finished 0yum-hourly.cron
Aug 04 20:01:02 centos72m anacron[1548]: Will run job `cron.daily' in 20 min.
Aug 04 20:01:02 centos72m anacron[1548]: Will run job `cron.weekly' in 40 min.
Aug 04 20:01:02 centos72m anacron[1548]: Will run job `cron.monthly' in 60 min.
Aug 04 20:01:02 centos72m anacron[1548]: Jobs will be executed sequentially
As and when new information is written to the journal, this information will be displayed to your screen.
Limiting the size of output
When displaying output, you may limit the number of lines that are displayed by passing either the "-n" or "--lines=" parameter.
Command Issued: journalctl -n 5
# journalctl -n 5
-- Logs begin at Thu 2016-08-04 19:04:49 BST, end at Thu 2016-08-04 20:01:02 BST
Aug 04 20:01:02 centos72m run-parts(/etc/cron.hourly)[1556]: finished 0yum-hour
Aug 04 20:01:02 centos72m anacron[1548]: Will run job `cron.daily' in 20 min.
Aug 04 20:01:02 centos72m anacron[1548]: Will run job `cron.weekly' in 40 min.
Aug 04 20:01:02 centos72m anacron[1548]: Will run job `cron.monthly' in 60 min.
Aug 04 20:01:02 centos72m anacron[1548]: Jobs will be executed sequentially
The above command displayed only the most recent journal entries and limited the number of lines displayed to the specified number passed.
Display events within a certain time
journalctl has the ability to allow you to showing entries on or newer than the specified date, or on or older than the specified date. These time frames are specified after the parameter "--since". Dates have to be specified in the format "YYYY-MM-DD HH:MM:SS". You may also pass a time frame as "1 hour ago", or "3 days ago".
Command Issued: journalctl --since 1 hour ago
# journalctl --since "1 hour ago"
-- Logs begin at Thu 2016-08-04 19:04:49 BST, end at Thu 2016-08-04 20:21:06 BST
Aug 04 20:01:01 centos72m systemd[1]: Started Session 3 of user root.
Aug 04 20:01:01 centos72m systemd[1]: Starting Session 3 of user root.
Aug 04 20:01:01 centos72m CROND[1539]: (root) CMD (run-parts /etc/cron.hourly)
Aug 04 20:01:01 centos72m run-parts(/etc/cron.hourly)[1542]: starting 0anacron
Aug 04 20:01:02 centos72m anacron[1548]: Anacron started on 2016-08-04
Aug 04 20:01:02 centos72m run-parts(/etc/cron.hourly)[1550]: finished 0anacron
Aug 04 20:01:02 centos72m run-parts(/etc/cron.hourly)[1552]: starting 0yum-hourl
Aug 04 20:01:02 centos72m run-parts(/etc/cron.hourly)[1556]: finished 0yum-hourl
Aug 04 20:01:02 centos72m anacron[1548]: Will run job `cron.daily' in 20 min.
Aug 04 20:01:02 centos72m anacron[1548]: Will run job `cron.weekly' in 40 min.
Aug 04 20:01:02 centos72m anacron[1548]: Will run job `cron.monthly' in 60 min.
Aug 04 20:01:02 centos72m anacron[1548]: Jobs will be executed sequentially
Aug 04 20:21:02 centos72m anacron[1548]: Job `cron.daily' started
Aug 04 20:21:02 centos72m run-parts(/etc/cron.daily)[1594]: starting 0yum-daily.
Aug 04 20:21:02 centos72m run-parts(/etc/cron.daily)[1598]: finished 0yum-daily.
Aug 04 20:21:02 centos72m run-parts(/etc/cron.daily)[1600]: starting logrotate
Aug 04 20:21:02 centos72m run-parts(/etc/cron.daily)[1609]: finished logrotate
Aug 04 20:21:02 centos72m run-parts(/etc/cron.daily)[1611]: starting man-db.cron
Aug 04 20:21:06 centos72m run-parts(/etc/cron.daily)[4264]: finished man-db.cron
Aug 04 20:21:06 centos72m anacron[1548]: Job `cron.daily' terminated
The above command shows only output from the last hour. To see the exact times, look at the line that begins "-- logs begin......"
Command Issued: journalctl --since "2016-08-04 20:00:00" --until "2016-08-04 20:15:00"
# journalctl --since "2016-08-04 20:00:00" --until "2016-08-04 20:15:00"
-- Logs begin at Thu 2016-08-04 19:04:49 BST, end at Thu 2016-08-04 20:41:02 BST
Aug 04 20:01:01 centos72m systemd[1]: Started Session 3 of user root.
Aug 04 20:01:01 centos72m systemd[1]: Starting Session 3 of user root.
Aug 04 20:01:01 centos72m CROND[1539]: (root) CMD (run-parts /etc/cron.hourly)
Aug 04 20:01:01 centos72m run-parts(/etc/cron.hourly)[1542]: starting 0anacron
Aug 04 20:01:02 centos72m anacron[1548]: Anacron started on 2016-08-04
Aug 04 20:01:02 centos72m run-parts(/etc/cron.hourly)[1550]: finished 0anacron
Aug 04 20:01:02 centos72m run-parts(/etc/cron.hourly)[1552]: starting 0yum-hourl
Aug 04 20:01:02 centos72m run-parts(/etc/cron.hourly)[1556]: finished 0yum-hourl
Aug 04 20:01:02 centos72m anacron[1548]: Will run job `cron.daily' in 20 min.
Aug 04 20:01:02 centos72m anacron[1548]: Will run job `cron.weekly' in 40 min.
Aug 04 20:01:02 centos72m anacron[1548]: Will run job `cron.monthly' in 60 min.
Aug 04 20:01:02 centos72m anacron[1548]: Jobs will be executed sequentially
lines 1-13/13 (END)
The above command will now only output information between the time frames specified.
Display events specific to a user
journalctl has the ability to filter the output by specifying a the "UID" of a particular user. A "UID" is a unique number that identifies a user on a system. To find out the id of a specific user, you can issue either the command "id" followed by the userid.
Command Issued: id john
$ id john
uid=1000(john) gid=1000(john) groups=1000(john),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),108(lpadmin),125(sambashare),126(vboxusers)
From the above we can see that the UID for the user "john" is "1000". We can now use this information to construct our next command.
Command Issued: journalctl _UID=1000 -n 5
# journalctl _UID=1000 -n 5
-- Logs begin at Thu 2016-08-04 18:50:26 BST, end at Thu 2016-08-04 21:01:50 BST
Aug 04 21:00:23 ubuntu su[8075]: Successful su for tux by john
Aug 04 21:00:23 ubuntu su[8075]: + /dev/pts/2 john:tux
Aug 04 21:00:23 ubuntu su[8075]: pam_unix(su:session): session opened for user t
Aug 04 21:00:23 ubuntu su[8075]: pam_systemd(su:session): Cannot create session:
Aug 04 21:00:33 ubuntu su[8075]: pam_unix(su:session): session closed for user t
Using journalctl we are able to show only the output for the user "john". In this particular example, the additional parameter of "-n 5" was also used to limit the size of the output.
Display events for a specific Unit
journalctl has the ability also to display events for a particular Unit (service). For example, if you are running a web server (Apache), you may use the parameter "-u" to specify the unit you wish to see output for.
Command Issued: journalctl -u httpd.service
# journalctl -u httpd.service
-- Logs begin at Fri 2016-08-05 09:07:14 BST, end at Fri 2016-08-05 09:17:07 BST
Aug 05 09:13:54 centos72m systemd[1]: Starting The Apache HTTP Server...
Aug 05 09:13:54 centos72m systemd[1]: Started The Apache HTTP Server.
Aug 05 09:14:51 centos72m systemd[1]: Stopping The Apache HTTP Server...
Aug 05 09:14:52 centos72m systemd[1]: Starting The Apache HTTP Server...
Aug 05 09:14:52 centos72m systemd[1]: Started The Apache HTTP Server.
From the above we can see that the "Apache" web-server has been started, stopped and then restarted from the output. To just check on the status of a unit, you may use the command:
"systemctl status httpd.service". This command will show whether the service is running, and the command "systemctl list-unit-files httpd.service" will show you whether the service is set to automatically start.
It is also possible to specify more than one unit for display.
Command Issued: journalctl -u httpd.service -u crond.service
# journalctl -u httpd.service -u crond.service
-- Logs begin at Fri 2016-08-05 09:07:14 BST, end at Fri 2016-08-05 09:22:14 BST
Aug 05 09:07:26 centos72m systemd[1]: Started Command Scheduler.
Aug 05 09:07:26 centos72m systemd[1]: Starting Command Scheduler...
Aug 05 09:07:26 centos72m crond[669]: (CRON) INFO (RANDOM_DELAY will be scaled w
Aug 05 09:07:26 centos72m crond[669]: (CRON) INFO (running with inotify support)
Aug 05 09:13:54 centos72m systemd[1]: Starting The Apache HTTP Server...
Aug 05 09:13:54 centos72m systemd[1]: Started The Apache HTTP Server.
Aug 05 09:14:51 centos72m systemd[1]: Stopping The Apache HTTP Server...
Aug 05 09:14:52 centos72m systemd[1]: Starting The Apache HTTP Server...
Aug 05 09:14:52 centos72m systemd[1]: Started The Apache HTTP Server.
From the above output you can see that both entries have now been included within the displayed output.
Formatting the Output
Various formatting options are available to the journalctl command. The option " -o" or "--output=" can be passed with one of the following formatting options. There are numerous formatting options available to the journalctl command. For a full list, please take a look at the man pages for "journalctl". Below are some examples of the journalctl formatting option being specified.
Command Issued: journalctl -o short
-- Logs begin at Fri 2016-08-05 09:07:14 BST, end at Fri 2016-08-05 09:22:14 BST
Aug 05 09:07:14 centos72m systemd-journal[87]: Runtime journal is using 8.0M (ma
Aug 05 09:07:14 centos72m systemd-journal[87]: Runtime journal is using 8.0M (ma
Aug 05 09:07:14 centos72m kernel: Initializing cgroup subsys cpuset
Aug 05 09:07:14 centos72m kernel: Initializing cgroup subsys cpu
Aug 05 09:07:14 centos72m kernel: Initializing cgroup subsys cpuacct
Aug 05 09:07:14 centos72m kernel: Linux version 3.10.0-327.28.2.el7.x86_64 (buil
Aug 05 09:07:14 centos72m kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-327.2
Aug 05 09:07:14 centos72m kernel: e820: BIOS-provided physical RAM map:
Aug 05 09:07:14 centos72m kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000
The above is an extract of the output from the journal. The "short" option is the default option when no formatting option is specified.
Command Issued: journalctl -o short-precise
-- Logs begin at Fri 2016-08-05 09:07:14 BST, end at Fri 2016-08-05 09:22:14 BST
Aug 05 09:07:14.542037 centos72m systemd-journal[87]: Runtime journal is using 8
Aug 05 09:07:14.542170 centos72m systemd-journal[87]: Runtime journal is using 8
Aug 05 09:07:14.542194 centos72m kernel: Initializing cgroup subsys cpuset
Aug 05 09:07:14.542203 centos72m kernel: Initializing cgroup subsys cpu
Aug 05 09:07:14.542209 centos72m kernel: Initializing cgroup subsys cpuacct
Aug 05 09:07:14.542214 centos72m kernel: Linux version 3.10.0-327.28.2.el7.x86_6
Aug 05 09:07:14.542221 centos72m kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.
Aug 05 09:07:14.542227 centos72m kernel: e820: BIOS-provided physical RAM map:
The "short-precise" formatting option will show the same output, however, it will included a full time stamp with microseconds.
Command Issued: journalctl -o verbose
-- Logs begin at Fri 2016-08-05 09:07:14 BST, end at Fri 2016-08-05 09:22:14 BST
Fri 2016-08-05 09:07:14.542037 BST [s=f89c99aa206a4b80bd030bb986c40d05;i=1;b=0ee
PRIORITY=6
_TRANSPORT=driver
MESSAGE=Runtime journal is using 8.0M (max allowed 100.0M, trying to leave 1
MESSAGE_ID=ec387f577b844b8fa948f33cad9a75e6
_PID=87
_UID=0
_GID=0
_COMM=systemd-journal
_EXE=/usr/lib/systemd/systemd-journald
The above is only a small extract of the "verbose" formatting option. The "verbose" option will show full entries with all fields.
Display Boot information
The "-b" parameter may be passed to journalctl to display information relating to a specific system boot. Various parameters may be passed after the "-b" which allow you to specify a particular boot. For a list of boot numbers, you may issue the command "journalctl --list-boots". This will show you a list of boot numbers that are relative to the current boot, their IDs and the timestamp of the first and last message of the boot.
Command Issued: journalctl --list-boots
0 fbb340e720074a4b9e086214127360fc Fri 2016-08-05 09:58:56 BST—Fri 2016-08-05 10:04:40 BST
From the above output, we can see that we only have one entry. The ID is displayed along with the relevant timestamp information.
To see ALL boot info, see the section enabling persistent journals.
Command Issued: journalctl -b
-- Logs begin at Fri 2016-08-05 09:58:56 BST, end at Fri 2016-08-05 10:04:40 BST. --
Aug 05 09:58:56 centos72m systemd-journal[87]: Runtime journal is using 8.0M (max allowed 100.0M, trying to leave 150.0M free of 9
Aug 05 09:58:56 centos72m systemd-journal[87]: Runtime journal is using 8.0M (max allowed 100.0M, trying to leave 150.0M free of 9
Aug 05 09:58:56 centos72m kernel: Initializing cgroup subsys cpuset
Aug 05 09:58:56 centos72m kernel: Initializing cgroup subsys cpu
Aug 05 09:58:56 centos72m kernel: Initializing cgroup subsys cpuacct
Aug 05 09:58:56 centos72m kernel: Linux version 3.10.0-327.28.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.3 20
Aug 05 09:58:56 centos72m kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-327.28.2.el7.x86_64 root=/dev/mapper/sysVG-rootLV ro rd
Aug 05 09:58:56 centos72m kernel: e820: BIOS-provided physical RAM map:
If no arguments are passed, the "journalctl -b" command will display information from the most recent boot.
Display kernel messages
The command "journalctl -k" or "journalctl --dmesg" may used to display only "kernel" messages.
Command Issued: journalctl --k
-- Logs begin at Fri 2016-08-05 09:58:56 BST, end at Fri 2016-08-05 10:16:03 BST. --
Aug 05 09:58:56 centos72m kernel: Initializing cgroup subsys cpuset
Aug 05 09:58:56 centos72m kernel: Initializing cgroup subsys cpu
Aug 05 09:58:56 centos72m kernel: Initializing cgroup subsys cpuacct
Aug 05 09:58:56 centos72m kernel: Linux version 3.10.0-327.28.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.3 20
Aug 05 09:58:56 centos72m kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-327.28.2.el7.x86_64 root=/dev/mapper/sysVG-rootLV ro rd
Aug 05 09:58:56 centos72m kernel: e820: BIOS-provided physical RAM map:
The above is an extract of the kernel messages from the journal.
Enabling persistent journals
By default the journalctl command retrieves its information from files held within memory or from a small ring buffer. To enable persistency, first you will need to create a directory where the journals will be stored. Once the directory is in place, you will then need to restart "journald".
Command Issued: mkdir -p /var/log/journals
# mkdir -p /var/log/journal
# systemctl restart systemd-journald
Once the above has been done, you will now have persistency. As an example I have rebooted my system and issued the "list--boots" option "journalctl --list-boots". This time we can see previous boots as well as the most recent boot of our system.
# journalctl --list-boots
-1 fbb340e720074a4b9e086214127360fc Fri 2016-08-05 09:58:56 BST—Fri 2016-08-05 10:30:23 BST
0 87dac0c7e2ed41d7b0761516bd013a38 Fri 2016-08-05 10:30:32 BST—Fri 2016-08-05 10:36:37 BST
