How to Configure Log Rotation, Journald, and Centralized Logging on Debian and Ubuntu Servers

Logs are files that record activity during an application’s execution. They are also sometimes used to analyze an application’s status. Log files are therefore important files within a system that continuously record all events (activities) occurring within the system. These files are crucial to a system, making it easier for us, especially administrators, to examine and trace various problems that arise. Log files help admins identify bugs, attack sources, and system damage, even if they don’t know how to fix them. In this article, we will show you how to configure log rotation, journald, and centralized logging on Debian and Ubuntu servers.

Log files, typically located in the /var/log directory, can grow in size, potentially filling your hard drive and causing system or service failures. Fortunately, this usually won’t happen because Linux systems generally come with a log rotation mechanism through the logrotate software package.

Prerequisites

• A Debian VPS or an Ubuntu server
• SSH root access or a regular system user with sudo privileges

Conventions

# – given commands should be executed with root privileges either directly as a root user or by use of sudo command
$ – given commands should be executed as a regular user

Step 1. Update the System

Let’s log in to your Debian 13 VPS through SSH as a root user or as a regular user with sudo privileges.

ssh root@IP_Address -p Port_number

If you cannot log in as root, remember to substitute “root” with a user that has sudo privileges. Additionally, change “IP_Address” and “Port_Number” make sure it matches your server’s respective IP address and SSH port.

Step 2. Logrotate

Logrotate is a tool used in Linux operating systems to manage log files. Logrotate’s primary function is to organize, rotate, and automatically delete log files. With logrotate, you can configure how often log files are rotated, how many log files are retained, and even how old log files are deleted. This helps keep your system light and clean.

Log rotation with logrotate is very simple: it moves or renames the old log file to a different name (e.g., file.log to file.log.1), then creates a new log file. This process is performed for a specified period of time, then repeated in subsequent periods (e.g., file.log.1 –> file.log.2, file.log –> file.log.1, create new file.log), and so on until the specified number of rotations is reached.

To get started, you need to understand some basic steps in configuring logrotate. Here they are:

• Install Logrotate: Make sure logrotate is installed on your system. It should be installed by default on standard Ubuntu/Debian installation. You can check with the logrotate –version command.
• Configuration File: Logrotate uses a configuration file typically located at /etc/logrotate.conf and within the /etc/logrotate.d/ folder.
• Specify Log Files: Specify the log files you want to rotate. For example, /var/log/syslog.
• Specify Rotation Frequency: You can set daily, weekly, or monthly rotations. For example, if you want daily rotations, add daily to the configuration file.
• Specify Retain Amount: You can also specify how many log files to retain. For example, rotate 7 will retain the last seven log files.

Configuration Example

Here is an example of a simple configuration for logrotate. You can add this to the /etc/logrotate.d/example file:

/var/log/example.log { 
   daily 
   rotate 7 
   compress 
   missingok 
   notifempty 
}

In the example above:

daily: The log file will be rotated daily.
rotate 7: Keeps the last seven log files.
compress: Compresses the already rotated log file.
missingok: Will not generate an error if the log file does not exist.
notifempty: Will not rotate the log file if it is empty.

Step 3. Journald

Journald is a system in Linux distributions responsible for collecting and storing logging data. It is part of systemd, responsible for controlling which processes or programs are activated during the Linux operating system’s bootup. The log files created by journald can grow rapidly, as they are constantly being added with information that may be useful for troubleshooting. So, don’t be surprised if these journald log files can become quite large.

Journald log files are not formatted in plaintext like typical log files. Journald logs in binary format, similar to authentication logs, which are typically found in /var/log/btmp.

Additionally, there is no log rotation mechanism for journald’s logging files. By default, log files are split and compressed when they exceed a specified size limit.

First, you can check the disk usage of the journal log using the following command:

# du -h /var/log/journal/

or

# journalctl --disk-usage

This is the log that has accumulated since the system was first installed (if it has never been cleared).

On Linux systems using systemd like Ubuntu and Debian, journal log configuration can be adjusted in /etc/systemd/journald.conf file.

To change these settings, you can open the file using an editor, such as nano:

# nano /etc/systemd/journald.conf

Some important parameters to note include:

SystemMaxUse: Determines the maximum total size of journal logs allowed on disk.
SystemKeepFree: Determines the amount of free disk space that must be kept available.
MaxRetentionSec: Determines how long logs will be retained before they are automatically deleted.

For example, here’s a fairly safe configuration for a VPS with a small disk capacity (please adjust as needed):

SystemMaxUse=500M
SystemKeepFree=1G
MaxRetentionSec=30day

After making these configuration changes, restart the journald service for the new settings to take effect:

# systemctl restart systemd-journald

Then you can check the log space usage again with the following command:

# journalctl --disk-usage

Step 4. Centralized Logging

Centralized log collection is one of the goals of log management. This centralized logging makes it easier to debug when a system malfunctions or an incident occurs. Furthermore, centralized logging is also crucial for a robust security strategy. Placing all logs in one central location simplifies log analysis and provides a more detailed picture of the current situation.

There are actually many free tools available for syslog server purposes, with popular ones being rsyslog and syslog-ng. Syslog-ng has more features and a more complex configuration, but rsyslog is sufficient for standard log collection needs. In this tutorial, we’ll discuss configuring a centralized log server using rsyslog. Rsyslog is considered installed; if it’s not, simply install it by running the command below.

# apt install ryslog

The main rsyslog configuration is located in /etc/rsyslog.conf. You can add configuration directly there, but since this file also reads configuration files from /etc/rsyslog.d/, it would be better to create a new configuration file there.

# nano /etc/rsyslog.d/rosehosting.conf

You can change rosehosting.conf to any file name you want. However, make sure it’s located in /etc/rsyslog.d/ and has the extension .conf, insert the following lines into the file:

$ModLoad imudp
$UDPServerRun 514
$ModLoad imtcp
$InputTCPServerRun 514

$AllowedSender TCP, 127.0.0.1, 192.168.1.0/24, *.domain.com, sub1.domain.com
$AllowedSender UDP, 127.0.0.1, 192.168.1.0/24, *.domain.com, sub1.domain.com

$template remote-incoming-logs,"/var/log/rsyslog/%HOSTNAME%/%PROGRAMNAME%.log" 
*.* ?remote-incoming-logs
& ~

The configuration above will tell rsyslog to run on port 514, UDP and TCP. It also tells which IP address, subnet, or domain/subdomain who can connect to this ryslog server.

Save the file, exit then restart rsyslog:

# systemctl restart rsyslog

That’s it! Rsyslog server has been comfigured, we can open the ports now.

# ufw allow 514/tcp
# ufw allow 514/udp

Configure rsyslog Client

This syslog client will be responsible for sending logs to the syslog server we provided above. On the client server, run this:

# apt update && apt install rsyslog -y

Configuration on the client side is relatively straightforward.

# nano /etc/rsyslog.d/rsyslog-client.conf

Then add the following lines:

#Send log over UDP
*.* @ubuntu.rosehosting.com:514

#Send log over TCP
*.* @@ubuntu.rosehosting.com:514

#Set disk queue
$ActionQueueFileName queue
$ActionQueueMaxDiskSpace 1g
$ActionQueueSaveOnShutdown on
$ActionQueueType LinkedList
$ActionResumeRetryCount -1

Make sure to replace ubuntu.rosehosting.com with your own rsyslog server’s hostname. You can also send the message directly to an IP address, for example, 192.168.1.2:514. Just adjust the host and port accordingly.

Now, to apply the changes, we need to restart and enable the rsyslog service.

# systemctl restart rsyslog

You’ve Finally Set Up Centralized Logging on Debian and Ubuntu Servers

That’s it; you can view this client’s logs on your rsyslog server, specifically in /var/log/rsyslog/

Congratulations! You have learned how to configure log rotation, journald, and centralized logging on Debian and Ubuntu servers.

Of course, you don’t have to configure log rotation, journald, and centralized logging on Debian and Ubuntu servers if you have an active hosting service with us in which case you can simply ask our expert Linux administrators to configure log rotation, journald, and centralized logging on Debian and Ubuntu servers for you for free. They are available 24×7 and will take care of your request immediately.

If you liked this post about how to configure log rotation, journald, and centralized logging on Debian and Ubuntu servers, please share it with your friends or leave a comment below.