💡

Before you start

You need a Linux machine where you can use sudo, and you should not practise on a server anyone depends on. The commands here change real system state. A spare machine, a virtual machine, or a cloud instance you can rebuild is the right place; a laptop you own is fine too. If the terminal itself is new to you, do Introduction to the Linux Terminal first — it takes about ten minutes and everything below assumes it. Every step tells you how to undo it.

Logs You Never Read Are Just Storage

Most Linux systems log far more than anyone looks at. The goal of this tutorial is not to collect more — it is to know which handful of events actually indicate a problem, and to be able to answer a specific question quickly when something looks wrong.

There are two systems in play on a modern distribution. journald collects everything services emit and is what you will use daily. auditd is a separate kernel-level subsystem that records syscall-level events — who touched which file, who executed what — and is what you add when you need forensic detail.

journalctl: The Daily Tool

journalctl -u ssh --since "1 hour ago"     # one service, recent
journalctl -p err -b                        # errors since this boot
journalctl -f                               # follow, like tail -f
journalctl _COMM=sudo -n 50                 # everything sudo did
journalctl --since "2026-08-01" --until "2026-08-02 12:00"
journalctl -u nginx -o json-pretty          # structured output for scripting

-b — this boot. -b -1 is the previous boot, which is how you investigate an unexpected reboot
-p — priority: err, warning, crit
-k — kernel messages only
-g PATTERN — grep within the journal, keeping the field structure intact

Authentication events specifically:

journalctl -u ssh | grep -i 'failed\|invalid'
sudo lastb | head -20        # failed logins
last -n 20                   # successful logins
who                          # who is on the machine right now

Make the Journal Persistent

On some installations the journal lives only in memory and is lost on reboot — which is exactly when you most want it.

journalctl --disk-usage
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

Then cap it so it cannot fill the disk, in /etc/systemd/journald.conf:

[Journal]
Storage=persistent
SystemMaxUse=1G
MaxRetentionSec=90day

auditd: Answering Specific Questions

Add auditd when you need to know exactly who did something. Its value comes entirely from targeted rules — enabling everything produces unreadable volume and real performance cost.

sudo apt install auditd audispd-plugins

Rules live in /etc/audit/rules.d/ and are loaded at start:

-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/sudoers -p wa -k privilege
-w /etc/sudoers.d/ -p wa -k privilege
-w /etc/ssh/sshd_config -p wa -k sshd
-w /var/log/auth.log -p wa -k logtamper
-a always,exit -F arch=b64 -S execve -F euid=0 -F auid>=1000 -F auid!=-1 -k rootcmd

-w — watch a path
-p wa — on write and attribute change (not reads, which are noisy)
-k — a key you can search by later. Always set one; a rule without a key is painful to query
The last rule records every command run as root by a human account — arguably the highest-value single audit rule on a multi-admin server

sudo augenrules --load
sudo auditctl -l              # confirm the rules are actually loaded

Searching:

sudo ausearch -k identity -i          # -i makes IDs human-readable
sudo ausearch -k rootcmd --start today -i
sudo ausearch -m USER_LOGIN --start today -i
sudo aureport --summary
sudo aureport --auth --failed

💡

auid is the field that makes auditing useful.

The audit UID is set at login and follows a user through sudo and any shell changes. So even when someone becomes root, auid still records which human logged in. Searching by euid alone loses exactly the attribution you wanted.

Protecting the Logs Themselves

An attacker with root can edit local logs. You cannot fully prevent that on the same machine — which is the whole argument for shipping logs elsewhere.

Forward to a separate host — a remote syslog target or a log service. Once a line has left the machine, local tampering cannot retract it. This is the single most effective measure
Make audit rules immutable — end the rules file with -e 2, which locks the configuration until reboot so rules cannot be quietly unloaded
Watch the log files themselves — the logtamper rule above
Check logrotate — retention that silently deletes at seven days is a gap you will discover at the worst moment (/etc/logrotate.d/)
Watch for gaps — a period with no logs is itself a finding

What Actually Deserves Attention

A short list beats a dashboard nobody reads.

A successful login after many failures from the same source — the pattern that means a guess landed
Any successful root login, if you have disabled it
Logins at unusual hours, or from unfamiliar countries
New accounts, or group changes — especially additions to sudo
Changes to sshd_config, sudoers, or crontabs
New setuid binaries appearing anywhere
Services restarting unexpectedly, or a reboot nobody scheduled
Outbound connections from a server that should only receive

A five-minute morning check, worth scripting:

sudo lastb | head -20
last -n 10
sudo ausearch -k privilege --start today -i
journalctl -p err -b | tail -20
sudo find / -perm -4000 -type f -newer /etc/hostname 2>/dev/null

Now Do It Yourself: Five Steps

Auditing is not "having logs" — every machine has those. It is being able to ask a log a question and get an answer. You will check who is on the machine now, review the login history, write and retrieve your own journal entry, and turn a wall of authentication noise into a short list of offenders. Steps 1 to 3 need no sudo at all.

See who is on the machine, and who has been

Go: open a terminal.

Do: run who, then last -n 5.

You should see: the current sessions with their terminal and start time, then the last five login events — each showing the user, where from, and either a duration or still logged in. Lines reading reboot system boot mark restarts, which is often the fastest way to spot an unplanned one.

If not: if last prints nothing, the record file has been rotated away, which is normal on a long-lived system. Filter to just restarts with last -x reboot. Note this data comes from /var/log/wtmp, which is world-readable — a useful reminder that "not secret" and "not important" are different things.

Write your own entry into the system journal, then find it

Go: same terminal.

Do: run these two commands.

logger -t audit-practice "testing the journal"
journalctl -t audit-practice -n 1 --no-pager

You should see: a line like Aug 20 00:47:39 hostname audit-practice[1378860]: testing the journal — timestamp, host, the tag you chose, the process id, and your message. This is the round trip every service on the machine uses, and -t is how you pull one source out of everything else.

If not: if journalctl prints Hint: You are currently not seeing messages from other users and the system, that is correct and expected — an ordinary account sees only its own. Your own message still appears. To read everything you need sudo journalctl, or membership of the systemd-journal group.

Find out which logs you can actually read

Go: same terminal.

Do: run ls -l /var/log/auth.log /var/log/syslog /var/log/wtmp /var/log/btmp.

You should see: wtmp readable by everyone, while auth.log, syslog and btmp are restricted to root and the adm group. That split is deliberate: btmp records failed logins, and failed logins frequently contain a password typed into the username box by mistake.

If not: on a systemd-only machine some of these files do not exist at all, because the journal replaced them — use journalctl _COMM=sshd instead of reading auth.log. ⚠️ Never cat a failed-login log onto a shared screen or into a ticket for exactly the reason above.

Turn noise into a question you can answer

Go: same terminal. So the numbers match, practise on a sample rather than the real log.

Do: create a four-line sample, then count the failures.

printf 'Aug 20 03:11:02 host sshd[111]: Failed password for invalid user admin from 203.0.113.9 port 51000 ssh2\nAug 20 03:11:05 host sshd[112]: Failed password for invalid user admin from 203.0.113.9 port 51002 ssh2\nAug 20 03:11:09 host sshd[113]: Failed password for root from 198.51.100.4 port 40222 ssh2\nAug 20 03:12:00 host sshd[114]: Accepted publickey for alice from 192.0.2.7 port 55010 ssh2\n' > auth-sample.log
grep -c "Failed password" auth-sample.log

You should see: 3. On a real server run the same idea against the live file with sudo grep -c "Failed password" /var/log/auth.log. A number in the thousands is not an emergency by itself — any machine with a public SSH port is scanned constantly. What matters is whether any of them succeeded.

If not: if you get 0, your distribution words it differently — look at a few real lines first with sudo tail -20 /var/log/auth.log and match what is actually there. Never assume a log format; read it.

Rank the offenders, then check whether anyone got in

Go: same folder.

Do: run these two commands.

grep "Failed password" auth-sample.log | grep -oE 'from [0-9.]+' | awk '{print $2}' | sort | uniq -c | sort -rn
grep "Accepted" auth-sample.log

You should see: a ranked tally — 2 203.0.113.9 then 1 198.51.100.4 — and then the one successful login, Accepted publickey for alice from 192.0.2.7. Read the pipeline left to right: find the failures, pull out the addresses, count each, sort by count descending. That is the whole of basic log analysis.

If not: 🔴 The second command is the one that matters. Thousands of failures are background noise; a single Accepted password from an address you do not recognise is an incident. Check the method too — Accepted publickey is expected on a hardened server, while Accepted password means password logins are still enabled and should not be.

🎉

Check yourself before moving on

Without scrolling up: your server shows 40,000 failed SSH logins this week. How worried should you be, and what single command decides it? Answer: not worried by the number itself — any public SSH port is scanned constantly. Search for Accepted, and check both the source address and whether it says publickey or password.

Now do it without the page: build one pipeline that lists the usernames attackers tried most often, rather than their addresses. Same shape as step 5 with a different field pulled out — and the answer tells you which accounts to be sure do not exist.

Summary

journald for daily work, auditd when you need attribution
Make the journal persistent and capped — memory-only logs vanish exactly when needed
Targeted audit rules with -k keys beat auditing everything
auid survives sudo — it is what ties a root action to a person
Ship logs off the machine — the only real defence against tampering
A missing period of logs is a finding, not a gap to ignore

🎉

Start with one rule and one habit.

Add the rootcmd audit rule so every command run as root is attributable to a human, and check sudo lastb | head once a day. That combination catches more real problems than any amount of log volume you never look at.

Security Auditing and Logging