· Scenario

Disk full at 2 a.m.: a calm, systematic triage

/var hits 100% on a production app server. A repeatable sequence to find the cause, free space safely, and stop it happening again — without deleting the wrong thing.

  • Linux
  • Troubleshooting
  • Runbooks

The scenario

Situation
A monitoring alert fires: /var on a production application server is at 100%. The application has started failing to write logs and requests are erroring.
Environment
RHEL with LVM and XFS; application logs under /var/log/app; logrotate configured but not for this log.
Goal
Restore service in minutes without losing data needed for the post-incident review, then remove the root cause.
Outcome
Space recovered from 100% to 16% in under ten minutes; logrotate policy fixed so the alert has not recurred.

Full filesystems are among the most common Linux incidents, and among the easiest to make worse. The instinct is to start deleting. The goal of this runbook is to replace instinct with a short, safe sequence.

The whole triage in 40 seconds

Animated terminal session showing df, du, find, lsof and truncate commands freeing space on /var
Recorded from a scripted, anonymised session — see the tools folder in this site's repository for how it's made.

Step 1 — confirm what is actually full

df -hT /var        # space
df -i  /var        # inodes — a filesystem can be "full" with free space left

If IUse% is 100% and space is not, you’re hunting millions of small files (session files, mail queue, cache), not one big one. That’s a different search: sudo find /var -xdev -type f | cut -d/ -f2-4 | sort | uniq -c | sort -rn | head.

Step 2 — find where the space went

sudo du -xh --max-depth=1 /var 2>/dev/null | sort -rh | head

-x keeps du on one filesystem, so a mounted /var/lib/docker or NFS share doesn’t pollute the answer. Drill down one level at a time until you reach the culprit, then list large files directly:

sudo find /var/log -xdev -type f -size +1G -exec ls -lh {} +

Step 3 — check for deleted-but-open files

If df says full but du can’t find the space, a process is probably holding a deleted file open. The space isn’t released until that file descriptor closes.

sudo lsof +L1            # open files with link count < 1, i.e. deleted

The fix is to restart (or signal) the owning process — or, if you can’t restart it yet, truncate through its descriptor: : > /proc/<pid>/fd/<fd>.

Step 4 — free space without destroying evidence

Safer options, in order of preference:

  1. Compress and move the old part if you need it for the post-incident review: sudo sh -c 'tail -n 50000 app.log > /tmp/app.log.tail' before truncating.
  2. Truncate in place: sudo truncate -s 0 /var/log/app/app.log — the process keeps its file handle and carries on.
  3. Extend the filesystem if the data is legitimately needed and the volume group has space: sudo lvextend -r -L +5G /dev/vg00/var (-r grows XFS online in the same step).

Step 5 — fix the cause, not the symptom

In this scenario the application wrote to a log that wasn’t covered by logrotate. The permanent fix was a policy file:

# /etc/logrotate.d/app
/var/log/app/*.log {
    daily
    rotate 14
    maxsize 1G
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

copytruncate is used because this application can’t reopen its log on a signal. If yours can, prefer create plus a postrotate reload — copytruncate can lose a few lines written during the copy.

Validate before you walk away: sudo logrotate -d /etc/logrotate.d/app (dry run) and then sudo logrotate -f /etc/logrotate.d/app.

Checklist

  • [ ] Space or inodes? (df -h, df -i)
  • [ ] Where? (du -x, find -size)
  • [ ] Deleted-but-open? (lsof +L1)
  • [ ] Free safely (truncate / move / extend — not rm on open files)
  • [ ] Root cause fixed and tested (logrotate dry run)
  • [ ] Monitor tuned to warn on trend, not just threshold

The scenario

Situation
An application admin reports that sudo suddenly fails with an 'expired' message, even though they are typing the correct password and SSH key login still works.
Environment
RHEL servers with local service-admin accounts governed by a password-ageing policy; some hosts also use SSSD.
Goal
Restore access quickly without weakening the security policy, and stop it surprising people again.
Outcome
Root cause identified as password ageing in the PAM account phase; access restored via a controlled reset, and expiry warnings added to monitoring.

This is one of those tickets that sounds like “user forgot password” and isn’t. The user’s password is correct. Their SSH key works. And yet:

$ sudo -i
[sudo] password for appadmin:
You are required to change your password immediately (password expired)
sudo: Account or password is expired, reset your password and try again

Why a correct password isn’t enough

sudo doesn’t check passwords itself — it hands the job to PAM (Pluggable Authentication Modules), using the service file /etc/pam.d/sudo. PAM runs separate management groups, in order:

How sudo walks the PAM stack sudo calls PAM using the service file /etc/pam.d/sudo. PAM runs four management groups in order: auth, account, password, session. The auth step can succeed while the account step fails because the account check reads password and account expiry from the shadow file or from SSSD, so a user with an expired password is rejected even though their password is correct. $ sudo -i user: appadmin /etc/pam.d/sudo 1 · authPASS pam_sss.so / pam_unix.so "Is the password correct?" → yes 2 · accountFAIL pam_unix.so → /etc/shadow (chage) "Is the account / password still valid?" → expired sudo: Account or password is expired… 3 · password only used when changing a password 4 · session never reached — limits, logging, mkhomedir Key point: a correct password is not enough. The account phase is a separate check, which is why "but my password works for SSH keys" is not evidence the account is healthy.
Authentication succeeds; the account phase fails. They are different questions.
  • auth — is this really the user? (password correct → pass)
  • account — is this account allowed right now? (expired, locked, outside allowed hours → fail)
  • password — used only when changing a password
  • session — set-up and tear-down: limits, logging, home directory creation

SSH key login skips the password check entirely, and depending on UsePAM and the PAM stack for sshd, may not trip the same account checks. That’s why “SSH works” is not evidence the account is healthy.

Step 1 — read the account’s ageing data

sudo chage -l appadmin
Last password change                : Aug 01, 2026
Password expires                    : Oct 30, 2026
Password inactive                   : never
Account expires                     : never
Minimum number of days between password change : 1
Maximum number of days between password change : 90
Number of days of warning before password expires : 7

90-day maximum, last changed 1 August — expired on 30 October. Mystery solved for local accounts.

Step 2 — confirm it in the logs

sudo journalctl -t sudo --since "1 hour ago"
sudo grep -E 'appadmin.*(expired|account)' /var/log/secure | tail

Look for pam_unix(sudo:account): expired password — note account, not auth. On SSSD-backed hosts you’ll see pam_sss(sudo:account) instead, and the expiry comes from the directory, so the fix belongs there, not on the server.

Step 3 — fix it without weakening the policy

For a human admin account, the clean fix is a controlled reset that forces a change at next login:

sudo passwd appadmin          # set a temporary password via your approved channel
sudo chage -d 0 appadmin      # force change on next login

For a service account that only runs automation, the better long-term fix is to remove password-based sudo altogether: key-based login, a narrowly scoped NOPASSWD rule for the specific commands it runs, and a locked password (passwd -l).

Step 4 — make sure nobody is surprised again

Expiry is predictable, so it should never be an incident. A small daily check, run from cron or your configuration management tool, can report accounts expiring soon:

#!/usr/bin/env bash
# expiring-accounts.sh — list local accounts whose password expires within N days.
set -euo pipefail
DAYS=${1:-14}
today=$(( $(date +%s) / 86400 ))
# /etc/shadow fields: 1=user 3=last change (days since epoch) 5=max age
sudo awk -F: -v today="$today" -v window="$DAYS" '
  $2 !~ /^[!*]/ && $5 != "" && $5 < 99999 && $3 != "" {
    left = ($3 + $5) - today
    if (left <= window) printf "%-20s expires in %d day(s)\n", $1, left
  }' /etc/shadow

Feed the output to your monitoring as a low-priority alert, or email the account owners. On SSSD/IdM, query expiry from the directory instead.

Takeaways

  • sudo failures with a correct password usually live in PAM’s account phase.
  • chage -l and the pam_*(<service>:account) log line give you the answer in under a minute.
  • Fix with a controlled reset, not by disabling expiry; remove passwords from service accounts entirely.
  • Predictable failures should be monitored, not discovered.

The scenario

Situation
Ahead of a subscription review, the number of hosts registered in Satellite didn't match the number of RHEL servers the CMDB said were running.
Environment
Mixed on-prem and AWS RHEL estate; hosts rebuilt from templates; Satellite with Dev, QA and Prod lifecycle environments.
Goal
Get to a trustworthy registered-host list, and keep it that way every month with minimal effort.
Outcome
A monthly export-and-reconcile routine that flags stale, duplicate and unregistered hosts before they become licence or patching problems.

In a large RHEL estate, registration drift happens quietly. A VM is rebuilt with the same hostname and now appears twice. A server is decommissioned but never unregistered. A rushed build skips registration and never receives patches. None of these break anything on the day — they surface later, during a licence true-up or when a vulnerability scan finds an unpatched host nobody knew about.

The routine

Host registration and monthly licence reporting with Red Hat Satellite A new RHEL host registers to Satellite using an activation key tied to an organisation, lifecycle environment and content view. Satellite tracks every registered host. Each month a scheduled report exports the host inventory, which is reconciled against the CMDB to find stale, duplicate or unregistered hosts before the licence true-up. New RHEL host built from template subscription-manager register --activationkey Red Hat Satellite organisation lifecycle env: Dev → QA → Prod content view (frozen repos) host inventory + facts Monthly report scheduled export hammer host list --csv Reconcile vs CMDB / cloud inventory Findings each month • stale hosts (no check-in > 30 days) → unregister • duplicates (rebuilt with same name) → remove old record • running but unregistered → register or decommission • wrong lifecycle env → move before next patch cycle
Register consistently, export monthly, reconcile against what should exist.

1. Register consistently — always with an activation key

Activation keys bind a host to the right organisation, lifecycle environment and content view in one step, so builds don’t depend on someone remembering the right options.

# Install the Satellite CA/consumer package, then register (run on the new host)
sudo rpm -Uvh https://satellite.example.internal/pub/katello-ca-consumer-latest.noarch.rpm
sudo subscription-manager register \
    --org="EXAMPLE_ORG" \
    --activationkey="rhel9-prod"
sudo subscription-manager identity    # confirm registration

Put this in your build template or provisioning playbook so it isn’t optional. On newer Satellite versions, the global registration command generated from the web UI does the same job with a single curl | bash.

2. Export the inventory monthly

hammer, Satellite’s CLI, makes the export scriptable:

hammer --output csv host list \
    --organization "EXAMPLE_ORG" \
    --fields "Name,Operating System,Lifecycle Environment,Content View,Last Checkin" \
    > "satellite-hosts-$(date +%Y-%m).csv"

Keeping one CSV per month gives you a simple history: you can diff months to see what was added and removed.

3. Reconcile against what should exist

Compare the Satellite list with your source of truth (CMDB export, VMware inventory, or aws ec2 describe-instances). A few lines of shell are enough to find the gaps:

#!/usr/bin/env bash
# reconcile.sh <satellite.csv> <cmdb_hosts.txt>  — hostnames one per line in the second file
set -euo pipefail
[[ $# -eq 2 ]] || { echo "usage: $0 satellite.csv cmdb_hosts.txt" >&2; exit 2; }
tail -n +2 "$1" | cut -d, -f1 | sed 's/\..*//' | tr 'A-Z' 'a-z' | sort -u > /tmp/sat.txt
sed 's/\..*//' "$2" | tr 'A-Z' 'a-z' | sort -u > /tmp/cmdb.txt
echo "== Running but NOT registered (no patches, no entitlement) =="
comm -13 /tmp/sat.txt /tmp/cmdb.txt
echo
echo "== Registered but NOT in CMDB (decommissioned? paying for nothing) =="
comm -23 /tmp/sat.txt /tmp/cmdb.txt

Short hostnames are compared, lower-cased, so APP01.example.internal and app01 match.

4. Act on the findings

Finding Typical cause Action
No check-in for 30+ days Host decommissioned or powered off Confirm, then hammer host delete
Same name registered twice Rebuilt from template Remove the older record
Running but unregistered Build skipped registration Register with the correct activation key
Wrong lifecycle environment Built with the wrong key Move it before the next patch cycle

Why bother?

  • Licensing: the registered-host count is what you’ll be asked to justify.
  • Security: an unregistered host doesn’t get content-view updates — it’s effectively unpatched.
  • Trust: when the inventory is right, every other report built on it (patch compliance, vulnerability exposure) is right too.