7. Linux Logs, System Monitoring, Scripting & Recovery
Modules 23-28: journalctl, top/vmstat/iostat, df vs du, backup tar/rsync, Bash scripting set -euo pipefail & server hardening
Module 1: Logs, journalctl & System Resource Monitoring
Diagnosing system issues using systemd logs and performance profiling tools.
🇬🇧 English Definition
Logs are essential for diagnosis. systemd systems use journalctl alongside traditional /var/log files. Monitor CPU, memory, disk I/O, and load using top, htop, free, vmstat, and iostat.
🇮🇳 Hinglish Definition
Logs failures aur errors diagnose karne ke liye main evidence hain. Resource monitoring ka goal resource bottleneck identify karna hai.
Core Concepts & Rules
- journalctl -u service: Filters logs specifically for a target systemd service.
- vmstat 1 & iostat -xz 1: Monitors CPU wait times, memory paging, and disk I/O latency live.
- df vs du: df reports filesystem space usage; du reports space consumed by directory trees. Deleted-open files can cause df to report higher usage than du.
🔍 Line-by-Line Command Breakdown
- Line 1: journalctl -u nginx --since '1 hour ago' queries recent nginx service logs.
- Line 2: free -h prints total, used, and available system memory in human units.
- Line 3: vmstat 1 outputs memory, swap, IO, and CPU statistics refreshed every 1 second.
🎯 Interview Focus & Questions
- Q: How do you inspect a service failure?
A: Use systemctl status, journalctl -u service, application log files, and syntax validation tests. - Q: Why can df be much higher than du?
A: Deleted files still held open by running processes, reserved filesystem space, or unmounted boundaries.
journalctl -u nginx --since "1 hour ago"
journalctl -f
tail -f /var/log/syslog
top
free -h
vmstat 1
iostat -xz 1
df -h
du -sh /var/*Module 2: Backup Strategies & Production Bash Shell Scripting
Creating verifiable backups and writing safe, robust administrative Bash scripts.
🇬🇧 English Definition
Backups protect against data loss using tar and rsync. Always test restore procedures. Production Bash scripts must include strict error flags (set -euo pipefail), quoted variables, and structured logging.
🇮🇳 Hinglish Definition
Backup file banana enough nahi hai; restore test karna compulsory hai. Bash scripts predictable, logged aur safe hone chahiye.
Core Concepts & Rules
- set -euo pipefail: Immediately aborts script execution on error, undefined variable, or pipeline failure.
- rsync -avh --delete: Mirrors directories while preserving permissions and removing deleted target files.
- Quoting Variables: Prevents word splitting and unwanted wildcard expansion bugs.
🔍 Line-by-Line Command Breakdown
- Line 1: // Shebang: /usr/bin/env bash specifies Bash interpreter path.
- Line 2: set -euo pipefail enforces strict error handling rules.
- Line 3-7: Automated service health check script logging status to file.
🎯 Interview Focus & Questions
- Q: What should a backup strategy include?
A: Scope, frequency, retention, off-site copy, monitoring, encryption, and regular restore testing. - Q: What is $?
A: Contains the exit status code (0 for success, non-zero for error) of the most recently executed command.
// Script: healthcheck.sh
// Shebang: /usr/bin/env bash
set -euo pipefail
LOG=/var/log/healthcheck.log
echo "Host: $(hostname)" | tee -a "$LOG"
if systemctl is-active --quiet nginx; then
echo "nginx is running" | tee -a "$LOG"
else
echo "nginx is down" | tee -a "$LOG"
fi