Last week, we explored scripting principles for modularity, parsing, and reporting in Scripting: Modularity, Parsing, and Reporting. These principles help you break down complex tasks into reusable scripts, whether in Bash, Perl, or Python. We also introduced regular expressions and best practices for parsing in Scripting with Regex in Bash, Perl, and Python.

Building on that foundation, this guide focuses on automating routine system maintenance and monitoring tasks. While scripting allows you to perform individual operations efficiently, automation ensures these tasks run consistently, reliably, and with minimal oversight.

This week, we cover practical examples of system maintenance tasks, automated log extraction, monitoring for system health, and log rotation strategies. These concepts form the foundation for more complex orchestration across multiple systems. If you’re interested in network monitoring, Mastering Network Tools provides complementary insight for integrating monitoring scripts with system and network logs.

Core System Maintenance Tasks

OS Updates

Regular updates keep your system secure and stable. We recommend a balanced approach:

• Minor updates: Can be automated through scripts run via cron. These typically include security patches and bug fixes.
• Major version upgrades: Should be manually reviewed to avoid unexpected changes.

Different distributions handle updates differently:

• Debian/Ubuntu (apt) – minor updates can be applied automatically.
• RedHat/CentOS (yum/dnf) – minor updates may be automated; major upgrades should remain manual.
• Arch Linux (pacman) – rolling release requires regular manual updates (Installing Arch Linux guide has details).

Tip: Always log all update actions for auditing and troubleshooting. A dedicated log file makes it easier to track success and failures.

A simple update script can select the distribution type with a variable and record output to ~/.logs/os-update.log.

#!/bin/bash
# Simple OS update script
# DIST=1 (Debian/Ubuntu), 2 (RedHat/CentOS), 3 (Arch Linux)
DIST=${1:-1}
LOG="$HOME/.logs/os-update.log"
mkdir -p "$(dirname "$LOG")"
case $DIST in
  1) sudo apt update && sudo apt upgrade -y >> "$LOG" 2>&1 ;;
  2) sudo dnf update -y >> "$LOG" 2>&1 ;;
  3) sudo pacman -Syu --noconfirm >> "$LOG" 2>&1 ;;
esac
echo "Update completed on $(date)" >> "$LOG"

/## Disk Usage Monitoring

Disk space surprises can be catastrophic. Regular monitoring is essential. Useful commands:

df -h      # Disk free space by partition
du -sh *   # Size of directories

Tip: Monitor user-space temporary directories such as ~/tmp, ~/.tmp, or ~/.temp. Regular cleanup prevents accidental disk space exhaustion. For additional cleanup strategies, see Stupid Bash Tricks, Part One.

Logging disk usage patterns to ~/.logs/disk-usage.log helps identify growth trends early and alerts you to potential problems before they affect operations.

Service Monitoring and Other Logs

Track key service logs to stay proactive:

• Cron job outputs
• Web server logs
• Custom scripts generating logs

Pull Quote: “Logging failures or unusual patterns provides insight into recurring errors and unusual activity, enabling proactive maintenance.”

Automation ensures consistent extraction and logging of relevant information, which can feed into monitoring dashboards or alerting systems.

Example: BBS Logs

For illustration, consider a software application generating logs automatically:

• crash.log
• events.log
• error.log
• hack.log
• sbbsecho.log

Automation can parse these logs to extract key information. For example, hack.log might feed hack-ips.log for tracking IP addresses of failed login attempts. Additional cron jobs may log output to sbbsecho-cron.log or hackips-cron.log.

Tip: Focus on automation: extract, log, and alert as needed. You do not need to understand the underlying software in depth—just how to process its logs. For more on BBS setups, see Setting Up a BBS.

Logging & Monitoring

Logging Best Practices

• Separate system and user logs to avoid permission issues and accidental data loss.
• Avoid running scripts as root unless absolutely necessary.
• Monitor log file growth, as files can expand unexpectedly and consume significant disk space.

Pull Quote: “Monitor log file growth: files can explode in size unexpectedly.”

Automated Log Extraction

Parsing logs for key information (IP addresses, errors, task results) can be automated with cron. Alerts can optionally be triggered for significant events.

#!/bin/bash
# Extract IPs from hack.log to hack-ips.log
LOG_DIR="$HOME/.logs"
mkdir -p "$LOG_DIR"
grep -oE '\b([0-9]{1,3}\.){3}[0-9]{1,3}\b' hack.log > "$LOG_DIR/hack-ips.log"

Tip: Running extraction via cron ensures you always have up-to-date data without manual intervention.

System Health Monitoring

Track essential system metrics regularly:

• CPU load: top, vmstat
• Memory usage: free, vmstat
• Disk space: df, du

Log metrics to ~/.logs/system-health.log and optionally alert via email if thresholds are exceeded:

#!/bin/bash
LOG="$HOME/.logs/system-health.log"
mkdir -p "$(dirname "$LOG")"
CPU_LOAD=$(uptime | awk -F'load average:' '{ print $2 }' | cut -d, -f1)
MEM_USED=$(free | awk '/Mem/ {printf("%.0f"), $3/$2*100}')
DISK_USED=$(df / | tail -1 | awk '{print $5}' | sed 's/%//')
echo "$(date) CPU:$CPU_LOAD MEM:$MEM_USED% DISK:$DISK_USED%" >> "$LOG"
if (( CPU_LOAD > 90 || MEM_USED > 85 || DISK_USED > 90 )); then
    echo "System threshold exceeded on $(hostname)" | mail -s "System Alert" admin@example.com
fi

Tip: Integrating system health checks with cron ensures you maintain visibility and can respond promptly to issues. For monitoring network interfaces and ports, see Ports for Everyone.

Log Rotation

Using System Logrotate

Add custom user logs (~/.logs) to logrotate. Configure frequency, compression, and retention policies in /etc/logrotate.d/custom-logs:

~/.logs/*.log {
    daily
    rotate 5
    compress
    missingok
    notifempty
}

Tip: Proper rotation prevents uncontrolled log growth and keeps historical data manageable.

Custom Rotation Script

For multiple directories or specialized requirements, a numeric .1/.2/.3 rotation is simple and predictable. Numeric rotation ensures you retain a fixed number of logs and avoids surprises caused by date-based rotations.

#!/bin/bash
# Rotate multiple log directories
LOGDIRS=("$HOME/.logs" "$HOME/otherlogs")
LOGDAYS=5
rotate_log() {
    local log="$1"
    local base=$(basename "$log" .log)
    # Shift older logs up
    for ((i=LOGDAYS; i>=1; i--)); do
        if [ -f "$log.$i" ]; then
            if [ $i -eq $LOGDAYS ]; then
                rm -f "$log.$i"
            else
                mv "$log.$i" "$log.$((i+1))"
            fi
        fi
    done
    # Rotate current log
    [ -s "$log" ] && mv "$log" "$log.1"
}
for dir in "${LOGDIRS[@]}"; do
    for log in "$dir"/*.log; do
        [ -f "$log" ] && rotate_log "$log"
    done
done

Apply this script to user-space logs, such as hack-ips.log and sbbsecho-cron.log, to maintain predictable retention.

Cron & Automation

Schedule scripts safely for recurring tasks like nightly OS updates, log extraction, and disk cleanup. Best practices:

• Log outputs to dedicated files
• Monitor failures for early intervention
• Avoid running everything as root

Tip: Cron provides a simple, reliable way to automate maintenance without manual oversight. For examples of automation beyond system tasks, see Email Server Management Script.

Conclusion

Automating routine system maintenance lays the groundwork for more complex multi-system orchestration. By maintaining system health through updates and disk monitoring, administrators can prevent unexpected failures and downtime.

Automated log extraction provides insight into service activity, task results, and potential errors. Coupled with thoughtful log rotation, this ensures logs remain manageable, actionable, and safe.

Using cron to schedule recurring tasks like health checks, log parsing, and cleanup reduces manual effort while improving reliability. Proper separation of user and system logs ensures security and ease of maintenance.

By following these practices, you can maintain system health efficiently and build a foundation for scaling automation across multiple systems with confidence.

Appendix: Reference Scripts

This appendix provides full, ready-to-use scripts for automating key system maintenance tasks. Each script includes descriptions, tips, and usage notes.

OS Update Script

Automates minor system updates while logging output. Supports Debian/Ubuntu, RedHat/CentOS, and Arch Linux.

#!/bin/bash
# Simple OS update script
# DIST=1 (Debian/Ubuntu), 2 (RedHat/CentOS), 3 (Arch Linux)
DIST=${1:-1}
LOG="$HOME/.logs/os-update.log"
mkdir -p "$(dirname "$LOG")"
case $DIST in
  1) sudo apt update && sudo apt upgrade -y >> "$LOG" 2>&1 ;;
  2) sudo dnf update -y >> "$LOG" 2>&1 ;;
  3) sudo pacman -Syu --noconfirm >> "$LOG" 2>&1 ;;
esac
echo "Update completed on $(date)" >> "$LOG"

Tips:

• Run this via cron for nightly minor updates.
• Review logs regularly for errors or failed updates.
• Major version upgrades should be handled manually.

Log Extraction Script (Example: Extract IPs)

Parses logs for key data, such as IP addresses or errors, and saves results to dedicated logs.

#!/bin/bash
# Extract IP addresses from hack.log to hack-ips.log
LOG_DIR="$HOME/.logs"
mkdir -p "$LOG_DIR"
grep -oE '\b([0-9]{1,3}\.){3}[0-9]{1,3}\b' hack.log > "$LOG_DIR/hack-ips.log"

Tips:

• Adapt grep patterns for different log formats.
• Schedule via cron to process logs automatically.
• Keeps sensitive data separate from original log files.

System Health Monitoring Script

Monitors CPU load, memory usage, and disk usage. Logs results and optionally sends alerts if thresholds are exceeded.

#!/bin/bash
LOG="$HOME/.logs/system-health.log"
mkdir -p "$(dirname "$LOG")"
CPU_LOAD=$(uptime | awk -F'load average:' '{ print $2 }' | cut -d, -f1)
MEM_USED=$(free | awk '/Mem/ {printf("%.0f"), $3/$2*100}')
DISK_USED=$(df / | tail -1 | awk '{print $5}' | sed 's/%//')
echo "$(date) CPU:$CPU_LOAD MEM:$MEM_USED% DISK:$DISK_USED%" >> "$LOG"
if (( CPU_LOAD > 90 || MEM_USED > 85 || DISK_USED > 90 )); then
    echo "System threshold exceeded on $(hostname)" | mail -s "System Alert" admin@example.com
fi

Tips:

• Adjust CPU, memory, and disk thresholds to suit your environment.
• Use cron to schedule regular checks (hourly or daily).
• Logs in ~/.logs/system-health.log allow trend analysis over time.

Custom Log Rotation Script

Rotates multiple log directories using numeric suffixes (.1, .2, .3) for predictable retention.

#!/bin/bash
# Rotate multiple log directories using numeric suffixes
LOGDIRS=("$HOME/.logs" "$HOME/otherlogs")
LOGDAYS=5
rotate_log() {
    local log="$1"
    local base=$(basename "$log" .log)
    # Shift older logs up
    for ((i=LOGDAYS; i>=1; i--)); do
        if [ -f "$log.$i" ]; then
            if [ $i -eq $LOGDAYS ]; then
                rm -f "$log.$i"
            else
                mv "$log.$i" "$log.$((i+1))"
            fi
        fi
    done
    # Rotate current log
    [ -s "$log" ] && mv "$log" "$log.1"
}
for dir in "${LOGDIRS[@]}"; do
    for log in "$dir"/*.log; do
        [ -f "$log" ] && rotate_log "$log"
    done
done

Tips:

• Numeric rotation is simple and predictable.
• Apply to user-space logs such as hack-ips.log or sbbsecho-cron.log.
• Combine with cron to automate rotation on a daily or weekly schedule.

Usage Note:
These scripts form a foundation for routine system maintenance. Together, they enable automated updates, monitoring, and log management — reducing manual effort and improving reliability.