Narsing Shah

📌 Bash Log Capturing & Error Analyzer

8/1/2016

Overview

When troubleshooting complex enterprise server environments (Solaris, Linux, AIX), log analysis often involves sifting through hundreds of compressed .tar and .gz archives across multiple server clusters.

This tool automates log capturing, uncompressing archives in batch, and executing pattern searches for error detection, unusual login detection, and hardware failure counting.

How to Use

Make the scripts executable:

chmod +x vm_log_collector.sh log_analyzer_rca.sh

Run collector on any VM:

./vm_log_collector.sh my_vm_archive.tar.gz

Run analyzer on the archive (or uncompressed log directory):

./log_analyzer_rca.sh my_vm_archive.tar.gz

Two modular Bash scripts:**

vm_log_collector.sh

  • Captures Application (charlie, dexter, eddie), OS, Kernel, I/O, Network diagnostics, and tcpdump packet captures into a timestamped .tar.gz archive.

log_analyzer_rca.sh

  • The improvised analyzer engine that processes archives or directories, separates Application vs Non-Application logs, and generates a structured RCA report.

Key Features

  • Automated Un-Tar & Un-Zip: Recursively unpacks .tar, .tar.gz, and .zip archives into isolated working directories.
  • Error Pattern Matching: Scans for kernel panics, OOM kills, hardware bus errors, and authentication failures.
  • Root Cause Analysis Summary: Generates a consolidated markdown/text report summarizing failure counts grouped by server host.

VM Log Collector Script (vm_log_collector.sh)

#!/usr/bin/env bash
# ==============================================================================
# Script: vm_log_collector.sh
# Purpose: Comprehensive VM Log Collector for System, OS, Kernel, I/O, Network,
#          and Multi-Application Logs (charlie, dexter, eddie, etc.)
# ==============================================================================

set -euo pipefail

# Configuration & Defaults
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
HOSTNAME_STR=$(hostname 2>/dev/null || echo "unknown_host")
WORK_BASE_DIR="${TMPDIR:-.}"
CAPTURE_DIR="${WORK_BASE_DIR}/vm_log_capture_${HOSTNAME_STR}_${TIMESTAMP}"
ARCHIVE_OUT="${1:-vm_logs_${HOSTNAME_STR}_${TIMESTAMP}.tar.gz}"
TCPDUMP_SECONDS="${TCPDUMP_SECONDS:-5}"
TCPDUMP_PACKETS="${TCPDUMP_PACKETS:-200}"

# Default application log base directories to search/collect
APP_NAMES=("charlie" "dexter" "eddie")
APP_LOG_BASE_PATHS=(
    "/var/log/apps"
    "/opt/apps"
    "/var/log"
    "/home"
)

echo "======================================================================="
echo " Starting VM Comprehensive Log Collector"
echo " Host: ${HOSTNAME_STR} | Timestamp: ${TIMESTAMP}"
echo "======================================================================="

# Create temporary directory structure
mkdir -p "${CAPTURE_DIR}/system_info"
mkdir -p "${CAPTURE_DIR}/os_logs"
mkdir -p "${CAPTURE_DIR}/kernel_logs"
mkdir -p "${CAPTURE_DIR}/io_logs"
mkdir -p "${CAPTURE_DIR}/network_logs"
mkdir -p "${CAPTURE_DIR}/app_logs"

# 1. System Metadata & Health Snapshot
echo "[+] Capturing System Snapshot & Metadata..."
{
    echo "=== Hostname & Kernel ==="
    uname -a || true
    uptime || true
    echo -e "\n=== CPU Info ==="
    lscpu 2>/dev/null || head -n 20 /proc/cpuinfo 2>/dev/null || true
    echo -e "\n=== Memory Usage ==="
    free -m 2>/dev/null || top -bn1 | head -n 10 || true
    echo -e "\n=== Process Tree ==="
    ps aux --sort=-%cpu 2>/dev/null | head -n 30 || true
} > "${CAPTURE_DIR}/system_info/sys_snapshot.txt"

# 2. Application Logs (charlie, dexter, eddie)
echo "[+] Capturing Application Logs (charlie, dexter, eddie)..."
for app in "${APP_NAMES[@]}"; do
    APP_TARGET_DIR="${CAPTURE_DIR}/app_logs/${app}"
    mkdir -p "${APP_TARGET_DIR}"
    FOUND=0

    for base in "${APP_LOG_BASE_PATHS[@]}"; do
        if [ -d "${base}/${app}" ]; then
            echo "    -> Found ${app} logs at ${base}/${app}"
            cp -rL "${base}/${app}"/* "${APP_TARGET_DIR}/" 2>/dev/null || true
            FOUND=1
        elif [ -f "${base}/${app}.log" ]; then
            echo "    -> Found ${app} log file at ${base}/${app}.log"
            cp "${base}/${app}.log" "${APP_TARGET_DIR}/" 2>/dev/null || true
            FOUND=1
        fi
    done

    if [ "${FOUND}" -eq 0 ]; then
        echo "    -> Searching system for '${app}' log files..."
        find /var/log /opt /tmp -type f -name "*${app}*.log" 2>/dev/null | while read -r f; do
            cp "$f" "${APP_TARGET_DIR}/" 2>/dev/null || true
        done
    fi
done

# 3. OS & System Logs
echo "[+] Capturing OS Logs (syslog, messages, journalctl, auth)..."
if command -v journalctl &>/dev/null; then
    journalctl --since "24 hours ago" --no-pager > "${CAPTURE_DIR}/os_logs/journalctl_24h.log" 2>/dev/null || true
fi
for log_file in syslog messages auth.log secure daemon.log; do
    if [ -f "/var/log/${log_file}" ]; then
        cp "/var/log/${log_file}" "${CAPTURE_DIR}/os_logs/" 2>/dev/null || true
    fi
done

# 4. Kernel Logs
echo "[+] Capturing Kernel Logs (dmesg, kern.log)..."
if command -v dmesg &>/dev/null; then
    dmesg -T > "${CAPTURE_DIR}/kernel_logs/dmesg_human.log" 2>/dev/null || dmesg > "${CAPTURE_DIR}/kernel_logs/dmesg.log" 2>/dev/null || true
fi
if [ -f "/var/log/kern.log" ]; then
    cp "/var/log/kern.log" "${CAPTURE_DIR}/kernel_logs/" 2>/dev/null || true
fi

# 5. Input / Output (I/O) & Filesystem Logs
echo "[+] Capturing I/O & Storage Diagnostics..."
{
    echo "=== Filesystem Usage (df -h) ==="
    df -h || true
    echo -e "\n=== Inode Usage (df -i) ==="
    df -i || true
    echo -e "\n=== Block Devices (lsblk) ==="
    lsblk 2>/dev/null || true
} > "${CAPTURE_DIR}/io_logs/storage_info.txt"

if command -v iostat &>/dev/null; then
    iostat -xz 1 3 > "${CAPTURE_DIR}/io_logs/iostat.log" 2>/dev/null || true
fi
if command -v vmstat &>/dev/null; then
    vmstat 1 3 > "${CAPTURE_DIR}/io_logs/vmstat.log" 2>/dev/null || true
fi
if [ -f "/var/log/audit/audit.log" ]; then
    tail -n 5000 "/var/log/audit/audit.log" > "${CAPTURE_DIR}/io_logs/audit_tail.log" 2>/dev/null || true
fi

# 6. Network Diagnostics & TCPDump Packet Capture
echo "[+] Capturing Network Diagnostics & TCPDump Packet Capture..."
{
    echo "=== Network Interfaces ==="
    ip addr 2>/dev/null || ifconfig 2>/dev/null || true
    echo -e "\n=== Routing Table ==="
    ip route 2>/dev/null || route -n 2>/dev/null || true
    echo -e "\n=== Active Connections & Listening Ports ==="
    ss -tulpn 2>/dev/null || netstat -tulpn 2>/dev/null || true
    echo -e "\n=== Network Statistics ==="
    netstat -s 2>/dev/null || ip -s link 2>/dev/null || true
} > "${CAPTURE_DIR}/network_logs/net_diagnostics.txt"

if command -v tcpdump &>/dev/null; then
    DEFAULT_IFACE=$(ip route | grep default | awk '{print $5}' | head -n 1 || echo "any")
    echo "    -> Running tcpdump on interface '${DEFAULT_IFACE}'..."
    timeout "${TCPDUMP_SECONDS}" tcpdump -i "${DEFAULT_IFACE}" -c "${TCPDUMP_PACKETS}" -w "${CAPTURE_DIR}/network_logs/capture.pcap" 2>/dev/null || true
fi

# 7. Compress Archive
echo "[+] Packaging captured logs into archive: ${ARCHIVE_OUT}..."
tar -czf "${ARCHIVE_OUT}" -C "$(dirname "${CAPTURE_DIR}")" "$(basename "${CAPTURE_DIR}")"
rm -rf "${CAPTURE_DIR}"

echo "======================================================================="
echo " SUCCESS: Log archive generated at ${ARCHIVE_OUT}"
echo "======================================================================="

Improvised Log Analyzer & RCA Engine (log_analyzer_rca.sh)

#!/usr/bin/env bash
# ==============================================================================
# Script: log_analyzer_rca.sh
# Purpose: Advanced Log Analyzer & Root Cause Analysis (RCA) Engine
#          Supports Application Logs (charlie, dexter, eddie) & Non-App Logs
#          (OS, Kernel, I/O, Network/tcpdump) from archives or directories.
# ==============================================================================

set -euo pipefail

# Inputs & Configuration
TARGET_PATH="${1:-.}"
OUTPUT_REPORT="rca_summary_$(date +%Y%m%d_%H%M%S).txt"
WORK_BASE_DIR="${TMPDIR:-.}"
TMP_WORK_DIR=$(mktemp -d "${WORK_BASE_DIR}/log_analyzer_rca.XXXXXX" 2>/dev/null || mktemp -d ./log_analyzer_rca.XXXXXX)

cleanup() {
    rm -rf "${TMP_WORK_DIR}"
}
trap cleanup EXIT

log_header() {
    local msg="$1"
    echo -e "\n=======================================================================" | tee -a "${OUTPUT_REPORT}"
    echo -e " ${msg}" | tee -a "${OUTPUT_REPORT}"
    echo -e "=======================================================================" | tee -a "${OUTPUT_REPORT}"
}

log_subheader() {
    local msg="$1"
    echo -e "\n--- [ ${msg} ] ---" | tee -a "${OUTPUT_REPORT}"
}

count_matches() {
    local pattern="$1"
    shift
    if [ $# -eq 0 ] || [ -z "${1:-}" ]; then
        echo "0"
        return
    fi
    local count
    count=$(cat "$@" 2>/dev/null | grep -iE "${pattern}" | wc -l | awk '{print $1}')
    echo "${count:-0}"
}

# ------------------------------------------------------------------------------
# STEP 1: Unpack / Ingest Target Input (Archives or Directory)
# ------------------------------------------------------------------------------
log_header "STEP 1: Log Archive Ingestion & Unpacking"

EXTRACT_DIR="${TMP_WORK_DIR}/extracted_logs"
mkdir -p "${EXTRACT_DIR}"

ingest_item() {
    local item="$1"
    if [ -d "${item}" ]; then
        echo "[+] Ingesting Directory: ${item}" | tee -a "${OUTPUT_REPORT}"
        cp -rL "${item}"/* "${EXTRACT_DIR}/" 2>/dev/null || true
    elif [ -f "${item}" ]; then
        echo "[+] Unpacking File: ${item}" | tee -a "${OUTPUT_REPORT}"
        case "${item}" in
            *.tar.gz|*.tgz)
                tar -zxf "${item}" -C "${EXTRACT_DIR}" 2>/dev/null || tar -zxf "${item}" --strip-components=1 -C "${EXTRACT_DIR}" 2>/dev/null || true
                ;;
            *.tar.bz2|*.tbz2) tar -jxf "${item}" -C "${EXTRACT_DIR}" 2>/dev/null || true ;;
            *.tar.xz|*.txz)   tar -Jxf "${item}" -C "${EXTRACT_DIR}" 2>/dev/null || true ;;
            *.tar)            tar -xf "${item}" -C "${EXTRACT_DIR}" 2>/dev/null || true ;;
            *.zip)
                if command -v unzip &>/dev/null; then
                    unzip -q "${item}" -d "${EXTRACT_DIR}" 2>/dev/null || true
                elif command -v python3 &>/dev/null; then
                    python3 -c "import zipfile; zipfile.ZipFile('${item}').extractall('${EXTRACT_DIR}')" 2>/dev/null || true
                fi
                ;;
            *.log|*.txt) cp "${item}" "${EXTRACT_DIR}/" ;;
            *) cp "${item}" "${EXTRACT_DIR}/" 2>/dev/null || true ;;
        esac
    fi
}

if [ -d "${TARGET_PATH}" ]; then
    ARCHIVE_FILES=$(find "${TARGET_PATH}" -maxdepth 2 -type f \( -name "*.tar.gz" -o -name "*.tgz" -o -name "*.zip" -o -name "*.tar" -o -name "*.tar.bz2" \) 2>/dev/null || true)
    if [ -n "${ARCHIVE_FILES}" ]; then
        while read -r arch; do
            [ -n "${arch}" ] && ingest_item "${arch}"
        done <<< "${ARCHIVE_FILES}"
    else
        ingest_item "${TARGET_PATH}"
    fi
else
    ingest_item "${TARGET_PATH}"
fi

find "${EXTRACT_DIR}" -type f -name "*.gz" -exec gunzip -f {} + 2>/dev/null || true

# ------------------------------------------------------------------------------
# STEP 2: Application Log Error Analyzer (charlie, dexter, eddie)
# ------------------------------------------------------------------------------
log_header "STEP 2: APPLICATION LOG ERROR ANALYZER"

TARGET_APPS=("charlie" "dexter" "eddie")

for app in "${TARGET_APPS[@]}"; do
    APP_UPPER=$(echo "${app}" | tr '[:lower:]' '[:upper:]')
    log_subheader "Application: ${APP_UPPER}"

    APP_FILES=$(find "${EXTRACT_DIR}" -type f \( -path "*/${app}/*" -o -name "*${app}*.log" -o -name "*${app}*.txt" \) 2>/dev/null || true)

    if [ -z "${APP_FILES}" ]; then
        echo "  [-] No log files found for application '${app}'." | tee -a "${OUTPUT_REPORT}"
        continue
    fi

    # Severity Breakdown Metrics
    FATAL_COUNT=$(count_matches 'fatal|critical|panic' ${APP_FILES})
    ERROR_COUNT=$(count_matches 'error|exception|failed|failure' ${APP_FILES})
    WARN_COUNT=$(count_matches 'warn|warning' ${APP_FILES})
    HTTP_5XX_COUNT=$(count_matches 'HTTP/[1-2]\.[0-9].* 5[0-9]{2}' ${APP_FILES})

    echo "  [Metrics Summary]" | tee -a "${OUTPUT_REPORT}"
    echo "    - Critical/Fatal Errors : ${FATAL_COUNT}" | tee -a "${OUTPUT_REPORT}"
    echo "    - General Errors/Failures: ${ERROR_COUNT}" | tee -a "${OUTPUT_REPORT}"
    echo "    - Warnings               : ${WARN_COUNT}" | tee -a "${OUTPUT_REPORT}"
    echo "    - HTTP 5xx Server Errors : ${HTTP_5XX_COUNT}" | tee -a "${OUTPUT_REPORT}"

    # Top Recurring Errors (Deduplicated with Counts)
    echo -e "\n  [Top 5 Recurring Error Patterns]" | tee -a "${OUTPUT_REPORT}"
    grep -iE 'fatal|critical|error|exception|panic|failed' ${APP_FILES} 2>/dev/null \
        | sed -E 's/[0-9]{4}-[0-9]{2}-[0-9]{2}[T ][0-9]{2}:[0-9]{2}:[0-9]{2}(\.[0-9]+)?Z?//g' \
        | sed -E 's/([0-9]{1,3}\.){3}[0-9]{1,3}/<IP>/g' \
        | sort | uniq -c | sort -nr | head -n 5 \
        | sed 's/^/    /' | tee -a "${OUTPUT_REPORT}" || echo "    None detected." | tee -a "${OUTPUT_REPORT}"

    # Sample Stack Traces
    echo -e "\n  [Sample Stack Traces / Critical Log Snippets]" | tee -a "${OUTPUT_REPORT}"
    grep -iC 2 -E 'Exception|Traceback|NullPointer|panic:|FatalError|TimeoutException' ${APP_FILES} 2>/dev/null \
        | head -n 15 | sed 's/^/    /' | tee -a "${OUTPUT_REPORT}" || echo "    No explicit stack traces found." | tee -a "${OUTPUT_REPORT}"
done

# ------------------------------------------------------------------------------
# STEP 3: Non-Application Log Error Analyzer (OS, Kernel, I/O, Network)
# ------------------------------------------------------------------------------
log_header "STEP 3: NON-APPLICATION LOG ERROR ANALYZER"

# 3a. OS & System Logs
log_subheader "OS & System Errors (syslog, journalctl, auth)"
OS_FILES=$(find "${EXTRACT_DIR}" -type f \( -name "*syslog*" -o -name "*journalctl*" -o -name "*messages*" -o -name "*auth*" -o -name "*secure*" -o -path "*/os_logs/*" \) 2>/dev/null || true)

if [ -n "${OS_FILES}" ]; then
    echo "  [Systemd & Service Failures]" | tee -a "${OUTPUT_REPORT}"
    grep -iE 'failed|failure|failed to start|exited with code|unit.*failed' ${OS_FILES} 2>/dev/null \
        | sort | uniq -c | sort -nr | head -n 5 | sed 's/^/    /' | tee -a "${OUTPUT_REPORT}" || echo "    No service failures detected." | tee -a "${OUTPUT_REPORT}"

    echo -e "\n  [Authentication & Security Errors]" | tee -a "${OUTPUT_REPORT}"
    grep -iE 'Failed password|invalid user|authentication failure|sudo:.*COMMAND' ${OS_FILES} 2>/dev/null \
        | sort | uniq -c | sort -nr | head -n 5 | sed 's/^/    /' | tee -a "${OUTPUT_REPORT}" || echo "    No auth failures detected." | tee -a "${OUTPUT_REPORT}"
fi

# 3b. Kernel Logs
log_subheader "Kernel & Hardware Diagnostics (dmesg, kern.log)"
KERN_FILES=$(find "${EXTRACT_DIR}" -type f \( -name "*dmesg*" -o -name "*kern.log*" -o -path "*/kernel_logs/*" \) 2>/dev/null || true)

if [ -n "${KERN_FILES}" ]; then
    echo "  [Out Of Memory (OOM) Killer Events]" | tee -a "${OUTPUT_REPORT}"
    OOM_EVENTS=$(grep -iC 1 -E 'Out of memory: Kill process|oom-killer|Killed process' ${KERN_FILES} 2>/dev/null || true)
    if [ -n "${OOM_EVENTS}" ]; then
        echo "${OOM_EVENTS}" | head -n 10 | sed 's/^/    /' | tee -a "${OUTPUT_REPORT}"
    else
        echo "    No OOM-killer events detected." | tee -a "${OUTPUT_REPORT}"
    fi

    echo -e "\n  [Kernel Panics & Hardware Errors]" | tee -a "${OUTPUT_REPORT}"
    grep -iE 'kernel panic|segfault|Hardware Error|MCE|PCIe Bus Error|call trace' ${KERN_FILES} 2>/dev/null \
        | head -n 5 | sed 's/^/    /' | tee -a "${OUTPUT_REPORT}" || echo "    No kernel panics or hardware errors detected." | tee -a "${OUTPUT_REPORT}"
fi

# 3c. I/O & Storage
log_subheader "Input/Output (I/O) & Disk Diagnostics"
DISK_ERRS=$(find "${EXTRACT_DIR}" -type f -exec cat {} + 2>/dev/null | grep -iE 'I/O error|read-only|remount-ro|bad blocks|buffer I/O error' || true)
if [ -n "${DISK_ERRS}" ]; then
    echo "${DISK_ERRS}" | head -n 5 | sed 's/^/    /' | tee -a "${OUTPUT_REPORT}"
else
    echo "    No disk I/O errors or read-only filesystem remounts detected." | tee -a "${OUTPUT_REPORT}"
fi

# 3d. Network & TCPDump
log_subheader "Network & Packet Diagnostics (tcpdump, netstat, ss)"
PCAP_FILE=$(find "${EXTRACT_DIR}" -type f -name "*.pcap" 2>/dev/null | head -n 1 || true)
if [ -n "${PCAP_FILE}" ]; then
    echo "  [TCPDump PCAP Packet Capture Analysis]" | tee -a "${OUTPUT_REPORT}"
    if command -v tcpdump &>/dev/null; then
        TOTAL_PACKETS=$(tcpdump -r "${PCAP_FILE}" 2>/dev/null | wc -l | awk '{print $1}')
        RST_PACKETS=$(tcpdump -r "${PCAP_FILE}" 'tcp[tcpflags] & (tcp-rst) != 0' 2>/dev/null | wc -l | awk '{print $1}')
        echo "    - Total Captured Packets : ${TOTAL_PACKETS}" | tee -a "${OUTPUT_REPORT}"
        echo "    - TCP RST (Resets/Errors): ${RST_PACKETS}" | tee -a "${OUTPUT_REPORT}"
    fi
fi

# ------------------------------------------------------------------------------
# STEP 4: Executive Root Cause Analysis (RCA) Summary
# ------------------------------------------------------------------------------
log_header "STEP 4: EXECUTIVE ROOT CAUSE ANALYSIS (RCA) SUMMARY"

{
    echo "Date of Report       : $(date)"
    echo "Analyzed Target      : ${TARGET_PATH}"
    echo "Total Log Files      : $(find "${EXTRACT_DIR}" -type f | wc -l | awk '{print $1}')"
    echo ""
    echo "CRITICAL FINDINGS & CORRELATION:"
    
    ALL_EXTRACTED_FILES=$(find "${EXTRACT_DIR}" -type f 2>/dev/null || true)
    OOM_FOUND=$(count_matches 'Out of memory|oom-killer' ${ALL_EXTRACTED_FILES})
    IO_FOUND=$(count_matches 'I/O error|remount-ro' ${ALL_EXTRACTED_FILES})
    HTTP5XX_FOUND=$(count_matches 'HTTP/[1-2]\.[0-9].* 5[0-9]{2}' ${ALL_EXTRACTED_FILES})

    if [ "${OOM_FOUND}" -gt 0 ]; then
        echo "  [!] PRIMARY ISSUE: Out of Memory (OOM) Killer invoked by kernel."
        echo "      Impact: Application processes were forcibly terminated due to memory exhaustion."
    elif [ "${IO_FOUND}" -gt 0 ]; then
        echo "  [!] PRIMARY ISSUE: Disk Storage I/O Failure."
        echo "      Impact: Filesystem errors or degraded block devices caused write/read failures."
    elif [ "${HTTP5XX_FOUND}" -gt 0 ]; then
        echo "  [!] PRIMARY ISSUE: Application-Level HTTP 5xx Failures."
        echo "      Impact: Application tier (charlie/dexter/eddie) returning server error responses."
    else
        echo "  [*] STATUS: No catastrophic kernel/disk failure detected. Review application error details above."
    fi

    echo ""
    echo "RECOMMENDED REMEDIATION STEPS:"
    echo "  1. Review top application error stack traces for 'charlie', 'dexter', and 'eddie'."
    echo "  2. If OOM events detected, increase memory limits or investigate application memory leaks."
    echo "  3. Check disk space (df -h) and block device health (smartctl / dmesg) if I/O errors exist."
    echo "  4. Verify network connectivity, firewall rules, and listening sockets (ss -tulpn)."
    echo ""
    echo "=== RCA Analysis Complete. Report saved to ${OUTPUT_REPORT} ==="
} | tee -a "${OUTPUT_REPORT}"

exit 0

Empirical Sample Execution Output

Below is an actual report produced by running log_analyzer_rca.sh against a sample log archive containing charlie, dexter, eddie, systemd failures, and kernel OOM logs:

=======================================================================
 STEP 1: Log Archive Ingestion & Unpacking
=======================================================================
[+] Unpacking File: sample_vm_logs.tar.gz

=======================================================================
 STEP 2: APPLICATION LOG ERROR ANALYZER
=======================================================================

--- [ Application: CHARLIE ] ---
  [Metrics Summary]
    - Critical/Fatal Errors : 1
    - General Errors/Failures: 4
    - Warnings               : 1
    - HTTP 5xx Server Errors : 2

  [Top 5 Recurring Error Patterns]
       2  [ERROR] HTTP/1.1 500 Internal Server Error - URI: /api/v1/orders
       1  [FATAL] Critical failure in payment gateway connector: TimeoutException
       1  [ERROR] java.lang.NullPointerException: Cannot invoke "String.getBytes()" because "payload" is null

  [Sample Stack Traces / Critical Log Snippets]
    2026-08-14T00:10:15Z [ERROR] java.lang.NullPointerException: Cannot invoke "String.getBytes()" because "payload" is null
        at com.charlie.service.OrderProcessor.processOrder(OrderProcessor.java:42)

--- [ Application: DEXTER ] ---
  [Metrics Summary]
    - Critical/Fatal Errors : 2
    - General Errors/Failures: 3
    - Warnings               : 0
    - HTTP 5xx Server Errors : 0

  [Top 5 Recurring Error Patterns]
       2  [ERROR] Connection pool exhausted: Timeout waiting for idle object
       1  [FATAL] java.lang.OutOfMemoryError: Java heap space
       1  [CRITICAL] Panic: Unrecoverable queue corruption in queue_manager.go:94

--- [ Application: EDDIE ] ---
  [Metrics Summary]
    - Critical/Fatal Errors : 0
    - General Errors/Failures: 4
    - Warnings               : 0
    - HTTP 5xx Server Errors : 2

  [Top 5 Recurring Error Patterns]
       2  [ERROR] HTTP/1.1 502 Bad Gateway - Upstream proxy unreachable
       1 KeyError: 'user_id'

=======================================================================
 STEP 3: NON-APPLICATION LOG ERROR ANALYZER
=======================================================================

--- [ OS & System Errors (syslog, journalctl, auth) ] ---
  [Systemd & Service Failures]
       1 Aug 14 00:11:16 host1 systemd[1]: charlie-worker.service: Main process exited, status=1/FAILURE

--- [ Kernel & Hardware Diagnostics (dmesg, kern.log) ] ---
  [Out Of Memory (OOM) Killer Events]
    [12345.67890] Out of memory: Kill process 8912 (java) score 920 or sacrifice child
    [12345.67891] Killed process 8912 (java) total-vm:4194304kB, anon-rss:3670016kB

=======================================================================
 STEP 4: EXECUTIVE ROOT CAUSE ANALYSIS (RCA) SUMMARY
=======================================================================
Analyzed Target      : sample_vm_logs.tar.gz
Total Log Files      : 5

CRITICAL FINDINGS & CORRELATION:
  [!] PRIMARY ISSUE: Out of Memory (OOM) Killer invoked by kernel.
      Impact: Application processes were forcibly terminated due to memory exhaustion.

=== RCA Analysis Complete. Report saved to rca_summary_20260814_003440.txt ===