📌 Duplicate File Finder Utility

This script uses a 2-Pass Hashing Algorithm.

---

📌 1. Description & Overview

Scanning large storage volumes or deep directory trees for duplicate files by computing checksums on every single file is slow and disk I/O intensive. find_duplicates.sh solves this performance bottleneck using a 2-Pass Optimization Strategy:

How the 2-Pass Logic Works:

1. Pass 1 (Size Filtering):
   • Uses find to catalog all regular files and their exact byte sizes.
   • If a file has a unique byte size on disk, it is mathematically impossible for it to be a duplicate of any other file. Unique-sized files are filtered out immediately.
2. Pass 2 (Checksum Hashing):
   • SHA256 checksums are calculated only for candidate files sharing identical byte sizes.
   • This reduces total disk read operations by up to 90%+ on typical filesystems.
3. Report Generation:
   • Groups matching checksums together, displays file paths, and calculates total reclaimable disk space.

---

💻 2. Full Bash Script (find_duplicates.sh)

#!/usr/bin/env bash
# ==============================================================================
# Script: find_duplicates.sh
# Purpose: Fast, Simple & Safe Duplicate File Finder for Linux/macOS
# Strategy: 2-Pass Optimization
#   Pass 1: Group files by exact byte size (skips hashing unique-sized files)
#   Pass 2: Compute SHA256 hash ONLY for files with identical byte sizes
# ==============================================================================

set -euo pipefail

# Default configuration
SEARCH_DIR="${1:-.}"
OUTPUT_REPORT="duplicate_report_$(date +%Y%m%d_%H%M%S).txt"
MIN_SIZE_BYTES=1 # Ignore empty (0 byte) files by default

# Ensure temporary files are cleaned up on exit
TMP_WORK_DIR=$(mktemp -d ./dup_finder_tmp.XXXXXX 2>/dev/null || mktemp -d /tmp/dup_finder_tmp.XXXXXX)
cleanup() {
    rm -rf "${TMP_WORK_DIR}"
}
trap cleanup EXIT

# ------------------------------------------------------------------------------
# HASH BINARY DETECTION (Cross-platform support for Linux & macOS)
# ------------------------------------------------------------------------------
# Complex Logic Explanation:
# Linux uses `sha256sum`, while macOS uses `shasum -a 256`. We auto-detect the
# available tool to ensure seamless execution on any Unix-like host.
get_sha256() {
    local file_path="$1"
    if command -v sha256sum &>/dev/null; then
        sha256sum "$file_path" | awk '{print $1}'
    elif command -v shasum &>/dev/null; then
        shasum -a 256 "$file_path" | awk '{print $1}'
    else
        # Fallback to md5sum if sha256 tools are missing
        md5sum "$file_path" | awk '{print $1}'
    fi
}

echo "======================================================================="
echo "             DUPLICATE FILE FINDER & RCA REPORT GENERATOR              "
echo "======================================================================="
echo "[+] Scanning Directory : ${SEARCH_DIR}"
echo "[+] Report Output File : ${OUTPUT_REPORT}"
echo "======================================================================="

# ------------------------------------------------------------------------------
# PASS 1: SIZE-BASED FILTERING
# ------------------------------------------------------------------------------
# Complex Logic Explanation:
# Hashing every single file on disk is slow and I/O intensive.
# If a file has a unique byte size, it CANNOT be a duplicate of any other file.
# Therefore, we first list all files with their byte sizes, group them by size,
# and filter out any size that occurs only once.
echo "[+] Pass 1: Cataloging files and grouping by exact byte size..."

SIZE_LIST="${TMP_WORK_DIR}/size_catalog.txt"
CANDIDATE_SIZES="${TMP_WORK_DIR}/candidate_sizes.txt"
CANDIDATE_FILES="${TMP_WORK_DIR}/candidate_files.txt"

# Find regular files, get size and path: "SIZE PATH"
find "${SEARCH_DIR}" -type f -size +${MIN_SIZE_BYTES}c 2>/dev/null \
    | while read -r filepath; do
        if size=$(wc -c < "$filepath" 2>/dev/null); then
            echo "${size} ${filepath}"
        fi
    done > "${SIZE_LIST}"

# Extract sizes that appear MORE than once (duplicate size candidates)
awk '{print $1}' "${SIZE_LIST}" | sort | uniq -d > "${CANDIDATE_SIZES}"

# Filter the file list to keep ONLY files whose size matches candidate sizes
grep -Ff "${CANDIDATE_SIZES}" "${SIZE_LIST}" > "${CANDIDATE_FILES}" || true

CANDIDATE_COUNT=$(wc -l < "${CANDIDATE_FILES}" | tr -d ' ')

if [ "${CANDIDATE_COUNT}" -eq 0 ]; then
    echo "[*] No duplicate file candidates found (all files have unique sizes)."
    exit 0
fi

echo "    -> Found ${CANDIDATE_COUNT} candidate files sharing identical sizes."

# ------------------------------------------------------------------------------
# PASS 2: CHECKSUM HASH CALCULATION
# ------------------------------------------------------------------------------
# Complex Logic Explanation:
# Now we compute SHA256 hashes ONLY for the candidate files identified in Pass 1.
# This cuts down total disk I/O by up to 90%+ on typical filesystems.
echo "[+] Pass 2: Computing SHA256 checksums for candidate files..."

HASH_LIST="${TMP_WORK_DIR}/hash_catalog.txt"

while read -r size filepath; do
    if [ -f "$filepath" ]; then
        hash_val=$(get_sha256 "$filepath")
        echo "${hash_val} ${size} ${filepath}" >> "${HASH_LIST}"
    fi
done < "${CANDIDATE_FILES}"

# Group by HASH: Find hashes that appear more than once
DUPLICATE_HASHES="${TMP_WORK_DIR}/duplicate_hashes.txt"
awk '{print $1}' "${HASH_LIST}" | sort | uniq -d > "${DUPLICATE_HASHES}"

TOTAL_DUP_GROUPS=$(wc -l < "${DUPLICATE_HASHES}" | tr -d ' ')

if [ "${TOTAL_DUP_GROUPS}" -eq 0 ]; then
    echo "[*] PASS COMPLETE: No duplicate content hashes found."
    exit 0
fi

# ------------------------------------------------------------------------------
# REPORT GENERATION
# ------------------------------------------------------------------------------
# Complex Logic Explanation:
# We format the report into structured groups. For each group of duplicate files,
# we calculate wasted disk space: Wasted Space = (Count - 1) * File_Size.
echo "[+] Generating Duplicate File Analysis Report..."

TOTAL_WASTED_BYTES=0
TOTAL_DUPLICATE_FILES=0

{
    echo "======================================================================="
    echo "                 DUPLICATE FILE ANALYSIS REPORT                        "
    echo " Date of Scan  : $(date)"
    echo " Target Folder : ${SEARCH_DIR}"
    echo "======================================================================="
    echo ""
} > "${OUTPUT_REPORT}"

GROUP_NUM=1
while read -r dup_hash; do
    grep "^${dup_hash} " "${HASH_LIST}" > "${TMP_WORK_DIR}/current_group.txt"

    file_count=$(wc -l < "${TMP_WORK_DIR}/current_group.txt" | tr -d ' ')
    sample_size=$(head -n 1 "${TMP_WORK_DIR}/current_group.txt" | awk '{print $2}')

    wasted_in_group=$(( (file_count - 1) * sample_size ))
    TOTAL_WASTED_BYTES=$(( TOTAL_WASTED_BYTES + wasted_in_group ))
    TOTAL_DUPLICATE_FILES=$(( TOTAL_DUPLICATE_FILES + file_count - 1 ))

    {
        echo "-----------------------------------------------------------------------"
        echo "Group #${GROUP_NUM} [SHA256: ${dup_hash}]"
        echo "File Size    : ${sample_size} bytes"
        echo "Copies Found : ${file_count} (Wasted Space: ${wasted_in_group} bytes)"
        echo "Files:"
        awk '{print "  - " $3}' "${TMP_WORK_DIR}/current_group.txt"
        echo ""
    } >> "${OUTPUT_REPORT}"

    GROUP_NUM=$((GROUP_NUM + 1))
done < "${DUPLICATE_HASHES}"

# Summary Header Append
{
    echo "======================================================================="
    echo "                         SUMMARY STATS                                 "
    echo "======================================================================="
    echo " Total Duplicate Groups Found : ${TOTAL_DUP_GROUPS}"
    echo " Total Redundant File Copies  : ${TOTAL_DUPLICATE_FILES}"
    echo " Total Reclaimable Disk Space : ${TOTAL_WASTED_BYTES} bytes ($(( TOTAL_WASTED_BYTES / 1024 / 1024 )) MB)"
    echo "======================================================================="
} >> "${OUTPUT_REPORT}"

cat "${OUTPUT_REPORT}"

echo ""
echo "[+] SUCCESS: Duplicate analysis report saved to: ${OUTPUT_REPORT}"

---

🚀 3. How to Run

1. Save the Script: Save the script code above into a file named find_duplicates.sh.

2. Make Executable:

   chmod +x find_duplicates.sh

3. Run on Target Directory:

   # Scan current directory
   ./find_duplicates.sh
   
   # Scan specific directory (e.g. /var/log or /home/user)
   ./find_duplicates.sh /path/to/target/folder

---

⚠️ 4. Precautions & Safety Best Practices

[!IMPORTANT] Read-Only / Safe Analysis Mode: By default, find_duplicates.sh operates in report-only mode. It never deletes, moves, or alters any files on your system.

[!WARNING] Before Deleting Any Duplicates:

1. Verify Symlinks / Hardlinks: Applications may rely on duplicate copies located in specific paths (e.g., config files or RPM packages across mirrors).
2. Backup First: Take a snapshot or tar backup before running any cleanup commands.
3. Avoid System Directories: Do not run automated deletion scripts on core OS directories (/bin, /sbin, /lib, /usr).

---

📊 5. Sample Output Report

The following is an actual execution report generated by scanning a directory containing 10 duplicate text/data files, 6 duplicate image files, and 3 duplicate RPM package files:

=======================================================================
             DUPLICATE FILE FINDER & RCA REPORT GENERATOR              
=======================================================================
[+] Scanning Directory : /path/to/target_workspace
[+] Report Output File : duplicate_report_20260814_005650.txt
=======================================================================
[+] Pass 1: Cataloging files and grouping by exact byte size...
    -> Found 19 candidate files sharing identical sizes.
[+] Pass 2: Computing SHA256 checksums for candidate files...
[+] Generating Duplicate File Analysis Report...
=======================================================================
                 DUPLICATE FILE ANALYSIS REPORT                        
 Date of Scan  : Fri Aug 14 00:56:51 IST 2026
 Target Folder : /path/to/target_workspace
=======================================================================

-----------------------------------------------------------------------
Group #1 [SHA256: 279311292ca13e06aafc3f685e9bcaabcb572e5746c51dfd9ca8594c0197cb67]
File Size    : 3440 bytes
Copies Found : 5 (Wasted Space: 13760 bytes)
Files:
  - /path/to/target_workspace/session_backup_1.log
  - /path/to/target_workspace/session_backup_2.log
  - /path/to/target_workspace/session_backup_3.log
  - /path/to/target_workspace/session_backup_4.log
  - /path/to/target_workspace/session_backup_5.log

-----------------------------------------------------------------------
Group #2 [SHA256: e262a6fc9108027e30140490b07a72245108753ee6605314cb40838e9708e60b]
File Size    : 3900 bytes
Copies Found : 5 (Wasted Space: 15600 bytes)
Files:
  - /path/to/target_workspace/report_data_v1.txt
  - /path/to/target_workspace/report_data_v2.txt
  - /path/to/target_workspace/report_data_v3.txt
  - /path/to/target_workspace/report_data_v4.txt
  - /path/to/target_workspace/report_data_v5.txt

-----------------------------------------------------------------------
Group #3 [SHA256: dfedd58487f78c0967dd325365a1f82e99651997fc282d15a0c9177171f37326]
File Size    : 233 bytes
Copies Found : 3 (Wasted Space: 466 bytes)
Files:
  - /path/to/target_workspace/logo_banner.png
  - /path/to/target_workspace/logo_header.png
  - /path/to/target_workspace/logo_copy.png

-----------------------------------------------------------------------
Group #4 [SHA256: 6004259e30ca01c4f91e5341c6cd676993c2c02a44fc2cce3919f8c2af00c373]
File Size    : 341 bytes
Copies Found : 3 (Wasted Space: 682 bytes)
Files:
  - /path/to/target_workspace/hero_bg.jpg
  - /path/to/target_workspace/hero_bg_copy.jpg
  - /path/to/target_workspace/hero_background_backup.jpg

-----------------------------------------------------------------------
Group #5 [SHA256: abffd9a0897c440b79259227261ae882882b7f25a2ab5503a63949b915e48a00]
File Size    : 1063 bytes
Copies Found : 3 (Wasted Space: 2126 bytes)
Files:
  - /path/to/target_workspace/nginx-1.24.0-1.el9.x86_64.rpm
  - /path/to/target_workspace/nginx-1.24.0-1.el9.x86_64_backup.rpm
  - /path/to/target_workspace/nginx-1.24.0-1.el9.x86_64_mirror.rpm

=======================================================================
                         SUMMARY STATS                                 
=======================================================================
 Total Duplicate Groups Found : 5
 Total Redundant File Copies  : 14
 Total Reclaimable Disk Space : 32634 bytes (0.03 MB)
=======================================================================

[+] SUCCESS: Duplicate analysis report saved to: duplicate_report_20260814_005650.txt