📢 Repository Restructure: Action Required by Jan 5th

[funilrys]

TL;DR

We are moving from a flat structure to a partitioned folder structure to reduce the strain on GitHub's infrastructure.

• Root files (*.txt, *.adblock) are now FROZEN. They will no longer receive updates.
• New data is now located in subdirectories (e.g., /phishing-domains-ACTIVE/).
• Deadline: All root-level files will be permanently deleted the week of January 5, 2026.
• phish.co.za users: You are not affected by this change.

---

To ensure our repository remains sustainable and to reduce the strain on GitHub's infrastructure, we are transitioning from a flat file structure to a partitioned directory structure.

---

Deprecation & Holiday Grace Period

We want to ensure this transition is smooth and doesn't disrupt operations over the holidays.

• Now: Root-level files (the *.txt and *.adblock files in the main folder) are frozen. They will no longer receive new updates.
• Holiday Period: No files will be deleted during the end-of-year break to avoid breaking automated systems while staff are away.
• January 5–11, 2026: All legacy root-level files will be permanently removed.

How to handle the new structure ?

Each new directory contains a *.manifest.txt file (or *.manifest.adblock - if applicable). This manifest serves as the index of all available segments for that folder.

Important

Users fetching data via phish.co.za are not affected and do not need to take any action. This only impacts those pulling data directly from the GitHub repository.

🗺️ Migration Mapping

Old Root File (Frozen)	New Directory	Manifest File
phishing-IPs-ACTIVE.txt	/phishing-IPs-ACTIVE/	phishing-IPs-ACTIVE.manifest.txt
phishing-IPs-INACTIVE.txt	/phishing-IPs-INACTIVE/	phishing-IPs-INACTIVE.manifest.txt
phishing-IPs-INVALID.txt	/phishing-IPs-INVALID/	phishing-IPs-INVALID.manifest.txt
phishing-domains-ACTIVE.adblock	/phishing-domains-ACTIVE/	phishing-domains-ACTIVE.manifest.adblock
phishing-domains-ACTIVE.txt	/phishing-domains-ACTIVE/	phishing-domains-ACTIVE.manifest.txt
phishing-domains-INACTIVE.txt	/phishing-domains-INACTIVE/	phishing-domains-INACTIVE.manifest.txt
phishing-domains-INVALID.txt	/phishing-domains-INVALID/	phishing-domains-INVALID.manifest.txt
phishing-domains-NEW-last-hour.txt	/phishing-domains-NEW-last-hour/	phishing-domains-NEW-last-hour.manifest.txt
phishing-domains-NEW-today.txt	/phishing-domains-NEW-today/	phishing-domains-NEW-today.manifest.txt
phishing-ips-NEW-last-hour.txt	/phishing-ips-NEW-last-hour/	phishing-ips-NEW-last-hour.manifest.txt
phishing-ips-NEW-today.txt	/phishing-ips-NEW-today/	phishing-ips-NEW-today.manifest.txt
phishing-links-ACTIVE-NOW.txt	/phishing-links-ACTIVE-NOW/	phishing-links-ACTIVE-NOW.manifest.txt
phishing-links-ACTIVE-today.txt	/phishing-links-ACTIVE-today/	phishing-links-ACTIVE-today.manifest.txt
phishing-links-ACTIVE.txt	/phishing-links-ACTIVE/	phishing-links-ACTIVE.manifest.txt
phishing-links-INACTIVE.txt	/phishing-links-INACTIVE/	phishing-links-INACTIVE.manifest.txt
phishing-links-INVALID.txt	/phishing-links-INVALID/	phishing-links-INVALID.manifest.txt
phishing-links-NEW-last-hour.txt	/phishing-links-NEW-last-hour/	phishing-links-NEW-last-hour.manifest.txt
phishing-links-NEW-today.txt	/phishing-links-NEW-today/	phishing-links-NEW-today.manifest.txt

---

Migration Helper

The following cross-platform Python script demonstrates how to programmatically fetch and combine all segments for a specific category based on the manifest file.

#!/usr/bin/env python3
"""
A script to fetch and combine segments based on a manifest file.

Usage: 
    python gh-mig-helper.py [category/folder-name]
"""

import sys

import requests

BASE_URL = (
    "https://github.com/Phishing-Database/Phishing.Database/raw/refs/heads/master"
)


def fetch_data(category: str) -> None:
    """
    A function to fetch and combine data segments based on a manifest file for a given category.

    :param str category:
        The category folder name (e.g., 'phishing-domains-ACTIVE').
    """

    manifest_url = f"{BASE_URL}/{category}/{category}.manifest.txt"
    output_file = f"{category}.txt"

    req = requests.get(manifest_url, timeout=60)
    req.raise_for_status()

    filenames = [line.strip() for line in req.text.splitlines() if line.strip()]

    print(f"Fetching manifest: {manifest_url}")

    try:
        with open(output_file, "wb") as file_stream:
            for filename in filenames:
                file_url = f"{BASE_URL}/{category}/{filename}"
                print(f"⚒️  Fetching segment: {filename}")

                with requests.get(file_url, stream=True, timeout=120) as req:
                    req.raise_for_status()

                    for line in req.iter_content(chunk_size=8192, decode_unicode=False):
                        file_stream.write(line)

        print(f"✅ Combined segments saved into: {output_file}")
    except requests.exceptions.RequestException as exception:
        print(f"❌ Network Error: {exception}")
    except Exception as exception:
        print(f"❌ Error: {exception}")


if __name__ == "__main__":
    if len(sys.argv) != 2:
        print(f"Usage: python {sys.argv[0]} [category-folder-name]")
        sys.exit(1)

    fetch_data(sys.argv[1])