Support Custom PyPI-Compatible Repositories for Package Metadata

[voidpetal]

Summary

This PR fixes python-inspector's ability to fetch package metadata from custom PyPI-compatible repositories (Artifactory, Nexus, DevPI, etc.). Previously, the tool would successfully resolve dependencies but fail to populate the packages array with metadata, resulting in missing licenses, checksums, and download URLs.

Problem Statement

When using python-inspector with custom repository indexes (via --index-url):

• ✅ Dependency resolution worked: All packages were correctly resolved
• ✅ Dependency graph was built: The resolved_dependencies_graph was correctly populated
• ❌ Package metadata was missing: The packages array remained empty
• ❌ ORT integration failed: ORT couldn't retrieve licenses, checksums, or download URLs

Root Cause

File: src/python_inspector/package_data.py

The get_pypi_data_from_purl() function hardcoded https://pypi.org/pypi instead of deriving the base URL from the custom index:

base_path = "https://pypi.org/pypi"  # Hardcoded
api_url = f"{base_path}/{name}/{version}/json"

When internal packages don't exist on PyPI.org, the fetch returns 404 and the package is silently excluded from output.

Additional Challenge: URL Path Variations

Custom repositories may use different URL structures between their PEP 503 Simple API and JSON API endpoints:

• Simple API: https://repo.example.com/simple/../packages/hash/file.whl
• JSON API: https://repo.example.com/pypi/hash/file.whl

Matching by full URL fails when paths differ between endpoints.

Solution

This PR implements three key improvements:

1. Dynamic Base URL Derivation

Modified get_pypi_data_from_purl() to derive the JSON API base URL from the provided repositories:

• Convert PEP 503 Simple API URLs (/simple) to JSON API URLs (/pypi)
• Fallback to PyPI.org for backward compatibility

if repos:
    repos_list = list(repos) if not isinstance(repos, list) else repos
    base_path = repos_list[0].index_url.replace("/simple", "/pypi")
else:
    base_path = "https://pypi.org/pypi"

2. Universal Filename-Based Matching

Instead of comparing full URLs (which vary by repository), extract and match by:

• Package filename (standardized by PEP 427/491, unique per version)
• SHA256 hash when available (immutable identifier)

This approach is:

• ✅ Compatible with PyPI.org (existing behavior preserved)
• ✅ Works with Artifactory, Nexus, DevPI
• ✅ Handles URL path variations between API endpoints
• ✅ Future-proof against CDN changes or repository migrations

3. Relative URL Resolution

Added proper URL resolution for repositories that return relative URLs in their JSON API:

from urllib.parse import urljoin
absolute_url = urljoin(api_url, url)

Changes Made

Modified Files

• src/python_inspector/package_data.py
  • Added get_file_match_key() utility function
  • Modified get_pypi_data_from_purl() to support custom repositories
  • Cleaned up imports (removed unused urlunparse)

New Files

• tests/test_package_data.py
  • 14 comprehensive unit tests for the new functionality
  • Covers PyPI.org URLs, Artifactory URLs, and edge cases

Testing

Unit Tests

pytest tests/test_package_data.py -v
# ===== 14 passed, 11 warnings in 0.60s =====

All tests cover:

• Filename extraction from various URL formats
• SHA256 hash extraction from URL fragments
• PyPI.org and Artifactory URL patterns
• Edge cases (empty fragments, relative URLs, etc.)

Integration Testing

Tested with custom Artifactory repository:

Before fix:

{
  "packages": [],
  "resolved_dependencies_graph": { "pkg:pypi/[email protected]": [...] }
}

After fix:

{
  "packages": [
    {
      "name": "internal-package",
      "version": "1.0.0",
      "license_expression": "Apache-2.0",
      "download_url": "https://repo.example.com/packages/...",
      "sha256": "abc123...",
      "md5": "def456..."
    }
  ],
  "resolved_dependencies_graph": { "pkg:pypi/[email protected]": [...] }
}

Tested scenarios:

• PyPI.org (public packages) - existing behavior preserved
• internal Artifactory virtual repositories - metadata now retrieved

Benefits

✅ Universal compatibility: Works with PyPI.org, Artifactory, Nexus, DevPI, and any PEP 503-compliant repository
✅ Backward compatible: Existing PyPI.org usage unchanged
✅ Complete metadata: Populates licenses, checksums, and download URLs for all packages
✅ ORT integration: Enables SBOM generation with custom repositories
✅ Future-proof: Independent of repository-specific URL structures

Compatibility

No breaking changes. Existing users with PyPI.org workflows are unaffected.