Skip to content

README.md

Latest commit

 

History

History
329 lines (257 loc) · 10.4 KB

README.md

File metadata and controls

329 lines (257 loc) · 10.4 KB
Raw

Repology API

A FastAPI-based service for package lookup with redirect handling, built for the Repology database.

Features

  • Package Lookup: Get detailed package information by name
  • Redirect Handling: Automatically follows single redirects, returns multiple redirect options
  • Repository Ordering: Packages sorted by active repository priority
  • Clean Architecture: Modular design with proper separation of concerns
  • Type Safety: Full type annotations with Pydantic models
  • Database Abstraction: Clean database layer hiding SQL complexity

Architecture

src/repologyapi/
├── main.py              # FastAPI application entry point
├── config.py            # Configuration management
├── models/              # Pydantic data models
│   ├── package.py       # Package and metapackage models
│   ├── repo_metadata.py # Repository metadata models
│   ├── link.py          # Link models
│   └── response.py      # API response models
├── database/            # Database layer
│   ├── connection.py    # Connection pool management
│   ├── service.py       # Database service aggregator
│   └── entities/        # Database entity classes
│       ├── metapackage.py
│       ├── package.py
│       ├── redirect.py
│       ├── repo_metadata.py
│       └── link.py
├── services/            # Business logic layer
│   ├── package_service.py # Main package service
│   └── sorting.py       # Package sorting utilities
└── api/                 # API routes
    └── routes.py        # FastAPI route definitions

Installation

  1. Prerequisites: Python 3.12+ and uv package manager

  2. Install dependencies:

    uv sync

  3. Configure database (optional): Create a .env file to override default settings:

    DATABASE__HOST=localhost
DATABASE__PORT=5432
DATABASE__DATABASE=repology
DATABASE__USER=repology
DATABASE__PASSWORD=repology

Usage

Start the API server

# Using the CLI command
uv run repologyapi

# Or with uvicorn directly
uv run uvicorn repologyapi.main:app --host 0.0.0.0 --port 8000 --reload

API Documentation

Once running, visit:

API Endpoints

Main Package Lookup

GET /api/v1/packages/{package_name}

Returns package details with automatic redirect handling:

  • Active package: Returns list of PackageDataDetailed objects
  • Single redirect: Automatically follows redirect and returns target packages
  • Multiple redirects: Returns RedirectResponse with redirect options
  • Not found: Returns empty list

Example responses:

// Active package
[
  {
    "repo": "alpine_3_15",
    "family": "alpine",
    "visiblename": "xz",
    "effname": "xz",
    "version": "5.8.1",
    "versionclass": 1,
    "flags": 0,
    "origversion": "5.8.1",
    "rawversion": "5.8.1-1",
    "shadow": false,
    // ... other fields
  }
]

// Multiple redirects
{
  "redirects": ["xz-utils", "xz-tools", "liblzma"]
}

Package Summary Endpoint

GET /api/v1/packages/{package_name}/summary

Returns aggregated summary data from all PackageDataDetailed objects for a package:

{
  "name": "xz",
  "alternative_names": ["xz-utils", "liblzma"],
  "upstream_homepage": "https://tukaani.org/xz/",
  "upstream_source": "https://github.com/tukaani-project/xz",
  "comments": ["XZ Utils compression library", "LZMA compression tools"],
  "source_links": ["https://github.com/tukaani-project/xz.git"],
  "repos": ["debian_12", "ubuntu_22_04", "alpine_3_15"]
}

Summary Fields:

  • name: Package name (primary visible name)
  • alternative_names: All unique visible names from different repositories
  • upstream_homepage: Homepage URL (link type 0 - UPSTREAM_HOMEPAGE)
  • upstream_source: Source repository URL (link type 2 - UPSTREAM_REPOSITORY)
  • comments: All package comments from different repositories
  • source_links: All source links (link type 7)
  • repos: All repository names containing this package

URL-Based Package Lookup

⚠️ Important Limitation: URL-based endpoints may return incomplete results because not all repositories store upstream URLs in their package metadata. Some repositories (like Debian) may use different URL formats than the original upstream URL.

Recommended Approach for complete results:

  1. Get package names: GET /packages/url/summary/{url}
  2. Get full packages: GET /packages/{package_name}

GET /packages/url/{url:path}

Find packages that reference a specific URL:

  • URL Normalization: Automatically removes query parameters from the URL
  • Exact Matching: Finds packages containing the exact normalized URL
  • Repository Sorting: Results sorted by active repository priority
  • Empty Results: Returns empty list if URL not found
  • ⚠️ May be incomplete: Some repositories may not store the URL

Example:

# URL-encode the URL for the path parameter
curl "http://localhost:8000/packages/url/https%3A%2F%2Fgithub.com%2Fuser%2Frepo"

Recommended two-step approach for complete results:

# Step 1: Get package names from URL
curl "http://localhost:8000/packages/url/summary/https%3A%2F%2Fgithub.com%2Fuser%2Frepo"
# Returns: [{"name": "example-package", ...}]

# Step 2: Get complete package information
curl "http://localhost:8000/packages/example-package"
# Returns: Complete package data from ALL repositories

GET /packages/repo/{repo_name}/url/{url:path}

Find packages from a specific repository that reference a URL (optimized for performance):

  • Repository Filtering: Filters by repository first, then by URL for better performance
  • Repository Validation: Validates repository exists and is active before searching
  • URL Normalization: Automatically removes query parameters from the URL
  • Exact Matching: Finds packages containing the exact normalized URL within the repository
  • No Sorting Needed: Results from single repository don't need cross-repository sorting
  • Performance Benefits: Much faster than global URL search when repository is known

Example:

# Search for URL within specific repository
curl "http://localhost:8000/packages/repo/debian_12/url/https%3A%2F%2Fgithub.com%2Fuser%2Frepo"

Performance comparison:

  • Global search: Searches all packages → Filters by URL (slower for large datasets)
  • Repository-filtered: Filters by repository → Searches within repository (much faster)

GET /packages/url/summary/{url:path}

Get package summary information for packages that reference a URL:

  • Consistent Array Response: Always returns an array of PackageSummary objects for consistency
  • Single Package: Returns array with one element if all packages belong to one effname
  • Multiple Packages: Returns array with multiple elements if packages belong to different effnames
  • 404 Response: Returns 404 if no packages reference the URL

Example response for single package:

[
  {
    "name": "example-package",
    "alternative_names": ["example", "pkg-example"],
    "upstream_homepage": "https://example.com",
    "upstream_source": "https://github.com/user/repo",
    "comments": ["Example package description"],
    "source_links": ["https://github.com/user/repo.git"],
    "repos": ["debian_12", "ubuntu_22_04"]
  }
]

Example response for multiple packages:

[
  {
    "name": "package-one",
    "alternative_names": [],
    "upstream_homepage": "https://example.com",
    // ... other fields
  },
  {
    "name": "package-two", 
    "alternative_names": [],
    "upstream_homepage": "https://example.com",
    // ... other fields
  }
]

Additional Endpoints

  • GET /api/v1/packages/{package_name}/repo/{repo_name} - Get packages from specific repository
  • GET /api/v1/packages/{package_name}/with-urls - Get packages with resolved URLs
  • GET /api/v1/packages/{package_name}/exists - Check if package exists
  • GET /api/v1/packages/{package_name}/redirects - Get redirect information
  • GET /api/v1/health - Health check endpoint
  • GET /api/v1/info - API information

Package Lookup Logic

The service implements the complete Repology package lookup logic:

  1. Look up metapackage by effective name
  2. Check if active (num_repos > 0)
  3. If active: Get packages, sort by repository order, return results
  4. If inactive/missing: Check for redirects
    • 0 redirects: Return empty list
    • 1 redirect: Recursively follow redirect
    • Multiple redirects: Return redirect options

Data Models

PackageDataDetailed

Complete package information including:

  • Repository and family information
  • Version details (original, raw, parsed)
  • Maintainer and licensing information
  • Links and URLs
  • CPE (Common Platform Enumeration) data

Repository Ordering

Packages are sorted according to active repository priority, ensuring consistent and meaningful ordering across all responses.

Testing

Run the test suite:

# Test API structure and models (no database required)
uv run python test_api_simple.py

# Test database connectivity (requires database access)
uv run python test_db_connection.py

Configuration

The service uses Pydantic Settings for configuration management:

  • Database: Connection details, pool settings
  • API: CORS, pagination limits
  • Logging: Level and format configuration

Environment variables use double underscore notation:

  • DATABASE__HOST - Database host
  • DATABASE__USER - Database user
  • API__TITLE - API title
  • LOGGING__LEVEL - Log level

Development

The codebase follows Python best practices:

  • Type annotations throughout
  • Pydantic models for data validation
  • Async/await for database operations
  • Dependency injection for services
  • Clean separation of concerns
  • Comprehensive error handling
  • Structured logging

Database Schema

The service works with the standard Repology database schema:

  • metapackages - Package metadata
  • packages - Individual package records
  • project_redirects - Automatic redirects
  • project_redirects_manual - Manual redirects
  • repositories - Repository metadata
  • links - Package links and URLs

License

This project implements the Repology API specification and follows the same architectural patterns as the original Repology codebase.