ingestr-lite

A minimal, flexible data ingestion engine written in Python. Inspired by ingestr but simplified for rapid development, better readability, and ease of extension.

Features

• Modular pipeline: Source → Transform → Destination
• Supports common file formats: CSV, JSON, Excel
• Pluggable transformation steps (drop columns, cast types, custom logic)
• Load into PostgreSQL, AzureSQL DB (work in progress)
• Configurable via YAML and .env for secrets
• CLI interface using Typer
• Easy to extend with new sources, transforms, or writers

---

Project Structure

ingester-lite/
│
├── ingest/                     # Core pipeline orchestration
│   ├── __init__.py
│   ├── config_loader.py        # Parses YAML + environment variables
│   ├── orchestrator.py         # Main job runner (source → transform → sink)
│   ├── registry.py             # Maps type names to implementations
│   ├── runner.py               # Callable from CLI or external script
│
├── sources/                    # Source connectors
│   ├── __init__.py
│   ├── csv_reader.py
│   ├── excel_reader.py
│   ├── json_reader.py
│
├── transforms/               # Data transformations
│   ├── __init__.py
│   ├── base.py                 # Abstract base class
│   ├── drop_columns.py         # drop columns
│   ├── change_type.py          # for type casting
│   ├── rename_columns.py       # renaming columns
│
├── destinations/               # Destination writers
│   ├── __init__.py
│   ├── postgres_writer.py
│
├── cli/                        # Command-line interface
│   ├── __init__.py
│   ├── app.py                  # Typer CLI entrypoint
│
├── config/                     # Example configs
│   └── csv_job.yaml
│   └── excel_job.yaml
│   └── json_job.yaml
│
├── tests/                      # Unit tests (pytest)
│   └── ...
|
├── main.py                     # For local execution
└── .gitignore


---

---

🚀 Getting Started

1. Clone the repo

git clone https://github.com/anzarwani/ingestr-lite.git
cd ingestr-lite

2. Install dependecies

pip install -r requirements.txt

3. Set up environment variables

Create a .env file in the root directory and add your database credentials:

DATABASE_URI=postgresql://username:password@localhost:5432/<dbname>

4. Run your ingestion job

python cli/app.py config/csv_job.yaml   # depends on what your source file is

---

Feel free to contribute:

• Add source connectors
• Add destinations sinks
• Add monitoring dashboard (Preferrably Streamlit)
• Logging