Upload local files to a data lakehouse as native tables — one command, no boilerplate.
Reads CSV, TSV, Excel, JSON, JSONL, Parquet, Avro, ORC, and SQLite. Streams data in chunks, infers (or accepts) a schema, and writes to the destination via a swappable connector. Currently ships with a Databricks connector (Unity Catalog Volumes → Delta table, M2M OAuth).
One-liner (macOS / Linux, requires Python 3.10+):
curl -fsSL https://raw.githubusercontent.com/JD-gokul/dbupload/main/install.sh | bashInstalls dbupload to ~/.local/bin in its own virtualenv. No global packages touched.
Or clone and install manually:
git clone https://github.com/JD-gokul/dbupload.git
cd databricks-file-upload
python3 -m venv .venv && source .venv/bin/activate
pip install -e .Requirements: Python 3.10 or later.
Copy .env.example to .env in the directory where you run dbupload (or any parent up to 3 levels up):
cp .env.example .envDATABRICKS_HOST=https://adb-xxxxxxxxxxxx.xx.azuredatabricks.net
DATABRICKS_CLIENT_ID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
DATABRICKS_CLIENT_SECRET=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
DATABRICKS_WAREHOUSE_ID=xxxxxxxxxxxxxxxx| Variable | Where to find it |
|---|---|
DATABRICKS_HOST |
Workspace URL in your browser |
DATABRICKS_CLIENT_ID |
Account Console → User Management → Service Principals → your SP → OAuth secrets |
DATABRICKS_CLIENT_SECRET |
Same page — generate a secret |
DATABRICKS_WAREHOUSE_ID |
Workspace → SQL → Warehouses → your warehouse → Connection Details |
The service principal needs WRITE VOLUME and CREATE TABLE on the target catalog/schema, and CAN USE on the warehouse.
dbupload FILE DESTINATION [OPTIONS]
DESTINATION format depends on the connector. For Databricks: catalog.schema.table.
# CSV → Delta table
dbupload sales.csv prod.finance.sales
# Preview inferred schema without uploading
dbupload sales.csv prod.finance.sales --dry-run
# Skip the confirmation prompt (for scripts)
dbupload sales.csv prod.finance.sales --yes# Force format when extension is ambiguous
dbupload data.txt prod.raw.dump --format csv
# CSV: semicolon delimiter, no header row
dbupload export.csv prod.raw.data --delimiter ";" --no-header
# CSV: explicit encoding
dbupload legacy.csv prod.raw.data --encoding latin-1
# Excel: specific sheet by name or index
dbupload report.xlsx prod.finance.q3 --sheet "Q3 Data"
dbupload report.xlsx prod.finance.q3 --sheet 2
# SQLite: specific table
dbupload app.db prod.raw.users --sqlite-table users# Override a single column's type
dbupload sales.csv prod.finance.sales --cast amount:DECIMAL(10,2)
dbupload sales.csv prod.finance.sales --cast amount:DECIMAL(10,2) --cast id:INT
# Provide a full schema file — columns not listed are dropped
dbupload sales.csv prod.finance.sales --schema-file schema.yaml
dbupload sales.csv prod.finance.sales --schema-file schema.sqlSchema file — YAML (.yaml):
columns:
- name: order_id
type: BIGINT
- name: amount
type: DECIMAL(10,2)Schema file — Spark DDL (.sql):
order_id BIGINT NOT NULL,
amount DECIMAL(10,2)A full CREATE TABLE name (...) wrapper is also accepted.
Priority (highest first): --cast → --schema-file → inferred from data.
When --schema-file is given it is authoritative: columns absent from the schema file are dropped.
# Tune chunk size (default: 100,000 rows per Parquet part)
dbupload big.csv prod.raw.data --chunk-size 50000
# Delete the staging Volume directory after the table is created
dbupload big.csv prod.raw.data --cleanupArguments:
FILE Path to the local file to upload
DESTINATION Connector-specific target (e.g. catalog.schema.table)
Options:
-c, --connector Destination connector (default: databricks)
-f, --format Override auto-detected format
-d, --delimiter Field delimiter for CSV/TSV (default: ,)
--no-header Treat row 1 as data, not column names
-e, --encoding File encoding (default: auto-detect via chardet)
-s, --sheet Excel sheet — name, 0-based index, or "all"
--sqlite-table SQLite table name (default: first table)
--schema-file Path to .yaml or .sql schema file
--cast Override one column's type: col:TYPE (repeatable)
--chunk-size Rows per Parquet part (default: 100,000)
--cleanup Delete staging directory after table creation
--dry-run Show resolved schema and exit without uploading
-y, --yes Skip confirmation prompt
--help Show this message and exit
| Format | Extensions | Schema in file? |
|---|---|---|
| CSV | .csv |
No — inference + chardet encoding detection |
| TSV | .tsv |
No |
| Excel | .xlsx, .xls |
Partial — cell types unreliable |
| JSON | .json |
Partial — no numeric precision |
| JSONL | .jsonl |
Partial |
| Parquet | .parquet |
Yes — full schema in file footer |
| Avro | .avro |
Yes — schema in file header |
| ORC | .orc |
Yes — type info in stripe footer |
| SQLite | .db, .sqlite |
Partial — loose type affinity |
For Parquet, Avro, and ORC you rarely need --schema-file.
For CSV and Excel it's recommended in production pipelines where inference surprises are unacceptable.
File
│
▼
[Reader] Detect format → stream file as pa.RecordBatch chunks
│
▼
[Schema] Infer types from 1,000-row sample → apply --schema-file / --cast overrides
│
▼
[Connector] Receive chunks → write to destination (stage + create table for Databricks)
│
▼
Destination table
The reader and connector are independent modules with defined interfaces — you can add new file formats or new destinations without touching anything else. See the contribution guides below.
uploader/
cli.py Typer CLI — wires everything together
detect.py Extension → Format enum
schema.py pandas dtype → Spark SQL type inference
schema_file.py .yaml / .sql external schema parser
readers/
base.py ChunkedReader ABC (chunks → Iterator[pa.RecordBatch])
csv.py excel.py json.py parquet.py avro.py orc.py sqlite.py
connectors/
base.py Connector ABC + WriteResult
__init__.py Registry: connector_for('databricks')
databricks/ DatabricksConnector — staging + Delta table creation
The interchange type between readers and connectors is pa.RecordBatch (Apache Arrow). Readers don't know about destinations; connectors don't know about file formats.
See docs/adding-a-reader.md for the full guide.
Short version: subclass ChunkedReader, implement chunks() → Iterator[pa.RecordBatch], register the extension in detect.py and the class in readers/__init__.py.
See docs/adding-a-connector.md for the full guide.
Short version: create uploader/connectors/<name>/, subclass Connector, implement from_env() and write(), call register().
MIT