ESPJsonDB

A lightweight document database for ESP32 devices. ESPJsonDB borrows the ergonomics of MongoDB/Mongoose while embracing embedded constraints: collections live as JSON on LittleFS, memory use is capped through an optional cache, and every API leans on ArduinoJson types so you can stay inside a single document representation.

CI / Release / License

Features

• Simple, mongoose-like API for embedded projects (create/update/remove/find with predicates or JSON filters).
• Optional in-memory cache with dirty-document tracking and change detection to avoid needless flash I/O.
• Automatic LittleFS synchronisation on a background FreeRTOS task (ESPJsonDBConfig controls interval, stack, priority, and cache usage).
• MessagePack compression + StreamUtils for efficient read/write pipelines.
• Schema registry with required fields, defaults, type validation, and collection-level unique constraints.
• Event + error callbacks so firmware can observe sync cycles or take action when validation fails.
• Snapshot/restore helpers for backups plus diagnostics that report per-collection counts and config details.
• Generic file storage helpers under /<baseDir>/_files with chunked stream read/write for any file type (text, binary, etc.).

Examples

Quick start:

#include <ESPJsonDB.h>

ESPJsonDB db;

void setup() {
    Serial.begin(115200);

    ESPJsonDBConfig cfg;
    cfg.intervalMs = 3000;  // autosync every 3s
    cfg.autosync = true;

    if (!db.init("/test_db", cfg).ok()) {
        Serial.println("DB init failed");
        return;
    }

    db.onEvent([](DBEventType evt){
        Serial.printf("Event: %s\n", dbEventTypeToString(evt));
    });
    db.onError([](const DbStatus &st){
        Serial.printf("Error: %s\n", st.message);
    });
}

Working with documents is intentionally JsonDocument-centric:

JsonDocument doc;
doc["email"] = "user@example.com";
doc["role"] = "admin";
auto createRes = db.create("users", doc.as<JsonObjectConst>());

if (createRes.status.ok()) {
    const std::string& id = createRes.value;
    auto found = db.findById("users", id);
    if (found.status.ok()) {
        Serial.printf("Role: %s\n", found.value["role"].as<const char*>());
    }
    db.updateById("users", id, [](DocView& view){
        view["role"].set("owner");
    });
    db.removeById("users", id);
}

See the sketches under examples/ for end-to-end flows:

• QuickStart – database initialisation and simple CRUD.
• Collections – create/drop collections at runtime.
• CacheDisabled – run without the RAM cache.
• BulkOperations – batch inserts, updates, and queries.
• SchemaValidation – enforce required fields and custom validators.
• UniqueFields – per-collection uniqueness guarantees.
• References – store one-to-many relations and populate them lazily.
• FileStreaming – store and stream txt / json / csv / bin / custom extension payloads.
• LargeFileStreaming – chunked upload + chunked verification for a large binary payload without full-buffer RAM copies.
• AsyncFileUpload – non-blocking, callback-driven chunk upload on a background task.
• AsyncLargeFileUpload – background chunk upload for a large binary payload with progress polling and streaming hash verification.

File storage example:

ESPJsonDBFileOptions fileOpts;
fileOpts.chunkSize = 256;
db.writeTextFile("notes/readme.txt", "hello from esp-jsondb");

db.writeFileFromPath("firmware/chunk.bin", "/fw/chunk.bin", fileOpts);

db.writeFileStream(
    "firmware/chunk_cb.bin",
    [](size_t requested, uint8_t* buffer, size_t& produced, bool& eof) -> DbStatus {
        // fill `buffer` with up to `requested` bytes, set produced/eof
        produced = 0;
        eof = true;
        return {DbStatusCode::Ok, ""};
    },
    fileOpts
);

Gotchas

• Each collection lives in RAM when the cache is enabled; disable the cache or add PSRAM when handling large documents.
• All payloads are JSON; converting to structs is optional but deserialisation still costs memory—size your JsonDocument objects carefully.
• Sync callbacks run on the background task; keep them short to avoid blocking periodic flushes.
• Unique constraints and validators run inside write operations. Long-running validators will increase latency for the calling task.
• writeFileStream() and readFileStream() hold the filesystem lock while processing the stream; use reasonable chunk sizes and avoid blocking stream sources/sinks.
• writeFileStreamAsync() runs producer callbacks on a background task; callbacks must be short and thread-safe.
• /_files is an internal reserved directory used for file storage and cannot be used as a collection name.
• getSnapshot() and restoreFromSnapshot() currently cover document collections only; file storage under /_files is not included.

API Reference

• DbStatus init(const char* baseDir = "/db", const ESPJsonDBConfig& cfg = {}) – mount LittleFS (cfg.initFileSystem) and create the autosync task (optional). Diagnostics are lightweight and served from in-memory counters.
• void onEvent(std::function<void(DBEventType)>) / void onError(std::function<void(const DbStatus&)>) – receive sync, CRUD, and validation events.
• Collection management: collection(name), dropCollection(name), dropAll(), getAllCollectionName().
• Document helpers:
  • Create: create, createMany (JSON array) plus direct Collection::create* variants.
  • Read: findById, findOne, findMany (predicate or JSON filter) returning DocView so you can read/write lazily.
  • Update/delete: updateOne, updateById, updateMany, removeById, removeMany (predicate or JSON filter).
• Schemas: registerSchema(name, Schema), unRegisterSchema(name); Schema exposes fields with type/default/unique flags plus optional custom validate callables.
• References: store { "collection": "authors", "_id": "..." } inside a document and call DocView::populate(fieldName) to expand the reference into an embedded object.
• Sync + diagnostics: syncNow(), getDiag() (JSON summary), getSnapshot() / restoreFromSnapshot() for backups.
  • getDiag() does not touch the filesystem; it reports cached counters overlaid with currently loaded collection sizes.
• File storage:
  • writeFileStream(path, in, bytesToWrite, opts) / readFileStream(path, out, chunkSize) for chunked stream transfer.
  • writeFileStream(path, pullCb, opts) for synchronous callback-driven chunk production.
  • writeFileFromPath(path, sourceFsPath, opts) to copy a source file path into DB-managed file storage.
  • writeFileStreamAsync(path, pullCb, opts, doneCb) for non-blocking producer-driven uploads.
  • cancelFileUpload(uploadId), getFileUploadState(uploadId) for async job control.
  • writeFile(path, data, size) / readFile(path) for direct byte buffers.
  • writeTextFile(path, text) / readTextFile(path) for UTF-8 or plain text payloads.
  • fileExists(path), fileSize(path), removeFile(path) for file lifecycle utilities.
  • File paths are relative to /<baseDir>/_files and path traversal segments are rejected.
  • ESPJsonDBFileOptions: overwrite and chunkSize controls for stream writes.
  • DbFileUploadPullCb: callback receives (requested, buffer, produced, eof) and fills bytes into buffer.

ESPJsonDBConfig knobs:

• intervalMs, stackSize, priority, coreId – background autosync cadence & FreeRTOS tuning.
• autosync, coldSync, cacheEnabled – enable/disable timers and caches.
• fs, initFileSystem, formatOnFail, partitionLabel, maxOpenFiles – file system integration; pass your own fs::FS if you mount LittleFS elsewhere.

Stack sizes are expressed in bytes.

Restrictions

• Designed for ESP32 + LittleFS. Other platforms/FSes are untested.
• Large documents are only practical on boards with PSRAM when the cache is enabled.
• Requires ArduinoJson 6+, StreamUtils, and a FreeRTOS-capable environment (Arduino-ESP32 or ESP-IDF with C++17).

Tests

An integration harness (test/) runs CRUD, bulk, schema, reference, and diagnostic scenarios via the DbTester class. Build it as a PlatformIO test or ESP-IDF component (include test/dbTest.cpp in your project) and run it on hardware to validate changes. Contributions that expand automated coverage are welcome.

License

MIT — see LICENSE.md.

ESPToolKit

• Check out other libraries: https://github.com/orgs/ESPToolKit/repositories
• Hang out on Discord: https://discord.gg/WG8sSqAy
• Support the project: https://ko-fi.com/esptoolkit
• Visit the website: https://www.esptoolkit.hu/