feat: LogosDB 1.0.0 — smart chunking, UTF-8 hardening & security audit

[jose-compu]

Summary

Production-ready 1.0.0 release. Delivers intelligent text chunking, Unicode hardening, path-traversal protections, a comprehensive security audit of the Python adapters, and 251 passing tests including a live smoke test.

Closes #98, #108, #109, #110, #111

Changes

• Smart Chunking (auto / line / section / legacy) — auto detects strategy from file extension: line-window for code, section-aware for Markdown/RST, paragraph-fallback for everything else
• UTF-8 NFC normalization + rejection of bidi override and zero-width characters in security.ts
• Incremental manifest fingerprints chunkMode — changing the chunking strategy on an existing namespace forces a clean re-index automatically
• Path traversal hardening in MCP (security.ts) and Python adapter (vibe.py): os.walk(followlinks=False) + per-entry confinement check
• splitLongText helper in chunker to correctly sub-split single paragraphs larger than targetChars
• sanitizeEntry fix in manifest: chunkMode is omitted (not set to undefined) when absent, preserving round-trip deepStrictEqual and backward compatibility
• Expanded test suite — 251 unit + smoke tests across chunker, security, manifest, hybrid, quota, store and gitignore-walker modules
• Documentation updates: env-var table, chunking strategies section, per-mode examples
• Bump logosdb-mcp-server to 1.0.0

Type of Change

• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Performance improvement
• Refactoring (no functional changes)

Breaking note: chunk(text, chunkSize) positional signature still works (legacy mode), but the preferred call is now chunk(text, { targetChars, mode, filePath }). Callers using the old two-argument form continue to work unchanged.

Testing

• Unit tests added/updated
• Integration tests pass
• Manual testing performed

npm test
# tests 251 | pass 251 | fail 0
smoke: ok — tools/list returned logosdb_search, logosdb_index_file, logosdb_list

New test files: mcp/src/chunker.test.ts (97 tests), extended mcp/src/security.test.ts (59 tests).
Edge cases covered: targetChars=1, overlapChars=0, CRLF, all Unicode bidi/invisible ranges, null-byte paths, double symlink escapes, all SKIP_DIRS, ATX + Setext headings, large-section sub-splitting.

Checklist

• Code compiles without warnings on GCC and Clang
• Tests pass (npm test — 251/251, smoke clean)
• Benchmarks run if performance-related
• CHANGELOG updated for user-facing changes
• Documentation updated (README, headers, guides)
• PR title follows conventional commit style (feat:, fix:, docs:, etc.)

Additional Notes

Migration for existing incremental namespaces: the default mode is auto. Old manifest entries (no chunkMode field) compare equal to auto, so files are not force-re-indexed on upgrade. To switch an existing namespace to the new smart chunker, run logosdb_index_file once with incremental: false.

LOGOSDB_CHUNK_MODE env var sets the global default; override per call via the chunking parameter of logosdb_index_file.

Made with Cursor