fix(knowledge): unify markdown chunking across ingestion paths

[014-code]

Summary

• Use the structure-aware Markdown splitter consistently for uploads, repository sync, and built-in vault sync.
• Add .markdown to the repository knowledge-file allow-list.
• Keep the existing fixed-window splitter for .txt and .rst files.
• Add regression coverage for extension-based splitter selection and repository scanning.

Why

Markdown headings provide useful document structure for retrieval. Previously, repository and built-in vault sync always used fixed-window chunking, while uploaded Markdown used the structure-aware splitter. This made the same Markdown content behave differently depending on its ingestion path.

This first version centralizes that decision while keeping the existing behavior for plain text and reStructuredText.

Scope

This PR does not change the search API, embedding model, Qdrant schema, point ID rules, or retrieval ranking logic.

Validation

• go test -race ./internal/manager/biz/knowledge -count=1
• go vet ./...
• go build ./...
• git diff --check

The full go test -race ./... run was attempted. It was blocked by the current container environment: the knowledge package encountered an embed-file cannot allocate memory failure, and two existing root/container-sensitive tests failed in aiops/chatruntime and server/edge.

Risk and rollback

This is an ingestion-only change with no API, schema, or deployment changes. Reverting commit 68a7875 restores the previous chunking behavior and repository extension allow-list.

Closes #247

Author confirmation

• I have read and agree to the project contribution guide. Guide: https://github.com/ongridio/ongrid/blob/main/CONTRIBUTING.md

[014-code]

Implementation details for this first version:

1. Upload ingestion

ingestUpload now delegates chunking to splitKnowledgeContent. Markdown, .markdown, DOCX, and PDF inputs use the existing structure-aware Markdown splitter. DOCX/PDF content has already been converted to Markdown by the upload extraction path.

2. Repository sync

Repository files now use the same helper before embedding. The repository allow-list also accepts .markdown. The existing exclusions for YAML, TOML, JSON, and other configuration/data formats remain unchanged.

3. Built-in vault sync

Built-in vault files now use the same extension-based strategy as uploads and repository files, so the three ingestion paths produce consistent Markdown chunks.

4. Compatibility

.txt and .rst continue to use the existing fixed-window splitter. No search API, embedding model, Qdrant schema, point ID rule, or ranking logic was changed.

5. Tests

Added coverage for:

• Markdown heading-aware splitting
• Case-insensitive .MARKDOWN handling
• Fixed-window fallback for .txt and .rst
• Repository discovery of .md, .markdown, .txt, and .rst
• Continued exclusion of .yaml

Validation completed:

• go test -race ./internal/manager/biz/knowledge -count=1
• go vet ./...
• go build ./...
• git diff --check

This is intentionally a small first version for #247. Could you please confirm whether this Markdown structure-aware chunking approach is acceptable?

In particular, should the next iteration add:

• real retrieval-quality evaluation data or a small RAG benchmark;
• support for additional Markdown-like extensions;
• different chunk-size/overlap or embedding-cost limits;
• separate policies for Markdown, plain text, and other document types?

I would appreciate feedback on whether this implementation direction is suitable and what changes should be made before expanding the scope.