Feat/new ingestions - #22
Merged
Merged
Conversation
kl pdf <file> and the PWA image/pdf capture tab now accept PDFs: stored in the vault media dir, text layer extracted at capture time (ledongthuc/pdf, pure Go — per-page skip so partially-scanned PDFs still capture), then the extracted text rides the standard enrichment pipeline as a pdf-type note (tags, summary, key ideas, entities, chunks, connections chaining). - internal/ingest: ExtractPDFText with 200k-char cap + IngestPDF; titles prefer the uploaded filename, fall back to first content line (blank-line tolerant) since media storage renames uploads to timestamps - capture API: multipart routing now peeks the uploaded filename from the parsed form (URL-query check never matched, PDFs fell through to the image path — caught immediately in live verification) - kl: new pdf command; PWA: img/pdf tab, pdf search filter, queue icon Live-verified end-to-end: fixture PDF captured, extracted 5k chars, enriched (treehouses/grandma tags, summary, entities), connections + memory chained, indexed and searchable.
Record in the PWA (MediaRecorder, mic-permission aware, re-record +
preview before submit) -> POST /v1/capture/audio -> server stores the
audio in vault media and transcribes synchronously via a pluggable STT
service -> transcript rides the standard pipeline as a voice-type note
(tags, summary, entities, chunks, connections chaining). Audio linked
via source_file; transcript titles the note.
- config: stt {enabled, endpoint, api, model, timeout_s} — works with
any OpenAI-compatible /v1/audio/transcriptions server
(faster-whisper-server, speaches) or a whisper.cpp server
(/inference, text response). Disabled by default: no endpoint, the
voice tab returns 503 with a setup hint.
- internal/stt: client for both formats; transcription is synchronous
and failure aborts the capture (502) — no transcript, no note.
- IngestVoice shares the captured-file pipeline with IngestPDF via a
common ingestWithFile helper.
Live-verified end-to-end with a fake STT service: wav upload ->
transcribed -> voice note (plant-care tags, transcript as raw,
source_file linked) -> connections + memory chained. 3 handler tests +
3 STT client tests; 94 vitest green; assets rebuilt.
README capture features + kl pdf command + PWA tabs; SPEC gains PDF and Voice capture sections (extraction rules, STT config, fail-loud semantics) and the v1.2 roadmap line marked shipped; openapi documents /v1/capture/audio with error codes; config.example gains the stt block; REPO_STRUCTURE lists stt/, media.go, capture_pdf.go, VoiceCapture.tsx; TECH_STACK adds ledongthuc/pdf; UI_SPEC covers the img/pdf + voice capture tabs.
…uage
PDF selection used to render a broken <img> (FileReader data-URL of a
PDF). The img/pdf tab now detects the file type: PDFs get a document
card — gold FileText icon in a bordered tile, filename, size, and a
'pdf · text will be extracted' mono subtitle; drop-zone sub-label
lists pdf.
VoiceCapture redesigned around the img-drop tokens: dashed-gold idle
zone with a mic medallion ('tap to record'), a recording panel with
pulsing red dot, live mono timer and animated equalizer bars
('tap to stop'), and a review card with labelled duration, styled
player, and re-record affordance.
94 vitest green, tsc clean, assets rebuilt and served.
The review step used the raw native <audio controls> and the duration label always read 00:00 — recorder.onstop captured the seconds value from when recording started (stale closure), and re-records leaked object URLs. - VoicePlayer: custom playback bar — gold play/pause medallion, seekable gradient progress track, current/total mono clock; styled to the product tokens - duration now read from a seconds ref at stop time (live value) - old object URLs revoked on re-record
The recording panel's bars container had flex:1 (stretching across the panel with a large empty middle) and percentage-height bars rendering as thin stretched lines. Bars are now fixed-size (4px wide, pixel heights, centered), the timer gets a stable min-width, and the idle/recording/review panels use height:fit-content so the h-full composer area can't stretch them.
Since the note_path-wipe fix, connections/memory jobs share the ingest
note's path. Every note_path-based join (keyword search, entity lookups,
follow-up candidates, completion checks, note content fetch) then
matched the enricher row too — the PDF note appeared twice in search
with type 'connections', and person matches duplicated.
All such joins now append a shared ingestTypesFilter
(type NOT IN ('connections','memory')); regression-tested at the
SearchKeyword level (exactly one result, ingest type).
Also:
- purged the two pre-title-fix test PDF notes (the raw
'1788541785035298000' filenames you saw) and their orphaned media
- recordings/ moved to testdata/recordings per repo layout
/v1/health now advertises an additive stt capability object
({enabled: bool}) alongside dependencies. The PWA capture view reads
it (health is already polled) and hides the voice tab when disabled —
falling back to text if the mode was active when the capability
disappears. The voice tab only appears once stt.enabled + endpoint
are configured.
config.example documents endpoint options (faster-whisper-server,
speaches, whisper.cpp) with a run-it-locally example. Handler test
asserts the health field; CaptureView tests mock the new hook.
Live-verified the full voice pipeline against a real speaches
container: spoken fixture ('remember to call grandma tomorrow about
the family dinner') -> faster-whisper-tiny transcript -> voice note
with Grandma entity, grandma/family-dinner tags, audio linked.
Findings folded in:
- speaches REQUIRES the model field per request (422 otherwise) —
stt.model documented as required-for-speaches in config.example
- models are not pre-baked into the image; preload via
POST /v1/models/{model_id} (the MODEL_ID env did not preload) —
compose comment corrected
- docker-compose gains an stt profile running speaches on :9001
…ile)
Answer to 'loaded conditionally?': speaches natively offloads after
300s idle, and keeps the model resident until then. The knob makes
khayal release it immediately after each transcription for small boxes.
- stt.unload_after (default false): after a successful transcription,
khayal fires DELETE {base}/api/ps/{model} (speaches contract);
weights stay in the service's disk cache — next capture reloads
from disk, never re-downloads
- fired async with a 3s timeout: the unload endpoint can hang under
racing calls (observed) and must never stall the capture response
- handler tests: unload fires when enabled (polled), never when
disabled
- docker-compose: speaches model cache mount corrected to
/home/ubuntu/.cache/huggingface (the /root path never received the
download — container recreate silently re-fetched)
Live-verified: capture -> decrement -> 'unloaded' 78ms later in
service logs. RAM release timing is governed by the service runtime
(allocator/GC); the offload itself is confirmed at the API level.
kl voice [file]: - with a file argument: uploads the recording (/v1/capture/audio) - without: records from the mic via the best available platform recorder (arecord on Linux, sox/rec elsewhere) — Go has no stdlib mic access, so recording shells out with ctrl+c to stop, temp file uploaded then removed - client gains CaptureAudio with voice-specific error mapping (unconfigured server, failed transcription reported as such) and a 180s request timeout — synchronous server-side transcription can exceed the default client timeout (model reload after idle) - kl no longer collapses server-side failures into 'unreachable': connection is probed first, real reasons shown themed Live-verified end-to-end through speaches: kl voice spoken.wav -> queued -> transcribed -> voice note. README + SPEC updated.
'Stuck on uploading' diagnosed with hard evidence: speaches' model
load wedged when a second load overlapped an in-flight one (two
'Loading model' lines 58s apart, neither completing), so the pipeline
ran the full 120s STT timeout while kl printed nothing.
- khayal: STT transcriptions now serialize through a mutex slot —
concurrent captures can never trigger the overlapping-load wedge
- closure bug from the slot refactor fixed (error branch referenced
the outer nil err instead of transcribeErr)
- kl: live elapsed feedback during the synchronous wait
('uploading · server transcribing… 42s', rewritten in place) — a
30-120s wait is now legible instead of frozen
- operational note: after ~300s idle speaches unloads the model and
the next transcription pays a disk reload; if a load wedges,
restart the speaches container and preload once
VPS evidence: a transcription that outlives the 120s timeout is
usually the model still loading from disk after idle — and the load
continues server-side after khayal gives up, so a blind retry
succeeds. Make that the contract:
- stt client returns a typed ErrTimeout (net.Error timeout detection)
- on timeout, khayal fires an async PreloadModel (speaches
POST /v1/models/{model}; generic servers just 404 harmlessly) and
returns 502 STT_TIMEOUT with an actionable 'try again in ~30
seconds' message
- handler test: slow STT -> 502 + retry hint + preload fired
testdata config gains unload_after, the six explicit connection-type toggles, and a cleaned stt block; SPEC audio strategy line aligned with the implementation (audio stored in vault media). AGENTS.md records the standing rule: testdata config, config.example, and SPEC config section update together whenever config surface changes.
…load Root cause of 'voice not working' against testdata: unload_after: true made khayal DELETE-unload the model after every capture, and speaches' next load wedged (>120s, never completing) — every voice capture then burned the full STT timeout. Container restart + preload restored 2.4s transcriptions. testdata config flips unload_after: false — speaches' native 300s idle offload releases RAM safely without the wedge. config.example and SPEC warn that the knob is unreliable against speaches' current version (keep it for other services; observe first). Live-verified: capture returns in 0.8s, voice note done with correct transcript.
PWA voice captures were producing '2 2 2 2 3 3 4 4…' and 'Hello? x30'
— small-whisper loop hallucinations on short/ambient clips. speaches
exposes per-segment quality metrics via response_format=verbose_json,
so the client now:
- requests verbose_json and drops segments above faster-whisper's own
thresholds (compression_ratio > 2.4, no_speech_prob > 0.9)
- joins only clean segments; an all-hallucinated recording yields an
empty transcript -> STT_EMPTY 502 ('transcription was empty') so
garbage never becomes a note
- falls back to plain text when the response isn't JSON
- whisper.cpp path unchanged (text format)
Live-verified: the previously hallucinating webm now returns a clean
rejection; the good recording still transcribes and ingests. 3 new
client tests; recommended model unchanged (tiny) — base helps but the
filter is what stops the loops.
Voice shipped experimentally in v1.2 and proved operationally fragile: small whisper models loop-hallucinate on short clips, model loads wedge under concurrent requests, and every capture became an ops incident. Mobile keyboard dictation through the text path covers the same flow today. Removed: PWA voice tab + recorder, kl voice, /v1/capture/audio, internal/stt, stt config block (testdata + example), health stt capability, speaches docker-compose profile (container stopped). The full v1.2 voice implementation (STT clients, segment-level hallucination filtering, preload/unload lifecycle) lives in git history for the revisit. Docs updated across README/SPEC (deferred section + not-building note)/openapi/UI_SPEC/TECH_STACK. Roadmap: v1.2 = connections 4-6 + PDF ingestion.
… clean - PLAN.md gains the v1.2 SHIPPED table (connections 4-6, variant joins, diversity ranking, PDF, notes UI) and corrects the stale deferral line - offline queue CaptureRequest type narrowed back to text/url/image (voice widened it during the feature's life) - full audit: zero lingering voice/stt/speaches references outside the deferred sections; testdata + config.example + compose clean
The pipeline card previously tracked only the ingest job with fake step labels and vanished the moment it saved — the connections pass was invisible. Now: - buildPipeline derives the true end-to-end pipeline from WS-patched job state: queued -> enriching (per-type label) -> embedding -> connecting, and the card PERSISTS through the connections phase that used to be silently swallowed by internal-type filtering - ActiveJobCard: done/active/future step states (check icons, pulsing active label), segmented progress bar (filled fraction + live shimmer on the active segment) - CaptureResult queued tile uses the same honest per-type labels (queued stage active, no more simulated 'saved=done') - PROCESSING_STEPS constants replaced by lib/pipeline.ts Voice remnants: lint's unused countJobs/sttMu/withSTTSlot removed, dead voice/vplayer CSS purged, CaptureRequest widened type narrowed. 98 vitest green, full Go suite + lint clean, assets rebuilt.
The four-stage row used space-between with long labels in a ~350px card: truncation ellipses everywhere, unreadable. Redesigned as a proper stepper following standard indicator principles: short single-word labels (details moved to tooltips), dot markers with per-state treatment (gold fill + check for done, pulsing gold ring for active, dim for future), connector segments that fill gold as stages complete. CaptureResult queued tile uses the same short labels with detail tooltips.
The label-per-dot stepper still overflowed at 4 stages in the card width (dot+label+connector math exceeds ~320px). Switched to the width-proof pattern: 12px dot markers with gold connectors (never overflow, they flex), and the ACTIVE stage's label + detail rendered as a text line beneath — ellipsized safely.
Screenshot diagnosis: rows were structured [connector][dot] with the last row flex:0, so the first dot sat far-left, connectors stretched, and the remaining dots clustered at the right edge. Flattened to the classic stepper — dot/connector pairs as direct flex siblings, each wrap flex:1 (last flex:0) — dots distribute evenly, connectors carry the progress between them.
…xact gaps User-verified one-liner: pipe-dot-wrap:first-child (not last-child) gets flex:0. Since each wrap is [connector][dot], collapsing the FIRST wrap (dot-only, no connector) makes wraps 2-4 split the remaining width equally — every connector then spans precisely the gap between its dot and the previous one. Dots land at 0/33/66/100%. My last-child version left wrap1's trailing space empty, drawing a detached line and clustering dots right (the reported screenshot).
The two upload previews were inconsistent and off-brand: images rendered as a 120px cropped thumbnail at opacity 0.5 with a dark gradient overlay, while PDFs got a differently-structured icon row. One attachment card now serves both: - image: full image object-contain up to 220px on a subtle checker backdrop — the whole photo, no crop, no wash - pdf: document medallion zone with mono 'PDF' type label - shared meta footer: gold type icon, filename (ellipsis), type chip (IMAGE/PDF) + size (+ 'text will be extracted' for pdf), ghost remove button with destructive hover Old img-filled/img-overlay/pdf-filled styles fully retired.
- preview heights reduced: image max 220->150px, pdf doc zone
110->88px (icon medallion 48->40px)
- the X button looked vertically stretched because the global
tap-target rule (button { min-height: 44px }) overrode its 26px
height while the icon stayed 14px — neutralized min-height on
.att-rm and pinned the svg to 14px
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.