Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -31,11 +31,16 @@ BIRDNETGO_URL=http://192.168.1.10:8080
# GEMINI_API_KEY=

# Seconds to wait between image-API calls, to stay within the Gemini free-tier
# rate limit. Default matches the pipeline (6s). Only used with GEMINI_API_KEY.
# rate limit (default 6). This is the throughput knob: art is acquired one pose at
# a time, so this gap is the only thing pacing the calls. Lower it on a paid tier,
# raise it if you get throttled, 0 to remove the gap entirely. Needs GEMINI_API_KEY.
# GENERATE_SLEEP=6

# Cap species generated per pipeline run (default 4). Only used with GEMINI_API_KEY.
# GENERATE_MAX_PER_CYCLE=4
# Prompt addenda for species the model keeps getting wrong, as a JSON object keyed
# by scientific name or slug. Defaults to _species-notes.json beside the art, so it
# lives in the persisted volume. Layered over the set bundled with the pipeline;
# only affects art Saezuri generates, never art downloaded from the repo.
# SPECIES_NOTES=/usr/share/nginx/html/assets/illustrations/_species-notes.json

# --- Free pre-made illustrations (on by default) ---
# A zero-cost alternative (or complement) to on-demand Gemini generation: the moment
Expand Down
13 changes: 13 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,19 @@ Where things live, so a change lands in the right place fast.
gates/aggregates species, and publishes `/snapshot.json`, `/layout-manifest.json`,
`/calls-manifest.json`, and the e-ink PNG frames (`render.ts`, reusing `computeLayout`).
Run it with `npm run refresh:dev`.
- **Art acquisition:** `src/server/generate.ts` (`Generator`). The unit of work is a **pose**,
not a species: a perched-only species already renders (`resolveArt`), so perched is what
lifts it off the fallback silhouette and flight only changes the 15% that roll for it.
Two independent lanes — repo download (concurrent, cheap) and Gemini generation (serial) —
so free art never queues behind a paid render. Precedence is fixed and deliberate: file on
disk > illustrations repo > local generation > fallback silhouette. The repo is the state of
the art, so `species-notes.json` (`notes.ts`) tunes the *generation* fallback only and never
replaces downloaded art; a changed note re-renders a pose only when `source: 'generated'`.
`_art-state.json` in `assetsDir` records per pose: repo 404s (7-day TTL, so the repo isn't
re-probed every publish), provenance, and the note version. It is a cache, never a source of
truth — a bad entry must degrade to "probe again", never to "skip forever". The generate lane
owns the rate limit (`GENERATE_SLEEP`): the pipeline is invoked once per pose, so its own
inter-call sleep never fires.
- **Reference calls:** `src/server/calls.ts` (`CallLibrary`) queues a lookup per newly-heard
species, mirroring `generate.ts`; `callProviders/` holds one provider per archive behind a
common interface — `find()` resolves null for "nothing here" (cacheable) and **throws** for
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ RUN mkdir -p /opt/canvas && cd /opt/canvas \
# We fetch only the pipeline.tar.gz asset (not the whole art repo), so the build
# stays lean. Bumping PIPELINE_VERSION (a normal app commit) adopts a new pipeline.
# NOTE: the illustrations repo must have published this release first.
ARG PIPELINE_VERSION=v1.0.0
ARG PIPELINE_VERSION=v1.1.0
RUN apk add --no-cache curl \
&& mkdir -p /build \
&& curl -fSL "https://github.com/vrwrts/saezuri-illustrations/releases/download/${PIPELINE_VERSION}/pipeline.tar.gz" \
Expand Down
59 changes: 50 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,8 @@ annotated copy of every setting.
| `ILLUSTRATIONS_REF` | `main` | Branch or release tag to pull art from. Pin a tag for a fixed art set. |
| `ILLUSTRATIONS_BASE_URL` | derived jsDelivr URL | Overrides the whole download base URL, and wins over the two above. For testing against a local file server. |
| `GEMINI_API_KEY` | unset | Google AI (Gemini) key. Set it to *also* generate art for species the repo lacks (see below); unset relies on downloads only. |
| `GENERATE_MAX_PER_CYCLE` | `4` | Cap on species generated per pipeline run. |
| `GENERATE_SLEEP` | pipeline default (`6`) | Seconds between image-API calls, to stay under the Gemini free tier. Read by the vendored pipeline, not by Saezuri — the default lives there. |
| `GENERATE_SLEEP` | `6` | Seconds between image-API calls, to stay under the Gemini free tier. **The throughput knob**: lower it on a paid tier, raise it if you get throttled, `0` to remove the gap. |
| `SPECIES_NOTES` | `_species-notes.json` beside the art | Prompt addenda for species that keep coming out wrong (see below). Layered over the set bundled with the pipeline. |

### Reference calls

Expand Down Expand Up @@ -183,26 +183,67 @@ docker run -d -p 8090:8080 \
```

The refresh service holds BirdNET-Go's detection SSE stream; the moment a new species is heard it
first tries the free download, and if the repo doesn't have it, generates a perched + flight cutout
(via the bundled pipeline), then refreshes the layout manifest the frontend polls. Silhouettes turn
into real birds on their own within seconds to hours.
first tries the free download, and if the repo doesn't have it, generates the cutout (via the
bundled pipeline), then refreshes the layout manifest the frontend polls. Silhouettes turn into
real birds on their own within seconds to hours.

Art is acquired **one pose at a time**, perched first. A species needs only its perched cutout to
stop being a silhouette, so that render lands and shows up before the flight pose is even started.
The free downloads and the paid generation run independently, so a species whose art is already in
the repo appears immediately rather than queueing behind someone else's render.

Things to know:

- **It uses the paid Gemini image API with _your_ key** — you pay for what it generates.
Only detected species not already downloaded are generated (typically a handful). Generation is
paced by `GENERATE_SLEEP` (default 6s) to stay under the free tier and capped per cycle by
`GENERATE_MAX_PER_CYCLE`.
Only detected species the repo doesn't already have are generated (typically a handful).
Generation is paced by `GENERATE_SLEEP` (default 6s) to stay under the free tier; that gap is
the only throughput control, since the limit here is the API's request rate.
- **A pose the model declines is left alone for a day** rather than re-attempted every
refresh, so a stubborn species can't quietly drain quota. See
[Free illustrations](#free-illustrations) for how gaps are remembered.
- **Persist the art** with the named volume above so container upgrades don't re-spend
those API calls. The manifest is rebuilt from the volume at startup.
- **The generator is bundled in every image** — vendored at build time from the
[saezuri-illustrations](https://github.com/vrwrts/saezuri-illustrations) pipeline at a pinned
version (numpy/scipy cutout, no ML model, so the `nginx:alpine` image stays modest).
`GEMINI_API_KEY` unset simply means the worker never generates; the container is otherwise identical.
- **A species that keeps coming out wrong** needs a better prompt, not more attempts — see
[Species notes](#species-notes).
- **Licensing.** Generating art locally for your own display is personal use. The style
derives from the CC-BY-NC-SA lineage (see below) — confirm the obligations before
publishing generated images.

## Species notes

Sometimes a species comes out wrong no matter how many times you regenerate it — the model's prior
is simply off, and re-rolling the dice won't fix it. A *note* is a sentence or two appended to that
species' prompt only:

```json
{
"Turdus merula": "Solid glossy black, no pale markings; bill and eye-ring bright orange-yellow.",
"parus-major": "Black crown and throat stripe, bright white cheeks, yellow underparts."
}
```

Save it as `_species-notes.json` beside the art (inside the persisted volume, so it survives
upgrades), or point `SPECIES_NOTES` anywhere you like. A key may be either the scientific name or
its slug — the slug is what you see in the illustration filenames. Keys beginning with `_` are
comments.

How it behaves:

- **Edits apply on their own.** When you change a species' note, its art is re-rendered on the next
cycle; you don't need to delete anything or restart the container.
- **It only affects art Saezuri generated.** A cutout downloaded from the illustrations repo is
left alone, because that repo is the state of the art and everyone benefits from it being right.
If a note fixes a species the repo gets wrong, [contribute it
upstream](https://github.com/vrwrts/saezuri-illustrations) rather than keeping the fix local — the
pipeline ships its own `species-notes.json` that yours is layered over, and that is the file to
send a PR to.
- **It needs `GEMINI_API_KEY`.** A note is an instruction to the generator; with no key there is
nothing to instruct.

## Free illustrations

You don't have to pay for generation to get real art. **On by default**, the moment BirdNET-Go
Expand Down Expand Up @@ -233,7 +274,7 @@ How it behaves:
- **Gaps it can't fill are remembered, not retried forever.** A cutout the repo doesn't have (or
that generation declined) is logged once and left alone for a while — a week for a repo miss, a
day for a generation miss — instead of being re-requested on every refresh. The record lives in
`_misses.json` in the illustrations volume; delete it to retry everything immediately.
`_art-state.json` in the illustrations volume; delete it to retry everything immediately.
- **Self-healing.** Delete a cutout from the volume and the service notices, re-downloads it (or
regenerates it), and rebuilds the manifest — which is also how you replace art you don't like.
This works for a single pose of a pair, and for a bird that hasn't been heard in weeks: every
Expand Down
28 changes: 26 additions & 2 deletions addon/DOCS.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,12 +72,36 @@ generic silhouette, still labelled and still sized by their real count.
| **Illustrations branch** | `main` | Branch or tag to download from. |
| **Illustrations base URL** | derived | Overrides the two above with a direct URL. |
| **Gemini API key** | unset | Optional. Set it to *also* generate art, in the same style, for species nobody has contributed yet. |
| **Generated illustrations per cycle** | `4` | How many to generate at a time. |
| **Pause between generations** | `6` | Seconds between generated illustrations. |
| **Pause between generations** | `6` | Seconds between generated illustrations. Lower on a paid tier, raise if rate-limited. |
| **Species notes** | none | Per-bird prompt corrections, see below. |

Generation costs money at Google's rates and is entirely optional. Everything works
without a key.

Illustrations are generated one pose at a time, perched first. A bird stops being a
silhouette as soon as its perched illustration lands, so it appears without waiting
for the flight one.

### Species notes

Some birds come out wrong no matter how often they are regenerated — the model's idea
of them is simply off, and trying again won't help. A note is a short description
added to that bird's prompt only. One entry per bird:

```
Turdus merula|Solid glossy black, orange-yellow bill and eye-ring.
Parus major|Black crown and throat stripe, white cheeks, yellow underparts.
```

Use the scientific name before the pipe (the slug, like `turdus-merula`, also works).
Change a note and that bird is redrawn on the next cycle — nothing to restart.

Notes only affect illustrations this app generates itself. One downloaded from the
illustrations repository is left as it is, because that repository is the shared set
everyone draws from. If a note fixes a bird the repository gets wrong, please
[contribute it there](https://github.com/vrwrts/saezuri-illustrations) so every
installation benefits.

### Reference recordings

When a species is heard, Saezuri looks up a freely-licensed recording of its call and
Expand Down
11 changes: 9 additions & 2 deletions addon/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ ports:
ports_description:
"8080/tcp": Direct web access, only needed for an e-ink panel fetching /24h.png
options:
generate_max_per_cycle: 4
calls_max_per_cycle: 4
frame_width: 800
frame_height: 480
Expand All @@ -40,8 +39,16 @@ schema:
illustrations_ref: str?
illustrations_base_url: str?
gemini_api_key: password?
generate_max_per_cycle: int(1,64)
generate_sleep: int(0,600)
# "Scientific name|note" per entry, e.g.
# Turdus merula|Solid glossy black, orange-yellow bill and eye-ring.
# run.sh materialises these into the notes file the generator reads.
species_notes:
- str?
# Ignored since 0.15: art is acquired one pose at a time, so there is no batch
# to cap. Kept in the schema only so an existing configuration still validates;
# remove it from your options at your convenience.
generate_max_per_cycle: int(1,64)?
call_providers: str?
calls_max_per_cycle: int(1,64)
frame_width: int(100,4000)
Expand Down
33 changes: 31 additions & 2 deletions addon/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,35 @@ export_opt() {
fi
}

# The generator reads prompt addenda from a JSON file, but a Home Assistant user has
# no way to edit a file inside the app's own /data. So the option is a list of
# "key|note" strings — editable in the app's configuration UI, and the same
# pipe-separated convention the pipeline already uses for "scientific|common" — and
# this turns it into the file. Split on the FIRST pipe only, since a note may
# contain one.
export_species_notes() {
opt_present species_notes || return 0
_notes_file=/data/species-notes.json
jq '[.species_notes[]?
| select(type == "string")
| (. | index("|")) as $i
| select($i != null)
| { key: (.[:$i] | sub("^\\s+"; "") | sub("\\s+$"; "")),
value: (.[$i+1:] | sub("^\\s+"; "") | sub("\\s+$"; "")) }
| select(.key != "" and .value != "")
] | from_entries' "$OPTIONS" > "$_notes_file"
_count=$(jq 'length' "$_notes_file")
if [ "$_count" -gt 0 ]; then
export SPECIES_NOTES="$_notes_file"
log "species_notes: ${_count} note(s) -> ${_notes_file}"
else
# Nothing usable: leave SPECIES_NOTES unset so the app falls back to its own
# default path, rather than pointing it at an empty file.
rm -f "$_notes_file"
log "species_notes: no usable entries (expected \"Scientific name|note\")"
fi
}

# FRAME_SHADOW is read as "anything but 0", so a YAML bool has to become 1 or 0;
# exporting the string "false" would silently enable the shadow.
export_bool_opt() {
Expand All @@ -60,8 +89,8 @@ export_opt ILLUSTRATIONS_REPO illustrations_repo
export_opt ILLUSTRATIONS_REF illustrations_ref
export_opt ILLUSTRATIONS_BASE_URL illustrations_base_url
export_opt GEMINI_API_KEY gemini_api_key
export_opt GENERATE_MAX_PER_CYCLE generate_max_per_cycle
export_opt GENERATE_SLEEP generate_sleep
export_species_notes
export_opt CALL_PROVIDERS call_providers
export_opt CALLS_MAX_PER_CYCLE calls_max_per_cycle
export_opt FRAME_WIDTH frame_width
Expand Down Expand Up @@ -174,7 +203,7 @@ log "BIRDNETGO_URL=${BIRDNETGO_URL}"
log "BIRDNETGO_TOKEN=$(redacted "${BIRDNETGO_TOKEN:-}")"
log "GEMINI_API_KEY=$(redacted "${GEMINI_API_KEY:-}")"
for _name in ILLUSTRATIONS_REPO ILLUSTRATIONS_REF ILLUSTRATIONS_BASE_URL \
GENERATE_MAX_PER_CYCLE GENERATE_SLEEP CALL_PROVIDERS CALLS_MAX_PER_CYCLE \
GENERATE_SLEEP SPECIES_NOTES CALL_PROVIDERS CALLS_MAX_PER_CYCLE \
FRAME_WIDTH FRAME_HEIGHT FRAME_BG FRAME_SHADOW FRAME_WINDOWS \
SPECIES_DICT_LOCALES PUBLISH_DEBOUNCE_MS AGING_INTERVAL_MS \
SUMMARY_INTERVAL_MS FRAME_HTML_DIR CACHE_DIR; do
Expand Down
19 changes: 15 additions & 4 deletions addon/translations/en.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -31,12 +31,23 @@ configuration:
description: >-
Optional. Set it to also generate art for species nobody has contributed
an illustration for yet.
generate_max_per_cycle:
name: Generated illustrations per cycle
description: How many illustrations to generate at a time.
generate_sleep:
name: Pause between generations
description: Seconds to wait between generated illustrations.
description: >-
Seconds to wait between generated illustrations. Lower it if you are on a
paid Gemini tier, raise it if you get rate-limited.
species_notes:
name: Species notes
description: >-
For birds that keep coming out wrong. One entry per bird, written as
"Scientific name|what it should look like" — for example "Turdus
merula|Solid glossy black, orange-yellow bill and eye-ring". Only affects
illustrations this app generates itself; downloaded ones are left alone.
generate_max_per_cycle:
name: Generated illustrations per cycle (unused)
description: >-
No longer used — illustrations are now generated one at a time. Safe to
remove from your configuration.
call_providers:
name: Recording archives
description: >-
Expand Down
9 changes: 6 additions & 3 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,13 @@ services:
# display-only: the refresh service still publishes, it just skips
# generation, so new species stay hidden until art exists.
GEMINI_API_KEY: ${GEMINI_API_KEY:-}
# Throttle between image-API calls (unset => pregen's 6s free-tier default).
# Seconds between image-API calls (default 6, the Gemini free-tier pace). The
# throughput knob: art is acquired one pose at a time and this gap is what keeps
# us under the request-rate limit.
GENERATE_SLEEP: ${GENERATE_SLEEP:-}
# Cap species generated per pipeline run.
GENERATE_MAX_PER_CYCLE: ${GENERATE_MAX_PER_CYCLE:-4}
# Prompt addenda for species that keep coming out wrong. Defaults to
# _species-notes.json beside the art, i.e. inside the persisted volume.
SPECIES_NOTES: ${SPECIES_NOTES:-}
# --- Free pre-made illustrations (see .env.example) ---
# Per detected species, the refresh service downloads a ready-made cutout from
# the saezuri-illustrations repo (via jsDelivr) — no Gemini key needed. Defaults
Expand Down
Loading
Loading