feat: Chat translation

[tiaL-ops]

Context

In this pull request, I am implementing a feature to translate chat messages.

Related Issues

• Related Issue Translate chat messages #771
• Closes Translate chat messages #771

Verification

• Ran npm run dev and npm run emu, then opened Collaborate and Chat to verify that the dropdown renders correctly.
• Checked localhost:4000 and the Firestore emulator to verify that chat messages are sent and that the schema is updated.
• run npm run check:now, npm test, , all test passed
• npm run locales : the x label are : 4 unwritten ("$?") string(s) would fall back to English. Run "npm run locales-translate" to fill them: ui.collaborate.translate.label, ui.collaborate.translate.language, ui.collaborate.translate.error, ui.collaborate.translate.messageError

Checklist

• Updated the Message V3 schema to include a language field (the language the message is written in).
• Verified the changes in the Firebase Emulator.
• Added a dropdown at the top of the chat box before sending, using getTranslatableLocales().
• Added another dropdown at the top to translate messages.
• Added a function that uses translateMarkupText with the LLM as the raw translator.
• Have a fallback design when chat not working
• Verify translation by npm run locale

[amyjko]

This is a solid start! This is going in the right direction; you're using the right translation infrastructure, the message schema looks right to me. The core architecture looks solid.

Here are a few things to work on:

• All the user-facing text you've added is hard-coded into English. You'll need to update UIText.ts, generate the new locale schema with npm run schemas, run npm run locales-fix to add them to the locales, then move the strings to en-US.json, then update the UI to use the various localized text components to render them. There are many examples across the platform for uses of <LocalizedText>.

• Make sure to run npx prettier to keep formatting consistent.

• Translation is has a large number of sequential network round-trips. translateMessages loops over every message and awaits one translateMarkupText call each. For a long chat that's going to be slow. The RawTranslator interface takes an array of texts and returns them 1:1 — so you can collect all messages sharing a source language, translate them in one call, and map the results back. Definitely add that performance improvement.

• The issue (#771) explicitly suggests optionally saving the translation so we don't re-request it. Right now translations live in transient $state and a message sent after you turn translation on won't get translated until you toggle the dropdown again. I think we should persist translations in the message schema rather than re-translating them, to keep costs down. They should be invalidated if the chat message is edited.

• In the translation block you render both msg.language (source) above the divider and translations[msg.id].language (target) below. Two bare language codes stacked around a rule reads a little cryptically. Consider a clearer "from X → Y" display, or dropping the redundant source tag.

• Remember to run npm run check:now, npm test, and npm run locales to verify everything.

[amyjko]

This is good progress! It's getting closer. The batching is done right
(grouped by source language, one call per group, groups running concurrently), the
translations are cached and reused, the error handling degrades per-batch instead of
failing the whole chat, and the strings are properly localized with the right format tags.
The architecture is sound overall.

I did a very careful review today, so the list is long, but everything should be within reach. Most of what's below is about making it actually run, and about some edge cases in chat that translation touches.

Why translation "is still in English"

ChatView.svelte imports functions directly from @db/firebase and bails when it's
falsy. But functions is undefined until something calls getFunctionsInstance()
(src/db/firebase.ts:109) — that's what lazily loads the Functions SDK and wires the
emulator. Nothing in the chat path calls it, so you're hitting the error branch almost
every time. It probably looked intermittent, because CreatorDatabase happens to call
getFunctionsInstance() when it fetches missing creators, so whether translation works
is a race with an unrelated fetch.

Every other call site in the codebase does await getFunctionsInstance() —
Translate.svelte:68, LocaleChooser.svelte:117, accountExists.ts:13, Join.svelte:45,
Login.svelte:84, localize/+page.svelte:802. None of them import functions directly.
Do the same and I think your last checkbox unblocks itself.

While you're there, drop the top-level import { httpsCallable } from 'firebase/functions'
and import it dynamically like src/db/projects/translate.ts does — the static import
defeats the lazy load and pulls the Functions SDK into every page that renders a chat, which is a lot of extra code for the initial bundle.

Rebase on current main before anything else

The branch is 58 commits behind and GitHub reports conflicts, and I think that's causing
most of the noise in this diff:

• static/schemas/LocaleText.json shows +22,520 / −21,806, but if I normalize both sides through prettier, only 36 lines actually differ — the rest is 4-space → 2-space reindentation. Same for zh-TW (1,478 lines, 20 real), ja-JP, hi-IN. Running npx prettier --write on every changed JSON, as CLAUDE.md asks, collapses this to a diff we can actually read.
• More seriously, some existing translations lost content. In es-MX, Paragraph.doc went from "\¶Párrafo 1.\n\nPárrafo 2.\n\nPárrafo 3.¶'tres párrafos'\" to just "\¶Párrafo 1." — text deleted, and an unterminated \…\ block left behind. Same in Refine.doc, PropertyReference.doc, StructureDefinition.doc. bn-BD has the same truncation plus whole strings quietly re-translated with markup markers dropped (/সীমাবদ্ধ/ came back as the untranslated /restrict/) and the register changed. There's smaller damage in mr-IN, ne-NP, zh-CN, ja-JP, hi-IN, vi-VN, id-ID.

None of that is your fault — you ran the tooling as documented, but on a branch predating the fixes. main has since made npm run locales read-only (it reports drift instead of repairing it) and fixed the array-splitting rules that caused those truncations. So: merge main, revert all the locale and schema changes, re-run npm run schemas and npm run locales-fix, then prettier. The only locale changes left should be your four new keys and their $? placeholders.

Correctness

• Translation bypasses moderation. translateMessages skips only msg.text === null, so pending and removed messages still get translated — and the {#if translations[msg.id]} block is a sibling of <div class="what">, so it renders outside the moderation guard just above it. That means turning on translation shows reported and removed content to people who shouldn't see it, and writes a translated copy of it to the database. Please mirror the same msg.moderation === undefined || msg.moderation === 'approved' condition.

• Deleted messages keep their translations. deleteMessage nulls out text but leaves translations intact, so a deleted message's content lives on in translated form. This is the invalidation point from my last review — there's no edit path in chat, so delete and moderate are the equivalents.

• Translations aren't counted against the chat size cap. Chat's constructor trims messages against MAX_CHAT_MESSAGES_BYTES (128 KB) using only message.text?.length. Cached translations are invisible to that estimate, so a chat translated into a few languages can be several times bigger than the trimmer thinks, and drift toward Firestore's hard 1 MB document limit. After that, sending a message starts failing.

• The write side isn't batched, even though the reads are. After batching the LLM calls, the caching loop calls saveMessageTranslation once per message, and each one runs a full transaction that reads and rewrites the entire chat document. Translating a 50-message chat is 50 serialized transactions on one doc. Do a single saveMessageTranslations(chat, language, Map<id, text>) that applies them all at once.

• A Small race condition translations = next is assigned after several awaits with no guard, so changing the dropdown twice quickly can let an older pass overwrite a newer one. Capture the target and bail if it changed.

A design question

Right now translations only recompute when the dropdown changes, so a message that arrives while translation is on stays untranslated until you toggle it off and on. We might consider an explicit translate button instead of triggering on drop down change.

Reuse

• The RawTranslator adapter in ChatView.svelte is a copy of the one in src/db/projects/translate.ts. Let's extract it once (something like getFirebaseTranslator(functions): RawTranslator) and use it from both.
• The ~130 lines of grouping and batching belong in src/db/translateMarkup.ts, not in a
  view component. Tat module already owns this layer, is deliberately backend-agnostic, and has tests. A plural sibling of translateMarkupText would fit naturally, and would be unit-testable.
• For the language pickers: Translatable has ~161 languages, so both dropdowns are ~161 items in a narrow chat panel. Translate.svelte already solves this with LocaleSearch + filterLocalesByQuery and renders entries with LocaleName — I'd reuse those instead of the hand-rolled languageOptions / languageName().
• Deduplicating the translatable locales down to bare language codes drops the region, which collapses zh-CN and zh-TW into zh — Simplified and Traditional become indistinguishable, and the same happens to pt-PT / pt-BR. Store the full locale string and compare with localesAreEqual, like Translate.svelte does.
• MessageSchemaV3 and V4 in one PR can just be a single V3. Also worth knowing: messages don't carry a v field, MessageSchemaLatestVersion isn't referenced anywhere, and the unknown-version union doesn't do anything since all the added fields are optional. I'd leave that machinery as you found it.
• src/db/chats/ChatDatabase.test.ts already tests addMessage, reportMessage,
  moderateMessage, and deleteMessage against a mocked Firestore. Tests for the language tag, the translation cache, and deletion clearing translations would be good.

Localization and accessibility

• The two role="status" divs are live regions, and we route all dynamic announcements
  through Announcer.svelte via getAnnouncer() — component-local live regions cross-talk with the central one on real screen readers.
• For the visible error, Notice.svelte is the component we use elsewhere (see Translate.svelte).
• The visible label and the Options label prop are the same string, so the control gets
  announced twice and has no distinct name. Pick one.
• A few string issues: the "off" option is a hard-coded '—' rather than a localized string; error and messageError are nearly identical, and messageError says "this chat" when it's about a single message; and the → between languages is hard-coded, which points the wrong way in RTL locales. A Template<['from','to']> would let translators control the wording and the direction — remember our template inputs are always named, never numbered.
• field.language is correctly tagged [plain], but field is meant for input fields — it
  probably belongs next to translate.label.

Housekeeping

Prettier hasn't run on the source files either: there's a stray blank line inside the
@db/translateMarkup import, and ChatDatabase.svelte.ts re-indents some pre-existing code it didn't otherwise touch.

Suggested order

1. Merge main, revert the locale/schema churn, regenerate, run prettier.
2. Fix getFunctionsInstance() and verify translation actually runs against the emulator.
3. Moderation guard and deletion invalidation.
4. Extract the shared translator, move batching into translateMarkup.ts, add tests.
5. Localization and accessibility cleanup.
6. npm run check:now, npm test, npm run locales — and read the locales output for x-marked errors rather than trusting the exit code alone.

This is close! Ping me when you're ready for another pass.

[tiaL-ops]

Thank you so much for all details! I will start working on it now and will let you know once its ready again

[tiaL-ops]

I reverted/copy pasted mutliple times , I m not sure why it keeps moving the import place

[tiaL-ops]

Hi @amyjko , I think I’ve made good progress on the work. I’d appreciate another review when you have time. I was thinking of updating the changelog once everything is finalized, or is it better to update it with every change instead?