Skip to content

testing.md

Latest commit

 

History

History
287 lines (200 loc) · 10.7 KB

testing.md

File metadata and controls

287 lines (200 loc) · 10.7 KB
Raw

Testing strategy

Goals

The test suite protects several distinct boundaries:

  1. pure correction-model behavior;
  2. the JavaScript bridge contract;
  3. the real embedded QuickJS/Grammalecte engine;
  4. vendored-engine reproducibility;
  5. Android framework integration;
  6. real-world compatibility across editors.

Keeping these layers separate makes failures easier to diagnose.

1. Pure JVM unit tests

Fast JVM tests cover correction logic that does not require the Android framework or the real QuickJS runtime.

Current coverage includes:

  • issue range validation;
  • exact-range issue merging and priority;
  • suggestion deduplication and limits;
  • UTF-16 offset behavior, including emoji;
  • asset path normalization;
  • correction-result invariants.

These tests should remain fast and focused.

2. JavaScript bridge contract test

tools/test-js-bridge.mjs executes android_bridge.js with synthetic Grammalecte-compatible stubs.

It verifies the bridge contract, including:

  • required data-file initialization;
  • grammar issue conversion;
  • spelling issue conversion;
  • issue kinds;
  • offsets;
  • suggestion deduplication;
  • word validation;
  • suggestion limits.

This test intentionally does not prove that the real embedded Grammalecte engine can perform a correction. It protects the JavaScript/Kotlin integration contract independently from upstream engine data.

3. Real QuickJS / Grammalecte JVM integration tests

The engine-grammalecte module also runs tests against the actual vendored Grammalecte assets through QuickJS.

RealEngineSmokeTest verifies a minimal real-engine path, including:

  • a spelling correction (magazin → magasin);
  • a grammar correction (aller → allé / allée).

RealEngineRegressionTest extends this with representative French regression fixtures covering:

  • spelling;
  • grammar agreements;
  • conjugation;
  • apostrophes and typography-related corrections;
  • multiple representative grammar rules;
  • false-positive checks on correct sentences;
  • validity of all returned issue ranges.

Regression fixtures must remain synthetic and should isolate the behavior being protected.

4. Vendored engine reproducibility

The Vendored engine smoke test GitHub Actions workflow:

  1. checks out the project;
  2. validates the committed upstream metadata;
  3. regenerates Grammalecte from the pinned upstream commit;
  4. validates the regenerated metadata;
  5. verifies that regeneration produces no Git diff;
  6. runs the real engine tests;
  7. builds the debug Android application.

The vendoring script normalizes known non-deterministic upstream dictionary metadata:

  • build timestamps are derived from the pinned upstream commit;
  • l2grams values are sorted before serialization.

This protects against:

  • upstream source-layout changes;
  • stale or manually modified vendored assets;
  • non-reproducible generated dictionaries;
  • missing generated assets;
  • engine-module regressions;
  • APK packaging regressions.

5. Android instrumentation

Instrumentation tests cover Android behavior that cannot be reliably validated with plain JVM tests.

Current application-level instrumentation verifies that:

  • the Grammalecte dictionary is packaged in the application;
  • the real packaged QuickJS/Grammalecte engine performs spelling correction;
  • the real packaged engine performs grammar correction;
  • returned UTF-16 ranges remain correct with emoji before an issue;
  • the spell-checker service is discoverable;
  • the spell-checker service is protected by android.permission.BIND_TEXT_SERVICE;
  • the IME is discoverable and protected by android.permission.BIND_INPUT_METHOD;
  • the IME exposes the expected android.view.im metadata;
  • the IME can be enabled and selected through Android's input-method framework;
  • a real editable EditText exposes selected text to the IME through a real InputConnection;
  • a selection made after the IME is already open triggers a fresh Grammalecte analysis;
  • the Grammalecte IME can analyze that selection and commit corrected text back into the real editor;
  • the ACTION_PROCESS_TEXT activity is discoverable for text/plain;
  • a real Android TextServicesManager session reaches the packaged SpellCheckerService and returns a Grammalecte correction end to end.

The spellchecker instrumentation suite verifies Android result mapping, including:

  • spelling suggestions and typo attributes;
  • grammar suggestions and grammar-specific attributes where supported;
  • UTF-16 sentence offsets with emoji;
  • apostrophe ranges;
  • multiple issues retaining their own ranges;
  • identical-range issue merging and grammar priority;
  • word suggestion limits;
  • valid dictionary-word attributes.

The instrumentation workflow runs:

  • API 35 for relevant pull requests;
  • API 26, API 30 and API 35 on scheduled/manual runs.

API 26 is the current minimum supported Android API. Additional device, ROM and application coverage is still planned.

6. Build and privacy checks

make check is the canonical local quality entry point and is also used by the main CI workflow.

It currently runs:

  • the source-manifest offline privacy check;
  • ShellCheck;
  • the JavaScript bridge contract test;
  • ktlint;
  • JVM tests;
  • Android lint.

make assemble builds the debug APK and then runs tools/check-apk-permissions.sh.

The APK permission check inspects the final packaged application with Android aapt2 and fails if android.permission.INTERNET appears after manifest merging.

This complements the faster source-manifest check and protects the project's offline-only invariant against permissions introduced through dependencies.

7. Physical-device validation

Some compatibility behavior depends on the target application's editor implementation and therefore requires real-device testing.

Development testing on a physical Android device has validated real local Grammalecte corrections through:

  • native Android text fields with SpellCheckerService;
  • ACTION_PROCESS_TEXT;
  • the Grammalecte IME through InputConnection.

The end-to-end IME instrumentation test has also been validated on a physical Samsung Galaxy S21 running Android 15. It uses a debug-only editable test activity, switches to the Grammalecte IME through Android's input-method framework, selects synthetic text, performs a real correction and verifies the resulting editor contents.

A debug-only WebViewCompatibilityTestActivity provides a local android.webkit.WebView fixture containing a synthetic editable <textarea>. It is used to reproduce editor-specific InputConnection, selection and PROCESS_TEXT behavior without relying on an external website or network access.

Physical WebView validation also checks that potentially slow selected-text reads do not block the IME UI and that the analyzed selection range can be restored before committing corrected text.

The IME replacement path has been manually validated in:

  • Samsung Notes;
  • Firefox;
  • SMS editing;
  • Android System WebView.

These checks are development validation, not a complete compatibility matrix.

Compatibility matrix

Before a stable public release, maintain a reproducible matrix that records for each tested application:

  • Android version;
  • application version;
  • whether SpellCheckerService is invoked;
  • whether PROCESS_TEXT is offered;
  • whether returned PROCESS_TEXT text is applied;
  • whether the IME receives a usable InputConnection;
  • whether selected-text replacement succeeds.

Compatibility claims should identify the integration path actually tested.

Fixture policy

Never use private messages, real emails, credentials or copied user content as test fixtures.

Use short synthetic French sentences designed to isolate one behavior.

Performance tests

Performance coverage uses two complementary JVM tests against the real embedded QuickJS/Grammalecte engine.

EnginePerformanceCharacterizationTest remains an opt-in measurement tool. It reports cold-start, sentence and paragraph timings without enforcing thresholds:

./gradlew \
  :engine-grammalecte:testDebugUnitTest \
  --tests '*EnginePerformanceCharacterizationTest*' \
  -PgrammalectePerf=1 \
  --rerun-tasks \
  --no-daemon

EnginePerformanceRegressionTest adds regression protection.

Its deterministic test verifies that repeated analyses reuse the initialized QuickJS runtime without reloading the embedded Grammalecte assets. This part runs with the normal engine test suite.

Its timing test is enabled by -PgrammalectePerf=1 and is run in CI through:

make performance-check

The timing budgets are intentionally wider than the development baseline so that normal host load does not make CI flaky:

  • cold first correction: at most 3000 ms;
  • sentence median: at most 75 ms;
  • sentence p95: at most 200 ms;
  • paragraph median: at most 150 ms;
  • paragraph p95: at most 400 ms.

A three-run development baseline measured:

  • cold first correction: 809.54–828.45 ms;
  • sentence median: 13.48–13.84 ms;
  • sentence p95: 14.94–16.45 ms;
  • paragraph median: 31.89–32.23 ms;
  • paragraph p95: 32.85–34.35 ms.

These JVM measurements are regression guards, not Android-device performance guarantees. Physical-device latency remains a separate compatibility and profiling concern.

Performance regression tests are intended to detect substantial regressions, such as accidental runtime reinitialization, repeated asset loading or major analysis slowdowns, rather than small timing variations.

Runtime and memory stress tests

EngineRuntimeStressTest exercises the real embedded QuickJS/Grammalecte runtime under sustained use.

The stress suite covers:

  • a long-lived engine performing hundreds of grammar analyses and repeated spelling checks;
  • retained QuickJS memory before and after sustained analysis, measured after explicit QuickJS garbage collection;
  • concurrent bursts of grammar and spelling requests sharing one serialized runtime;
  • repeated engine creation, use and closure cycles.

Run the suite with:

make stress-check

The stress tests are opt-in through -PgrammalecteStress=1 and are executed by CI through the dedicated stress-check target.

The initial development measurement after 400 long-lived analyses reported:

  • QuickJS memory before stress: 30.00 MiB;
  • QuickJS memory after stress: 30.00 MiB;
  • retained-memory growth: 0.00 MiB;
  • QuickJS allocated memory after stress: 40.62 MiB;
  • objects after stress: 91,502;
  • strings after stress: 36,620.

The retained-memory regression budget is 8 MiB. This intentionally allows normal runtime and host variation while detecting substantial accumulation across repeated corrections.

These measurements concern the embedded QuickJS runtime used by the JVM test environment. They protect engine lifecycle and memory behavior but are not a claim about total Android process memory or device-specific RSS.