refactor(web): simplify inputSegment spec + related SearchSpace property 🚂

[jahorton]

These are a few spec changes I wished to do sooner, but the previous code for ContextToken splitting and merging complicated matters and prevented the changes. Now that splitting and merging lie entirely on SearchPath and SearchSpace, these changes can now be made cleanly.

Once we support corrections from tokenizations that don't match the current context state, application of any suggestion not based on the true context's actual tokenization (say, due to fat-fingering a whitespace) will cause the context's "true transform" to become invalidated, with a different one (from the best correction) taking its place. The transform ID field is still valid, but trueTransform will not reflect the actual context state - thus, there's little reason to keep it around.

Furthermore, SearchSpace should not expose the PathInputProperties.subsetId fields corresponding to its inputs; it will be possible for one or for more such "subsets" to converge on the same tokenization, and this cannot be reflected well when returning PathInputProperties from the .inputSegments property. The InputSegment portion itself will be shared by all converging paths, but not the subsetId.

Build-bot: skip release:web release:android release:ios

User Testing

Run all tests via Keyman for Android. Use the keyboard in-app.

TEST_PRED_CHAIN_APPLY: Apply numerous suggestions in sequence and verify that they apply properly.

TEST_PRED_TEXT_SUGGEST_ANDROID: Verify that predictive text produces appropriate suggestions and that they apply properly.

• Test with gff_amharic.
• Verify that reversions work properly. (Tap backspace after applying a suggestion, then the left-most "suggestion".)

TEST_CLEAR_CONTEXT: Verify that predictive text throws no errors when clearing the context / deleting all text.

TEST_PRED_LONG_TEXT_REVERSIONS: Verify that predictive text produces appropriate suggestions and that they apply properly.

• Test with sil_euro_latin.

• Type the following text, in full:

  applesauce and orange juice don't seem like they'd make for the best breakf
  

• Tap the breakfast suggestion.

• Tap backspace.

• Tap the reversion ("breakf")

• Verify that the reversion is properly applied and that Android does not signal any errors (via toasts, notifications, or otherwise).

TEST_PERIOD_APPLY_OVERWRITE: Using the sil_euro_latin keyboard, check if punctuation typed immediately after applying a suggestion erases the whitespace.

• Type testi
• Tap the testing suggestion
• Then tap the . key.
• Verify that testing. is the result, with no appended whitespace.

TEST_SPACE_APPLY_OVERWRITE: Using the sil_euro_latin keyboard, check if punctuation typed immediately after applying a suggestion erases the whitespace.

• Type testi
• Tap the testing suggestion
• Then tap the key.
• Verify that testing is the result.

TEST_PERIOD_APPLICATION: Using the sil_euro_latin keyboard, check if punctuation typed with a highlighted suggestion applies it and appears afterward.

• Type testi and verify that it is selected / highlighted. (If it isn't, delete its text and retype - tap accuracy may help.)
• Then tap the . key.
• Verify that testing. is the result, with no appended whitespace.
• Tap . again and verify that it outputs a new ..

TEST_SPACE_APPLICATION: Using the sil_euro_latin keyboard, check if punctuation typed with a highlighted suggestion applies it and appears afterward.

• Type testi and verify that it is selected / highlighted. (If it isn't, delete its text and retype - tap accuracy may help.)
• Then tap the key.
• Verify that testing is the result, with no appended whitespace.
• Tap and verify that it outputs a new .

TEST_PERIOD_OVERWRITE_REVERSION: Using the sil_euro_latin keyboard, check if punctuation typed immediately after applying a suggestion erases the whitespace.

• Type testi
• Tap the testing suggestion
• Then tap the . key.
• Verify that testing. is the result, with no appended whitespace.
• Backspace over the . and verify that a proper reversion is displayed: "testi"
• Apply the reversion and verify that no errors result.
• Clear the context and type 3 words, then a space, and then repeat this test until reaching this line.

TEST_PERIOD_APPLICATION_REVERSION: Using the sil_euro_latin keyboard, check if punctuation typed immediately after applying a suggestion erases the whitespace.

• Type testi and verify that it is selected / highlighted. (If it isn't, delete its text and retype - tap accuracy may help.)
• Then tap the . key.
• Verify that testing. is the result, with no appended whitespace.
• Backspace over the ., landing at testing, and verify that a proper reversion is displayed: "testi"
• Apply the reversion and verify that no errors result.
• Clear the context and type 3 words, then a space, and then repeat this test until reaching this line.

TEST_PERIOD_APPLICATION_DELAYED_REVERSION: Using the sil_euro_latin keyboard, check if punctuation typed immediately after applying a suggestion erases the whitespace.

• Type testi and verify that it is selected / highlighted. (If it isn't, delete its text and retype - tap accuracy may help.)
• Then tap the . key.
• Verify that testing. is the result, with no appended whitespace.
• Tap . again and verify that it outputs a new ..
• Backspace over the . and verify that a proper reversion is displayed: "testi"
• Apply the reversion and verify that no errors result.
• Clear the context and type 3 words, then a space, and then repeat this test until reaching this line.

TEST_MULTITAPS: Verify that the . key on english_punct_rota rotates through the desired punctuation marks.

• Install the keyboard package found at https://jahorton.github.io/english_punct_rota.kmp.

  It may be installed via the following QR link as well:

  

1. Erase all text.
2. Tap the . key four times, rotating through ., ,, and ?, landing on !.
3. Type the following text: Testing is important . (Include the space mark.)
4. Repeat step 2 above and verify that only the punctuation mark is altered.

TEST_MULTITAP_OVERWRITE: Using a keyboard with a multitap punctuation rota key, check if punctuation typed immediately after applying a suggestion erases the whitespace, even on repeated taps.

• Install the keyboard package found at https://jahorton.github.io/english_punct_rota.kmp.

  It may be installed via the following QR link as well:

  

• Type testi

• Manually apply the testing suggestion.

• Then tap the . key four times, rotating through ., ,, and ?, landing on !.

• Verify that the results are testing., testing,, testing?, and the final result is testing!, with no appended whitespace.

TEST_MULTITAP_APPLICATION: Using a keyboard with a multitap punctuation rota key, check if punctuation typed with a highlighted suggestion applies it and appears afterward, even on repeated taps.

• Install the keyboard package found at https://jahorton.github.io/english_punct_rota.kmp.

  It may be installed via the following QR link as well:

  

• Type testi and verify that it is selected / highlighted. (If it isn't, delete its text and retype - tap accuracy may help.)

• Then tap the . key four times, rotating through ., ,, and ?, landing on !.

• Verify that the results are testing., testing,, testing?, and the final result is testing!, with no appended whitespace.

TEST_SUGGESTIONS_ON_BKSP: Using Keyman for Android, verify that reasonable suggestions are displayed for text after using BKSP to reach the end of a previous word.

• Type term
• Type
• Tap backspace once.
• Multiple suggestions should be displayed, all based on 'term'.

TEST_DELAYED_ON_WORD: Using the sil_euro_latin keyboard, check if deleting characters up to the end of an applied suggestion's main body displays a reversion.

• Type trials .
• Tap the and suggestion.
• Type testi, then tap space to automatically apply it. (Restart this test if it is not automatically highlighted somehow.)
• Type ab.
• Delete ab so that you end up immediately after testing.
• Verify that a "testi" reversion is displayed.
• Apply it and verify that the full text reverts to trials and testi.
• Verify that the original suggestions are displayed, but no suggestion is automatically selected / highlighted.
• Type 'a' and verify that no errors, notifications, or warnings are displayed.

TEST_DELAYED_ON_SPACE: Using the sil_euro_latin keyboard, check if deleting characters up to the end of an applied suggestion's main body displays a reversion.

• Type trials .
• Tap the and suggestion.
• Type testi, then tap space to automatically apply it. (Restart this test if it is not automatically highlighted somehow.)
• Type ab.
• Delete ab so that you end up immediately after the whitespace after testing.
• Verify that a "testi" reversion is displayed.
• Apply it and verify that the full text reverts to trials and testi.
• Verify that the original suggestions are displayed, but no suggestion is automatically selected / highlighted.
• Type 'a' and verify that no errors, notifications, or warnings are displayed.

TEST_DOUBLE_DELAYED: Using the sil_euro_latin keyboard, check if two suggestions can be reverted in rapid succession.

• With the context empty, type that.
• Type a p, then apply the point suggestion.
• Apply the of suggestion, giving you that point of .
• Tap backspace and verify that a "" suggestion is displayed.
• Tap the "" suggestion and verify that the text becomes that point .
• Tap backspace again and verify that a "p" suggestion is displayed.
• Tap the "p" suggestion and verify that the text becomes that p.

TEST_ROBUSTNESS: Spend at least 5 minutes trying to "break" predictive text and/or cause it to work improperly. Report back on any issues discovered.

[keymanapp-test-bot]

User Test Results

Test specification and instructions

• ⬜ TEST_PRED_CHAIN_APPLY (OPEN)
• ⬜ TEST_PRED_TEXT_SUGGEST_ANDROID (OPEN)
• ⬜ TEST_CLEAR_CONTEXT (OPEN)
• ⬜ TEST_PRED_LONG_TEXT_REVERSIONS (OPEN)
• ⬜ TEST_PERIOD_APPLY_OVERWRITE (OPEN)
• ⬜ TEST_SPACE_APPLY_OVERWRITE (OPEN)
• ⬜ TEST_PERIOD_APPLICATION (OPEN)
• ⬜ TEST_SPACE_APPLICATION (OPEN)
• ⬜ TEST_PERIOD_OVERWRITE_REVERSION (OPEN)
• ⬜ TEST_PERIOD_APPLICATION_REVERSION (OPEN)
• ⬜ TEST_PERIOD_APPLICATION_DELAYED_REVERSION (OPEN)
• ⬜ TEST_MULTITAPS (OPEN)
• ⬜ TEST_MULTITAP_OVERWRITE (OPEN)
• ⬜ TEST_MULTITAP_APPLICATION (OPEN)
• ⬜ TEST_SUGGESTIONS_ON_BKSP (OPEN)
• ⬜ TEST_DELAYED_ON_WORD (OPEN)
• ⬜ TEST_DELAYED_ON_SPACE (OPEN)
• ⬜ TEST_DOUBLE_DELAYED (OPEN)
• ⬜ TEST_ROBUSTNESS (OPEN)

Results Template

# Test Results

* **TEST_PRED_CHAIN_APPLY (OPEN):** notes
* **TEST_PRED_TEXT_SUGGEST_ANDROID (OPEN):** notes
* **TEST_CLEAR_CONTEXT (OPEN):** notes
* **TEST_PRED_LONG_TEXT_REVERSIONS (OPEN):** notes
* **TEST_PERIOD_APPLY_OVERWRITE (OPEN):** notes
* **TEST_SPACE_APPLY_OVERWRITE (OPEN):** notes
* **TEST_PERIOD_APPLICATION (OPEN):** notes
* **TEST_SPACE_APPLICATION (OPEN):** notes
* **TEST_PERIOD_OVERWRITE_REVERSION (OPEN):** notes
* **TEST_PERIOD_APPLICATION_REVERSION (OPEN):** notes
* **TEST_PERIOD_APPLICATION_DELAYED_REVERSION (OPEN):** notes
* **TEST_MULTITAPS (OPEN):** notes
* **TEST_MULTITAP_OVERWRITE (OPEN):** notes
* **TEST_MULTITAP_APPLICATION (OPEN):** notes
* **TEST_SUGGESTIONS_ON_BKSP (OPEN):** notes
* **TEST_DELAYED_ON_WORD (OPEN):** notes
* **TEST_DELAYED_ON_SPACE (OPEN):** notes
* **TEST_DOUBLE_DELAYED (OPEN):** notes
* **TEST_ROBUSTNESS (OPEN):** notes

Test Artifacts

• Android
  • Keyman for Android apk
  • FirstVoices Keyboards for Android apk
  • FirstVoices Keyboards for Android apk (old PRs)
  • KeyboardHarness apk
  • Keyman for Android apk (old PRs)
  • KMSample1 apk
  • KMSample2 apk
• iOS
  • Keyman for iOS (simulator image)
  • FirstVoices Keyboards for iOS (simulator image)
  • FirstVoices Keyboards for iOS (simulator image) (old PRs)
  • Keyman for iOS (simulator image) (old PRs)
  • TestFlight internal PR build version - 19.0.137 (0.15141.14892)
• Web
  • KeymanWeb Test Home