CI: Comprehensive UI screenshot walkthrough with data seeding#59
Merged
Conversation
Turn the Android UI Screenshot workflow into a verification gallery of every screen in the app, each shown with realistic data. - Pre-seed the app's SharedPreferences before launch (via run-as, SELinux-safe): a curated set of home apps picked from packages actually installed on the emulator, and a rich notes history (text, a done to-do, an urgent reminder, an image and a voice memo). First-run flags are flipped off so no onboarding dialog hides the UI. seed_data.py builds both prefs XML files; find_node.py lets the driver tap real on-screen elements located from a live `uiautomator dump` instead of hard-coded coordinates. - Walk and screenshot 24 states across Home, App drawer (list, live search, per-app long-press menu, inline rename), Settings (home/appearance/DND/ gestures sections plus their sub-selectors), Notes (list, actions menu, inline edit, full-screen image, search, empty state), and a couple of live theme/alignment variations. - Publish step renders the gallery into the run's Job Summary grouped by section (images embed inline via the ci-screenshots branch). - Also run on PRs into main/master so the gallery shows up during review; bump the job timeout to 60m for the longer walkthrough. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PTbv7mVssh3FXvhdVAMnzN
The first run of the walkthrough captured the home screen for nearly every step. Three root causes, all fixed here: 1. Seeding silently failed. `adb shell run-as PKG sh -c 'cat > file'` has adb flatten argv into one string, so the device shell (uid shell, cwd /) ran the `>` redirect instead of the app-context sh — "No such file or directory", and nothing got seeded. The app ran in first-run state with default apps and no notes. Fixed by passing the whole run-as invocation as one double-quoted arg (new appwrite/appwrite_b64 helpers) so the redirect stays inside the single quotes and runs in the app's data dir. 2. Navigation never happened. `input swipe … 120ms` wasn't detected as a fling, so swipe-up/left didn't open the drawer/notes (long-press worked, hence Settings did). Replaced with open_drawer/open_notes that swipe and then verify via uiautomator, retrying across several durations. 3. Variation + option taps hit off-screen targets. Inline selectors render below low settings rows; reveal_row now scrolls the row up first, and the home theme/alignment variations are produced by re-seeding prefs and restarting (seed_main) instead of driving the off-screen selectors. seed_data.py gains optional theme/alignment/apps-count overrides for the re-seeded variation screenshots. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PTbv7mVssh3FXvhdVAMnzN
… drawer Seeding works now, but reviewing the captured images showed the walkthrough still misfired in several places. Root causes and fixes: - Notes section (every shot was actually the home screen): swipe-left never fires onSwipeLeft on this emulator. Open the Notes page deterministically via MainActivity's ACTION_SEND handler (`am start -a SEND -t text/plain`), which drops shared text onto the notes page and navigates there — no gesture. - App search launched Google Calendar instead of filtering: AppDrawerAdapter auto-launches when the query narrows to exactly one app. Search "c" (matches several) instead of "ca" (only Calendar). - Settings pickers never expanded: each row's click handler is on the *value* view (homeAppsNum, dateTime, iconShape, alignment, appThemeText, dndDuration, swipeLeftAction, tvGestures), not the label I was tapping. New expand_setting taps the value id; find_node now prefers exact id matches (so "alignment" doesn't hit "alignmentLeft"). - Drawer menu/rename shots were home: the drawer didn't reopen after the search excursion. Restart fresh before reopening (a cold launch is the one state where the swipe-up fling reliably registers). - Home slot 4 showed "App": com.android.camera2 has no launchable activity under -camera-back none. Dropped Camera from the preferred home apps. - Image shot opened the notification shade: the reveal swipe was a home swipe-down; scroll within the notes list instead. Also drop the notes empty-state shot (couldn't be reached without the broken swipe) and add diag() logging on navigation misses. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PTbv7mVssh3FXvhdVAMnzN
…ImageMagick Viewing run #3: home, drawer, search, all Settings pickers, the notes list, notes search and both variations now render correctly. Three things still misfired: - App-menu/rename shots were the home screen: the swipe-up fling won't register after a restart, so reopening the drawer failed. open_drawer now falls back to long-pressing a home app (onLongClick -> showAppList), which opens the same searchable app list reliably and without the single-match auto-launch. - Note action dialog never opened (long-press selected the note text instead): notesText is textIsSelectable while the row's long-click is on the root view. New longpress_note presses the row gap just right of the bubble (find_node gained --bounds) so the root long-click fires -> Copy/Share/Edit/Delete. - Image note was missing ("9 entries", no thumbnail): the runner has no ImageMagick, so the sample PNG was never created and the image entry was skipped. Generate it with pure-stdlib Python (make_png.py) and log the pushed size to confirm it lands. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PTbv7mVssh3FXvhdVAMnzN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Overhaul the Android UI screenshot CI pipeline to capture a complete, data-driven walkthrough of every screen, sub-menu, and theme variation. The new approach pre-seeds realistic data (home apps, notes history with mixed media) directly into the app's storage, then drives the full UI with resolution-independent gesture automation, producing a gallery that doubles as verification testing.
Key Changes
Enhanced
ui_screenshots.sh: Expanded from a basic 5-screen walkthrough to a comprehensive 24-screenshot gallery covering:New
seed_data.py: Generates pre-seeded SharedPreferences XML files:New
find_node.py: UIAutomator-based node locator for resolution-independent taps:uiautomator dumpXML to find nodes by text, content-desc, resource-id, or classGesture automation improvements:
tap()andlongpress()helpers that locate elements via UIAutomator before tappingscroll_to_text()for navigating long settings pagestype_text()for search inputset -eto allow best-effort walkthrough (one failed tap doesn't abort the gallery)Data seeding workflow:
adb shell run-asto respect app security contextManifest format: Updated from 2-column to 3-column TSV (
filename | section | caption) to group screenshots by feature area in the published galleryWorkflow updates (
android-ui-screenshots.yml):Gallery publishing (
publish_screenshots.sh):Implementation Details
adb shell run-asto preserve app ownership and SELinux labelsThe gallery now serves dual purpose: a visual showcase of every feature and a regression test that verifies the UI renders correctly with data.
https://claude.ai/code/session_01PTbv7mVssh3FXvhdVAMnzN