You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: DECISIONS.md
+143Lines changed: 143 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,6 +10,9 @@ Resolves the open decisions from the plan (planning/annotkit in the cli repo). P
10
10
| Versioning | SemVer, 0.x pre-1.0 | Breaking changes allowed while 0.x; 1.0 marks a stable public API. |
11
11
| Default element source | Accessibility hierarchy | The only strategy that surfaces SwiftUI `accessibilityIdentifier` values. |
12
12
| Annotation target rule | Deepest actionable, else deepest meaningful; anchor the selector to the nearest identifier | One rule on both platforms. Supersedes the earlier macOS "deepest meaningful" and iOS "nearest identified" split. See below. |
13
+
| Marquee target rule | Largest meaningful element ≥85% surrounded; else the tightest element enclosing the drawn frame | Rect selection, the deliberate inverse of the point rule's deepest-wins. See below. |
14
+
| Selection navigation | Bidirectional Parent/Child over one path; descent replays history and only queries the source at the deepest rung | Replaces the one-way "Widen". Prepending the frontier child shifts every rung, so a note's `component` is the first SEEDED rung above the BOUND one. Whether the parent chain stays seeded-only is OPEN. See below. |
15
+
| Frame mode anchoring | The frame the user DREW anchors the overlay until they navigate; the resolved element is NAMED in the composer, not drawn on the canvas | Hover is point-mode-only, gated in the session rather than the view. See below. |
13
16
| Opt-in element source | View tree (NSView/UIView) | Surfaces concrete view class names; richer for AppKit/UIKit hosts. Collapses to hosting views in pure SwiftUI. |
14
17
|`pathname` mapping | Host-supplied route, inferred fallback | A native app has no URL routes; the host sets a route, else infer from the key window title or identifier. |
15
18
| Overlay coverage | Primary screen (MVP) | The overlay covers the primary display; SwiftUI-local points map to AX screen coordinates there. Full multi-display placement is deferred (cli-a99qm.4.2). |
@@ -70,6 +73,146 @@ Selector *anchoring* (`#Card >> …`) still requires a true ancestor because the
70
73
selector may be positional or text-based while the `component` field still names
71
74
the card, so the note locates the right code either way.
72
75
76
+
## Marquee target rule (VRT-cne0.5)
77
+
78
+
The user press-drags a rectangle around what they mean. Given every element the
79
+
adapter can see (a flat array, not an ancestor chain — a marquee sweeps across
80
+
siblings and unrelated subtrees), the note binds by two passes over the
81
+
*standardized* rect, using the same eligibility as the point rule (never the
82
+
window, the application, chrome, or a window-spanning ghost group, and never a
83
+
zero-area frame):
84
+
85
+
1.**Surrounded** — every eligible element the frame covers to ≥85% of that
86
+
element's OWN area; the **largest** wins.
87
+
2.**Enclosing** — nothing was surrounded, so the frame was drawn inside
88
+
something: every eligible element whose frame contains the whole rect; the
89
+
**smallest** wins.
90
+
3. Neither — nil, and the session falls back to a region note anchored near the
91
+
frame, exactly as a point that hit-tests to nothing does.
92
+
93
+
Ties inside a pass: seeded beats unseeded, then shallower (surrounded) / deeper
94
+
(enclosing), then lowest index. Areas compare with exact `==`, no epsilon.
95
+
96
+
**Why largest-wins**, when the point rule is deepest-wins? Because the gestures
97
+
mean opposite things. A click means "this exact spot", so it descends. Drawing a
98
+
box around a card means "I mean this *whole* thing", so it must ascend past the
99
+
labels and buttons the box also swallowed. Same tree, opposite intent — hence a
100
+
separate rule rather than a mode flag on `AnnotationTargetRule`.
101
+
102
+
**Why 0.85 and not strict containment.** A hand-drawn rect clips edges. Users
103
+
drag roughly around a card and routinely shave a corner or slice through a
104
+
trailing chevron; at 1.0 that silently demotes to the enclosing fallback and
105
+
binds the note to the panel instead of the card — the exact failure marquee
106
+
exists to remove. 0.85 absorbs that sloppiness and still sits far above the
107
+
coverage a neighbouring card picks up when a drag merely overlaps its edge.
108
+
109
+
**Why seeded-beats-unseeded exists at all.** It is not a general preference for
110
+
identified elements (that is the mistake the point rule documents above). It is
111
+
narrowly for the coextensive case: `.axCardSurface(id)` hangs the card's
112
+
identifier on a clear `Color.clear` background leaf that is *exactly* the same
113
+
frame as the card's content group. Both are surrounded identically, and only the
114
+
seeded one carries the identifier that locates code. `AXIntrospection.deepestChild`
115
+
already resolves this same pattern by exact equal-area comparison, which is why
116
+
no epsilon is used here — the two frames come from one layout computation, so the
117
+
arithmetic is bit-identical, and an epsilon would instead start collapsing
118
+
genuinely different elements into a seeding decision.
119
+
120
+
**Why the enclosing fallback.** It is the rect generalization of the point-region
121
+
note (rule 5 above): a scribble over a card's padding surrounds nothing, and
122
+
dropping it would be the same lost-click bug `RegionAnchorSource` was added to
123
+
fix. Smallest-wins there because the tightest enclosure is the most specific — a
124
+
scribble inside a card must not resolve to the window-spanning panel that also
125
+
contains it.
126
+
127
+
Depth and index are determinism-only tie-breaks; they exist so the same drag
128
+
always resolves to the same element. The pure decision lives in
129
+
`MarqueeTargetRule` and is unit-tested independent of AX; adapters expose it via
130
+
the optional `MarqueeTargetSource` capability, which returns a component-widening
131
+
ladder identical in contract to `ComponentLadderSource`, so widening and the
132
+
note's `component` field work unchanged.
133
+
134
+
## Selection navigation (VRT-mijf.1)
135
+
136
+
The composer's one-way "Widen" button is replaced by Parent and Child over a
137
+
single path: index 0 is the deepest rung known so far, ascending indices are
138
+
progressively broader, and one index marks the rung the note is bound to.
139
+
140
+
The rename is not cosmetic. "Widen" named the MECHANISM — the highlighted area
141
+
gets bigger — when the act is choosing which component the note is filed against.
142
+
A button that makes things bigger implies no inverse, so a user who overshot, or
143
+
whom the target rule bound coarser than they meant, had nothing to press.
144
+
145
+
**Descent prefers HISTORY over re-querying.** Above the deepest rung, Child just
146
+
steps the index back down what the user climbed. Only AT the deepest rung does it
147
+
ask the source for children, and it then PREPENDS the one it takes, so index 0
148
+
still means "deepest known rung". Re-querying on every press would be less code
149
+
and wrong: the source's answer is a heuristic over a LIVE tree, so a hover state
150
+
resolving or a list reflowing between two presses makes the same key produce a
151
+
different result. Prepending is what makes the round trip hold in BOTH
152
+
directions — after descending to child C, Parent returns to the original target
153
+
and Child returns to C ITSELF rather than re-running the heuristic against a tree
154
+
that has moved on.
155
+
156
+
**The consequence that bit us.** Prepending shifts every existing rung up one, so
157
+
"the note's `component` is the rung above the target" stopped being true: index 1
158
+
is now the ORIGINAL target, which is frequently unseeded. `component` is
159
+
therefore the first SEEDED rung strictly above the BOUND rung, and it is read
160
+
from that rung's IDENTIFIER, never from its `Element.id`. An unseeded element's
161
+
`id` is a slash-joined path (`AXWindow[0]/AXGroup[0]/AXStaticText[1]`); exported
162
+
as a `component` it hands the consuming agent a grep target that matches nothing
163
+
while looking entirely plausible in the note — a silent miss, not a visible one.
164
+
The same path is rooted differently depending on which entry point produced the
165
+
element (`snapshot()` roots at the window, the hit-test and marquee paths at the
166
+
application), so the id is not even stable for one node, which is a second reason
167
+
it can never be a code locator.
168
+
169
+
**Open, pending dogfood: should the parent chain stay seeded-only?** It is today —
170
+
every rung above the target is an identified component, so every rung locates
171
+
code and no press can bind a note to something that names nothing. The cost is
172
+
that it skips structural levels the user can SEE: a row inside an unseeded stack
173
+
offers no rung for the stack, so Parent jumps from the row straight to the card
174
+
and the level the user was aiming at is unreachable. Admitting unseeded rungs
175
+
would fix the navigation and degrade the notes. Which failure is worse is not
176
+
decidable from the design; it needs real use, so this is recorded as unresolved
177
+
rather than settled.
178
+
179
+
## Frame mode anchoring (VRT-mijf.2)
180
+
181
+
When a drawn frame resolves to a real element, the overlay anchors its highlight,
182
+
composer and pin to the FRAME the user drew — not to the element — until the user
183
+
presses Parent or Child, at which point the bound element becomes the anchor and
184
+
the frame stays on screen, dimmed.
185
+
186
+
**Why the frame outranks the resolved element.** The user drew a box, so the box
187
+
is the truth of the selection until they say otherwise. Anchoring to the
188
+
resolution instead makes the rectangle vanish the instant the mouse comes up and
189
+
the highlight snap to a card that was never swept, which reads as the tool having
190
+
ignored the gesture.
191
+
192
+
**Why the element is NAMED rather than DRAWN.** A note must never be captured
193
+
against a target the user could not see, so the binding has to appear somewhere.
194
+
But a second rectangle on the canvas is exactly what "show me only the frame I
195
+
drew" rules out, and two boxes of different shapes leave it ambiguous which one
196
+
the note records. The composer header carries the name behind a `Frame →` prefix,
197
+
so it reads as what the frame RESOLVED to rather than as a label for the
198
+
rectangle, and the prefix disappears the moment navigation puts a named element
199
+
back on the canvas — the name is never qualified in two places at once.
200
+
201
+
**Why navigating reveals the element.** Pressing Parent or Child IS the question
202
+
"which element is this filed against?", so the answer has to become visible;
203
+
moving the binding while the highlight stays on the drawn rect would give no
204
+
feedback at all. The frame survives, weaker, because it is still what the note
205
+
records (`regionRect`) even once it no longer decides the binding.
206
+
207
+
**Why hover is gated in the SESSION, not the view.** Frame mode selects from a
208
+
swept rectangle, so a hover highlight there advertises a click-selection no press
209
+
in that mode can produce — the dogfooding report was a whole card lit up with its
210
+
name tag while nothing had been drawn. The view keeps its own guard for the
211
+
narrower during-the-drag case; the MODE gate belongs one level down because there
212
+
it is unit-testable without a window, no future UI path can reintroduce it, and
213
+
it removes a cross-process AX hit-test per pointer-motion event. It is a cost
214
+
decision as much as a visual one.
215
+
73
216
## IP hygiene (carried into the F7 legal gate)
74
217
75
218
- Do not copy original Agentation source (PolyForm Shield 1.0.0, non-compete). Only the `AGENTATION_NOTES.md` file format is reused, reimplemented clean-room.
Copy file name to clipboardExpand all lines: PARITY.md
+6Lines changed: 6 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -13,6 +13,12 @@ row; each asymmetry is closed by code or has a tracked mitigation.
13
13
| Hit test primitive |`AXUIElementCopyElementAtPosition` + NSView `hitTest`|`UIView.hitTest(_:with:)`| iOS has no global AX point query; uses view hitTest. Tracked: F5.2 |
14
14
| Annotation target rule | shared `AnnotationTargetRule` over an AX candidate chain | shared `AnnotationTargetRule` over a UIView candidate chain | none — both build a `[TargetCandidate]` chain and apply the SAME rule (deepest actionable, else deepest meaningful). Closes the earlier split (macOS "deepest meaningful" vs iOS "nearest identified"), cli-got28.2 |
15
15
| Component widening |`ComponentLadderSource` (AX chain) |`ComponentLadderSource` (UIView chain) | none — same ladder (target, then enclosing identified components) |
16
+
| Marquee selection (drawn frame → element) |`MarqueeTargetSource`: shared `MarqueeTargetRule` over `[MarqueeCandidate]` read from the AX tree |`MarqueeTargetSource`: shared `MarqueeTargetRule` over `[MarqueeCandidate]` read from the UIView tree | none — the DECISION is one pure rule (largest ≥85%-surrounded element, else the tightest enclosing one); the adapters differ only in how they read candidates out of their own tree. Both do ONE walk from a single root so depth (the rule's tie-break) is numbered comparably, both collect the subtree WHOLE (an intersects-the-frame filter would discard the enclosing-pass candidates), and both return the SAME target-first, broadest-last ladder as `ComponentLadderSource`, so the session's widening and `component` field work unchanged from a framed selection |
17
+
| Child navigation (select child) | `ChildNavigationSource` over the AX tree; the opt-in view-tree source implements it over `NSView` | `ChildNavigationSource` over the `UIView` tree | none — the ORDERING is the one pure `ChildNavigationRule` (contains the gesture's hint, then seeded, then larger area, then lowest index) on all three adapters, which supply only `[ChildCandidate]`. All three also share the cost shape the protocol mandates: re-find the bound element by descending its containing window along its own frame centre (bounded by tree DEPTH, not tree size), then collect the NEAREST meaningful descendants, each branch stopping at its first meaningful node — so a SwiftUI host's unidentified layout wrappers are descended THROUGH rather than offered as children, which is what keeps the Child control alive under pure SwiftUI on both platforms. macOS additionally skips window chrome and its own overlay window during that descent; UIKit has no chrome, and the overlay is already excluded by the shared window lookup |
18
+
| Frame-mode anchoring + hover gating | shared `AnnotationSession` (`selectionAnchorFrame`, the `tool == .point` hover gate, `setTool` clearing `hovered`) rendered by the shared `OverlayView`| same | none in the code — all of it is session-level and platform-free, and one SwiftUI view renders it. ASYMMETRIC VERIFICATION, recorded as a gap rather than closed: `AnnotKitOverlayProbe` Phase 8 drives navigation, the note's `component`, the hover gate and the frame anchor against a REAL accessibility tree, and it is macOS-only (`#if os(macOS)`, AppKit + `AXUIElement`), so the iOS adapter's live behaviour is covered only by unit tests over the pure rules. Mitigated, not fixed, by the fact that everything Phase 8 asserts about anchoring and hover lives in the shared session; what remains unverified on iOS is the ADAPTER's candidate collection. Note the hover gate is also moot on touch-only iOS — hover exists there only with a trackpad or pencil — so the reported symptom cannot arise without a pointer |
19
+
| Marquee drag threshold | cursor slop (a mouse does not move on a deliberate click) | larger touch slop | ASYMMETRIC BY DESIGN, owned by the drag UI, not the adapters: a finger rolls several points on a deliberate tap, so the macOS threshold on iOS would turn taps into marquees. Below the threshold both platforms route the gesture to the point path (`select(atAXPoint:)`), per the caller contract on `select(inAXRect:)`|
20
+
| Escape (back out one level) | `NSEvent.addLocalMonitorForEvents(matching: .keyDown)` owned by `OverlayController`, resolving the shared pure `EscapeRule` (drag → card → mode, pass-through when idle) | none — no Escape key exists on iOS | ASYMMETRIC BY THE HARDWARE, not by the code: a touch device has no Escape key, so there is nothing to bind. The DECISION is platform-free and unit-tested (`EscapeRule`), so an iOS back-out affordance (a swipe, a hardware-keyboard binding on iPad) can adopt it without re-deriving the precedence; only the macOS delivery mechanism is `#if os(macOS)`. The mechanism is a LOCAL KEY MONITOR rather than a SwiftUI modifier (`.onExitCommand`) because a panel-scoped modifier only fires while the overlay panel is KEY, and the panel is made key solely by a card focusing its text field — so in annotate mode with nothing open, the state a user most wants to leave, the HOST window is key and no view in the panel ever sees the keystroke. A local monitor works precisely because AnnotKit is in-process with its host: the Escape headed for the host window passes through it first, and it can swallow what it acted on (`EscapeAction.consumesEvent`), which a global monitor cannot. It is installed in `start()` and removed in BOTH `stop()` and `unmount()` — a monitor outliving the overlay would keep eating the host app's own Escape for the life of the process |
21
+
| Overlay excluded from element lookup | AX window identifier (`AXIntrospection.overlayWindowIdentifier`) filtered out of every `kAXWindows` read |`PassThroughWindow` TYPE identity filtered out of `IOSElementSource.windows()`| ASYMMETRIC BY NECESSITY — the hosts are different window kinds. macOS's overlay is a separate `NSPanel` matched by the identifier the controller stamps on it; iOS's is a `UIWindow` in the HOST's scene sharing its pid, so no pid/scene filter separates it and a type check (internal to the module) cannot drift the way an identifier convention can. Both filter in the shared window lookup, so snapshot / hit-test / region-anchor / marquee agree; leaving it in would let a marquee bind the user's note to AnnotKit's own UI |
16
22
| Coordinate space | Cocoa bottom-left to AX top-left flip | UIKit top-left native | iOS needs no flip; shared `ScreenSpace` used only on macOS |
17
23
| Screenshot | ScreenCaptureKit / `cacheDisplay`|`UIGraphicsImageRenderer` + `drawHierarchy`| both capture own hierarchy only; no cross-window or secure overlays |
18
24
| Overlay host | resizing `NSPanel` (toolbar corner idle, full screen annotating) | pass-through `UIWindow`| both interactive; selection via the shared SwiftUI catcher, not a global monitor |
0 commit comments