Skip to content

Commit 5da04a4

Browse files
angusbezzinaclaude
andcommitted
feat(ios): marquee frame selection + exclude AnnotKit's overlay window
Port marquee selection to iOS: `IOSElementSource` now conforms to `MarqueeTargetSource`, walking the UIView tree once from a single window root to build `[MarqueeCandidate]` and resolving with the shared, pure `MarqueeTargetRule` — so an identical drag over an identical layout resolves identically on both platforms by construction, not by two implementations being kept in step by hand. Returns the same target-first, broadest-last ladder `componentLadder(at:)` produces, so the session's widening and the note's `component` field work unchanged. Also excludes AnnotKit's own overlay from `IOSElementSource.windows()` by `PassThroughWindow` TYPE identity. Unlike macOS (a separate NSPanel already unreachable via kAXWindows), the iOS overlay is a UIWindow in the HOST's scene sharing its pid, and its chrome is genuinely identified and meaningful — a marquee spans the area it draws across, so a large overlay surface could win the rule's first pass outright and bind the user's note to our own UI. Filtering in the shared helper makes snapshot, hitTest, keyWindow, componentLadder and the marquee path agree. Verified: `xcrun --sdk iphoneos swiftc -typecheck -target arm64-apple-ios17.0 -swift-version 6` clean (no warnings), and a Mac Catalyst build (the os(iOS) path) succeeds. macOS unaffected: 99 tests green, AnnotKitOverlayProbe all-PASS. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent d94920d commit 5da04a4

2 files changed

Lines changed: 174 additions & 0 deletions

File tree

PARITY.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,9 @@ row; each asymmetry is closed by code or has a tracked mitigation.
1313
| Hit test primitive | `AXUIElementCopyElementAtPosition` + NSView `hitTest` | `UIView.hitTest(_:with:)` | iOS has no global AX point query; uses view hitTest. Tracked: F5.2 |
1414
| Annotation target rule | shared `AnnotationTargetRule` over an AX candidate chain | shared `AnnotationTargetRule` over a UIView candidate chain | none — both build a `[TargetCandidate]` chain and apply the SAME rule (deepest actionable, else deepest meaningful). Closes the earlier split (macOS "deepest meaningful" vs iOS "nearest identified"), cli-got28.2 |
1515
| Component widening | `ComponentLadderSource` (AX chain) | `ComponentLadderSource` (UIView chain) | none — same ladder (target, then enclosing identified components) |
16+
| Marquee selection (drawn frame → element) | `MarqueeTargetSource`: shared `MarqueeTargetRule` over `[MarqueeCandidate]` read from the AX tree | `MarqueeTargetSource`: shared `MarqueeTargetRule` over `[MarqueeCandidate]` read from the UIView tree | none — the DECISION is one pure rule (largest ≥85%-surrounded element, else the tightest enclosing one); the adapters differ only in how they read candidates out of their own tree. Both do ONE walk from a single root so depth (the rule's tie-break) is numbered comparably, both collect the subtree WHOLE (an intersects-the-frame filter would discard the enclosing-pass candidates), and both return the SAME target-first, broadest-last ladder as `ComponentLadderSource`, so the session's widening and `component` field work unchanged from a framed selection |
17+
| Marquee drag threshold | cursor slop (a mouse does not move on a deliberate click) | larger touch slop | ASYMMETRIC BY DESIGN, owned by the drag UI, not the adapters: a finger rolls several points on a deliberate tap, so the macOS threshold on iOS would turn taps into marquees. Below the threshold both platforms route the gesture to the point path (`select(atAXPoint:)`), per the caller contract on `select(inAXRect:)` |
18+
| Overlay excluded from element lookup | AX window identifier (`AXIntrospection.overlayWindowIdentifier`) filtered out of every `kAXWindows` read | `PassThroughWindow` TYPE identity filtered out of `IOSElementSource.windows()` | ASYMMETRIC BY NECESSITY — the hosts are different window kinds. macOS's overlay is a separate `NSPanel` matched by the identifier the controller stamps on it; iOS's is a `UIWindow` in the HOST's scene sharing its pid, so no pid/scene filter separates it and a type check (internal to the module) cannot drift the way an identifier convention can. Both filter in the shared window lookup, so snapshot / hit-test / region-anchor / marquee agree; leaving it in would let a marquee bind the user's note to AnnotKit's own UI |
1619
| Coordinate space | Cocoa bottom-left to AX top-left flip | UIKit top-left native | iOS needs no flip; shared `ScreenSpace` used only on macOS |
1720
| Screenshot | ScreenCaptureKit / `cacheDisplay` | `UIGraphicsImageRenderer` + `drawHierarchy` | both capture own hierarchy only; no cross-window or secure overlays |
1821
| Overlay host | resizing `NSPanel` (toolbar corner idle, full screen annotating) | pass-through `UIWindow` | both interactive; selection via the shared SwiftUI catcher, not a global monitor |

Sources/AnnotKit/iOS/IOSElementSource.swift

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -260,10 +260,30 @@ public final class IOSElementSource: ElementSource, ComponentLadderSource {
260260

261261
// MARK: - Window helpers
262262

263+
/// Every inspectable HOST window — AnnotKit's own overlay is never one.
264+
///
265+
/// On macOS the overlay is a separate `NSPanel` filtered out of `kAXWindows`
266+
/// by identifier; on iOS the risk is live rather than theoretical, because
267+
/// ``PassThroughWindow`` sits in the SAME scene as the host and is returned by
268+
/// `UIWindowScene.windows` like any other window. Its chrome is genuinely
269+
/// identified and genuinely meaningful, so it passes
270+
/// ``TargetCandidate/isEligibleMeaningful`` cleanly: leave it in and a marquee
271+
/// (which by construction spans screen area the overlay draws across) can bind
272+
/// the user's note to AnnotKit's own UI, and a snapshot/selector lists our
273+
/// toolbar as if it were app content.
274+
///
275+
/// Excluded by TYPE identity, not by name, identifier, or window level: the
276+
/// type is internal to this module so the check cannot be defeated by a naming
277+
/// convention drifting, and a pid filter is useless here because AnnotKit runs
278+
/// in the host's process. Filtering in this ONE helper is what makes
279+
/// ``snapshot()``, ``keyWindow``, `hitTest`, `componentLadder(at:)`, and the
280+
/// marquee path agree — matching the macOS side, which excludes the overlay in
281+
/// every window lookup rather than only in the one that motivated it.
263282
private static func windows() -> [UIWindow] {
264283
UIApplication.shared.connectedScenes
265284
.compactMap { $0 as? UIWindowScene }
266285
.flatMap(\.windows)
286+
.filter { !($0 is PassThroughWindow) }
267287
}
268288

269289
static var keyWindow: UIWindow? {
@@ -276,4 +296,155 @@ public final class IOSElementSource: ElementSource, ComponentLadderSource {
276296
return window.convert(inWindow, to: nil)
277297
}
278298
}
299+
300+
// MARK: - Marquee (drawn frame -> view)
301+
302+
extension IOSElementSource: MarqueeTargetSource {
303+
/// The ladder for a frame the user DREW, over the `UIView` tree: the view the
304+
/// frame binds to per ``MarqueeTargetRule`` first, then each enclosing
305+
/// identified component, broadest last — the SAME target-first contract as
306+
/// ``componentLadder(at:)``, because the session assumes `ladder[0]` IS the
307+
/// bound target for both widening and the note's `component` field.
308+
///
309+
/// Structurally identical to the macOS path (`AXIntrospection.marqueeLadder`):
310+
/// both platforms only differ in how they read candidates out of their tree,
311+
/// and the decision itself is the one shared pure rule, so an identical drag
312+
/// over an identical layout resolves identically by construction rather than
313+
/// by two implementations being kept in step by hand.
314+
///
315+
/// Cost: one full walk of the hit window's view tree per drag RELEASE — never
316+
/// during the drag and never on touch-move. That rate is what makes a
317+
/// whole-tree walk affordable here, where the point path must stay on the
318+
/// ancestor chain.
319+
public func marqueeLadder(in rect: CGRect) -> [Element] {
320+
// Standardize before anything geometric: a right-to-left / bottom-to-top
321+
// drag arrives with negative extents, where `contains` degenerates and the
322+
// window lookup below would silently find nothing.
323+
let marquee = rect.standardized
324+
guard let root = Self.marqueeRoot(containing: CGPoint(x: marquee.midX, y: marquee.midY)) else { return [] }
325+
326+
// ONE recursive walk from that single root, so every candidate's depth is
327+
// measured from the SAME origin (window = 0, its children = 1, …). Depth is
328+
// the rule's tie-break between geometrically indistinguishable candidates;
329+
// numbering assembled from several differently-rooted traversals would turn
330+
// that tie-break into noise.
331+
//
332+
// The subtree is collected WHOLE — deliberately not pre-filtered to views
333+
// intersecting the drawn frame. The rule's second pass needs the candidates
334+
// whose frames CONTAIN the frame (the user drew INSIDE something), and an
335+
// intersects-the-marquee filter is exactly what discards them.
336+
var views: [UIView] = []
337+
var candidates: [MarqueeCandidate] = []
338+
Self.collectMarqueeCandidates(root, depth: 0, views: &views, candidates: &candidates)
339+
340+
guard let resolution = MarqueeTargetRule.resolve(marquee: marquee, in: candidates) else { return [] }
341+
let target = views[resolution.index]
342+
343+
// The widening rungs are anchored at the TARGET's frame centre, not the
344+
// drawn frame's: a sloppy marquee can spill outside the element it bound
345+
// to, and a container that does not contain the target is not a component
346+
// the user could widen to. Same value the point path passes.
347+
let targetFrame = Self.screenFrame(of: target)
348+
let targetCentre = CGPoint(x: targetFrame.midX, y: targetFrame.midY)
349+
return [Self.element(for: target)]
350+
+ Self.enclosingComponents(of: target, containing: targetCentre).map { Self.element(for: $0) }
351+
}
352+
353+
/// The frontmost visible non-overlay window containing `point`, or nil when the
354+
/// drag happened over nothing of ours.
355+
///
356+
/// `UIWindowScene.windows` has no documented front-to-back order, so the
357+
/// frontmost is derived rather than assumed: highest `windowLevel` first, ties
358+
/// broken by the LATER array position, which is UIKit's own within-level
359+
/// ordering. Getting this wrong on a host that presents an alert or
360+
/// share-sheet window would walk the window BEHIND the one the user is looking
361+
/// at and bind the note to an element they cannot see.
362+
///
363+
/// AnnotKit's own overlay cannot be picked here because ``windows()`` never
364+
/// returns it — see that helper for why the exclusion lives there and not in
365+
/// this method.
366+
private static func marqueeRoot(containing point: CGPoint) -> UIWindow? {
367+
windows()
368+
.filter { !$0.isHidden && $0.alpha > 0.01 }
369+
.enumerated()
370+
.sorted { lhs, rhs in
371+
lhs.element.windowLevel.rawValue == rhs.element.windowLevel.rawValue
372+
? lhs.offset > rhs.offset
373+
: lhs.element.windowLevel.rawValue > rhs.element.windowLevel.rawValue
374+
}
375+
.first { screenFrame(of: $0.element).contains(point) }?
376+
.element
377+
}
378+
379+
/// Depth-first walk collecting a PARALLEL pair per view — the live `UIView` and
380+
/// its pure ``MarqueeCandidate`` — so ``MarqueeTargetRule/Resolution/index``
381+
/// maps straight back to a live view.
382+
///
383+
/// The candidate is built with the same ``candidate(for:)`` the point path
384+
/// uses, so container-root classification — which is what the rule's
385+
/// eligibility filter reads — is identical for a tap and a drag by
386+
/// construction, not by two call sites agreeing today.
387+
///
388+
/// Hidden and effectively transparent subtrees are skipped WHOLE: they keep
389+
/// real frames, so a marquee would happily "surround" a view the user cannot
390+
/// see, and being a large such frame it could win pass 1 outright. The point
391+
/// path gets this filtering free from `UIView.hitTest`, which a whole-tree walk
392+
/// never goes through.
393+
private static func collectMarqueeCandidates(
394+
_ view: UIView,
395+
depth: Int,
396+
views: inout [UIView],
397+
candidates: inout [MarqueeCandidate]
398+
) {
399+
views.append(view)
400+
candidates.append(
401+
MarqueeCandidate(element: candidate(for: view), frame: screenFrame(of: view), depth: depth)
402+
)
403+
guard depth < maxDepth else { return }
404+
for subview in view.subviews where !subview.isHidden && subview.alpha > 0.01 {
405+
collectMarqueeCandidates(subview, depth: depth + 1, views: &views, candidates: &candidates)
406+
}
407+
}
408+
409+
/// The identified components that geometrically ENCLOSE `target` at `point`,
410+
/// smallest-first — the widening rungs above a bound target, mirroring the
411+
/// macOS `enclosingComponents(of:containing:in:)` so a drag widens through the
412+
/// same kind of components on both platforms.
413+
///
414+
/// The scan is GEOMETRIC, not pure ancestry (DECISIONS.md → "Component
415+
/// containment is GEOMETRIC"): a card's identified background surface is
416+
/// routinely a SIBLING of the card's content rather than its ancestor, so it
417+
/// never appears in an ancestor chain. Scanning each ancestor PLUS its direct
418+
/// subviews reaches those surfaces without a second whole-tree walk. Deduped by
419+
/// identifier because the same surface is reachable from several ancestors, and
420+
/// the `>= targetArea` floor keeps a SMALLER identified sibling that merely
421+
/// happens to cover the target's centre out of a ladder that is supposed to
422+
/// only ever widen.
423+
///
424+
/// Container roots are excluded, matching
425+
/// ``AnnotationTargetRule/wideningLadder(in:)`` stopping at the window: the
426+
/// window encloses everything, so it would be the top rung of every ladder
427+
/// while naming no component an agent could act on.
428+
private static func enclosingComponents(of target: UIView, containing point: CGPoint) -> [UIView] {
429+
let targetFrame = screenFrame(of: target)
430+
let targetArea = targetFrame.width * targetFrame.height
431+
var containers: [(view: UIView, area: CGFloat)] = []
432+
var seen = Set<String>()
433+
for ancestor in ancestorChain(from: target) {
434+
for view in [ancestor] + ancestor.subviews {
435+
guard !(view is UIWindow), view !== target else { continue }
436+
guard !view.isHidden, view.alpha > 0.01 else { continue }
437+
let identifier = view.accessibilityIdentifier ?? ""
438+
guard !identifier.isEmpty, !seen.contains(identifier) else { continue }
439+
let frame = screenFrame(of: view)
440+
let area = frame.width * frame.height
441+
guard frame.width > 0, frame.height > 0, frame.contains(point), area >= targetArea else { continue }
442+
seen.insert(identifier)
443+
containers.append((view, area))
444+
}
445+
}
446+
containers.sort { $0.area < $1.area }
447+
return containers.map(\.view)
448+
}
449+
}
279450
#endif

0 commit comments

Comments
 (0)