Automatically filter out duplicate-sync sleep data - #58
Draft
gsbernstein wants to merge 9 commits into
Draft
Conversation
Detects same-source, same-night HealthKit sleep samples that overlap in
time -- the signature of a source (e.g. Oura) re-syncing a night it
already wrote -- and surfaces a small warning button next to the
affected source's row in the sleep history list.
Tapping it opens a cleanup sheet with:
- a timeline of the overlapping entries, split live into "keep" and
"delete" lanes
- a histogram of when the entries were added to HealthKit ("Date
Added"), with a draggable divider defaulting to the largest gap
between sync batches
- a live preview of how many entries/hours would be kept vs. deleted
- a submit button that deletes only the older, superseded batch
"Date Added" isn't exposed by any public HKSample API, so it's read
via HealthKit's internal creationTimestamp using Key-Value Coding,
guarded by a small Objective-C safe-KVC trampoline (KVCSafeAccessor)
so an undefined key degrades to nil instead of crashing.
HealthKitManager gains deleteDuplicateSamples(_:), which requests
write access to sleep analysis (previously only needed by DEBUG fake
data helpers) and deletes by sample UUID.
Co-authored-by: Greg <gsbernstein@users.noreply.github.com>
Clang's importer maps '+safeValue:forObject:' to 'safeValue(_:for:)' in Swift (dropping the redundant 'Object' since the parameter type is generic id), which broke the call site. Update it to match, and add an explicit NS_SWIFT_NAME so the imported signature can't drift again. Co-authored-by: Greg <gsbernstein@users.noreply.github.com>
SleepSourceComparisonView.shouldShow only renders when there's more than one source (or an excluded one) to compare -- a different concern from duplicate detection. A night with just a single (duplicated) source could fail to render the comparison view at all, hiding the cleanup button with it. Move the warning into its own DuplicateCleanupRow, rendered unconditionally in SleepDayGroup whenever duplicate groups exist for that night, independent of how many sources are present. Co-authored-by: Greg <gsbernstein@users.noreply.github.com>
A debugger dump of HKObject's ivars confirms _creationTimestamp is a raw double stored right alongside _startTimestamp/_endTimestamp, not an NSDate -- so 'as? Date' silently failed for every sample, making hasUsableCreationData always false and hiding every duplicate group regardless of overlap or source count (explains the reported case where an obviously-duplicated Oura night showed no warning at all: 12h15m of stage data crammed into a 10h12m bed-to-wake window). Cast to NSNumber instead, and convert the raw interval using Foundation's reference-date epoch (confirmed by the dump: the creation timestamp's magnitude matches _startTimestamp's), with a plausibility check against 'now' and a Unix-epoch fallback in case a future OS version changes the convention. Co-authored-by: Greg <gsbernstein@users.noreply.github.com>
requestAuthorization() (called once, at app launch) now requests both read and share/write access to sleep analysis in a single prompt, instead of only read. HKHealthStore.requestAuthorization can silently fail to present its system sheet when triggered from a context nested inside an already-presented modal -- exactly the situation the delete button in DuplicateCleanupSheet was in (a sheet presented from SleepDayGroup inside the main scroll view), since requireWriteAuthorization(for:) was calling it lazily at delete time. That's the most likely reason tapping Delete appeared to do nothing: the completion for the write prompt may simply never have fired. Asking for write access up front means the authorization decision already exists (granted or denied) by the time the user reaches the nested sheet, so requireWriteAuthorization(for:)'s later call resolves instantly with no UI to present -- avoiding the nested-modal presentation entirely for the common case. Co-authored-by: Greg <gsbernstein@users.noreply.github.com>
… source's samples The delete button appeared to work (no error, sheet dismissed) but never actually removed anything for real duplicates. The cause isn't authorization -- it's a hard, unbypassable HealthKit platform rule: HKHealthStore.deleteObjects(of:predicate:) only deletes objects the calling app itself wrote. Since duplicate samples always come from the other source (e.g. Oura, never Bedger), the delete call matched zero of Bedger's own objects and silently 'succeeded' with nothing deleted -- exactly the reported symptom. - DuplicateCandidateSample now carries sourceBundleID/sourceName so ownership can be checked per-sample. - HealthKitManager.deleteDuplicateSamples checks ownership up front and throws a new ForeignSourceDeletionError with a clear, actionable message instead of attempting a delete that would quietly match nothing. - DuplicateCleanupSheet now detects this case (group.sourceBundleID != Bundle.main.bundleIdentifier, true for every real-world duplicate) and swaps the Delete button for a manual-deletion notice plus an 'Open Health App' shortcut, while still using the same timeline/histogram to show exactly which entries to remove. Direct deletion only remains possible for the rare case where Bedger itself wrote the duplicate samples (e.g. debug data). - Updated README to describe this limitation accurately. Co-authored-by: Greg <gsbernstein@users.noreply.github.com>
HealthKit fundamentally can't let a third-party app delete samples another app wrote (see the previous commit's writeup), so the whole 'detect + review + delete' UI can never actually work for the real case (Oura-written duplicates) -- only for the near-impossible case where Bedger wrote the duplicate samples itself. Rather than ship a feature that can only offer 'go delete it yourself in the Health app,' revert it entirely: DuplicateSleepDetector, the cleanup sheet/ histogram/timeline/row views, the KVC-based creationTimestamp reader and its Obj-C bridging support, and the write-authorization/delete plumbing in HealthKitManager. Kept SleepSession.dateForGrouping's static extraction -- a harmless, reusable refactor that upcoming duplicate-filtering work (deduplicating overlapping samples in the app's own calculations, rather than trying to delete them from HealthKit) will also want. Next: improve in-app filtering so overlapping same-source duplicate samples don't skew sleep duration/stats, instead of trying to delete them. Co-authored-by: Greg <gsbernstein@users.noreply.github.com>
…ete it Adds SleepSampleDeduplicator: for each (night, source) group of raw sleep samples, detects same-source time overlaps -- which never happen within a single sync, since stage segments are sequential -- and, for any overlapping span, keeps only the most complete generation (the one covering the most total sleep), dropping the rest. A re-sync's non-overlapping new tail, and any night with no overlap at all, passes through untouched. This replaces the reverted delete-based cleanup feature: since HealthKit can only let an app delete samples it wrote itself, and real duplicates always come from another source (e.g. Oura), deletion could never actually work for the real case. Filtering the app's own calculations sidesteps that restriction entirely -- Bedger never needs write access, and a double-synced night no longer inflates Sleep Bank totals, per-night durations, or per-source comparisons. Wired into HealthKitManager.processSleepSamples, ahead of both allSleepSessions and the source-filtered sleepSessions, so every downstream consumer benefits without needing its own dedup logic. Preserves the original reverse-chronological ordering sessions are fetched in, which SleepDayGroup/LastNightCard rely on for wake/bed times. Co-authored-by: Greg <gsbernstein@users.noreply.github.com>
…h sync to drop The previous version picked whichever overlapping 'generation' covered more total sleep -- but that biases toward whichever sync happens to produce longer stage segments, which isn't necessarily the newer one. Switch to the same signal a person would use looking in the Health app: 'Date Added.' Restores the KVC-based HealthKitCreationDateReader (with its safe Objective-C accessor and bridging header) from the reverted delete-feature branch -- this time used purely for internal filtering, never surfaced in any UI or write path, so the risk profile of relying on an undocumented property is much lower: if it ever stops working, filtering just silently no-ops for that sample instead of anything user-visible breaking. SleepSampleDeduplicator now operates on raw HKCategorySamples (before they become SleepSessions, since 'creation date' isn't otherwise needed downstream) and, for a same-source night that shows overlap, uses the largest gap between distinct creation timestamps across the whole group -- not just the contested overlap span -- as the cutoff, dropping every sample created before it. Samples with no readable creation date, or groups without enough distinct timestamps to place a meaningful cutoff, are always kept. No new HealthKit authorization needed -- this only reads a property on samples already covered by existing read access. Co-authored-by: Greg <gsbernstein@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Some sources (notably Oura) have a "double sync" bug: syncing a night, then falling back asleep and syncing again, re-writes the same stretch of stage samples a second time instead of only writing the newly-recorded tail. Left alone, that inflates every duration total the app computes (Sleep Bank balance, per-night totals, source comparisons) by roughly 2x for the affected night.
History
This PR originally added an in-app UI to detect these duplicates and delete them from HealthKit on confirmation (a histogram + timeline + draggable divider + delete button). That turned out to be fundamentally impossible:
HKHealthStore.deleteObjects(of:predicate:)(anddelete(_:)) are documented to only ever delete objects the calling app itself wrote — a third-party app can never delete another app's HealthKit samples, no matter what write authorization it has. Since real duplicates always come from the other source (Oura), not Bedger, that delete call matched zero of Bedger's own objects and silently "succeeded" with nothing actually removed.That delete-based feature (detection UI, histogram/timeline views, and the write-authorization/delete plumbing in
HealthKitManager) has been reverted. In its place,SleepSampleDeduplicatorfilters the older, superseded sync out of Bedger's own calculations instead of trying to touch HealthKit at all.An earlier version of this picked which sync to drop by comparing total covered duration of overlapping "generations" — but that biases toward whichever sync happens to produce longer/coarser stage segments, not necessarily the newer one. It's now based on "Date Added" instead — the same signal a person would use looking at a sample in the Health app.
What this PR does now
HealthKitCreationDateReaderreads a sample's internalcreationTimestampvia undocumented Key-Value Coding (guarded by a small Objective-C safe-KVC trampoline,KVCSafeAccessor, wired in via a bridging header) — the only way to know when HealthKit received a sample, sincestartDate/endDateonly describe when the sleep happened. This is the same reader used (and already debugged) in the original delete-feature attempt, now used purely for internal filtering — never surfaced in any UI or write path, so if it ever stops working on some future OS, filtering just silently no-ops for that sample instead of anything user-visible breaking. No HealthKit write authorization needed; it only reads a property on samples already covered by existing read access.SleepSampleDeduplicator.deduplicate(_:)groups rawHKCategorySamples by (night, source). For a group that shows the structural signature of a duplicate re-sync — same-source samples overlapping in time, which never happens within a single healthy sync since stage segments are sequential — it finds the largest gap between distinct creation timestamps across the whole group (not just the contested overlap span) and drops every sample created before that cutoff. Samples with no readable creation date, or groups without enough distinct timestamps to place a meaningful cutoff, are always kept — never guessed away.HealthKitManager.processSleepSamples, before samples becomeSleepSessions, so every downstream consumer (Sleep Bank, per-night totals, source comparison view, etc.) benefits automatically. Preserves the original reverse-chronological sample ordering thatSleepDayGroup/LastNightCardrely on for wake/bed times.Notes for verification (macOS/Xcode only — see AGENTS.md)
Bedtime/Bedtime-Bridging-Header.h) is wired into both Debug and Release build settings (SWIFT_OBJC_BRIDGING_HEADER) inproject.pbxproj; the.h/.m/bridging-header files live under the existing file-system-synchronizedBedtime/group so Xcode should pick them up automatically without a manual "Add Files" step.DebugDataGenerator, then write an overlapping/extended second batch a bit later under the same source) and confirm the Sleep Bank total andSleepDayGroupduration reflect only the newer batch.