Skip to content

Automatically filter out duplicate-sync sleep data - #58

Draft
gsbernstein wants to merge 9 commits into
masterfrom
cursor/duplicate-sleep-cleanup-f961
Draft

Automatically filter out duplicate-sync sleep data#58
gsbernstein wants to merge 9 commits into
masterfrom
cursor/duplicate-sleep-cleanup-f961

Conversation

@gsbernstein

@gsbernstein gsbernstein commented Aug 24, 2026

Copy link
Copy Markdown
Owner

What

Some sources (notably Oura) have a "double sync" bug: syncing a night, then falling back asleep and syncing again, re-writes the same stretch of stage samples a second time instead of only writing the newly-recorded tail. Left alone, that inflates every duration total the app computes (Sleep Bank balance, per-night totals, source comparisons) by roughly 2x for the affected night.

History

This PR originally added an in-app UI to detect these duplicates and delete them from HealthKit on confirmation (a histogram + timeline + draggable divider + delete button). That turned out to be fundamentally impossible: HKHealthStore.deleteObjects(of:predicate:) (and delete(_:)) are documented to only ever delete objects the calling app itself wrote — a third-party app can never delete another app's HealthKit samples, no matter what write authorization it has. Since real duplicates always come from the other source (Oura), not Bedger, that delete call matched zero of Bedger's own objects and silently "succeeded" with nothing actually removed.

That delete-based feature (detection UI, histogram/timeline views, and the write-authorization/delete plumbing in HealthKitManager) has been reverted. In its place, SleepSampleDeduplicator filters the older, superseded sync out of Bedger's own calculations instead of trying to touch HealthKit at all.

An earlier version of this picked which sync to drop by comparing total covered duration of overlapping "generations" — but that biases toward whichever sync happens to produce longer/coarser stage segments, not necessarily the newer one. It's now based on "Date Added" instead — the same signal a person would use looking at a sample in the Health app.

What this PR does now

  • HealthKitCreationDateReader reads a sample's internal creationTimestamp via undocumented Key-Value Coding (guarded by a small Objective-C safe-KVC trampoline, KVCSafeAccessor, wired in via a bridging header) — the only way to know when HealthKit received a sample, since startDate/endDate only describe when the sleep happened. This is the same reader used (and already debugged) in the original delete-feature attempt, now used purely for internal filtering — never surfaced in any UI or write path, so if it ever stops working on some future OS, filtering just silently no-ops for that sample instead of anything user-visible breaking. No HealthKit write authorization needed; it only reads a property on samples already covered by existing read access.
  • SleepSampleDeduplicator.deduplicate(_:) groups raw HKCategorySamples by (night, source). For a group that shows the structural signature of a duplicate re-sync — same-source samples overlapping in time, which never happens within a single healthy sync since stage segments are sequential — it finds the largest gap between distinct creation timestamps across the whole group (not just the contested overlap span) and drops every sample created before that cutoff. Samples with no readable creation date, or groups without enough distinct timestamps to place a meaningful cutoff, are always kept — never guessed away.
  • Wired into HealthKitManager.processSleepSamples, before samples become SleepSessions, so every downstream consumer (Sleep Bank, per-night totals, source comparison view, etc.) benefits automatically. Preserves the original reverse-chronological sample ordering that SleepDayGroup/LastNightCard rely on for wake/bed times.

Notes for verification (macOS/Xcode only — see AGENTS.md)

  • The bridging header (Bedtime/Bedtime-Bridging-Header.h) is wired into both Debug and Release build settings (SWIFT_OBJC_BRIDGING_HEADER) in project.pbxproj; the .h/.m/bridging-header files live under the existing file-system-synchronized Bedtime/ group so Xcode should pick them up automatically without a manual "Add Files" step.
  • No new authorization prompts — this only reads a KVC property on already-read data.
  • To exercise this, seed HealthKit with two overlapping batches of stage samples for the same source/night at different times (e.g. write a night via DebugDataGenerator, then write an overlapping/extended second batch a bit later under the same source) and confirm the Sleep Bank total and SleepDayGroup duration reflect only the newer batch.
Open in Web Open in Cursor 

cursoragent and others added 8 commits August 24, 2026 21:34
Detects same-source, same-night HealthKit sleep samples that overlap in
time -- the signature of a source (e.g. Oura) re-syncing a night it
already wrote -- and surfaces a small warning button next to the
affected source's row in the sleep history list.

Tapping it opens a cleanup sheet with:
- a timeline of the overlapping entries, split live into "keep" and
  "delete" lanes
- a histogram of when the entries were added to HealthKit ("Date
  Added"), with a draggable divider defaulting to the largest gap
  between sync batches
- a live preview of how many entries/hours would be kept vs. deleted
- a submit button that deletes only the older, superseded batch

"Date Added" isn't exposed by any public HKSample API, so it's read
via HealthKit's internal creationTimestamp using Key-Value Coding,
guarded by a small Objective-C safe-KVC trampoline (KVCSafeAccessor)
so an undefined key degrades to nil instead of crashing.

HealthKitManager gains deleteDuplicateSamples(_:), which requests
write access to sleep analysis (previously only needed by DEBUG fake
data helpers) and deletes by sample UUID.

Co-authored-by: Greg <gsbernstein@users.noreply.github.com>
Clang's importer maps '+safeValue:forObject:' to 'safeValue(_:for:)' in
Swift (dropping the redundant 'Object' since the parameter type is
generic id), which broke the call site. Update it to match, and add an
explicit NS_SWIFT_NAME so the imported signature can't drift again.

Co-authored-by: Greg <gsbernstein@users.noreply.github.com>
SleepSourceComparisonView.shouldShow only renders when there's more
than one source (or an excluded one) to compare -- a different concern
from duplicate detection. A night with just a single (duplicated)
source could fail to render the comparison view at all, hiding the
cleanup button with it.

Move the warning into its own DuplicateCleanupRow, rendered
unconditionally in SleepDayGroup whenever duplicate groups exist for
that night, independent of how many sources are present.

Co-authored-by: Greg <gsbernstein@users.noreply.github.com>
A debugger dump of HKObject's ivars confirms _creationTimestamp is a
raw double stored right alongside _startTimestamp/_endTimestamp, not
an NSDate -- so 'as? Date' silently failed for every sample, making
hasUsableCreationData always false and hiding every duplicate group
regardless of overlap or source count (explains the reported case
where an obviously-duplicated Oura night showed no warning at all:
12h15m of stage data crammed into a 10h12m bed-to-wake window).

Cast to NSNumber instead, and convert the raw interval using
Foundation's reference-date epoch (confirmed by the dump: the
creation timestamp's magnitude matches _startTimestamp's), with a
plausibility check against 'now' and a Unix-epoch fallback in case a
future OS version changes the convention.

Co-authored-by: Greg <gsbernstein@users.noreply.github.com>
requestAuthorization() (called once, at app launch) now requests
both read and share/write access to sleep analysis in a single
prompt, instead of only read.

HKHealthStore.requestAuthorization can silently fail to present its
system sheet when triggered from a context nested inside an
already-presented modal -- exactly the situation the delete button
in DuplicateCleanupSheet was in (a sheet presented from SleepDayGroup
inside the main scroll view), since requireWriteAuthorization(for:)
was calling it lazily at delete time. That's the most likely reason
tapping Delete appeared to do nothing: the completion for the write
prompt may simply never have fired.

Asking for write access up front means the authorization decision
already exists (granted or denied) by the time the user reaches the
nested sheet, so requireWriteAuthorization(for:)'s later call
resolves instantly with no UI to present -- avoiding the nested-modal
presentation entirely for the common case.

Co-authored-by: Greg <gsbernstein@users.noreply.github.com>
… source's samples

The delete button appeared to work (no error, sheet dismissed) but
never actually removed anything for real duplicates. The cause isn't
authorization -- it's a hard, unbypassable HealthKit platform rule:
HKHealthStore.deleteObjects(of:predicate:) only deletes objects the
calling app itself wrote. Since duplicate samples always come from
the other source (e.g. Oura, never Bedger), the delete call matched
zero of Bedger's own objects and silently 'succeeded' with nothing
deleted -- exactly the reported symptom.

- DuplicateCandidateSample now carries sourceBundleID/sourceName so
  ownership can be checked per-sample.
- HealthKitManager.deleteDuplicateSamples checks ownership up front
  and throws a new ForeignSourceDeletionError with a clear,
  actionable message instead of attempting a delete that would
  quietly match nothing.
- DuplicateCleanupSheet now detects this case (group.sourceBundleID
  != Bundle.main.bundleIdentifier, true for every real-world
  duplicate) and swaps the Delete button for a manual-deletion
  notice plus an 'Open Health App' shortcut, while still using the
  same timeline/histogram to show exactly which entries to remove.
  Direct deletion only remains possible for the rare case where
  Bedger itself wrote the duplicate samples (e.g. debug data).
- Updated README to describe this limitation accurately.

Co-authored-by: Greg <gsbernstein@users.noreply.github.com>
HealthKit fundamentally can't let a third-party app delete samples
another app wrote (see the previous commit's writeup), so the whole
'detect + review + delete' UI can never actually work for the real
case (Oura-written duplicates) -- only for the near-impossible case
where Bedger wrote the duplicate samples itself. Rather than ship a
feature that can only offer 'go delete it yourself in the Health
app,' revert it entirely: DuplicateSleepDetector, the cleanup sheet/
histogram/timeline/row views, the KVC-based creationTimestamp reader
and its Obj-C bridging support, and the write-authorization/delete
plumbing in HealthKitManager.

Kept SleepSession.dateForGrouping's static extraction -- a
harmless, reusable refactor that upcoming duplicate-filtering work
(deduplicating overlapping samples in the app's own calculations,
rather than trying to delete them from HealthKit) will also want.

Next: improve in-app filtering so overlapping same-source duplicate
samples don't skew sleep duration/stats, instead of trying to
delete them.

Co-authored-by: Greg <gsbernstein@users.noreply.github.com>
…ete it

Adds SleepSampleDeduplicator: for each (night, source) group of raw
sleep samples, detects same-source time overlaps -- which never
happen within a single sync, since stage segments are sequential --
and, for any overlapping span, keeps only the most complete
generation (the one covering the most total sleep), dropping the
rest. A re-sync's non-overlapping new tail, and any night with no
overlap at all, passes through untouched.

This replaces the reverted delete-based cleanup feature: since
HealthKit can only let an app delete samples it wrote itself, and
real duplicates always come from another source (e.g. Oura),
deletion could never actually work for the real case. Filtering the
app's own calculations sidesteps that restriction entirely --
Bedger never needs write access, and a double-synced night no
longer inflates Sleep Bank totals, per-night durations, or
per-source comparisons.

Wired into HealthKitManager.processSleepSamples, ahead of both
allSleepSessions and the source-filtered sleepSessions, so every
downstream consumer benefits without needing its own dedup logic.
Preserves the original reverse-chronological ordering sessions are
fetched in, which SleepDayGroup/LastNightCard rely on for wake/bed
times.

Co-authored-by: Greg <gsbernstein@users.noreply.github.com>
@cursor cursor Bot changed the title Add duplicate sleep data cleanup feature Automatically filter out duplicate-sync sleep data Aug 25, 2026
…h sync to drop

The previous version picked whichever overlapping 'generation'
covered more total sleep -- but that biases toward whichever sync
happens to produce longer stage segments, which isn't necessarily
the newer one. Switch to the same signal a person would use looking
in the Health app: 'Date Added.'

Restores the KVC-based HealthKitCreationDateReader (with its safe
Objective-C accessor and bridging header) from the reverted
delete-feature branch -- this time used purely for internal
filtering, never surfaced in any UI or write path, so the risk
profile of relying on an undocumented property is much lower: if it
ever stops working, filtering just silently no-ops for that sample
instead of anything user-visible breaking.

SleepSampleDeduplicator now operates on raw HKCategorySamples
(before they become SleepSessions, since 'creation date' isn't
otherwise needed downstream) and, for a same-source night that
shows overlap, uses the largest gap between distinct creation
timestamps across the whole group -- not just the contested overlap
span -- as the cutoff, dropping every sample created before it.
Samples with no readable creation date, or groups without enough
distinct timestamps to place a meaningful cutoff, are always kept.

No new HealthKit authorization needed -- this only reads a property
on samples already covered by existing read access.

Co-authored-by: Greg <gsbernstein@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants