Skip to content

feat: Add pre_extraction_item_counts to EventData for sync duration estimation - #224

Merged
gasperzgonec merged 4 commits into
mainfrom
ASFND-360/propagate-counts
Aug 12, 2026
Merged

feat: Add pre_extraction_item_counts to EventData for sync duration estimation#224
gasperzgonec merged 4 commits into
mainfrom
ASFND-360/propagate-counts

Conversation

@i-arnavra3

@i-arnavra3 i-arnavra3 commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Description

Adds an optional pre_extraction_item_counts field to EventData so ADaaS snap-ins can report per-record-type counts on the MetadataExtractionDone event. Adds the ItemTypeCount interface and ItemInputType enum (main or users) and exports both. The field is optional, so existing snap-ins are unaffected.

Design doc: ADaaS estimated duration extension

Connected Issues

ASFND-360

Checklist

  • Tests added/updated and ran with npm run test OR no tests needed.
  • Ran backwards compatibility tests with npm run test:backwards-compatibility.
  • Code formatted and checked with npm run lint.
  • Tested airdrop-template linked to this PR.
  • Documentation updated and provided a link to PR / new docs OR no docs needed.

…stimation

Add ItemInputType enum ('main'/'users') and ItemTypeCount interface, and an
optional pre_extraction_item_counts field on EventData so ADaaS snap-ins can
report per-record-type counts on the metadata-done event. The field is optional
and passes through emit unchanged, so existing snap-ins are unaffected.

ASFND-360

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@radovanjorgic
radovanjorgic marked this pull request as draft July 22, 2026 08:52
Aakkash-Suresh
Aakkash-Suresh previously approved these changes Aug 4, 2026

@Aakkash-Suresh Aakkash-Suresh left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gasperzgonec

Copy link
Copy Markdown
Contributor

This was tested with the airdrop-template, and it returns this during the METADATA_EXTRACTION_DONE, which seems correct.

  event_data: {
    pre_extraction_item_counts: [
      {
        record_type: 'todos',
        count: 15,
        model_input_type: 'main'
      },
      {
        record_type: 'users',
        count: 10,
        model_input_type: 'users'
      }
    ],
    artifacts: [
      {
        id: 'don:core:dvrv-us-1:devo/1smXA7GSnn:artifact/97012',
        item_type: 'external_domain_metadata',
        item_count: 1
      }
    ]
  },

No errors or warnings, so platform seems to be accepting this on the QA.

This, however, still fails on PROD (METADATA_EXTRACTION_ERROR) with the following message: Invalid field: pre_extraction_item_counts.

@gasperzgonec

Copy link
Copy Markdown
Contributor

This has been promoted to PROD environment and should be ready to be promoted in the SDK as well.

@gasperzgonec
gasperzgonec marked this pull request as ready for review August 10, 2026 12:37
@gasperzgonec
gasperzgonec merged commit 23a5b86 into main Aug 12, 2026
9 checks passed
@gasperzgonec
gasperzgonec deleted the ASFND-360/propagate-counts branch August 12, 2026 10:14
radovanjorgic added a commit that referenced this pull request Aug 14, 2026
The behavioral half of v2. Everything here can break a connector.

Adapters:
- WorkerAdapter is split into ExtractionAdapter and LoadingAdapter over
  a shared BaseAdapter, so each phase only carries the surface it needs
- processTask is split into processExtractionTask and processLoadingTask

Emit contract:
- a worker's task returns a TaskResult
  ({ status: 'success' | 'progress' | 'delay' | 'error' }) and the SDK
  emits from it in emitFromResult
- adapter.emit is now protected; connectors no longer emit by hand, and
  double-emitting is no longer possible

State:
- split into base, extraction, and loading state, mirroring the adapters

Public API:
- the root barrel now names every export explicitly instead of
  re-exporting whole modules, so the public surface is one reviewable
  file and internals stay internal
- formatAxiosError and serializeAxiosError are replaced by a single
  serializeError

Carries #224 (pre_extraction_item_counts) over to the new contract.
Since emit is no longer public, the counts are supplied through the
ExtractionAdapter.preExtractionItemCounts setter, mirroring the existing
artifacts accessor, and the SDK attaches them to the metadata-done
event. Covered by a dummy-connector test asserting the emitted body.

Tests are rewritten against the new adapters and state.

BREAKING CHANGE: connectors must migrate to the split adapters and
process functions, return a TaskResult from their task instead of
emitting, and adopt the split state. Symbols that were previously
reachable through `export *` but are not in the new barrel are no
longer importable from the package root. Connectors that reported
pre_extraction_item_counts via adapter.emit must set
adapter.preExtractionItemCounts instead. See MIGRATION.md.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants