Upgrade to synapse 0.39.0. Closes #170 by juliangruber · Pull Request #186 · FilOzone/dealbot

juliangruber · 2026-02-03T11:59:15Z

~~Targets FilOzone/synapse-sdk#555 (not installed because currently failing tests)~~
Closes #170

Update to 38
Update to 39

SgtPooki · 2026-02-03T17:13:07Z

@juliangruber we should update filecoin-pin first for sure and then pull in both filecoin-pin and synapse-sdk here at the same time

rjan90 · 2026-02-10T13:58:55Z

Assigning to @SgtPooki to take over this PR, as per Slack thread

juliangruber · 2026-03-17T09:47:19Z

apps/backend/src/ipni/ipni-verification.service.ts

-          } as any,
-        },
+      isActive: storageProvider.isActive,
+      pdp: {


waitForIpniProviderResults() from filecoin-pin expects PDPProviders as input. We here only have StorageProviders, from the deal object. We can fake a PDPProvider, it doesn't matter what we use as .pdp (apparently). Any advice @hugomrdias @rvagg?

@juliangruber filecoin-pin uses the pdp provider details to build the expected multiaddr and then uses name and some other fields for logging/debug output. We can fake it, but we might need to shore those types up in filecoin-pin and here in Dealbot eventually, but I wouldn't take that on in this PR unless we need to

juliangruber · 2026-03-17T09:51:41Z

apps/backend/src/wallet-sdk/wallet-sdk.service.ts

-        const supportsPDP = !!info.products?.PDP;
-        // In order to support ipniIpfs, the provider must have PDP product
-        const supportsIpniIpfs = !!info.products?.PDP?.data?.ipniIpfs;
-        const isDevTagged = info.products?.PDP?.capabilities?.service_status === DEV_TAG;


this tag isn't exported by the PDPprovider type 🤔 Any advice @hugomrdias @rvagg?

i don't know if we need "isDevTagged" anymore honestly.. we shouldn't be testing against them, and isActive/isEndorsed (whatever the endorsed flag is) should be enough for dealbot.. as long as devTagged SPs are never "isActive" then we should be good

juliangruber · 2026-03-17T11:39:48Z

@rjan90 could you add back the copilot review request? I want to see if there's anything valuable in there

Copilot

Pull request overview

This PR upgrades the backend to Synapse SDK 0.39.0 (and filecoin-pin ^0.18.0) and updates Dealbot’s integration to match the new provider types, storage APIs, and upload progress callback/event naming.

Changes:

Update Synapse SDK imports/usage (chain/account setup via viem, new sp-registry/storage APIs, provider model changes).
Rename and propagate upload progress events (onUploadComplete → onStored, onPieceAdded/Confirmed → onPiecesAdded/Confirmed) and align schema fields (piece_* → pieces_*, region → location, IDs → bigint).
Add new DB migrations and update tests/docs/metrics DTOs to reflect the renamed fields and BigInt IDs.

Reviewed changes

Copilot reviewed 34 out of 35 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
docs/checks/events-and-metrics.md	Updates event documentation to reflect renamed upload completion handler/event.
docs/checks/data-storage.md	Updates docs for renamed callback (`onStored`).
apps/backend/test/migrations.e2e-spec.ts	Adds new migrations to the integration test migration list and adjusts rollback steps.
apps/backend/src/wallet-sdk/wallet-sdk.types.ts	Updates wallet/provider types to new Synapse/filecoin-pin models and BigInt-centric structures.
apps/backend/src/wallet-sdk/wallet-sdk.service.ts	Refactors wallet services initialization and provider syncing for Synapse 0.39 APIs and BigInt IDs.
apps/backend/src/wallet-sdk/wallet-sdk.service.spec.ts	Updates tests for BigInt provider IDs and updated provider shape.
apps/backend/src/retrieval-addons/strategies/ipfs-block.strategy.ts	Switches provider URL access to the new `pdp.serviceURL` shape.
apps/backend/src/retrieval-addons/strategies/ipfs-block.strategy.spec.ts	Updates retrieval strategy tests and asserts new per-block TTFB output shape.
apps/backend/src/metrics/utils/check-metric-labels.ts	Changes metric label input `providerId` to `bigint` (still normalized to string labels).
apps/backend/src/metrics/services/failed-deals.service.ts	Updates failed-deal mapping to renamed `piecesAddedTime` field.
apps/backend/src/metrics/dto/provider-performance.dto.ts	Renames API field `region` → `location`.
apps/backend/src/metrics/dto/failed-retrievals.dto.ts	Updates provider ID type to `bigint`.
apps/backend/src/metrics/dto/failed-deals.dto.ts	Updates dataset ID type to `bigint` and renames `pieceAddedTime` → `piecesAddedTime`.
apps/backend/src/jobs/jobs.service.spec.ts	Updates job context expectations for BigInt provider IDs.
apps/backend/src/ipni/ipni-verification.service.ts	Switches expected provider model to `PDPProvider` (filecoin-pin) for IPNI verification.
apps/backend/src/ipni/ipni-verification.service.spec.ts	Updates IPNI verification test data for `location`.
apps/backend/src/dev-tools/dev-tools.service.ts	Aligns “active” flag checks with new provider shape (`isActive`).
apps/backend/src/deal/deal.service.ts	Refactors deal creation/upload flow for new storage context API and renamed upload progress callbacks; updates timing fields.
apps/backend/src/deal/deal.service.spec.ts	Updates deal service tests for new event names, BigInt IDs, and provider shape.
apps/backend/src/deal-addons/strategies/ipni.strategy.ts	Renames addon hook to `onStored`.
apps/backend/src/deal-addons/strategies/ipni.strategy.spec.ts	Updates test fixtures for BigInt IDs and `location`.
apps/backend/src/deal-addons/interfaces/deal-addon.interface.ts	Renames addon interface hook `onUploadComplete` → `onStored`.
apps/backend/src/deal-addons/deal-addons.service.ts	Renames and rewires addon execution from upload-complete to stored hook.
apps/backend/src/database/migrations/1761500000006-DataSetIdBigInt.ts	Adds migration changing `deals.data_set_id` type.
apps/backend/src/database/migrations/1761500000005-ProviderIdBigInt.ts	Adds migration changing `storage_providers.providerId` type.
apps/backend/src/database/migrations/1761500000004-RenameRegionToLocation.ts	Adds migration renaming `region` → `location` in `storage_providers`.
apps/backend/src/database/migrations/1761500000003-RenameEvents.ts	Adds migration renaming `piece__time` columns to `pieces__time` in `deals`.
apps/backend/src/database/entities/storage-provider.entity.ts	Updates entity fields/indexes for `location` and `providerId: bigint`.
apps/backend/src/database/entities/deal.entity.ts	Updates entity `dataSetId` to `bigint` and renames `piece__time` fields to `pieces__time`.
apps/backend/src/data-retention/data-retention.service.ts	Updates provider type references and BigInt provider IDs for metrics/logging.
apps/backend/src/data-retention/data-retention.service.spec.ts	Updates tests to pass BigInt provider IDs to metric label builder.
apps/backend/src/config/app.config.ts	Tightens private key typing to `0x${string}` and updates env casting.
apps/backend/src/common/logging.ts	Updates log-context provider ID typing to `bigint`.
apps/backend/package.json	Bumps `@filoz/synapse-sdk` to `0.39.0`, updates `filecoin-pin`, removes `ethers`.

Comments suppressed due to low confidence (1)

apps/backend/src/common/logging.ts:109

DataSetLogContext.dataSetId is still typed as number, but Synapse/DB dataset IDs are now treated as bigint elsewhere in this PR (e.g., Deal.dataSetId?: bigint). This mismatch makes it easy to accidentally log/pass the wrong type and can reintroduce BigInt JSON serialization issues. Update dataSetId to bigint | string (whichever is canonical) to keep log-context typing consistent.

export type DataSetLogContext = {
  jobId?: string;
  providerAddress: string;
  providerId?: bigint;
  providerName?: string;
  dataSetId?: number;
  dataSetIndex?: number;
  metadata?: Record<string, string>;

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

docs/checks/events-and-metrics.md

apps/backend/test/migrations.e2e-spec.ts

apps/backend/src/ipni/ipni-verification.service.ts

+      pdp: {
+        // TODO
+        serviceURL: storageProvider.serviceUrl,
+        minPieceSizeInBytes: 0n,
+        maxPieceSizeInBytes: 0n,


apps/backend/src/deal/deal.service.ts

apps/backend/src/database/migrations/1761500000006-DataSetIdBigInt.ts

apps/backend/src/wallet-sdk/wallet-sdk.service.ts

apps/backend/src/database/migrations/1761500000005-ProviderIdBigInt.ts

SgtPooki

overall LGTM but I left some comments/questions.

I know we're waiting for some input from synapse maintainers too. Please re-request review when you need me again

SgtPooki · 2026-03-17T12:28:22Z

apps/backend/src/config/app.config.ts

@@ -293,7 +293,7 @@ export function loadConfig(): IConfig {
    blockchain: {
      network: (process.env.NETWORK || "calibration") as Network,
      walletAddress: process.env.WALLET_ADDRESS || "0x0000000000000000000000000000000000000000",
-      walletPrivateKey: process.env.WALLET_PRIVATE_KEY || "",
+      walletPrivateKey: process.env.WALLET_PRIVATE_KEY as "0x${string}",


just a note that before we merge this we need to swap our old private key formats in dealbot infra repo to 0x format

Makes sense! Where is the dealbot infra repo?

https://github.com/filozone/infra

SgtPooki · 2026-03-17T12:35:43Z

apps/backend/src/database/entities/deal.entity.ts

+  @Column({ name: "pieces_added_time", type: "timestamp", nullable: true })
+  piecesAddedTime: Date;
+  @Column({ name: "pieces_confirmed_time", type: "timestamp", nullable: true })
+  piecesConfirmedTime: Date;


we are only ever adding a single piece, do we need to change this?

We don't need to, it keeps it tidy to match the entity name to the event name, otherwise this easily leads to confusion. The SDK event has pieces plural even if just a single piece was confiemd.

SgtPooki · 2026-03-17T12:36:21Z

apps/backend/src/database/entities/storage-provider.entity.ts

@@ -29,7 +29,7 @@ export class StorageProvider {
  isApproved!: boolean;

  @Column()
-  region!: string;
+  location!: string;


do we need to change region to location, or can we just have a comment indicating that it comes from .location value of X type?

same thoughts as in #186 (comment)

SgtPooki · 2026-03-17T12:45:50Z

apps/backend/src/deal/deal.service.ts

    }
  }

-  private async cleanupSynapseInstance(synapse: Synapse): Promise<void> {


do we no longer need to cleanup the synapse instance? I know @rvagg had some comments about this with the filecoin-pin upgrade to latest synapse.

@rvagg's comment filecoin-project/filecoin-pin#343 (comment) was about process exit handing.

We could add back the cleanup logic, and add integration tests for module#close() properly exiting. I think that should be safer for a long running process like this, than just ignoring any opened resources. Will fix.

Filecoin-pin doesn't export any cleanup logic any more.

I started with closing the WS connection, if there is one: 0a37f33

We should also clean up any watch()ers, but Filecoin-pin doesn't use those, so I think we should be good. @rvagg should be best to confirm

apps/backend/src/wallet-sdk/wallet-sdk.service.ts

SgtPooki · 2026-03-17T13:03:35Z

apps/backend/src/wallet-sdk/wallet-sdk.service.ts

-        const supportsPDP = !!info.products?.PDP;
-        // In order to support ipniIpfs, the provider must have PDP product
-        const supportsIpniIpfs = !!info.products?.PDP?.data?.ipniIpfs;
-        const isDevTagged = info.products?.PDP?.capabilities?.service_status === DEV_TAG;


i don't know if we need "isDevTagged" anymore honestly.. we shouldn't be testing against them, and isActive/isEndorsed (whatever the endorsed flag is) should be enough for dealbot.. as long as devTagged SPs are never "isActive" then we should be good

BigLep · 2026-03-19T04:08:08Z

@juliangruber : What are the next steps here?

juliangruber added 2 commits February 3, 2026 12:57

upgrade to synapse 0.37.0 (untested). closes FilOzone#170

15638d1

pnpm run format

d6a6ff2

FilOzzy added this to FOC Feb 3, 2026

github-project-automation bot moved this to 📌 Triage in FOC Feb 3, 2026

rjan90 assigned juliangruber Feb 3, 2026

rjan90 moved this from 📌 Triage to ⌨️ In Progress in FOC Feb 3, 2026

rjan90 added this to the M4.0: mainnet staged milestone Feb 3, 2026

juliangruber changed the title ~~Upgrade to synapse 0.37.0 Closes #170~~ Upgrade to synapse 0.37.0. Closes #170 Feb 3, 2026

rjan90 mentioned this pull request Feb 9, 2026

Update to Synapse 0.38.0 (and corresponding FilecoinPin) #170

Open

rjan90 unassigned juliangruber Feb 9, 2026

BigLep moved this from ⌨️ In Progress to 🐱 Todo in FOC Feb 10, 2026

rjan90 assigned SgtPooki Feb 10, 2026

BigLep unassigned SgtPooki Feb 14, 2026

BigLep changed the title ~~Upgrade to synapse 0.37.0. Closes #170~~ Upgrade to synapse 0.38.0. Closes #170 Feb 27, 2026

rjan90 assigned juliangruber Mar 4, 2026

juliangruber moved this from 🐱 Todo to ⌨️ In Progress in FOC Mar 10, 2026

juliangruber added 7 commits March 10, 2026 11:04

add back synapse

ed91f5a

fixup! add back synapse

a76247e

Merge branch 'main' into synapse-0-37-0

66d7e74

clean up

b994101

clean up

0de0f39

progress

e2dc59d

Merge branch 'main' into synapse-0-37-0

3e12964

BigLep modified the milestones: M4.0: mainnet staged, M4.1: mainnet ready Mar 11, 2026

juliangruber added 3 commits March 11, 2026 08:12

progress

3c5f3d3

progress

d37af1f

progress

6b8e2e5

juliangruber commented Mar 17, 2026

View reviewed changes

juliangruber added 2 commits March 17, 2026 10:48

clean up

c7a959a

clean up

3cc9961

juliangruber commented Mar 17, 2026

View reviewed changes

juliangruber added 4 commits March 17, 2026 11:05

refactor using new costs apis

d0ff1fb

clean up

31719ff

more onUploadComplete -> onStored

ee5e916

only one copy

f23c846

juliangruber marked this pull request as ready for review March 17, 2026 11:34

Copilot AI review requested due to automatic review settings March 17, 2026 11:34

Copilot started reviewing on behalf of juliangruber March 17, 2026 11:34 View session

rjan90 requested review from SgtPooki, rvagg and silent-cipher and removed request for Copilot March 17, 2026 11:39

rjan90 moved this from ⌨️ In Progress to 🔎 Awaiting review in FOC Mar 17, 2026

SgtPooki requested a review from Copilot March 17, 2026 12:21

Copilot started reviewing on behalf of SgtPooki March 17, 2026 12:21 View session

Copilot AI reviewed Mar 17, 2026

View reviewed changes

SgtPooki reviewed Mar 17, 2026

View reviewed changes

This was referenced Mar 17, 2026

Parse bigints as native BigInt instead of string typeorm/typeorm#11981

Open

fix: add typeorm bigint column #370

Draft

juliangruber added 5 commits March 18, 2026 10:12

clean up websocket if exists

0a37f33

Merge branch 'main' into synapse-0-37-0

f3ac960

clean up

4b02980

clean up

15f7f07

fix typo

837a6d4

SgtPooki mentioned this pull request Mar 18, 2026

test: validate typeorm entities #375

Open

3 tasks

Conversation

juliangruber commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SgtPooki commented Feb 3, 2026

Uh oh!

rjan90 commented Feb 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliangruber commented Mar 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SgtPooki left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliangruber Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliangruber Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BigLep commented Mar 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

juliangruber commented Feb 3, 2026 •

edited

Loading

juliangruber Mar 18, 2026 •

edited

Loading

juliangruber Mar 18, 2026 •

edited

Loading