feat: competitor intelligence ingest + seedance2 fixes#15
Closed
mtmtian wants to merge 11 commits into
Closed
Conversation
Adds the competitor-intel data model per docs/growth/07 §4 (+ 09 §3's segmentPlan addition): CompetitorCreative caches AppGrowing ad metadata and AI analysis, idempotent on (orgId, source, externalId); RemixJob is a queryable ledger for turning competitor creatives into our-own-IP assets (not wired to an engine yet — Phase 2). Purely additive, both relate to Organization like the rest of the schema. Migration generated via `prisma migrate diff --from-schema <base> --to-schema prisma/schema.prisma --script` (no local Postgres available in this worktree — see .claude/rules/schema.md).
POST /api/ingest/competitor?org=<orgId> — clones the ingest/scenes HMAC + idempotent-upsert pattern for competitor ad creatives pushed by the local creative-pipeline/AppGrowing bridge. Accepts a single item or an array; mediaUrl already on our GCS bucket is linked directly, an external mediaUrl is fetched once and uploaded to GCS. Per-item created/updated/failed result. Connector parsing/mapping logic lives in src/lib/platforms/appgrowing.ts so the ingest source is isolated from the route, per docs/growth/07 §5. GET /api/competitors — session-scoped listing (requireAuthWithOrg, never trusts a client orgId), filterable by appName/level, orderable by adDays/impressions. Ref: docs/growth/09-pipeline-adex-integration.md §3
Real-world doubao-seedance-2-0-260128 task responses return the generated video URL at content[].video_url.url, not output.video_url — add resolveVideoUrl() that prefers content, falls back to output for older/other model shapes. Also round the requested duration to whole seconds before submitting to Ark; a fractional duration gets a 400 InvalidParameter. Found via first real Seedance2 calls in the local creative-pipeline tooling (commit e06e215/1224f1c). Ref: docs/growth/09-pipeline-adex-integration.md §3
Register→login→GET /api/orgs→POST /api/ingest/competitor round trip: HMAC-signed ingest creates then idempotently updates a CompetitorCreative row, a bad signature 401s, GET /api/competitors requires a session and reflects the upsert. playwright.config.ts's webServer.env gains INVITE_CODES_DISABLED + INGEST_WEBHOOK_SECRET so the spec can self-provision a user/org and sign requests without a PlatformAuth row. Verified the code path against a build with no DATABASE_URL (fails at the expected `DATABASE_URL is not set` boundary, confirming registration/ingest wiring is correct) — no live Postgres was available in this worktree to run the full assertions end-to-end.
Adds the design docs this PR implements: 07 (competitor-intel + remix proposal, incl. the CompetitorCreative/RemixJob model proposal), 08 (creative pipeline stages), 09 (this PR's scope — pipeline↔adex integration, PR #1 boundaries).
WHY: product decision — the whole pipeline lands on adex, not local tooling. Control plane stays in the Next.js service; minute-scale audio/video batch work moves to a Cloud Run Jobs worker whose image packages the already-verified creative-pipeline scripts. Adds per-stage design table, PR roadmap (#1-oratis#4), cost + model-tier policy.
WHY: verified against live task cgt-20260710153030-s5t5x — real
doubao-seedance-2-0-260128 returns content as an OBJECT {video_url:string}
(not an array) and duration at TOP LEVEL. Prior resolveVideoUrl threw
TypeError (content.find is not a function), leaving Assets stuck
'generating' — the PR's core fix did not work against the target model.
WHAT: resolveVideoUrl handles object/array/legacy shapes (array form skips
echoed reference_video inputs); resolveDuration reads top-level→output→
usage; generate route rounds duration before the Int? persist (was a 500
after a paid Ark job started).
WHY: code review — the competitors list filtered level via a rawMeta JSON path that nothing writes (dead filter), while its sibling segmentPlan got a real column. level is the primary routing axis (09 §3), so it warrants a first-class indexed column, not an untyped blob key.
WHY/WHAT (xhigh review findings): - SSRF: ingest fetched caller-supplied mediaUrl with no validation → new storage.uploadFromUrl blocks non-http(s) + private/link-local hosts. - OOM/timeout: whole video buffered, unbounded batch → 100MB cap + 50-item batch cap (413). - competitors list masked all errors as empty 200 → returns 500 on failure; level filter now hits the real column. - parseHumanNumber fixes impressions/adDays silently dropping '710.4K' / '1,942' (the ranking signals); mergeRawMeta no longer corrupts array payloads; asset type prefers content-type over extension; isOwnGcsUrl + gcsPublicPrefix consolidated into storage.ts (was duplicated).
WHY: register is rate-limited 5/hour/IP; registering per-test flaked across retries/reruns. Register once in beforeAll and share the authed context; the anon test uses the default unauthenticated request fixture (config baseURL, no hardcoded localhost).
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
竞品情报接入的第一层：把 AppGrowing 竞品素材落进 adex，并修复 seedance2 客户端对真实 API 的解析。与排队中的 #4–#7 完全独立（不互为祖先、改动文件零交集），可任意顺序合并。
范围
CompetitorCreative（含segmentPlan/level列 +(orgId,level)索引）、RemixJob，均为新增无破坏；迁移与 schema 同 commit。POST /api/ingest/competitor：HMAC 幂等入库（克隆ingest/scenes模式），媒体经服务端uploadFromUrl拉取入 GCS。GET /api/competitors：org 作用域列表/筛选（session 鉴权）。doubao-seedance-2-0-260128响应形状（content为对象、duration在顶层）重写resolveVideoUrl+ 新增resolveDuration。docs/growth/07/08/09。经过 xhigh code review 并修复的问题
resolveVideoUrl把content当数组.find()，对真实响应抛TypeError，Asset 永远卡在 generating——已按实测响应重写，对对象/数组/legacy 三形状防御。mediaUrl；新增 scheme/内网主机拦截 + 100MB 上限 + 批量 50 上限。level由死 JSON 筛选改真列；710.4K/1,942排序信号被判 null；等。验证（真实 Postgres 16）
🤖 Generated with Claude Code