feat(engine): Batch trigger reloaded #2779
Conversation
|
|
Note Other AI code review bot(s) detectedCodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review. WalkthroughThis pull request introduces a comprehensive two-phase batch processing system for Trigger.dev, including a new Redis-backed fair queue with Deficit Round Robin (DRR) scheduling, batch creation and item streaming APIs, database schema extensions to track batch processing states and errors, new UI routes for batch inspection, and observability infrastructure via Grafana dashboards. The work also removes the legacy run number incrementer pattern and extends environment configuration for batch-specific limits and rate limiting. Database changes add new batch status values (PROCESSING, PARTIAL_FAILED), tracking columns for processing timestamps and run counts, a new BatchTaskRunError table for failed items, and per-organization batch concurrency and rate limit configuration. Estimated code review effort🎯 5 (Critical) | ⏱️ ~110 minutes This review spans multiple interconnected systems with substantial complexity: the fair queue implementation introduces Redis-backed distributed scheduling with atomic Lua operations, concurrency management, and multi-group isolation; the batch API adds stateful request handling with idempotency and two-phase streaming; database migrations introduce new tracking structures; and integration points across services require verification of consistent state transitions.
Pre-merge checks and finishing touches❌ Failed checks (2 warnings)
✅ Passed checks (1 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
This is the final PR Bugbot will review for you during this billing cycle
Your free Bugbot reviews will reset on January 3
Details
You are on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle.
To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 18
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/trigger-sdk/src/v3/batch.ts (1)
26-46: Update JSDoc return type to matchRetrieveBatchV2Response
retrieveBatchnow correctly returnsApiPromise<RetrieveBatchV2Response>, but the JSDoc still documentsRetrieveBatchResponse. This can mislead consumers and tooling that read JSDoc.Consider updating the comment:
- * @returns {ApiPromise<RetrieveBatchResponse>} A promise that resolves with the batch details + * @returns {ApiPromise<RetrieveBatchV2Response>} A promise that resolves with the batch details
🧹 Nitpick comments (48)
apps/webapp/seed.mts (1)
113-156: Consider dynamically building the tenants array.The tenants array construction is repetitive. A refactored approach would reduce duplication and make it easier to maintain when adding/removing orgs or projects.
- console.log("tenants.json"); - console.log( - JSON.stringify({ - apiUrl: "http://localhost:3030", - tenants: [ - { - id: org1Project1.project.externalRef, - secretKey: org1Project1.environments.find((e) => e.type === "DEVELOPMENT")?.apiKey, - }, - { - id: org1Project2.project.externalRef, - secretKey: org1Project2.environments.find((e) => e.type === "DEVELOPMENT")?.apiKey, - }, - { - id: org1Project3.project.externalRef, - secretKey: org1Project3.environments.find((e) => e.type === "DEVELOPMENT")?.apiKey, - }, - { - id: org2Project1.project.externalRef, - secretKey: org2Project1.environments.find((e) => e.type === "DEVELOPMENT")?.apiKey, - }, - { - id: org2Project2.project.externalRef, - secretKey: org2Project2.environments.find((e) => e.type === "DEVELOPMENT")?.apiKey, - }, - { - id: org2Project3.project.externalRef, - secretKey: org2Project3.environments.find((e) => e.type === "DEVELOPMENT")?.apiKey, - }, - { - id: org3Project1.project.externalRef, - secretKey: org3Project1.environments.find((e) => e.type === "DEVELOPMENT")?.apiKey, - }, - { - id: org3Project2.project.externalRef, - secretKey: org3Project2.environments.find((e) => e.type === "DEVELOPMENT")?.apiKey, - }, - { - id: org3Project3.project.externalRef, - secretKey: org3Project3.environments.find((e) => e.type === "DEVELOPMENT")?.apiKey, - }, - ], - }) - ); + const allProjects = [ + org1Project1, org1Project2, org1Project3, + org2Project1, org2Project2, org2Project3, + org3Project1, org3Project2, org3Project3, + ]; + + const tenants = allProjects.map((p) => ({ + id: p.project.externalRef, + secretKey: p.environments.find((e) => e.type === "DEVELOPMENT")?.apiKey, + })); + + console.log("tenants.json"); + console.log(JSON.stringify({ apiUrl: "http://localhost:3030", tenants }));internal-packages/run-engine/src/engine/tests/batchTriggerAndWait.test.ts (3)
816-842: Loop iterates overcreatedRunsbut ignores therunIdwhen dequeuing.The loop variable
runIdfromcreatedRunsisn't used to select which run to dequeue—it relies on queue ordering. IfdequeuedChild.length === 0, the loop silently continues, which could mask timing issues in the test. Consider either:
- Failing the test if no run is dequeued when expected, or
- Adding a comment clarifying that the loop count just controls iteration count and ordering is assumed
for (const { runId } of createdRuns) { // Dequeue and start child await setTimeout(300); const dequeuedChild = await engine.dequeueFromWorkerQueue({ consumerId: "test_12345", workerQueue: "main", }); - if (dequeuedChild.length === 0) continue; + // Expect a run to be dequeued for each created run + expect(dequeuedChild.length).toBeGreaterThan(0);
1159-1185: Same dequeue loop pattern as the success test.Consider adding an assertion that a run is dequeued for each iteration, or documenting why silent continuation is acceptable. This would make test failures more debuggable if timing issues occur.
877-1221: Comprehensive partial failure test.Good coverage of the partial failure scenario. The test verifies:
PARTIAL_FAILEDstatus when some items fail- Error records created in
BatchTaskRunError- Parent still resumes after successful runs complete
- Batch eventually transitions to
COMPLETEDConsider adding a test case for when all items fail (the
ABORTEDbranch at line 983) to complete the status matrix coverage.Would you like me to help generate a test case for the all-items-fail (
ABORTED) scenario?packages/trigger-sdk/src/v3/shared.ts (3)
1578-1599: Streaming path fully buffers items, negating memory benefits.The
executeBatchTwoPhaseStreamingfunction collects all items into an array before processing, which eliminates the memory efficiency advantages of streaming for large batches. This is acknowledged in the comment, but users may expect true streaming behavior when passing anAsyncIterable.Consider documenting this limitation in the public API JSDoc comments (e.g., for
batchTriggerById) so users understand that streaming inputs are currently buffered before transmission.
1663-1961: Significant code duplication across transform functions.The six
transform*Stream*functions share ~90% identical logic for building theBatchItemNDJSONoptions object. The main variations are:
- Task ID extraction (
item.idvsitem.task.idvs parameter)lockToVersionsource (env var vstaskContext.worker?.version)- Optional queue parameter
Consider extracting the common options-building logic into a shared helper to reduce the ~300 lines of duplication:
function buildBatchItemOptions( item: { options?: BatchItemOptions }, payloadPacket: { dataType: string }, batchItemIdempotencyKey: string | undefined, options?: { idempotencyKeyTTL?: string }, overrides?: { lockToVersion?: string; queue?: string } ): BatchItemNDJSON['options'] { return { queue: item.options?.queue ? { name: item.options.queue } : overrides?.queue ? { name: overrides.queue } : undefined, // ... common fields lockToVersion: overrides?.lockToVersion ?? item.options?.version ?? getEnvVar("TRIGGER_VERSION"), }; }
611-730: Array path duplicates the streaming transform logic.The array processing (lines 613-652) duplicates the logic in
transformBatchItemsStream. Both paths could use the same transformation logic:// Unified approach const asyncItems = Array.isArray(items) ? (async function* () { for (const item of items) yield item; })() : normalizeToAsyncIterable(items); const ndJsonItems: BatchItemNDJSON[] = []; for await (const item of transformBatchItemsStream(asyncItems, options)) { ndJsonItems.push(item); }This would reduce duplication and ensure both paths stay in sync. However, the current explicit approach is also valid for clarity.
packages/redis-worker/src/fair-queue/tests/workerQueue.test.ts (1)
26-29: Consider guarding non-null assertions.While the test expects
popto return a result, using a non-null assertion without a preceding null check can make test failures less informative.Consider this pattern for clearer test failures:
const result = await manager.pop("worker-1"); expect(result).not.toBeNull(); -expect(result!.messageKey).toBe("msg-1:queue-1"); -expect(result!.queueLength).toBe(0); +if (result) { + expect(result.messageKey).toBe("msg-1:queue-1"); + expect(result.queueLength).toBe(0); +}docs/batch-queue-metrics.md (2)
74-78: Add language specifier to fenced code block.The code block lacks a language specifier. Based on the content (mathematical relationships), consider using
textorplaintextto satisfy the linter.-``` +```text batches_enqueued × avg_items_per_batch ≈ items_enqueued items_enqueued = items_processed + items_failed + items_pending batches_completed ≤ batches_enqueued (lag indicates processing backlog)--- `180-212`: **Add `promql` language specifier to PromQL code blocks.** The PromQL query blocks at lines 180, 194, and 205 are missing language specifiers. Adding `promql` will enable syntax highlighting and satisfy the linter. ```diff ### Processing Health -``` +```promql # Throughput rate(batch_queue_items_processed_total[5m])Apply similar changes to the code blocks starting at lines 194 and 205.
packages/redis-worker/src/fair-queue/retry.ts (1)
33-42: Minor redundancy in maxAttempts assignment.The
maxAttemptsvalue is already guaranteed to be set in thethis.optionsobject on line 35, making the null coalescing on line 41 redundant.this.options = { maxAttempts: options?.maxAttempts ?? 12, factor: options?.factor ?? 2, minTimeoutInMs: options?.minTimeoutInMs ?? 1_000, maxTimeoutInMs: options?.maxTimeoutInMs ?? 3_600_000, // 1 hour randomize: options?.randomize ?? true, }; - this.maxAttempts = this.options.maxAttempts ?? 12; + this.maxAttempts = this.options.maxAttempts!;apps/webapp/app/v3/batchTriggerWorker.server.ts (1)
24-25: Clarify the initialization pattern.While the comment on lines 9-10 explains the import, the
void enginestatement on line 25 could be more explicit about forcing initialization. Consider:- // Ensure the engine (and its BatchQueue) is initialized - void engine; + // Force engine initialization (triggers BatchQueue setup for v2 batches) + // The void operator discards the value while ensuring the side effect + void engine;packages/redis-worker/src/fair-queue/tests/drr.test.ts (1)
8-356: DRR scheduler test coverage is solid; consider renaming one test for clarityThe tests exercise deficit lifecycle, queue selection semantics, ordering by deficit, and aggregate deficit retrieval against a real Redis instance, which gives good confidence in
DRRScheduler.Minor nit: the test
redisTest("should skip tenants with insufficient deficit", ...);ultimately asserts that both tenants are returned after quantum is added, so the name doesn’t quite describe what’s being verified. Renaming it to something like
"should include tenants once quantum has been added"would make intent clearer for future maintainers.internal-packages/database/prisma/migrations/20251205135152_add_columns_for_run_engine_batch_trigger_v2/migration.sql (1)
17-30: Consider adding a unique constraint on(batchTaskRunId, index).The
BatchTaskRunErrortable allows multiple error entries with the samebatchTaskRunIdandindexcombination. If each batch item (identified by index) should have at most one error record, adding a unique constraint would enforce data integrity and prevent duplicate error entries.-- CreateIndex CREATE INDEX "BatchTaskRunError_batchTaskRunId_idx" ON "public"."BatchTaskRunError"("batchTaskRunId"); + +-- CreateIndex (optional: enforce one error per batch item) +CREATE UNIQUE INDEX "BatchTaskRunError_batchTaskRunId_index_key" ON "public"."BatchTaskRunError"("batchTaskRunId", "index");apps/webapp/app/presenters/v3/BatchPresenter.server.ts (1)
69-71: Consider using a more specific error or returning null.Throwing a generic
Error("Batch not found")may make it harder for the caller to distinguish between different failure modes. Consider using a custom error type or returningnulland letting the caller handle the not-found case, which is a common pattern in presenters.packages/redis-worker/src/fair-queue/tests/fairQueue.test.ts (1)
472-473: Fixed delay before DLQ check could cause flaky tests.Using a fixed 500ms delay to wait for DLQ processing may be insufficient under load or on slower CI machines. Consider using
vi.waitForwith a condition that checks DLQ length, similar to how other assertions in this file are structured.- // Give time for DLQ processing - await new Promise((resolve) => setTimeout(resolve, 500)); - - // Check DLQ - const dlqMessages = await queue.getDeadLetterMessages("t1"); - expect(dlqMessages).toHaveLength(1); + // Wait for DLQ processing and verify + let dlqMessages: Awaited<ReturnType<typeof queue.getDeadLetterMessages>> = []; + await vi.waitFor( + async () => { + dlqMessages = await queue.getDeadLetterMessages("t1"); + expect(dlqMessages).toHaveLength(1); + }, + { timeout: 5000 } + );internal-packages/run-engine/src/engine/tests/batchTwoPhase.test.ts (2)
484-508: Child run completion loop may skip runs if timing varies.The loop continues silently when
dequeuedChild.length === 0. While this handles timing issues gracefully, if a child run is never dequeued due to a bug, the test would still pass (it waits for waitpoints to clear, which could happen for other reasons).Consider adding an assertion after the loop to verify all expected child runs were processed:
// After the loop expect(createdRuns.filter(r => /* was completed */).length).toBe(2);
18-56: Consider extracting common engine configuration.The
RunEngineconfiguration is duplicated across all four tests with minor variations. Extract a helper function to reduce duplication and make tests easier to maintain.function createTestEngine(prisma: PrismaClient, redisOptions: RedisOptions, overrides?: Partial<RunEngineOptions>) { return new RunEngine({ prisma, worker: { redis: redisOptions, workers: 1, tasksPerWorker: 10, pollIntervalMs: 20, }, queue: { redis: redisOptions, masterQueueConsumersDisabled: true, processWorkerQueueDebounceMs: 50, }, runLock: { redis: redisOptions }, machines: { defaultMachine: "small-1x", machines: { "small-1x": { name: "small-1x" as const, cpu: 0.5, memory: 0.5, centsPerMs: 0.0001 }, }, baseCostInCents: 0.0001, }, batchQueue: { redis: redisOptions, consumerCount: 2, consumerIntervalMs: 50, drr: { quantum: 10, maxDeficit: 100 }, }, tracer: trace.getTracer("test", "0.0.0"), ...overrides, }); }Also applies to: 164-202, 278-316, 547-584
apps/webapp/app/routes/api.v3.batches.ts (1)
84-92: Consider redacting idempotencyKey in logs.The
idempotencyKeyis logged directly. If users include sensitive information in their idempotency keys (e.g., user IDs, email-based keys), this could leak PII to logs.logger.debug("Create batch request", { runCount: body.runCount, parentRunId: body.parentRunId, resumeParentOnCompletion: body.resumeParentOnCompletion, - idempotencyKey: body.idempotencyKey, + hasIdempotencyKey: !!body.idempotencyKey, triggerVersion, isFromWorker, triggerClient, });internal-packages/run-engine/src/engine/index.ts (1)
973-975:isBatchQueueEnabled()always returns true.Since
batchQueueis always instantiated in the constructor (lines 325-346), this method will always returntrue. If the intent is to conditionally enable the BatchQueue based on configuration, consider checking a config flag instead.isBatchQueueEnabled(): boolean { - return this.batchQueue !== undefined; + return this.options.batchQueue !== undefined && this.batchQueue !== undefined; }apps/webapp/app/v3/runEngine.server.ts (1)
354-366: Consider usingcreateManyfor batch error insertion.Creating error records in a sequential loop can be slow for batches with many failures. Consider using Prisma's
createManyfor better performance.- if (failures.length > 0) { - for (const failure of failures) { - await prisma.batchTaskRunError.create({ - data: { - batchTaskRunId: batchId, - index: failure.index, - taskIdentifier: failure.taskIdentifier, - payload: failure.payload, - options: failure.options as Prisma.InputJsonValue | undefined, - error: failure.error, - errorCode: failure.errorCode, - }, - }); - } - } + if (failures.length > 0) { + await prisma.batchTaskRunError.createMany({ + data: failures.map((failure) => ({ + batchTaskRunId: batchId, + index: failure.index, + taskIdentifier: failure.taskIdentifier, + payload: failure.payload, + options: failure.options as Prisma.InputJsonValue | undefined, + error: failure.error, + errorCode: failure.errorCode, + })), + }); + }packages/redis-worker/src/fair-queue/schedulers/roundRobin.ts (2)
79-89: Sequential capacity checks may impact performance.With many tenants, calling
context.isAtCapacitysequentially in a loop could introduce latency. Consider batching these checks or usingPromise.allwith a reasonable concurrency limit if the underlying implementation supports it.// Example: batch capacity checks const capacityResults = await Promise.all( rotatedTenants.map(async (tenantId) => ({ tenantId, atCapacity: await context.isAtCapacity("tenant", tenantId), })) ); const eligibleTenants: TenantQueues[] = capacityResults .filter((r) => !r.atCapacity) .map(({ tenantId }) => ({ tenantId, queues: queuesByTenant.get(tenantId) ?? [], }));
146-149: Consider adding TTL to the lastServed key.The
lastServedindex persists indefinitely in Redis. If shards are removed or renamed, stale keys accumulate. Consider setting an expiration or implementing periodic cleanup.async #setLastServedIndex(shardKey: string, index: number): Promise<void> { const key = this.#lastServedKey(shardKey); - await this.redis.set(key, index.toString()); + // Expire after 24 hours - will be refreshed on each selectQueues call + await this.redis.set(key, index.toString(), "EX", 86400); }apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam.batches.$batchParam/route.tsx (2)
51-73: Redundant error handling:tryCatchwrapped intry/catch.The
tryCatchutility already returns[error, data]tuple, making the outertry/catchunnecessary unlessBatchPresenterconstructor can throw. The current pattern double-handles errors.- try { - const presenter = new BatchPresenter(); - const [error, data] = await tryCatch( - presenter.call({ - environmentId: environment.id, - batchId: batchParam, - userId, - }) - ); - - if (error) { - throw new Error(error.message); - } - - return typedjson({ batch: data }); - } catch (error) { - console.error(error); - throw new Response(undefined, { - status: 400, - statusText: "Something went wrong, if this problem persists please contact support.", - }); - } + const presenter = new BatchPresenter(); + const [error, data] = await tryCatch( + presenter.call({ + environmentId: environment.id, + batchId: batchParam, + userId, + }) + ); + + if (error) { + console.error(error); + throw new Response(undefined, { + status: 400, + statusText: "Something went wrong, if this problem persists please contact support.", + }); + } + + return typedjson({ batch: data });
75-75: Use named function declaration instead of default export.Per coding guidelines for
**/*.{ts,tsx,js,jsx}, prefer function declarations over default exports.-export default function Page() { +function Page() { // ... component body } + +export { Page as default };Or rename to a more descriptive name like
BatchDetailPage.apps/webapp/app/runEngine/concerns/batchPayloads.server.ts (1)
148-163: Silent fallback on serialization failure may hide issues.When
JSON.stringifyfails for non-JSON types, returning an empty packet without logging could make debugging difficult.Consider adding debug logging for the catch case:
// For other types, try to stringify try { return { data: JSON.stringify(payload), dataType: payloadType }; } catch { + logger.debug("Failed to stringify payload, returning empty packet", { + payloadType, + }); return { dataType: payloadType }; }packages/redis-worker/src/fair-queue/schedulers/drr.ts (2)
184-187: Fragile deficit key construction.The key is built by splitting the master queue key and taking the first segment. This creates tight coupling to the key format and could break if the key structure changes.
Consider adding a dedicated method to the
FairQueueKeyProducerinterface for the DRR deficit key:#deficitKey(): string { - // Use a fixed key for DRR deficit tracking - return `${this.keys.masterQueueKey(0).split(":")[0]}:drr:deficit`; + // Consider adding a deficitKey() method to FairQueueKeyProducer + // For now, use a fixed prefix approach + const prefix = this.keys.masterQueueKey(0).split(":")[0] ?? "fairqueue"; + return `${prefix}:drr:deficit`; }Or better, add
drrDeficitKey()to theFairQueueKeyProducerinterface for consistency with other key methods.
189-215: Hardcoded limit of 1000 queues per shard.The limit is hardcoded without documentation or configuration option. This could silently drop queues in high-volume scenarios.
Consider making this configurable:
+ private queueFetchLimit: number; + constructor(private config: DRRSchedulerConfig) { // ... existing code + this.queueFetchLimit = config.queueFetchLimit ?? 1000; } async #getQueuesFromShard(shardKey: string): Promise<QueueWithScore[]> { const now = Date.now(); const results = await this.redis.zrangebyscore( shardKey, "-inf", now, "WITHSCORES", "LIMIT", 0, - 1000 // Limit for performance + this.queueFetchLimit );apps/webapp/app/runEngine/services/streamBatchItems.server.ts (2)
34-37: Constructor doesn't pass the engine option to base class.The
WithRunEnginebase class accepts an optionalengineparameter, but here onlyprismais passed. While this works because the base class has a default, explicitly passing both options would be more consistent with the class design.- constructor(protected readonly _prisma: PrismaClientOrTransaction = prisma) { - super({ prisma }); + constructor(protected readonly _prisma: PrismaClientOrTransaction = prisma, engine?: RunEngine) { + super({ prisma: _prisma, engine }); this.payloadProcessor = new BatchPayloadProcessor(); }
96-148: Consider adding a timeout or item limit to prevent unbounded processing.The
for awaitloop processes items indefinitely without any timeout or maximum item count check. If a client sends items slowly or never closes the stream, this could hold resources indefinitely.Consider adding a timeout mechanism or enforcing the
batch.runCountlimit during streaming:+ const maxItems = batch.runCount; // Process items from the stream for await (const rawItem of itemsIterator) { + // Safety check: don't accept more items than expected + if (itemsAccepted + itemsDeduplicated >= maxItems) { + throw new ServiceValidationError( + `Received more items than expected runCount ${maxItems}` + ); + }packages/redis-worker/src/fair-queue/keyProducer.ts (1)
94-102: Consider logging or documenting when fallback is used in extractTenantId.The fallback to
parts[0] ?? ""when the queue ID doesn't match the expectedtenant:{tenantId}:...format could silently return incorrect tenant IDs. This might make debugging difficult in production.The current behavior is reasonable for flexibility, but documenting the expected format clearly or logging when fallback is triggered would help with troubleshooting.
packages/core/src/v3/apiClient/index.ts (2)
427-432: Consider using ApiError for consistency in parse failure.When the response fails to parse, a generic
Erroris thrown. Other methods in this class throwApiErrorfor server-related issues. Consider wrapping this inApiErrorfor consistent error handling by callers.if (!parsed.success) { - throw new Error(`Invalid response from server: ${parsed.error.message}`); + throw ApiError.generate( + response.status, + { error: "Invalid response format", details: parsed.error.message }, + undefined, + Object.fromEntries(response.headers.entries()) + ); }
1545-1548: JSON.stringify can throw on circular references.If a
BatchItemNDJSONcontains circular references (unlikely but possible with user data),JSON.stringifywill throw. The error would propagate but might be confusing. Consider wrapping with try-catch for a clearer error message.- const line = JSON.stringify(item) + "\n"; - controller.enqueue(encoder.encode(line)); + try { + const line = JSON.stringify(item) + "\n"; + controller.enqueue(encoder.encode(line)); + } catch (err) { + controller.error(new Error(`Failed to serialize batch item at index ${index - 1}: ${(err as Error).message}`)); + }Also applies to: 1562-1564
packages/redis-worker/src/fair-queue/scheduler.ts (2)
1-1: Unused import: QueueDescriptor.
QueueDescriptoris imported from./types.jsbut never used in this file.-import type { FairScheduler, SchedulerContext, TenantQueues, QueueDescriptor } from "./types.js"; +import type { FairScheduler, SchedulerContext, TenantQueues } from "./types.js";
77-92: Consider parallelizing capacity checks for better performance.The sequential
awaitinside the loop means N tenants require N round-trips. For large tenant counts, parallel checks would be faster:protected async filterAtCapacity( tenants: TenantQueues[], context: SchedulerContext, groupName: string = "tenant" ): Promise<TenantQueues[]> { - const filtered: TenantQueues[] = []; - - for (const tenant of tenants) { - const isAtCapacity = await context.isAtCapacity(groupName, tenant.tenantId); - if (!isAtCapacity) { - filtered.push(tenant); - } - } - - return filtered; + const capacityChecks = await Promise.all( + tenants.map(async (tenant) => ({ + tenant, + isAtCapacity: await context.isAtCapacity(groupName, tenant.tenantId), + })) + ); + return capacityChecks.filter(({ isAtCapacity }) => !isAtCapacity).map(({ tenant }) => tenant); }packages/redis-worker/src/fair-queue/masterQueue.ts (1)
24-30: Consider adding error handling for Redis connection failure.The constructor creates a Redis client but doesn't handle connection errors. If Redis is unavailable, errors will occur on first operation. Consider adding connection verification or at least documenting that the caller should handle connection errors.
constructor(private options: MasterQueueOptions) { this.redis = createRedisClient(options.redis); this.keys = options.keys; this.shardCount = Math.max(1, options.shardCount); this.#registerCommands(); + // Note: Redis connection errors will surface on first operation }packages/redis-worker/src/fair-queue/concurrency.ts (3)
10-14: Consider usingtypeinstead ofinterfaceper coding guidelines.Per the TypeScript coding guidelines for this repository, prefer types over interfaces.
-export interface ConcurrencyManagerOptions { - redis: RedisOptions; - keys: FairQueueKeyProducer; - groups: ConcurrencyGroupConfig[]; -} +export type ConcurrencyManagerOptions = { + redis: RedisOptions; + keys: FairQueueKeyProducer; + groups: ConcurrencyGroupConfig[]; +};
47-62: Sequential capacity check may cause unnecessary round-trips.The
canProcessmethod performs sequential async checks for each group. Consider batching these checks using a pipeline or the Lua script approach used inreservefor better performance with many groups.
94-107: Release operation is not atomic across groups.Unlike
reserve, thereleasemethod uses a pipeline which executes commands sequentially but not atomically. If a failure occurs mid-pipeline, some groups may have the message removed while others retain it.Consider using a Lua script for atomic release similar to the reserve operation, or document that partial release is acceptable:
async release(queue: QueueDescriptor, messageId: string): Promise<void> { + // Note: Pipeline is sufficient here as partial release is recoverable + // (worst case: message is removed from some groups, which is still safe) const pipeline = this.redis.pipeline();packages/redis-worker/src/fair-queue/visibility.ts (1)
5-14: Consider usingtypeinstead ofinterfaceper coding guidelines.-export interface VisibilityManagerOptions { - redis: RedisOptions; - keys: FairQueueKeyProducer; - shardCount: number; - defaultTimeoutMs: number; - logger?: { - debug: (message: string, context?: Record<string, unknown>) => void; - error: (message: string, context?: Record<string, unknown>) => void; - }; -} +export type VisibilityManagerOptions = { + redis: RedisOptions; + keys: FairQueueKeyProducer; + shardCount: number; + defaultTimeoutMs: number; + logger?: { + debug: (message: string, context?: Record<string, unknown>) => void; + error: (message: string, context?: Record<string, unknown>) => void; + }; +};apps/webapp/app/runEngine/services/createBatch.server.ts (1)
173-203: Error handling for P2002 could be more precise.The current logic assumes that if the target doesn't include "oneTimeUseToken", it must be an idempotency key violation. However, there could be other unique constraints on the table.
Consider checking for the specific constraint name or including additional checks:
if ( Array.isArray(target) && target.length > 0 && typeof target[0] === "string" && target[0].includes("oneTimeUseToken") ) { throw new ServiceValidationError( "Cannot create batch with a one-time use token as it has already been used." ); - } else { + } else if ( + Array.isArray(target) && + target.some((t) => typeof t === "string" && t.includes("idempotencyKey")) + ) { throw new ServiceValidationError( "Cannot create batch as it has already been created with the same idempotency key." ); + } else { + // Unknown unique constraint violation - re-throw original error + throw error; }packages/redis-worker/src/fair-queue/workerQueue.ts (3)
4-11: Consider usingtypeinstead ofinterfaceper coding guidelines.-export interface WorkerQueueManagerOptions { - redis: RedisOptions; - keys: FairQueueKeyProducer; - logger?: { - debug: (message: string, context?: Record<string, unknown>) => void; - error: (message: string, context?: Record<string, unknown>) => void; - }; -} +export type WorkerQueueManagerOptions = { + redis: RedisOptions; + keys: FairQueueKeyProducer; + logger?: { + debug: (message: string, context?: Record<string, unknown>) => void; + error: (message: string, context?: Record<string, unknown>) => void; + }; +};
93-150: Blocking pop has a potential resource leak with abort signal.The event listener added to the abort signal is never explicitly removed if the operation completes normally before abort. While
{ once: true }helps, the listener remains attached until either abort or GC.Consider explicitly removing the listener in the finally block:
async blockingPop( workerQueueId: string, timeoutSeconds: number, signal?: AbortSignal ): Promise<string | null> { const workerQueueKey = this.keys.workerQueueKey(workerQueueId); const blockingClient = this.redis.duplicate(); + let cleanup: (() => void) | undefined; try { if (signal) { - const cleanup = () => { + cleanup = () => { blockingClient.disconnect(); }; signal.addEventListener("abort", cleanup, { once: true }); if (signal.aborted) { return null; } } // ... rest of implementation } finally { + if (signal && cleanup) { + signal.removeEventListener("abort", cleanup); + } await blockingClient.quit().catch(() => {}); } }
231-274: Lua script duplication between private and public methods.The
popWithLengthLua script is defined identically in both#registerCommands()andregisterCommands(). Consider extracting the script to a constant to avoid drift.+const POP_WITH_LENGTH_LUA = ` +local workerQueueKey = KEYS[1] +local messageKey = redis.call('LPOP', workerQueueKey) +if not messageKey then + return nil +end +local queueLength = redis.call('LLEN', workerQueueKey) +return {messageKey, queueLength} +`; #registerCommands(): void { this.redis.defineCommand("popWithLength", { numberOfKeys: 1, - lua: ` -local workerQueueKey = KEYS[1] -... - `, + lua: POP_WITH_LENGTH_LUA, }); } registerCommands(redis: Redis): void { redis.defineCommand("popWithLength", { numberOfKeys: 1, - lua: ` -local workerQueueKey = KEYS[1] -... - `, + lua: POP_WITH_LENGTH_LUA, }); }internal-packages/run-engine/src/batch-queue/completionTracker.ts (1)
101-110: Consider validating parsed JSON against BatchMeta schema.The
getMetamethod parses JSON and casts toBatchMetawithout validation. If corrupted data exists in Redis, this could cause unexpected runtime errors.Consider using Zod validation for defense-in-depth:
async getMeta(batchId: string): Promise<BatchMeta | null> { const key = this.metaKey(batchId); const metaJson = await this.redis.get(key); if (!metaJson) { return null; } const result = BatchMeta.safeParse(JSON.parse(metaJson)); if (!result.success) { this.logger.error("Invalid batch metadata", { batchId, error: result.error.message }); return null; } return result.data; }packages/redis-worker/src/fair-queue/index.ts (2)
786-795: Rate limiting waits indefinitely if resetAt is in the future.When rate limited, the code waits until
resetAtbefore proceeding. This could cause a consumer to block for an extended period. Consider adding a maximum wait time or checking abort signal during the wait.if (this.globalRateLimiter) { const result = await this.globalRateLimiter.limit(); if (!result.allowed && result.resetAt) { - const waitMs = Math.max(0, result.resetAt - Date.now()); + const waitMs = Math.min(5000, Math.max(0, result.resetAt - Date.now())); // Cap at 5s if (waitMs > 0) { this.logger.debug("Global rate limit reached, waiting", { waitMs, loopId }); await new Promise((resolve) => setTimeout(resolve, waitMs)); } } }
1026-1036: Duplicate rate limiting code could be extracted.The rate limiting logic at lines 786-795 and 1026-1036 is duplicated. Consider extracting to a private helper method.
private async waitForRateLimit(loopId: string): Promise<void> { if (!this.globalRateLimiter) return; const result = await this.globalRateLimiter.limit(); if (!result.allowed && result.resetAt) { const waitMs = Math.max(0, result.resetAt - Date.now()); if (waitMs > 0) { this.logger.debug("Global rate limit reached, waiting", { waitMs, loopId }); await new Promise((resolve) => setTimeout(resolve, waitMs)); } } }packages/redis-worker/src/fair-queue/types.ts (1)
15-21: Consider usingtypeinstead ofinterfaceper coding guidelines.The coding guidelines specify "Use types over interfaces for TypeScript". However, this is a new file with extensive interface usage. Given the scope and that interfaces provide better error messages for complex types, this may be acceptable.
If strict adherence to guidelines is required:
export type GlobalRateLimiter = { limit(): Promise<{ allowed: boolean; resetAt?: number }>; };
| #parseMember(member: string): { messageId: string; queueId: string } { | ||
| const colonIndex = member.indexOf(":"); | ||
| if (colonIndex === -1) { | ||
| return { messageId: member, queueId: "" }; | ||
| } | ||
| return { | ||
| messageId: member.substring(0, colonIndex), | ||
| queueId: member.substring(colonIndex + 1), | ||
| }; | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# First, find where member strings are created
rg -n "member\s*=" packages/redis-worker/src/fair-queue/visibility.ts -A 2 -B 2 | head -40Repository: triggerdotdev/trigger.dev
Length of output: 1528
🏁 Script executed:
#!/bin/bash
# Search for the member string creation pattern with messageId and queueId concatenation
rg -n "\`.*messageId.*queueId\`|\$\{.*messageId.*queueId" packages/redis-worker/src/ -A 1 -B 1Repository: triggerdotdev/trigger.dev
Length of output: 51
🏁 Script executed:
#!/bin/bash
# Find where messageId is generated/defined
rg -n "messageId" packages/redis-worker/src/fair-queue/visibility.ts -B 2 -A 2 | head -50Repository: triggerdotdev/trigger.dev
Length of output: 1747
🏁 Script executed:
#!/bin/bash
# Check the entire visibility.ts file to understand the member format
wc -l packages/redis-worker/src/fair-queue/visibility.tsRepository: triggerdotdev/trigger.dev
Length of output: 123
🏁 Script executed:
#!/bin/bash
# Find the #makeMember method definition
rg -n "#makeMember" packages/redis-worker/src/fair-queue/visibility.ts -A 5Repository: triggerdotdev/trigger.dev
Length of output: 1121
🏁 Script executed:
#!/bin/bash
# Find where messageId values come from - check imports and types
rg -n "messageId" packages/redis-worker/src/fair-queue/visibility.ts | head -20Repository: triggerdotdev/trigger.dev
Length of output: 1024
🏁 Script executed:
#!/bin/bash
# Check test files to understand expected formats
find packages/redis-worker -name "*.test.ts" -o -name "*.spec.ts" | xargs rg -l "parseMember\|messageId" | head -5Repository: triggerdotdev/trigger.dev
Length of output: 51
🏁 Script executed:
#!/bin/bash
# Search for messageId generation across the package
rg -n "messageId\s*[:=]" packages/redis-worker/src --type ts | grep -v "const.*:" | head -20Repository: triggerdotdev/trigger.dev
Length of output: 2192
🏁 Script executed:
#!/bin/bash
# Check nanoid usage and see if there are format constraints
rg -n "nanoid" packages/redis-worker/src/fair-queue/index.ts -B 2 -A 2Repository: triggerdotdev/trigger.dev
Length of output: 720
🏁 Script executed:
#!/bin/bash
# Look for test cases that test the parseMember method
rg -n "parseMember\|#parseMember" packages/redis-worker/src --type ts -A 5Repository: triggerdotdev/trigger.dev
Length of output: 51
🏁 Script executed:
#!/bin/bash
# Check if there are any tests for visibility.ts
find packages/redis-worker/src -path "*visibility*test*" -o -path "*visibility*spec*"Repository: triggerdotdev/trigger.dev
Length of output: 51
🏁 Script executed:
#!/bin/bash
# Look at the index.ts file to see messageId generation context
sed -n '235,245p' packages/redis-worker/src/fair-queue/index.ts
sed -n '348,358p' packages/redis-worker/src/fair-queue/index.tsRepository: triggerdotdev/trigger.dev
Length of output: 1065
🏁 Script executed:
#!/bin/bash
# Check the type definition for messageId in EnqueueOptions
rg -n "EnqueueOptions\|messageId" packages/redis-worker/src/fair-queue/index.ts -B 3 -A 3 | head -60Repository: triggerdotdev/trigger.dev
Length of output: 51
🏁 Script executed:
#!/bin/bash
# Look at the types file for messageId constraints
rg -n "messageId" packages/redis-worker/src/fair-queue/types.ts -B 2 -A 2Repository: triggerdotdev/trigger.dev
Length of output: 537
🏁 Script executed:
#!/bin/bash
# Check if there's any documentation about message ID format
rg -n "messageId.*format\|messageId.*constraint\|messageId.*uuid" packages/redis-worker/src --type ts -iRepository: triggerdotdev/trigger.dev
Length of output: 51
🏁 Script executed:
#!/bin/bash
# Look for the actual nanoid package info - does it generate colons?
# Standard nanoid uses: 0-9a-z_- (URL-safe)
# Let's verify there's no custom configuration
find packages/redis-worker -name "*.config.*" -o -name "package.json" | xargs grep -l "nanoid" 2>/dev/nullRepository: triggerdotdev/trigger.dev
Length of output: 140
🏁 Script executed:
#!/bin/bash
# Check package.json for nanoid version
cat packages/redis-worker/package.json | grep -A 2 -B 2 "nanoid"Repository: triggerdotdev/trigger.dev
Length of output: 218
🏁 Script executed:
#!/bin/bash
# Look for any input validation on messageId before it's used
rg -n "messageId.*validate\|validateMessage" packages/redis-worker/src/fair-queue/ --type tsRepository: triggerdotdev/trigger.dev
Length of output: 51
🏁 Script executed:
#!/bin/bash
# Check if there are any guards or constraints on messageId usage
rg -n "messageId\s*\?" packages/redis-worker/src/fair-queue/index.ts -B 2 -A 2 | head -40Repository: triggerdotdev/trigger.dev
Length of output: 495
🏁 Script executed:
#!/bin/bash
# Let's trace a messageId through the code to see if it's validated
# Check the enqueue function more carefully
sed -n '235,280p' packages/redis-worker/src/fair-queue/index.tsRepository: triggerdotdev/trigger.dev
Length of output: 1925
Member parsing assumes messageId contains no colons.
The #parseMember method uses indexOf(":") to split the member string. If a custom messageId contains colons, parsing will fail. The default behavior (nanoid generation) avoids this, but users can provide arbitrary messageId values.
Consider using lastIndexOf(":") instead, or document that messageId must not contain colons.
🤖 Prompt for AI Agents
In packages/redis-worker/src/fair-queue/visibility.ts around lines 363 to 372,
the #parseMember method currently splits the member string using indexOf(":"),
which breaks if messageId contains colons; change the split to use
lastIndexOf(":") so the queueId is parsed from the last colon and the messageId
is everything before it, preserving messageIds that include colons; keep the
existing fallback when no colon is found (return messageId as the whole member
and queueId as ""), and ensure behavior is clearly tested or documented if you
prefer the alternative constraint that messageId must not contain colons.
ericallam
force-pushed
the
feat/batch-trigger-v2
branch
from
e8388a5 to
3ed008d
Compare
There was a problem hiding this comment.
Actionable comments posted: 6
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/trigger-sdk/src/v3/batch.ts (1)
31-46: JSDoc return type is outdated.The JSDoc comment on line 31 still references
RetrieveBatchResponse, but the actual return type has been updated toRetrieveBatchV2Response. Update the JSDoc to maintain consistency.- * @returns {ApiPromise<RetrieveBatchResponse>} A promise that resolves with the batch details + * @returns {ApiPromise<RetrieveBatchV2Response>} A promise that resolves with the batch details
♻️ Duplicate comments (13)
internal-packages/BATCH_TRIGGER_LIMITS.md (2)
1-6: Remove AI conversation artifacts from documentation.Lines 1–6 contain conversational AI artifacts (
[10 tools called], "Now I have a comprehensive picture") that should be removed so the document begins directly with the technical analysis.-Let me analyze the potential issues with increasing batch size limits by examining the key components involved in batch processing: - -[10 tools called] - -Now I have a comprehensive picture. Let me compile a detailed analysis of the potential issues with increasing batch size limits: - ## Analysis: Potential Issues with Larger Batch Sizes
220-220: Remove trailing AI prompt artifact.Line 220 contains an AI prompt question that doesn't belong in technical documentation.
9. **Batch checkpointing**: For batches > 1,000 runs, checkpoint progress to allow recovery without reprocessing - -Would you like me to start implementing any of these recommendations?apps/webapp/app/env.server.ts (1)
943-950: VerifyBATCH_QUEUE_DRR_QUANTUMandBATCH_QUEUE_MAX_DEFICIThave safe fallbacks.These optional env vars are used by the DRR scheduler. Ensure the consuming code provides safe defaults when these are undefined, otherwise a runtime error may occur when the scheduler tries to use them.
#!/bin/bash # Check how these env vars are consumed and if defaults are provided at the usage site rg "BATCH_QUEUE_DRR_QUANTUM|BATCH_QUEUE_MAX_DEFICIT" -t ts -B 2 -A 5 | head -80.env.example (1)
88-94: Add trailing blank line.The local observability documentation is well-structured. However, the file is missing a trailing newline at the end.
docker/config/grafana/provisioning/dashboards/batch-queue.json (1)
113-118: NaN when no items are processed - previously flagged.The success rate expression
sum(rate(...processed...)) / (sum(rate(...processed...)) + sum(rate(...failed...)))returns NaN when both rates are 0. This was noted in a previous review.packages/redis-worker/src/fair-queue/schedulers/weighted.ts (3)
246-257: Division by zero risk when all tenants have zero avgAge.When all queues have a score of 0,
maxAgewill be 0, causingt.avgAge / maxAgeto produceNaN. This was flagged in a previous review.// Weighted shuffle to select top N tenants const maxAge = Math.max(...tenantAges.map((t) => t.avgAge)); + if (maxAge === 0) { + // All tenants have same age, use equal weights + const selectedTenants = new Set(tenantAges.slice(0, this.maximumTenantCount).map(t => t.tenantId)); + return queues.filter((q) => selectedTenants.has(q.tenantId)); + } const weightedTenants = tenantAges.map((t) => ({
302-314: Division by zero risk when maxLimit is 0.If all tenants have a concurrency limit of 0, the weight calculation at line 313 will divide by zero. This was flagged in a previous review.
// Calculate weights const maxLimit = Math.max( ...tenantIds.map((id) => snapshot.tenants.get(id)!.concurrency.limit) ); + if (maxLimit === 0) { + return this.#shuffle(tenantIds); + }
344-349: Division by zero risk when all queues have age 0.If
maxAgeis 0 (all queues are brand new), the weight calculation will produceNaN. This was flagged in a previous review.// Weighted random based on age const maxAge = Math.max(...queues.map((q) => q.age)); + if (maxAge === 0) { + return queues.map((q) => q.queueId); + } const weightedQueues = queues.map((q) => ({packages/redis-worker/src/fair-queue/visibility.ts (1)
363-372: Member parsing assumes messageId contains no colons.The
#parseMembermethod usesindexOf(":")to split. If a custom messageId contains colons, parsing produces incorrect results. UselastIndexOf(":")instead since queueId is appended last, or document the constraint.packages/redis-worker/src/fair-queue/index.ts (1)
1337-1351: Remove unusedshardIdwhen DLQ is disabled (minor).In the
!deadLetterQueueEnabledbranch of#moveToDeadLetterQueue,shardIdis computed but never used:if (!this.deadLetterQueueEnabled) { // Just complete and discard const shardId = this.masterQueue.getShardForQueue(storedMessage.queueId); await this.visibilityManager.complete(storedMessage.id, storedMessage.queueId); return; }You can safely drop the
shardIdline to avoid linter warnings.packages/trigger-sdk/src/v3/shared.ts (1)
1517-1564: ExposebatchIdon Phase 2 failures to aid recovery (non-blocking).If
createBatchsucceeds butstreamBatchItemsthrows, the error surfaced fromexecuteBatchTwoPhasedoesn’t carry thebatch.id, making post-failure inspection or manual recovery harder for callers. Consider attaching thebatchIdto the thrown error (or wrapping it) so higher-level code can decide whether to inspect/cleanup the batch:- const batch = await apiClient.createBatch( - { - runCount: items.length, - parentRunId: options.parentRunId, - resumeParentOnCompletion: options.resumeParentOnCompletion, - idempotencyKey: options.idempotencyKey, - }, - { spanParentAsLink: options.spanParentAsLink }, - requestOptions - ); - - // If the batch was cached (idempotent replay), skip streaming items - if (!batch.isCached) { - // Phase 2: Stream items - await apiClient.streamBatchItems(batch.id, items, requestOptions); - } + const batch = await apiClient.createBatch( + { + runCount: items.length, + parentRunId: options.parentRunId, + resumeParentOnCompletion: options.resumeParentOnCompletion, + idempotencyKey: options.idempotencyKey, + }, + { spanParentAsLink: options.spanParentAsLink }, + requestOptions + ); + + // If the batch was cached (idempotent replay), skip streaming items + if (!batch.isCached) { + try { + // Phase 2: Stream items + await apiClient.streamBatchItems(batch.id, items, requestOptions); + } catch (error) { + (error as any).batchId = batch.id; + throw error; + } + }This keeps existing behavior but gives callers more context when Phase 2 fails.
packages/redis-worker/src/fair-queue/types.ts (1)
40-79: Consider unifyingmetadatatypes across queue models (optional).
QueueMessage.metadatais typed asRecord<string, unknown>, whileStoredMessage.metadataand the metadata inEnqueueOptions/EnqueueBatchOptionsareRecord<string, string>. SinceStoredMessage.metadatais directly passed through toQueueMessage.metadata, this asymmetry is safe but a bit surprising.If metadata is intended to hold arbitrary JSON-like values everywhere, you could simplify by making these all
Record<string, unknown>:-export interface StoredMessage<TPayload = unknown> { +export interface StoredMessage<TPayload = unknown> { ... - metadata?: Record<string, string>; + metadata?: Record<string, unknown>; } -export interface EnqueueOptions<TPayload = unknown> { +export interface EnqueueOptions<TPayload = unknown> { ... - metadata?: Record<string, string>; + metadata?: Record<string, unknown>; } -export interface EnqueueBatchOptions<TPayload = unknown> { +export interface EnqueueBatchOptions<TPayload = unknown> { ... - metadata?: Record<string, string>; + metadata?: Record<string, unknown>; }Not required for correctness, but it keeps the type surface consistent.
Also applies to: 458-492
internal-packages/run-engine/src/batch-queue/index.ts (1)
63-67: Fix logger initialization to honoroptions.logger.The constructor still ignores a provided
options.logger; both ternary branches create a newLoggerinstance:this.logger = options.logger ? new Logger("BatchQueue", "info") : new Logger("BatchQueue", "info");You likely want to use the supplied logger when present:
- this.logger = options.logger - ? new Logger("BatchQueue", "info") - : new Logger("BatchQueue", "info"); + this.logger = options.logger ?? new Logger("BatchQueue", "info");
🧹 Nitpick comments (15)
internal-packages/BATCH_TRIGGER_LIMITS.md (1)
212-212: Replace overused intensifier "very" with more specific language.Lines 212 and 216 use "very" which can be more precisely expressed.
-7. **Streaming batch completion**: Instead of waiting for all runs, allow partial results callback for very large batches +7. **Streaming batch completion**: Instead of waiting for all runs, allow partial results callback for large batches (>1,000 runs) -8. **Separate batch runs table**: For very large batches, consider a denormalized `BatchRun` junction table optimized for batch queries +8. **Separate batch runs table**: For batches exceeding 1,000 runs, consider a denormalized `BatchRun` junction table optimized for batch queriesAlso applies to: 216-216
internal-packages/run-engine/src/engine/systems/batchSystem.ts (1)
66-78: Consider extracting the version string to a shared constant.The magic string
"runengine:v2"is used in multiple places across the codebase (batchSystem.ts:68, BatchPresenter.server.ts:74, and test files). Extracting it to a constant (e.g.,BATCH_VERSION_RUNENGINE_V2) would improve maintainability and prevent typo-related bugs if this string is referenced elsewhere.apps/webapp/seed.mts (1)
113-156: Consider simplifying tenant array construction and guarding against missing environments.The repetitive tenant object construction could be simplified, and the
secretKeylookup may silently produceundefinedif no DEVELOPMENT environment exists.- console.log("tenants.json"); - console.log( - JSON.stringify({ - apiUrl: "http://localhost:3030", - tenants: [ - { - id: org1Project1.project.externalRef, - secretKey: org1Project1.environments.find((e) => e.type === "DEVELOPMENT")?.apiKey, - }, - { - id: org1Project2.project.externalRef, - secretKey: org1Project2.environments.find((e) => e.type === "DEVELOPMENT")?.apiKey, - }, - // ... remaining entries - ], - }) - ); + const allProjects = [ + org1Project1, org1Project2, org1Project3, + org2Project1, org2Project2, org2Project3, + org3Project1, org3Project2, org3Project3, + ]; + + const tenants = allProjects.map((p) => { + const devEnv = p.environments.find((e) => e.type === "DEVELOPMENT"); + if (!devEnv) { + console.warn(`⚠️ No DEVELOPMENT environment found for project ${p.project.name}`); + } + return { + id: p.project.externalRef, + secretKey: devEnv?.apiKey, + }; + }); + + console.log("tenants.json"); + console.log(JSON.stringify({ apiUrl: "http://localhost:3030", tenants }));docs/batch-queue-metrics.md (1)
180-212: Add language specifiers to PromQL code blocks.The fenced code blocks containing PromQL queries should have a language specifier for proper syntax highlighting. This was flagged by markdownlint.
-``` +```promql # Throughput rate(batch_queue_items_processed_total[5m])Apply the same change to the code blocks at lines 194 and 205.
apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam.batches.$batchParam/route.tsx (1)
51-72: Redundant error handling pattern.The code uses
tryCatchfrom@trigger.dev/corewhich returns[error, data], but then wraps it in a try/catch block. The innerthrow new Error(error.message)at line 62 is caught by the outer catch at line 66, which logs and returns a generic 400. This works but is slightly redundant.Consider simplifying to use only one error handling approach, or making the error response more specific (e.g., 404 for "Batch not found" vs 400 for other errors).
try { const presenter = new BatchPresenter(); const [error, data] = await tryCatch( presenter.call({ environmentId: environment.id, batchId: batchParam, userId, }) ); if (error) { - throw new Error(error.message); + console.error("BatchPresenter error:", error); + if (error.message === "Batch not found") { + throw new Response("Batch not found", { status: 404 }); + } + throw new Response(undefined, { + status: 400, + statusText: error.message, + }); } return typedjson({ batch: data }); } catch (error) { + if (error instanceof Response) { + throw error; + } console.error(error); throw new Response(undefined, { status: 400, statusText: "Something went wrong, if this problem persists please contact support.", }); }apps/webapp/app/runEngine/concerns/batchPayloads.server.ts (1)
86-104: Consider documenting the inline storage limitation.When object storage is unavailable but payload exceeds the threshold, the code logs a warning and continues with inline storage. The comment notes this "may fail downstream for very large payloads."
This is a reasonable fallback, but consider whether this should be a hard failure in production environments where R2 is expected to be configured. As-is, large payloads could silently cause issues downstream.
packages/redis-worker/src/fair-queue/retry.ts (1)
29-52: Minor redundancy in maxAttempts assignment.
maxAttemptsis assigned twice - once within theoptionsobject (line 35) and again directly on line 41. This works but is slightly redundant.constructor(options?: Partial<RetryOptions>) { this.options = { maxAttempts: options?.maxAttempts ?? 12, factor: options?.factor ?? 2, minTimeoutInMs: options?.minTimeoutInMs ?? 1_000, maxTimeoutInMs: options?.maxTimeoutInMs ?? 3_600_000, // 1 hour randomize: options?.randomize ?? true, }; - this.maxAttempts = this.options.maxAttempts ?? 12; + this.maxAttempts = this.options.maxAttempts!; }packages/core/src/v3/apiClient/index.ts (1)
394-435: streamBatchItems lacks retry logic available to other API methods.Unlike other methods that use
zodfetchwith built-in retry support viamergeRequestOptions, this method uses rawfetch. For large batch streams, transient network failures could cause the entire stream to fail without retry.Consider documenting this limitation or implementing retry at a higher level for the streaming use case.
internal-packages/run-engine/src/batch-queue/tests/index.test.ts (1)
338-340: Avoid fixed sleep in tests; prefer event-driven waits.Using
setTimeoutwith a fixed 200ms delay is a potential source of flakiness. Consider using a more reliable mechanism like pollinggetBatchRemainingCountor checking queue state.- // Wait a bit - nothing should be processed - await new Promise((resolve) => setTimeout(resolve, 200)); - expect(processedItems).toHaveLength(0); + // Verify items are enqueued but not processed (consumers not started) + const remaining = await queue.getBatchRemainingCount("batch1"); + expect(remaining).toBe(3); + expect(processedItems).toHaveLength(0);packages/redis-worker/src/fair-queue/concurrency.ts (2)
47-62: TOCTOU race between canProcess and reserve.
canProcesschecks capacity non-atomically beforereserveis called. Between these calls, another consumer could reserve slots, causingreserveto fail unexpectedly. This is acceptable if callers handlereservereturningfalse, but consider documenting this or removingcanProcessifreservealready handles the atomic check.
97-107: Pipeline exec result not checked for errors.
pipeline.exec()returns an array of[error, result]tuples. If anySREMfails, the error is silently ignored. Consider checking for errors or at minimum logging them.- await pipeline.exec(); + const results = await pipeline.exec(); + if (results) { + for (const [err] of results) { + if (err) { + console.error("Error releasing concurrency:", err); + } + } + }packages/redis-worker/src/fair-queue/telemetry.ts (1)
248-318: Consider error handling in gauge callbacks.The gauge callbacks iterate over queues/shards and make async Redis calls. If any call fails, the entire callback throws and potentially disrupts metric collection for other dimensions.
Consider wrapping individual calls in try-catch to ensure partial failures don't prevent other metrics from being observed:
if (callbacks.getQueueLength && callbacks.observedQueues) { const getQueueLength = callbacks.getQueueLength; const queues = callbacks.observedQueues; this.metrics.queueLength.addCallback(async (observableResult) => { for (const queueId of queues) { - const length = await getQueueLength(queueId); - observableResult.observe(length, { - [FairQueueAttributes.QUEUE_ID]: queueId, - }); + try { + const length = await getQueueLength(queueId); + observableResult.observe(length, { + [FairQueueAttributes.QUEUE_ID]: queueId, + }); + } catch { + // Skip this queue on error, continue with others + } } }); }apps/webapp/app/runEngine/services/streamBatchItems.server.ts (1)
213-270: Reuse TextEncoder instance for better performance.A new
TextEncoderis created for each line (lines 234, 255). Since the stream may process many items, consider hoisting the encoder outside the transform callbacks.export function createNdjsonParserStream( maxItemBytes: number ): TransformStream<Uint8Array, unknown> { const decoder = new TextDecoder(); + const encoder = new TextEncoder(); let buffer = ""; let lineNumber = 0; return new TransformStream<Uint8Array, unknown>({ transform(chunk, controller) { buffer += decoder.decode(chunk, { stream: true }); // Split on newlines const lines = buffer.split("\n"); buffer = lines.pop() ?? ""; for (const line of lines) { lineNumber++; const trimmed = line.trim(); if (!trimmed) continue; // Check byte size before parsing - const lineBytes = new TextEncoder().encode(trimmed).length; + const lineBytes = encoder.encode(trimmed).length;packages/redis-worker/src/fair-queue/workerQueue.ts (1)
231-274: Duplicate Lua script definition.The
popWithLengthLua script is defined twice - in#registerCommands()andregisterCommands(). Extract the script to a constant to avoid duplication and potential drift.+const POP_WITH_LENGTH_LUA = ` +local workerQueueKey = KEYS[1] + +-- Pop the first message +local messageKey = redis.call('LPOP', workerQueueKey) +if not messageKey then + return nil +end + +-- Get remaining queue length +local queueLength = redis.call('LLEN', workerQueueKey) + +return {messageKey, queueLength} +`; + export class WorkerQueueManager { // ... #registerCommands(): void { this.redis.defineCommand("popWithLength", { numberOfKeys: 1, - lua: ` -local workerQueueKey = KEYS[1] -... - `, + lua: POP_WITH_LENGTH_LUA, }); } registerCommands(redis: Redis): void { redis.defineCommand("popWithLength", { numberOfKeys: 1, - lua: ` -local workerQueueKey = KEYS[1] -... - `, + lua: POP_WITH_LENGTH_LUA, }); } }apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam.batches/route.tsx (1)
212-264: Unify row selection styling by passingisSelectedinto cells (optional).Right now selection is applied via
classNameonTableRowand onlyTableCellMenureceivesisSelected, so hover/focus styles may differ between the menu cell and the others. For more consistent UX and keyboard focus outlines, consider passingisSelectedinto the data cells as well and letting the table primitives handle styling:- <TableRow key={batch.id} className={isSelected ? "bg-grid-dimmed" : undefined}> - <TableCell to={inspectorPath} isTabbableCell> + <TableRow key={batch.id}> + <TableCell to={inspectorPath} isTabbableCell isSelected={isSelected}> ... - <TableCell to={inspectorPath}> + <TableCell to={inspectorPath} isSelected={isSelected}> ... - <BatchActionsCell runsPath={runsPath} /> + <BatchActionsCell runsPath={runsPath} isSelected={isSelected} />and update
BatchActionsCellto accept/forwardisSelectedintoTableCellMenu.Also applies to: 281-291
| this.batchQueue = new BatchQueue({ | ||
| redis: { | ||
| host: options.batchQueue?.redis.host ?? "localhost", | ||
| port: options.batchQueue?.redis.port ?? 6379, | ||
| username: options.batchQueue?.redis.username, | ||
| password: options.batchQueue?.redis.password, | ||
| keyPrefix: `${options.batchQueue?.redis.keyPrefix ?? ""}batch-queue:`, | ||
| enableAutoPipelining: options.batchQueue?.redis.enableAutoPipelining ?? true, | ||
| tls: options.batchQueue?.redis.tls !== undefined, | ||
| }, |
There was a problem hiding this comment.
TLS configuration is incorrectly converted to boolean.
Line 333 converts the TLS options object to a boolean true instead of passing the actual TLS configuration. This will enable TLS but lose any custom TLS settings (certificates, ciphers, etc.).
keyPrefix: `${options.batchQueue?.redis.keyPrefix ?? ""}batch-queue:`,
enableAutoPipelining: options.batchQueue?.redis.enableAutoPipelining ?? true,
- tls: options.batchQueue?.redis.tls !== undefined,
+ tls: options.batchQueue?.redis.tls,
},📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| this.batchQueue = new BatchQueue({ | |
| redis: { | |
| host: options.batchQueue?.redis.host ?? "localhost", | |
| port: options.batchQueue?.redis.port ?? 6379, | |
| username: options.batchQueue?.redis.username, | |
| password: options.batchQueue?.redis.password, | |
| keyPrefix: `${options.batchQueue?.redis.keyPrefix ?? ""}batch-queue:`, | |
| enableAutoPipelining: options.batchQueue?.redis.enableAutoPipelining ?? true, | |
| tls: options.batchQueue?.redis.tls !== undefined, | |
| }, | |
| this.batchQueue = new BatchQueue({ | |
| redis: { | |
| host: options.batchQueue?.redis.host ?? "localhost", | |
| port: options.batchQueue?.redis.port ?? 6379, | |
| username: options.batchQueue?.redis.username, | |
| password: options.batchQueue?.redis.password, | |
| keyPrefix: `${options.batchQueue?.redis.keyPrefix ?? ""}batch-queue:`, | |
| enableAutoPipelining: options.batchQueue?.redis.enableAutoPipelining ?? true, | |
| tls: options.batchQueue?.redis.tls, | |
| }, |
🤖 Prompt for AI Agents
internal-packages/run-engine/src/engine/index.ts around lines 325 to 334: the
current code sets tls: options.batchQueue?.redis.tls !== undefined which coerces
the TLS options object into a boolean true and discards any custom TLS settings;
change it to pass the actual TLS value (e.g., tls:
options.batchQueue?.redis.tls) so the TLS configuration object (or undefined) is
forwarded intact and adjust types if necessary to match the downstream Redis
client's expected tls shape.
…e trigger pipeline
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
apps/webapp/app/runEngine/services/triggerTask.server.ts (1)
347-359: UsesafeParseto prevent validation errors from bypassing the ServiceValidationError path.
TaskRunError.parse(taskRun.error)will throw iftaskRun.errordoesn't match the expected schema, which exits the try block before reaching your explicit error handling at lines 355–359. This changes the error surface from ServiceValidationError to ZodError. UsesafeParsewith an explicit success check instead:- const error = taskRun.error ? TaskRunError.parse(taskRun.error) : undefined; + const parsedError = taskRun.error ? TaskRunError.safeParse(taskRun.error) : undefined; + const error = parsedError?.success ? parsedError.data : undefined; + + if (taskRun.error && !parsedError?.success) { + logger.error("Failed to parse taskRun.error", { + taskId, + runFriendlyId, + }); + throw new ServiceValidationError("Run failed with an unparseable error payload."); + }
♻️ Duplicate comments (2)
internal-packages/run-engine/src/engine/index.ts (2)
333-333: TLS configuration is incorrectly converted to boolean.This issue was already identified in a previous review. Line 333 converts the TLS options object to a boolean
trueinstead of passing the actual TLS configuration.
968-973:isBatchQueueEnabled()always returns true.This issue was already identified in a previous review. Since
batchQueueis unconditionally initialized in the constructor (lines 325-346), this method will always returntrue.
🧹 Nitpick comments (1)
internal-packages/run-engine/src/engine/systems/batchSystem.ts (1)
66-88: Consider extracting the batch version string to a constant.The hardcoded string
"runengine:v2"at line 68 could be extracted to a named constant (e.g.,BATCH_VERSION_V2) to prevent typos and improve maintainability if this version string is referenced elsewhere in the codebase.For example, at the top of the file or in a shared constants file:
const BATCH_VERSION_V2 = "runengine:v2";Then use it in the comparison:
- const isNewBatch = batch.batchVersion === "runengine:v2"; + const isNewBatch = batch.batchVersion === BATCH_VERSION_V2;
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (8)
apps/webapp/app/runEngine/concerns/runNumbers.server.ts(0 hunks)apps/webapp/app/runEngine/services/triggerTask.server.ts(1 hunks)apps/webapp/app/runEngine/types.ts(0 hunks)apps/webapp/app/v3/services/triggerTask.server.ts(0 hunks)internal-packages/run-engine/src/engine/index.ts(5 hunks)internal-packages/run-engine/src/engine/systems/batchSystem.ts(2 hunks)internal-packages/run-engine/src/engine/tests/batchTwoPhase.test.ts(1 hunks)internal-packages/run-engine/src/engine/types.ts(3 hunks)
💤 Files with no reviewable changes (3)
- apps/webapp/app/runEngine/types.ts
- apps/webapp/app/v3/services/triggerTask.server.ts
- apps/webapp/app/runEngine/concerns/runNumbers.server.ts
🚧 Files skipped from review as they are similar to previous changes (2)
- internal-packages/run-engine/src/engine/tests/batchTwoPhase.test.ts
- internal-packages/run-engine/src/engine/types.ts
🧰 Additional context used
📓 Path-based instructions (6)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
**/*.{ts,tsx}: Use types over interfaces for TypeScript
Avoid using enums; prefer string unions or const objects instead
Files:
internal-packages/run-engine/src/engine/index.tsinternal-packages/run-engine/src/engine/systems/batchSystem.tsapps/webapp/app/runEngine/services/triggerTask.server.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Use function declarations instead of default exports
Files:
internal-packages/run-engine/src/engine/index.tsinternal-packages/run-engine/src/engine/systems/batchSystem.tsapps/webapp/app/runEngine/services/triggerTask.server.ts
**/*.{js,ts,jsx,tsx,json,md,css,scss}
📄 CodeRabbit inference engine (AGENTS.md)
Format code using Prettier
Files:
internal-packages/run-engine/src/engine/index.tsinternal-packages/run-engine/src/engine/systems/batchSystem.tsapps/webapp/app/runEngine/services/triggerTask.server.ts
{packages/core,apps/webapp}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Use zod for validation in packages/core and apps/webapp
Files:
apps/webapp/app/runEngine/services/triggerTask.server.ts
apps/webapp/app/**/*.{ts,tsx}
📄 CodeRabbit inference engine (.cursor/rules/webapp.mdc)
Access all environment variables through the
envexport ofenv.server.tsinstead of directly accessingprocess.envin the Trigger.dev webapp
Files:
apps/webapp/app/runEngine/services/triggerTask.server.ts
apps/webapp/**/*.{ts,tsx}
📄 CodeRabbit inference engine (.cursor/rules/webapp.mdc)
apps/webapp/**/*.{ts,tsx}: When importing from@trigger.dev/corein the webapp, use subpath exports from the package.json instead of importing from the root path
Follow the Remix 2.1.0 and Express server conventions when updating the main trigger.dev webapp
Files:
apps/webapp/app/runEngine/services/triggerTask.server.ts
🧠 Learnings (29)
📓 Common learnings
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `tasks.batchTrigger()` to trigger multiple runs of a single task with different payloads
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.trigger()` to trigger multiple different tasks at once from backend code
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.triggerByTaskAndWait()` to batch trigger tasks by passing task instances and wait for results
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.triggerByTask()` to batch trigger tasks by passing task instances for static task sets
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.triggerAndWait()` to batch trigger multiple different tasks and wait for results
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `yourTask.batchTrigger()` to trigger multiple runs of a task from inside another task
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `runs.subscribeToBatch()` to subscribe to changes for all runs in a batch
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `runs.subscribeToBatch()` to subscribe to changes for all runs in a batch
Applied to files:
internal-packages/run-engine/src/engine/index.tsinternal-packages/run-engine/src/engine/systems/batchSystem.ts
📚 Learning: 2025-11-27T16:26:58.661Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/webapp.mdc:0-0
Timestamp: 2025-11-27T16:26:58.661Z
Learning: Use the Run Engine 2.0 from `internal/run-engine` for new run lifecycle code in the webapp instead of the legacy run engine
Applied to files:
internal-packages/run-engine/src/engine/index.ts
📚 Learning: 2025-08-14T18:35:44.370Z
Learnt from: nicktrn
Repo: triggerdotdev/trigger.dev PR: 2390
File: apps/webapp/app/env.server.ts:764-765
Timestamp: 2025-08-14T18:35:44.370Z
Learning: The BoolEnv helper in apps/webapp/app/utils/boolEnv.ts uses z.preprocess with inconsistent default value types across the codebase - some usages pass boolean defaults (correct) while others pass string defaults (incorrect), leading to type confusion. The helper should enforce boolean-only defaults or have clearer documentation.
Applied to files:
internal-packages/run-engine/src/engine/index.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.trigger()` to trigger multiple different tasks at once from backend code
Applied to files:
internal-packages/run-engine/src/engine/index.tsapps/webapp/app/runEngine/services/triggerTask.server.ts
📚 Learning: 2025-11-27T16:26:37.432Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-11-27T16:26:37.432Z
Learning: Applies to internal-packages/database/**/*.{ts,tsx} : Use Prisma for database interactions in internal-packages/database with PostgreSQL
Applied to files:
internal-packages/run-engine/src/engine/index.ts
📚 Learning: 2025-11-27T16:26:58.661Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/webapp.mdc:0-0
Timestamp: 2025-11-27T16:26:58.661Z
Learning: Leverage the PostgreSQL database through the `trigger.dev/database` Prisma 6.14.0 client in the webapp for all data access patterns
Applied to files:
internal-packages/run-engine/src/engine/index.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger.config.ts : Specify runtime environment (node or bun) in trigger.config.ts using the `runtime` property
Applied to files:
internal-packages/run-engine/src/engine/index.ts
📚 Learning: 2025-11-27T16:26:37.432Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-11-27T16:26:37.432Z
Learning: Applies to packages/trigger-sdk/**/*.{ts,tsx} : In the Trigger.dev SDK (packages/trigger-sdk), prefer isomorphic code like fetch and ReadableStream instead of Node.js-specific code
Applied to files:
internal-packages/run-engine/src/engine/index.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.triggerAndWait()` to batch trigger multiple different tasks and wait for results
Applied to files:
internal-packages/run-engine/src/engine/index.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `yourTask.batchTrigger()` to trigger multiple runs of a task from inside another task
Applied to files:
internal-packages/run-engine/src/engine/index.tsinternal-packages/run-engine/src/engine/systems/batchSystem.tsapps/webapp/app/runEngine/services/triggerTask.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `tasks.batchTrigger()` to trigger multiple runs of a single task with different payloads
Applied to files:
internal-packages/run-engine/src/engine/index.tsapps/webapp/app/runEngine/services/triggerTask.server.ts
📚 Learning: 2025-10-08T11:48:12.327Z
Learnt from: nicktrn
Repo: triggerdotdev/trigger.dev PR: 2593
File: packages/core/src/v3/workers/warmStartClient.ts:168-170
Timestamp: 2025-10-08T11:48:12.327Z
Learning: The trigger.dev runners execute only in Node 21 and 22 environments, so modern Node.js APIs like AbortSignal.any (introduced in v20.3.0) are supported.
Applied to files:
internal-packages/run-engine/src/engine/index.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `idempotencyKeyTTL` option to define a time window during which duplicate triggers return the original run
Applied to files:
internal-packages/run-engine/src/engine/index.tsapps/webapp/app/runEngine/services/triggerTask.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `idempotencyKeys.create()` to create idempotency keys for preventing duplicate task executions
Applied to files:
internal-packages/run-engine/src/engine/index.tsapps/webapp/app/runEngine/services/triggerTask.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.triggerByTaskAndWait()` to batch trigger tasks by passing task instances and wait for results
Applied to files:
internal-packages/run-engine/src/engine/systems/batchSystem.tsapps/webapp/app/runEngine/services/triggerTask.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Export tasks with unique IDs within the project to enable proper task discovery and execution
Applied to files:
apps/webapp/app/runEngine/services/triggerTask.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `tasks.trigger()` with type-only imports to trigger tasks from backend code without importing the task implementation
Applied to files:
apps/webapp/app/runEngine/services/triggerTask.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use the `task()` function from `trigger.dev/sdk/v3` to define tasks with id and run properties
Applied to files:
apps/webapp/app/runEngine/services/triggerTask.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `schedules.task()` for scheduled/cron tasks instead of regular `task()`
Applied to files:
apps/webapp/app/runEngine/services/triggerTask.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.triggerByTask()` to batch trigger tasks by passing task instances for static task sets
Applied to files:
apps/webapp/app/runEngine/services/triggerTask.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use metadata methods (set, del, replace, append, remove, increment, decrement, stream, flush) to update metadata during task execution
Applied to files:
apps/webapp/app/runEngine/services/triggerTask.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Subscribe to run updates using `runs.subscribeToRun()` for realtime monitoring of task execution
Applied to files:
apps/webapp/app/runEngine/services/triggerTask.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Attach metadata to task runs using the metadata option when triggering, and access/update it inside runs using metadata functions
Applied to files:
apps/webapp/app/runEngine/services/triggerTask.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger.config.ts : Configure OpenTelemetry instrumentations and exporters in trigger.config.ts for enhanced logging
Applied to files:
apps/webapp/app/runEngine/services/triggerTask.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `schemaTask()` from `trigger.dev/sdk/v3` with Zod schema for payload validation
Applied to files:
apps/webapp/app/runEngine/services/triggerTask.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `yourTask.trigger()` to trigger a task from inside another task with specified payload
Applied to files:
apps/webapp/app/runEngine/services/triggerTask.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `yourTask.batchTriggerAndWait()` to batch trigger tasks and wait for all results from a parent task
Applied to files:
apps/webapp/app/runEngine/services/triggerTask.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `yourTask.triggerAndWait()` to trigger a task and wait for its result from a parent task
Applied to files:
apps/webapp/app/runEngine/services/triggerTask.server.ts
🧬 Code graph analysis (1)
internal-packages/run-engine/src/engine/systems/batchSystem.ts (1)
apps/webapp/app/runEngine/services/batchTrigger.server.ts (2)
batch(521-664)batch(666-722)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (24)
- GitHub Check: Cursor Bugbot
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (8, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (1, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (6, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (4, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (5, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (7, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (5, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (2, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (3, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (3, 8)
- GitHub Check: typecheck / typecheck
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (1, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (6, 8)
- GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - npm)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (8, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (7, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (4, 8)
- GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - npm)
- GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - pnpm)
- GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - pnpm)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (2, 8)
- GitHub Check: units / packages / 🧪 Unit Tests: Packages (1, 1)
- GitHub Check: Analyze (javascript-typescript)
🔇 Additional comments (7)
internal-packages/run-engine/src/engine/systems/batchSystem.ts (1)
47-49: LGTM!The Prisma selection correctly includes the new fields needed to support v2 batch processing with success/failure counts.
internal-packages/run-engine/src/engine/index.ts (5)
23-23: LGTM!The import additions correctly bring in the BatchQueue infrastructure and related types needed for the v3 two-phase batch processing system.
Also applies to: 31-38
84-84: LGTM!The
batchQueuefield is appropriately declared as private and used consistently throughout the class.
321-332: LGTM!The BatchQueue initialization is well-structured with sensible defaults and helpful logging. The configuration correctly ties consumer startup to the worker disabled flag.
Also applies to: 334-353
975-1056: LGTM!The new BatchQueue methods are well-documented and provide a clean API surface for the two-phase batch processing system. The JSDoc comments clearly explain the purpose and usage of each method, especially the distinction between Phase 1 (initialization) and Phase 2 (item streaming).
1320-1321: LGTM!The BatchQueue cleanup is correctly integrated into the shutdown sequence, ensuring proper resource cleanup and Redis connection closure.
apps/webapp/app/runEngine/services/triggerTask.server.ts (1)
270-274: Good: trace/span attributes are set consistently (queueName + runId).
| const payloadPacket = await this.payloadProcessor.process(triggerRequest); | ||
|
|
||
| const taskRun = await this.engine.trigger( | ||
| { | ||
| friendlyId: runFriendlyId, | ||
| environment: environment, | ||
| idempotencyKey, | ||
| idempotencyKeyExpiresAt: idempotencyKey ? idempotencyKeyExpiresAt : undefined, | ||
| taskIdentifier: taskId, | ||
| payload: payloadPacket.data ?? "", | ||
| payloadType: payloadPacket.dataType, | ||
| context: body.context, | ||
| traceContext: this.#propagateExternalTraceContext( | ||
| event.traceContext, | ||
| parentRun?.traceContext, | ||
| event.traceparent?.spanId | ||
| ), | ||
| traceId: event.traceId, | ||
| spanId: event.spanId, | ||
| parentSpanId: | ||
| options.parentAsLinkType === "replay" ? undefined : event.traceparent?.spanId, | ||
| replayedFromTaskRunFriendlyId: options.replayedFromTaskRunFriendlyId, | ||
| lockedToVersionId: lockedToBackgroundWorker?.id, | ||
| taskVersion: lockedToBackgroundWorker?.version, | ||
| sdkVersion: lockedToBackgroundWorker?.sdkVersion, | ||
| cliVersion: lockedToBackgroundWorker?.cliVersion, | ||
| concurrencyKey: body.options?.concurrencyKey, | ||
| queue: queueName, | ||
| lockedQueueId, | ||
| workerQueue, | ||
| isTest: body.options?.test ?? false, | ||
| delayUntil, | ||
| queuedAt: delayUntil ? undefined : new Date(), | ||
| maxAttempts: body.options?.maxAttempts, | ||
| taskEventStore: store, | ||
| ttl, | ||
| tags, | ||
| oneTimeUseToken: options.oneTimeUseToken, | ||
| parentTaskRunId: parentRun?.id, | ||
| rootTaskRunId: parentRun?.rootTaskRunId ?? parentRun?.id, | ||
| batch: options?.batchId | ||
| ? { | ||
| id: options.batchId, | ||
| index: options.batchIndex ?? 0, | ||
| } | ||
| : undefined, | ||
| resumeParentOnCompletion: body.options?.resumeParentOnCompletion, | ||
| depth, | ||
| metadata: metadataPacket?.data, | ||
| metadataType: metadataPacket?.dataType, | ||
| seedMetadata: metadataPacket?.data, | ||
| seedMetadataType: metadataPacket?.dataType, | ||
| maxDurationInSeconds: body.options?.maxDuration | ||
| ? clampMaxDuration(body.options.maxDuration) | ||
| : undefined, | ||
| machine: body.options?.machine, | ||
| priorityMs: body.options?.priority ? body.options.priority * 1_000 : undefined, | ||
| queueTimestamp: | ||
| options.queueTimestamp ?? | ||
| (parentRun && body.options?.resumeParentOnCompletion | ||
| ? parentRun.queueTimestamp ?? undefined | ||
| : undefined), | ||
| scheduleId: options.scheduleId, | ||
| scheduleInstanceId: options.scheduleInstanceId, | ||
| createdAt: options.overrideCreatedAt, | ||
| bulkActionId: body.options?.bulkActionId, | ||
| planType, | ||
| realtimeStreamsVersion: options.realtimeStreamsVersion, | ||
| }, | ||
| this.prisma | ||
| ); |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Find PayloadProcessor definition and usage
echo "=== Searching for PayloadProcessor interface/type definition ==="
rg -n "interface PayloadProcessor|type PayloadProcessor|class PayloadProcessor" --max-count=20
echo ""
echo "=== Searching for PayloadProcessor.process() method ==="
rg -n "\.process\(" -B2 -A5 | head -100
echo ""
echo "=== Checking where PayloadProcessor is imported/defined ==="
fd "payloadProcessor" --type f
echo ""
echo "=== Searching for RunDuplicateIdempotencyKeyError and retry logic ==="
rg -n "RunDuplicateIdempotencyKeyError" -B2 -A2 | head -50Repository: triggerdotdev/trigger.dev
Length of output: 7603
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Find the file and look around line 365 for error handling
echo "=== Looking for error handling around line 365 ==="
sed -n '360,380p' apps/webapp/app/runEngine/services/triggerTask.server.ts
echo ""
echo "=== Search for idempotency key usage in payload processing ==="
rg -n "idempotencyKey" apps/webapp/app/runEngine/services/triggerTask.server.ts -C2
echo ""
echo "=== Look for PayloadProcessor initialization ==="
rg -n "payloadProcessor" apps/webapp/app/runEngine/services/triggerTask.server.ts -B2 -A2Repository: triggerdotdev/trigger.dev
Length of output: 3729
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Find PayloadProcessor implementation - likely in core packages or the webapp
echo "=== Finding PayloadProcessor files ==="
find . -name "*.ts" -type f | xargs rg -l "PayloadProcessor" | head -20
echo ""
echo "=== Search for the complete PayloadProcessor interface ==="
rg -n "interface PayloadProcessor|type PayloadProcessor" -A10Repository: triggerdotdev/trigger.dev
Length of output: 1259
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Check the PayloadProcessor implementation
echo "=== PayloadProcessor implementation in payloads.server.ts ==="
cat -n apps/webapp/app/runEngine/concerns/payloads.server.ts | head -150
echo ""
echo "=== Check for storage/R2 offload logic ==="
rg -n "offload|storage|r2|s3" apps/webapp/app/runEngine/concerns/payloads.server.ts -iRepository: triggerdotdev/trigger.dev
Length of output: 2899
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Get full file size and read the entire payloads.server.ts
echo "=== File size ==="
wc -l apps/webapp/app/runEngine/concerns/payloads.server.ts
echo ""
echo "=== Full content ==="
cat -n apps/webapp/app/runEngine/concerns/payloads.server.tsRepository: triggerdotdev/trigger.dev
Length of output: 2529
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Check the complete retry flow to understand when process() is called
echo "=== Viewing the complete call() method flow ==="
sed -n '130,230p' apps/webapp/app/runEngine/services/triggerTask.server.ts
echo ""
echo "=== Check the catch block and retry logic ==="
sed -n '360,385p' apps/webapp/app/runEngine/services/triggerTask.server.tsRepository: triggerdotdev/trigger.dev
Length of output: 4149
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Get the complete sequence with context
echo "=== Lines 190-280 to see exact flow ==="
sed -n '190,280p' apps/webapp/app/runEngine/services/triggerTask.server.tsRepository: triggerdotdev/trigger.dev
Length of output: 3021
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Check what happens when isCached is true
echo "=== IdempotencyKeyConcern to understand caching ==="
rg -n "class IdempotencyKeyConcern|handleTriggerRequest" apps/webapp/app/runEngine/concerns/idempotencyKeys.server.ts -A15 | head -60Repository: triggerdotdev/trigger.dev
Length of output: 1068
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Get the complete IdempotencyKeyConcern implementation
echo "=== Full IdempotencyKeyConcern file ==="
wc -l apps/webapp/app/runEngine/concerns/idempotencyKeys.server.ts
echo ""
cat -n apps/webapp/app/runEngine/concerns/idempotencyKeys.server.tsRepository: triggerdotdev/trigger.dev
Length of output: 6058
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Find where RunDuplicateIdempotencyKeyError is thrown in the engine
echo "=== Where RunDuplicateIdempotencyKeyError is thrown ==="
rg -n "RunDuplicateIdempotencyKeyError" -B10 -A5 internal-packages/run-engine/src/engine/index.ts | head -100
echo ""
echo "=== Check the transaction context around the error ==="
sed -n '540,570p' internal-packages/run-engine/src/engine/index.tsRepository: triggerdotdev/trigger.dev
Length of output: 3236
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Look for where the idempotency key is inserted/saved
echo "=== Search for where idempotencyKey is set in the trigger method ==="
rg -n "idempotencyKey" internal-packages/run-engine/src/engine/index.ts -B2 -A2 | head -150Repository: triggerdotdev/trigger.dev
Length of output: 2848
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Check the transaction context and where the taskRun is created
echo "=== Looking for the transaction block where taskRun is created ==="
sed -n '430,550p' internal-packages/run-engine/src/engine/index.tsRepository: triggerdotdev/trigger.dev
Length of output: 4149
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Check if there's prisma.$transaction usage
echo "=== Searching for transaction patterns ==="
rg -n "prisma.\\\$transaction" internal-packages/run-engine/src/engine/index.ts -B3 -A3Repository: triggerdotdev/trigger.dev
Length of output: 111
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Check how engine.trigger is called from triggerTask.server.ts
echo "=== Context around engine.trigger call ==="
sed -n '265,280p' apps/webapp/app/runEngine/services/triggerTask.server.ts
echo ""
echo "=== Check what this.prisma is in the context ==="
sed -n '50,90p' apps/webapp/app/runEngine/services/triggerTask.server.ts
echo ""
echo "=== Examine the exact retry call ==="
sed -n '363,375p' apps/webapp/app/runEngine/services/triggerTask.server.tsRepository: triggerdotdev/trigger.dev
Length of output: 2869
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Verify if the call is wrapped in a transaction at the service level
echo "=== Check the service's call method signature ==="
sed -n '100,135p' apps/webapp/app/runEngine/services/triggerTask.server.ts
echo ""
echo "=== Look for transaction usage in the main call method ==="
rg -n "prisma.\\\$transaction" apps/webapp/app/runEngine/services/triggerTask.server.ts -B3 -A3Repository: triggerdotdev/trigger.dev
Length of output: 1194
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Check if runFriendlyId is generated fresh or reused on retry
echo "=== Check runFriendlyId generation ==="
sed -n '115,125p' apps/webapp/app/runEngine/services/triggerTask.server.ts
echo ""
echo "=== Check if it's passed in options ==="
rg -n "runFriendlyId" apps/webapp/app/runEngine/services/triggerTask.server.ts | head -20Repository: triggerdotdev/trigger.dev
Length of output: 690
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Check the call signature to see what options contains
echo "=== Check the call method signature and what options are ==="
sed -n '100,110p' apps/webapp/app/runEngine/services/triggerTask.server.ts
echo ""
echo "=== Check what TriggerTaskServiceOptions contains ==="
rg -n "type.*TriggerTaskServiceOptions|interface.*TriggerTaskServiceOptions" -A30Repository: triggerdotdev/trigger.dev
Length of output: 13971
Ensure payload processing is idempotent when retrying on idempotency key race conditions.
On retry after RunDuplicateIdempotencyKeyError, payloadProcessor.process() executes again. The issue: If runFriendlyId is not preserved in options across retries (line 104 generates a new one when not in options), a new payload is uploaded to R2 with a different filename, orphaning the first upload. Additionally, the idempotency key concern cannot short-circuit the re-processing on retry because the initial taskRun creation failed (constraint violation), leaving no cached record. To fix this, either (1) preserve runFriendlyId in options across retries, (2) key payload offloads by idempotency key instead of friendlyId, or (3) add deduplication to uploadPacketToObjectStore() based on idempotency key.
🤖 Prompt for AI Agents
In apps/webapp/app/runEngine/services/triggerTask.server.ts around lines
275-345, payloadProcessor.process() is re-run on idempotency key retry which can
create a different runFriendlyId and orphan the first payload upload; preserve
the runFriendlyId across retries by setting options.runFriendlyId (or another
stable identifier) before calling payloadProcessor.process() so the same
filename is used on retries; alternatively, change uploadPacketToObjectStore()
to deduplicate by idempotencyKey (or key off idempotencyKey instead of
friendlyId) so repeated uploads reuse the same object rather than creating new
ones.
There was a problem hiding this comment.
Actionable comments posted: 3
♻️ Duplicate comments (2)
apps/webapp/app/runEngine/services/createBatch.server.ts (1)
119-153: Race condition: idempotency may be saved before batch is fully initialized.The event at line 121 triggers idempotency key saving asynchronously via
Evt.post(), which doesn't await handlers. IfinitializeBatch(line 153) fails after the idempotency key is saved, subsequent retries will return a cached batch that isn't fully initialized in Redis.Consider moving
onBatchTaskRunCreated.post(batch)afterinitializeBatchcompletes, or ensuring idempotency is only saved after all initialization succeeds.internal-packages/run-engine/src/engine/index.ts (1)
325-346: TLS configuration is incorrectly converted to boolean.Line 333 converts the TLS options object to
trueinstead of passing the actual TLS configuration, losing any custom settings (certificates, ciphers, etc.).Apply this diff:
keyPrefix: `${options.batchQueue?.redis.keyPrefix ?? ""}batch-queue:`, enableAutoPipelining: options.batchQueue?.redis.enableAutoPipelining ?? true, - tls: options.batchQueue?.redis.tls !== undefined, + tls: options.batchQueue?.redis.tls, },
🧹 Nitpick comments (1)
apps/webapp/app/entry.server.tsx (1)
1-25: Minor: remove stale inline comment on Remix node import
import { createReadableStreamFromReadable, type EntryContext } from "@remix-run/node"; // or cloudflare/deno(Line 1) reads like leftover guidance; prefer removing to reduce noise.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (8)
apps/webapp/app/entry.server.tsx(2 hunks)apps/webapp/app/routes/api.v3.batches.$batchId.items.ts(1 hunks)apps/webapp/app/routes/api.v3.batches.ts(1 hunks)apps/webapp/app/runEngine/services/createBatch.server.ts(1 hunks)apps/webapp/app/v3/runEngine.server.ts(2 hunks)apps/webapp/app/v3/runEngineHandlers.server.ts(2 hunks)apps/webapp/test/engine/triggerTask.test.ts(0 hunks)internal-packages/run-engine/src/engine/index.ts(5 hunks)
💤 Files with no reviewable changes (1)
- apps/webapp/test/engine/triggerTask.test.ts
🚧 Files skipped from review as they are similar to previous changes (3)
- apps/webapp/app/routes/api.v3.batches.ts
- apps/webapp/app/routes/api.v3.batches.$batchId.items.ts
- apps/webapp/app/v3/runEngine.server.ts
🧰 Additional context used
📓 Path-based instructions (6)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
**/*.{ts,tsx}: Use types over interfaces for TypeScript
Avoid using enums; prefer string unions or const objects instead
Files:
apps/webapp/app/v3/runEngineHandlers.server.tsapps/webapp/app/runEngine/services/createBatch.server.tsapps/webapp/app/entry.server.tsxinternal-packages/run-engine/src/engine/index.ts
{packages/core,apps/webapp}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Use zod for validation in packages/core and apps/webapp
Files:
apps/webapp/app/v3/runEngineHandlers.server.tsapps/webapp/app/runEngine/services/createBatch.server.tsapps/webapp/app/entry.server.tsx
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Use function declarations instead of default exports
Files:
apps/webapp/app/v3/runEngineHandlers.server.tsapps/webapp/app/runEngine/services/createBatch.server.tsapps/webapp/app/entry.server.tsxinternal-packages/run-engine/src/engine/index.ts
apps/webapp/app/**/*.{ts,tsx}
📄 CodeRabbit inference engine (.cursor/rules/webapp.mdc)
Access all environment variables through the
envexport ofenv.server.tsinstead of directly accessingprocess.envin the Trigger.dev webapp
Files:
apps/webapp/app/v3/runEngineHandlers.server.tsapps/webapp/app/runEngine/services/createBatch.server.tsapps/webapp/app/entry.server.tsx
apps/webapp/**/*.{ts,tsx}
📄 CodeRabbit inference engine (.cursor/rules/webapp.mdc)
apps/webapp/**/*.{ts,tsx}: When importing from@trigger.dev/corein the webapp, use subpath exports from the package.json instead of importing from the root path
Follow the Remix 2.1.0 and Express server conventions when updating the main trigger.dev webapp
Files:
apps/webapp/app/v3/runEngineHandlers.server.tsapps/webapp/app/runEngine/services/createBatch.server.tsapps/webapp/app/entry.server.tsx
**/*.{js,ts,jsx,tsx,json,md,css,scss}
📄 CodeRabbit inference engine (AGENTS.md)
Format code using Prettier
Files:
apps/webapp/app/v3/runEngineHandlers.server.tsapps/webapp/app/runEngine/services/createBatch.server.tsapps/webapp/app/entry.server.tsxinternal-packages/run-engine/src/engine/index.ts
🧠 Learnings (26)
📓 Common learnings
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `tasks.batchTrigger()` to trigger multiple runs of a single task with different payloads
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.trigger()` to trigger multiple different tasks at once from backend code
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.triggerByTaskAndWait()` to batch trigger tasks by passing task instances and wait for results
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.triggerAndWait()` to batch trigger multiple different tasks and wait for results
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `runs.subscribeToBatch()` to subscribe to changes for all runs in a batch
Applied to files:
apps/webapp/app/v3/runEngineHandlers.server.tsapps/webapp/app/runEngine/services/createBatch.server.tsinternal-packages/run-engine/src/engine/index.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.trigger()` to trigger multiple different tasks at once from backend code
Applied to files:
apps/webapp/app/v3/runEngineHandlers.server.tsapps/webapp/app/runEngine/services/createBatch.server.tsinternal-packages/run-engine/src/engine/index.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `tasks.batchTrigger()` to trigger multiple runs of a single task with different payloads
Applied to files:
apps/webapp/app/v3/runEngineHandlers.server.tsapps/webapp/app/runEngine/services/createBatch.server.tsinternal-packages/run-engine/src/engine/index.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.triggerByTask()` to batch trigger tasks by passing task instances for static task sets
Applied to files:
apps/webapp/app/v3/runEngineHandlers.server.tsapps/webapp/app/runEngine/services/createBatch.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `yourTask.batchTrigger()` to trigger multiple runs of a task from inside another task
Applied to files:
apps/webapp/app/v3/runEngineHandlers.server.tsapps/webapp/app/runEngine/services/createBatch.server.tsinternal-packages/run-engine/src/engine/index.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.triggerByTaskAndWait()` to batch trigger tasks by passing task instances and wait for results
Applied to files:
apps/webapp/app/v3/runEngineHandlers.server.tsapps/webapp/app/runEngine/services/createBatch.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.triggerAndWait()` to batch trigger multiple different tasks and wait for results
Applied to files:
apps/webapp/app/v3/runEngineHandlers.server.tsapps/webapp/app/runEngine/services/createBatch.server.tsinternal-packages/run-engine/src/engine/index.ts
📚 Learning: 2025-11-27T16:26:58.661Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/webapp.mdc:0-0
Timestamp: 2025-11-27T16:26:58.661Z
Learning: Use the Run Engine 2.0 from `internal/run-engine` for new run lifecycle code in the webapp instead of the legacy run engine
Applied to files:
apps/webapp/app/v3/runEngineHandlers.server.tsapps/webapp/app/entry.server.tsxinternal-packages/run-engine/src/engine/index.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `yourTask.batchTriggerAndWait()` to batch trigger tasks and wait for all results from a parent task
Applied to files:
apps/webapp/app/v3/runEngineHandlers.server.tsapps/webapp/app/runEngine/services/createBatch.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Attach metadata to task runs using the metadata option when triggering, and access/update it inside runs using metadata functions
Applied to files:
apps/webapp/app/v3/runEngineHandlers.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use the `task()` function from `trigger.dev/sdk/v3` to define tasks with id and run properties
Applied to files:
apps/webapp/app/v3/runEngineHandlers.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `idempotencyKeys.create()` to create idempotency keys for preventing duplicate task executions
Applied to files:
apps/webapp/app/runEngine/services/createBatch.server.tsinternal-packages/run-engine/src/engine/index.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `idempotencyKeyTTL` option to define a time window during which duplicate triggers return the original run
Applied to files:
apps/webapp/app/runEngine/services/createBatch.server.tsinternal-packages/run-engine/src/engine/index.ts
📚 Learning: 2025-11-27T16:26:58.661Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/webapp.mdc:0-0
Timestamp: 2025-11-27T16:26:58.661Z
Learning: Applies to apps/webapp/**/*.{ts,tsx} : Follow the Remix 2.1.0 and Express server conventions when updating the main trigger.dev webapp
Applied to files:
apps/webapp/app/entry.server.tsx
📚 Learning: 2025-11-27T16:26:58.661Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/webapp.mdc:0-0
Timestamp: 2025-11-27T16:26:58.661Z
Learning: Applies to apps/webapp/app/**/*.{ts,tsx} : Access all environment variables through the `env` export of `env.server.ts` instead of directly accessing `process.env` in the Trigger.dev webapp
Applied to files:
apps/webapp/app/entry.server.tsx
📚 Learning: 2025-11-27T16:26:58.661Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/webapp.mdc:0-0
Timestamp: 2025-11-27T16:26:58.661Z
Learning: Applies to apps/webapp/**/*.{ts,tsx} : When importing from `trigger.dev/core` in the webapp, use subpath exports from the package.json instead of importing from the root path
Applied to files:
apps/webapp/app/entry.server.tsx
📚 Learning: 2025-11-27T16:26:58.661Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/webapp.mdc:0-0
Timestamp: 2025-11-27T16:26:58.661Z
Learning: Applies to apps/webapp/app/services/**/*.server.{ts,tsx} : Separate testable services from configuration files; follow the pattern of `realtimeClient.server.ts` (testable service) and `realtimeClientGlobal.server.ts` (configuration) in the webapp
Applied to files:
apps/webapp/app/entry.server.tsx
📚 Learning: 2025-11-27T16:26:58.661Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/webapp.mdc:0-0
Timestamp: 2025-11-27T16:26:58.661Z
Learning: Applies to apps/webapp/app/v3/services/**/*.server.{ts,tsx} : Organize services in the webapp following the pattern `app/v3/services/*/*.server.ts`
Applied to files:
apps/webapp/app/entry.server.tsx
📚 Learning: 2025-11-27T16:26:37.432Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-11-27T16:26:37.432Z
Learning: The webapp at apps/webapp is a Remix 2.1 application using Node.js v20
Applied to files:
apps/webapp/app/entry.server.tsx
📚 Learning: 2025-08-14T18:35:44.370Z
Learnt from: nicktrn
Repo: triggerdotdev/trigger.dev PR: 2390
File: apps/webapp/app/env.server.ts:764-765
Timestamp: 2025-08-14T18:35:44.370Z
Learning: The BoolEnv helper in apps/webapp/app/utils/boolEnv.ts uses z.preprocess with inconsistent default value types across the codebase - some usages pass boolean defaults (correct) while others pass string defaults (incorrect), leading to type confusion. The helper should enforce boolean-only defaults or have clearer documentation.
Applied to files:
internal-packages/run-engine/src/engine/index.ts
📚 Learning: 2025-11-27T16:26:37.432Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-11-27T16:26:37.432Z
Learning: Applies to internal-packages/database/**/*.{ts,tsx} : Use Prisma for database interactions in internal-packages/database with PostgreSQL
Applied to files:
internal-packages/run-engine/src/engine/index.ts
📚 Learning: 2025-11-27T16:26:58.661Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/webapp.mdc:0-0
Timestamp: 2025-11-27T16:26:58.661Z
Learning: Leverage the PostgreSQL database through the `trigger.dev/database` Prisma 6.14.0 client in the webapp for all data access patterns
Applied to files:
internal-packages/run-engine/src/engine/index.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger.config.ts : Specify runtime environment (node or bun) in trigger.config.ts using the `runtime` property
Applied to files:
internal-packages/run-engine/src/engine/index.ts
📚 Learning: 2025-11-27T16:26:37.432Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-11-27T16:26:37.432Z
Learning: Applies to packages/trigger-sdk/**/*.{ts,tsx} : In the Trigger.dev SDK (packages/trigger-sdk), prefer isomorphic code like fetch and ReadableStream instead of Node.js-specific code
Applied to files:
internal-packages/run-engine/src/engine/index.ts
📚 Learning: 2025-10-08T11:48:12.327Z
Learnt from: nicktrn
Repo: triggerdotdev/trigger.dev PR: 2593
File: packages/core/src/v3/workers/warmStartClient.ts:168-170
Timestamp: 2025-10-08T11:48:12.327Z
Learning: The trigger.dev runners execute only in Node 21 and 22 environments, so modern Node.js APIs like AbortSignal.any (introduced in v20.3.0) are supported.
Applied to files:
internal-packages/run-engine/src/engine/index.ts
🧬 Code graph analysis (2)
apps/webapp/app/v3/runEngineHandlers.server.ts (4)
apps/webapp/app/v3/runEngine.server.ts (1)
engine(11-11)internal-packages/tracing/src/index.ts (1)
SpanKind(34-34)apps/webapp/app/v3/services/triggerTask.server.ts (1)
TriggerTaskService(51-120)apps/webapp/app/db.server.ts (1)
prisma(101-101)
apps/webapp/app/entry.server.tsx (1)
apps/webapp/app/v3/runEngineHandlers.server.ts (1)
setupBatchQueueCallbacks(645-794)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (24)
- GitHub Check: Cursor Bugbot
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (4, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (5, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (4, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (7, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (1, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (2, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (8, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (8, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (3, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (5, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (2, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (6, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (3, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (7, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (6, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (1, 8)
- GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - npm)
- GitHub Check: units / packages / 🧪 Unit Tests: Packages (1, 1)
- GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - pnpm)
- GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - pnpm)
- GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - npm)
- GitHub Check: typecheck / typecheck
- GitHub Check: Analyze (javascript-typescript)
🔇 Additional comments (11)
apps/webapp/app/runEngine/services/createBatch.server.ts (4)
1-12: LGTM!Imports are properly structured with subpath imports from
@trigger.dev/core/v3as per coding guidelines.
14-20: LGTM!Type definition uses
typeoverinterfaceand properly types optional fields with string union forrealtimeStreamsVersion.
34-44: LGTM!Class structure follows the existing service pattern in this codebase, extending
WithRunEngineand initializing dependencies in the constructor.
173-203: LGTM!Error handling properly distinguishes between
oneTimeUseTokenandidempotencyKeyunique constraint violations, providing clear user-facing error messages viaServiceValidationError.internal-packages/run-engine/src/engine/index.ts (3)
31-38: LGTM!BatchQueue imports and types are properly structured with
.jsextensions for ESM compatibility.
964-1048: LGTM!BatchQueue wrapper methods are well-documented with clear JSDoc comments and provide appropriate delegation to the underlying
BatchQueueinstance.
1303-1318: LGTM!BatchQueue is properly closed during engine shutdown, ensuring Redis connections are cleaned up.
apps/webapp/app/v3/runEngineHandlers.server.ts (2)
645-727: Reduce per-item overhead: reuseTriggerTaskServiceinstance
Creatingnew TriggerTaskService()per item (Line 662) will add avoidable overhead at high throughput. Prefer instantiating once insetupBatchQueueCallbacks()and closing over it.export function setupBatchQueueCallbacks() { + const triggerTaskService = new TriggerTaskService(); // Item processing callback - creates a run for each batch item engine.setBatchProcessItemCallback(async ({ batchId, friendlyId, itemIndex, item, meta }) => { … - const triggerTaskService = new TriggerTaskService(); - // Normalize payload - for application/store (R2 paths), this passes through as-is const payload = normalizePayload(item.payload, item.payloadType);[ suggest_recommended_refactor ]
1-19: Fix incomplete environment object passed to determineEngineVersionThe constructed environment object at line 675 is missing the
project.enginefield required bydetermineEngineVersion(). This will cause a runtime crash when the batch queue tries to determine the engine version.The object currently provides only
project: { id: ... }, butdetermineEngineVersionaccessesenvironment.project.engineto determine the RunEngineVersion. Additionally, theas AuthenticatedEnvironmentcast masks this incompatibility.Fetch the full environment from the database (via
findEnvironmentFromRunalready imported at line 10, or use the prisma query directly) to ensure all required fields are available, rather than constructing a partial object.⛔ Skipped due to learnings
Learnt from: CR Repo: triggerdotdev/trigger.dev PR: 0 File: .cursor/rules/writing-tasks.mdc:0-0 Timestamp: 2025-11-27T16:27:35.304Z Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `yourTask.batchTrigger()` to trigger multiple runs of a task from inside another taskLearnt from: CR Repo: triggerdotdev/trigger.dev PR: 0 File: .cursor/rules/writing-tasks.mdc:0-0 Timestamp: 2025-11-27T16:27:35.304Z Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.trigger()` to trigger multiple different tasks at once from backend codeLearnt from: CR Repo: triggerdotdev/trigger.dev PR: 0 File: .cursor/rules/writing-tasks.mdc:0-0 Timestamp: 2025-11-27T16:27:35.304Z Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.triggerByTaskAndWait()` to batch trigger tasks by passing task instances and wait for resultsLearnt from: CR Repo: triggerdotdev/trigger.dev PR: 0 File: .cursor/rules/writing-tasks.mdc:0-0 Timestamp: 2025-11-27T16:27:35.304Z Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.triggerAndWait()` to batch trigger multiple different tasks and wait for resultsLearnt from: CR Repo: triggerdotdev/trigger.dev PR: 0 File: .cursor/rules/writing-tasks.mdc:0-0 Timestamp: 2025-11-27T16:27:35.304Z Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `tasks.batchTrigger()` to trigger multiple runs of a single task with different payloadsLearnt from: CR Repo: triggerdotdev/trigger.dev PR: 0 File: .cursor/rules/writing-tasks.mdc:0-0 Timestamp: 2025-11-27T16:27:35.304Z Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `yourTask.batchTriggerAndWait()` to batch trigger tasks and wait for all results from a parent taskLearnt from: CR Repo: triggerdotdev/trigger.dev PR: 0 File: .cursor/rules/writing-tasks.mdc:0-0 Timestamp: 2025-11-27T16:27:35.304Z Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `runs.subscribeToBatch()` to subscribe to changes for all runs in a batchLearnt from: CR Repo: triggerdotdev/trigger.dev PR: 0 File: .cursor/rules/writing-tasks.mdc:0-0 Timestamp: 2025-11-27T16:27:35.304Z Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.triggerByTask()` to batch trigger tasks by passing task instances for static task setsLearnt from: CR Repo: triggerdotdev/trigger.dev PR: 0 File: .cursor/rules/writing-tasks.mdc:0-0 Timestamp: 2025-11-27T16:27:35.304Z Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Attach metadata to task runs using the metadata option when triggering, and access/update it inside runs using metadata functionsapps/webapp/app/entry.server.tsx (2)
241-242: Good: exportingrunWithHttpContextenables request-scoped async context
This is a nice plumbing improvement for propagating per-request context across async boundaries.
196-203: This concern is based on a misunderstanding of the execution model. WhileWorker.init()andbootstrap()are invoked before thesingleton()calls, neither is awaited, meaning they return promises that execute asynchronously. Thesingleton()calls on lines 236-237 execute synchronously and immediately, registering the handlers before the async work inWorker.init()andbootstrap()begins. Since neither function contains synchronous code that emits events, the handlers will always be registered before any events can fire. The proposed reordering is unnecessary.Likely an incorrect or invalid review comment.
| /** | ||
| * Normalize the payload from BatchQueue. | ||
| * | ||
| * Handles different payload types: | ||
| * - "application/store": Already offloaded to R2, payload is the path - pass through as-is | ||
| * - "application/json": May be a pre-serialized JSON string - parse to avoid double-stringification | ||
| * - Other types: Pass through as-is | ||
| * | ||
| * @param payload - The raw payload from the batch item | ||
| * @param payloadType - The payload type (e.g., "application/json", "application/store") | ||
| */ | ||
| function normalizePayload(payload: unknown, payloadType?: string): unknown { | ||
| // For non-JSON payloads (including application/store for R2-offloaded payloads), | ||
| // return as-is - no normalization needed | ||
| if (payloadType !== "application/json" && payloadType !== undefined) { | ||
| return payload; | ||
| } | ||
|
|
||
| // For JSON payloads, if payload is a string, try to parse it | ||
| // This handles pre-serialized JSON from the SDK | ||
| if (typeof payload === "string") { | ||
| try { | ||
| return JSON.parse(payload); | ||
| } catch { | ||
| // If it's not valid JSON, return as-is | ||
| return payload; | ||
| } | ||
| } | ||
|
|
||
| return payload; | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -e
rg "normalizePayload" -n || true
echo "---- File content ----"
sed -n '1,240p' apps/webapp/app/v3/runEngineHandlers.server.ts | sed -n '760,860p'Repository: triggerdotdev/trigger.dev
Length of output: 346
🏁 Script executed:
#!/bin/bash
set -e
echo "Showing lines around normalizePayload in apps/webapp/app/v3/runEngineHandlers.server.ts"
nl -ba apps/webapp/app/v3/runEngineHandlers.server.ts | sed -n '780,840p'Repository: triggerdotdev/trigger.dev
Length of output: 214
🏁 Script executed:
#!/bin/bash
set -e
file="apps/webapp/app/v3/runEngineHandlers.server.ts"
wc -l "$file" || true
sed -n '640,720p' "$file"
sed -n '780,840p' "$file"
sed -n '800,820p' "$file"Repository: triggerdotdev/trigger.dev
Length of output: 5687
🏁 Script executed:
#!/bin/bash
set -e
rg "payloadType" -n || true
rg "BatchQueue" -n || true
rg "batch.*payload" -n || true
sed -n '1,260p' apps/webapp/app/v3/runEngineHandlers.server.tsRepository: triggerdotdev/trigger.dev
Length of output: 50381
Only parse payloads when payloadType === "application/json"
Current logic will attempt JSON.parse when payloadType is undefined, which can change plain string payloads. Change the condition to return early unless payloadType is explicitly "application/json":
- if (payloadType !== "application/json" && payloadType !== undefined) {
+ if (payloadType !== "application/json") {
return payload;
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| /** | |
| * Normalize the payload from BatchQueue. | |
| * | |
| * Handles different payload types: | |
| * - "application/store": Already offloaded to R2, payload is the path - pass through as-is | |
| * - "application/json": May be a pre-serialized JSON string - parse to avoid double-stringification | |
| * - Other types: Pass through as-is | |
| * | |
| * @param payload - The raw payload from the batch item | |
| * @param payloadType - The payload type (e.g., "application/json", "application/store") | |
| */ | |
| function normalizePayload(payload: unknown, payloadType?: string): unknown { | |
| // For non-JSON payloads (including application/store for R2-offloaded payloads), | |
| // return as-is - no normalization needed | |
| if (payloadType !== "application/json" && payloadType !== undefined) { | |
| return payload; | |
| } | |
| // For JSON payloads, if payload is a string, try to parse it | |
| // This handles pre-serialized JSON from the SDK | |
| if (typeof payload === "string") { | |
| try { | |
| return JSON.parse(payload); | |
| } catch { | |
| // If it's not valid JSON, return as-is | |
| return payload; | |
| } | |
| } | |
| return payload; | |
| } | |
| /** | |
| * Normalize the payload from BatchQueue. | |
| * | |
| * Handles different payload types: | |
| * - "application/store": Already offloaded to R2, payload is the path - pass through as-is | |
| * - "application/json": May be a pre-serialized JSON string - parse to avoid double-stringification | |
| * - Other types: Pass through as-is | |
| * | |
| * @param payload - The raw payload from the batch item | |
| * @param payloadType - The payload type (e.g., "application/json", "application/store") | |
| */ | |
| function normalizePayload(payload: unknown, payloadType?: string): unknown { | |
| // For non-JSON payloads (including application/store for R2-offloaded payloads), | |
| // return as-is - no normalization needed | |
| if (payloadType !== "application/json") { | |
| return payload; | |
| } | |
| // For JSON payloads, if payload is a string, try to parse it | |
| // This handles pre-serialized JSON from the SDK | |
| if (typeof payload === "string") { | |
| try { | |
| return JSON.parse(payload); | |
| } catch { | |
| // If it's not valid JSON, return as-is | |
| return payload; | |
| } | |
| } | |
| return payload; | |
| } |
🤖 Prompt for AI Agents
In apps/webapp/app/v3/runEngineHandlers.server.ts around lines 796 to 826, the
function currently attempts to JSON.parse when payloadType is undefined which
can mutate plain string payloads; change the early-return condition so it only
returns early when payloadType is explicitly not "application/json" (i.e., if
payloadType !== "application/json" return payload), then keep the existing logic
that for payloadType === "application/json" you attempt JSON.parse on string
payloads with a try/catch and otherwise return payload unchanged.
| host: options.batchQueue?.redis.host ?? "localhost", | ||
| port: options.batchQueue?.redis.port ?? 6379, | ||
| username: options.batchQueue?.redis.username, | ||
| password: options.batchQueue?.redis.password, | ||
| keyPrefix: `${options.batchQueue?.redis.keyPrefix ?? ""}batch-queue:`, | ||
| enableAutoPipelining: options.batchQueue?.redis.enableAutoPipelining ?? true, | ||
| tls: options.batchQueue?.redis.tls !== undefined, | ||
| }, |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, let's look at the specific file and context around lines 327-334
head -n 346 internal-packages/run-engine/src/engine/index.ts | tail -n +320Repository: triggerdotdev/trigger.dev
Length of output: 1283
🏁 Script executed:
# Search for batchQueue configuration patterns
rg -n "batchQueue" internal-packages/run-engine/src/engine/index.ts -B2 -A2Repository: triggerdotdev/trigger.dev
Length of output: 2942
🏁 Script executed:
# Check how other Redis configs (runLock, queue, worker) are handled
rg -n "redis\.host|redis\.port" internal-packages/run-engine/src/engine/index.ts -B2 -A2Repository: triggerdotdev/trigger.dev
Length of output: 369
🏁 Script executed:
# Look for tests or documentation about batchQueue defaults
fd -e ts -e js -e md -x rg -l "batchQueue.*redis" {} \;Repository: triggerdotdev/trigger.dev
Length of output: 119
🏁 Script executed:
# Check if there's any documentation or comments about this default behavior
rg -n "batchQueue.*localhost|batchQueue.*6379" --type=ts --type=mdRepository: triggerdotdev/trigger.dev
Length of output: 289
🏁 Script executed:
# Find RunEngineOptions or EngineOptions type definition
rg -n "type.*RunEngineOptions|interface.*RunEngineOptions|export.*RunEngineOptions" --type=ts -B2 -A30Repository: triggerdotdev/trigger.dev
Length of output: 10356
🏁 Script executed:
# Search for where RunEngine is instantiated
rg -n "new RunEngine|new engine" --type=ts -B3 -A5 | head -150Repository: triggerdotdev/trigger.dev
Length of output: 13906
🏁 Script executed:
# Check if batchQueue is optional in options
rg -n "batchQueue\?" --type=ts -B2 -A2Repository: triggerdotdev/trigger.dev
Length of output: 3127
🏁 Script executed:
# Look for environment variables or config defaults
rg -n "process\.env.*redis|process\.env.*REDIS" --type=ts | head -30Repository: triggerdotdev/trigger.dev
Length of output: 3342
🏁 Script executed:
# Check if tests pass batchQueue options to RunEngine
rg -n "batchQueue" internal-packages/run-engine/src/engine/tests/ -A2 -B2Repository: triggerdotdev/trigger.dev
Length of output: 3068
🏁 Script executed:
# Look at a specific test to see the full engine initialization
head -n 100 internal-packages/run-engine/src/engine/tests/utils/engineTest.ts | tail -n 40Repository: triggerdotdev/trigger.dev
Length of output: 1141
🏁 Script executed:
# Check if there's initialization code in the webapp that might provide batchQueue
rg -n "new RunEngine" apps/webapp -B5 -A15 | head -80Repository: triggerdotdev/trigger.dev
Length of output: 5832
🏁 Script executed:
# Check BatchQueue initialization and error handling
cat -n internal-packages/run-engine/src/batch-queue/index.ts | head -100Repository: triggerdotdev/trigger.dev
Length of output: 4410
🏁 Script executed:
# Look for any checks or guards around localhost connection
rg -n "localhost" internal-packages/run-engine/src/batch-queue/ -B3 -A3Repository: triggerdotdev/trigger.dev
Length of output: 51
🏁 Script executed:
# See if there's validation that batchQueue must be provided
rg -n "batchQueue.*required|batchQueue.*must|batchQueue.*should" internal-packages/run-engine/src/ --type=tsRepository: triggerdotdev/trigger.dev
Length of output: 51
BatchQueue should require explicit Redis configuration instead of defaulting to localhost:6379.
The code unconditionally initializes BatchQueue even when options.batchQueue is undefined, which causes it to silently connect to localhost:6379 instead of the application's actual Redis. Since batchQueue is optional in the type definition but always initialized, this creates a dangerous mismatch: production deployments without explicit batchQueue configuration will immediately fail. Either mark batchQueue as required and ensure callers provide it, or skip BatchQueue initialization when the option is not provided.
🤖 Prompt for AI Agents
In internal-packages/run-engine/src/engine/index.ts around lines 327 to 334, the
BatchQueue is being initialized with default Redis values (localhost:6379) even
when options.batchQueue is undefined; update the code to only initialize
BatchQueue when options.batchQueue is provided (or if you prefer to require it,
validate and throw a clear error at startup). Concretely: add a guard that
checks if options.batchQueue is truthy before building the redis config and
creating the BatchQueue, remove the fallback host/port defaults so no implicit
localhost connection occurs, and if you choose the required approach, change the
type/validation and throw an explicit error instructing callers to supply
batchQueue configuration.
| batchIndex: itemIndex, | ||
| skipChecks: true, // Already validated at batch level | ||
| realtimeStreamsVersion: meta.realtimeStreamsVersion, | ||
| }, |
There was a problem hiding this comment.
Bug: Missing planType causes billing/usage attribution issues for batch runs
The batch processing callback passes skipChecks: true to TriggerTaskService.call but does not provide the planType option. The code in RunEngineTriggerTaskService explicitly expects planType to be passed when skipChecks is enabled (line 147: "When skipChecks is enabled, planType should be passed via options"). Without it, planType will be undefined for all batch-created runs, affecting billing accuracy and usage tracking. The planType needs to be captured during batch creation and stored in the batch metadata for use during item processing.
Additional Locations (1)
There was a problem hiding this comment.
Actionable comments posted: 14
♻️ Duplicate comments (9)
apps/webapp/app/runEngine/concerns/batchLimits.server.ts (1)
56-75: Duration format validation still missing.The type assertion
as Durationon line 60 bypasses validation, as previously flagged. The environment variableBATCH_RATE_LIMIT_REFILL_INTERVALis only validated as a string without format checking. An invalid duration format would pass validation but fail at runtime.apps/webapp/app/v3/runEngineHandlers.server.ts (3)
734-796: Make batch completion writes idempotent + atomic (transaction + createMany/skipDuplicates)This is the same issue previously flagged: the callback updates
batchTaskRunthen inserts errors one-by-one (Lines 749-777). If the callback is re-invoked or partially fails, you can get partial updates and/or duplicatebatchTaskRunErrorrows.
812-831:normalizePayload()should not JSON.parse whenpayloadTypeis undefinedAs previously noted: the current condition (Line 815) allows parsing when
payloadTypeisundefined, which can unintentionally transform plain string payloads.- if (payloadType !== "application/json" && payloadType !== undefined) { + if (payloadType !== "application/json") { return payload; }
645-732: PassplanTypewhen callingTriggerTaskService.call()withskipChecks: true(billing/usage correctness)
skipChecks: trueis set (line 698), but noplanTypeis provided in the options. TheRunEngineTriggerTaskServiceexplicitly requiresplanTypewhenskipChecksis enabled to ensure proper billing/usage attribution—when omitted, it logs a warning but processes the run withplanType === undefined, breaking usage tracking.The v2 batch trigger (
batchTrigger.server.ts) correctly passesplanTypefrom batch-level entitlement checks. Adopt the same pattern here: ensureplanTypeis included in the batchmetaobject during batch creation, then passmeta.planTypein theTriggerTaskService.call()options.const result = await triggerTaskService.call( item.task, environment, { payload, options: { ...(item.options as Record<string, unknown>), payloadType: item.payloadType, parentRunId: meta.parentRunId, resumeParentOnCompletion: meta.resumeParentOnCompletion, parentBatch: batchId, }, }, { triggerVersion: meta.triggerVersion, traceContext: meta.traceContext as Record<string, unknown> | undefined, spanParentAsLink: meta.spanParentAsLink, batchId, batchIndex: itemIndex, skipChecks: true, // Already validated at batch level + planType: meta.planType, realtimeStreamsVersion: meta.realtimeStreamsVersion, }, "V2" );apps/webapp/app/runEngine/services/streamBatchItems.server.ts (1)
150-170: Response does not indicate whether the batch was sealed (client can’t distinguish partial ingestion)
Line 154-169 returns the same shape as the success path (Line 194-198), so clients can’t tell “sealed & processing” vs “needs retry with missing items”. This matches the prior review feedback.Also: set span attributes before the early return so telemetry isn’t missing counts on partial ingestions.
if (enqueuedCount !== batch.runCount) { + span.setAttribute("itemsAccepted", itemsAccepted); + span.setAttribute("itemsDeduplicated", itemsDeduplicated); logger.warn("Batch item count mismatch", { batchId: batchFriendlyId, expected: batch.runCount, received: enqueuedCount, itemsAccepted, itemsDeduplicated, });packages/trigger-sdk/src/v3/shared.ts (1)
1533-1585: Good: two-phase errors now preserve phase + batchId context. That addresses a big chunk of “Phase 2 failed after Phase 1 succeeded” debuggability. Remaining question is whether you want any explicit server-side cleanup/cancel on stream failure.packages/redis-worker/src/fair-queue/index.ts (2)
798-832: Incorrect access toworkerQueuefield (should not be underpayload).
This matches a previously reported issue:workerQueueis a top-level field onStoredMessage, not insidepayload.- const workerQueueId = message.payload.workerQueue ?? queueId; + const workerQueueId = message.workerQueue ?? queueId;
1090-1103: Release reserved concurrency when dequeue-time payload validation fails.
This matches a previously reported issue: the early-return path moves to DLQ but doesn’t release the previously reserved concurrency slot.if (!result.success) { @@ // Move to DLQ await this.#moveToDeadLetterQueue(storedMessage, "Payload validation failed"); + // Release reserved concurrency so the queue/group doesn't get stuck + if (this.concurrencyManager) { + await this.concurrencyManager.release(descriptor, storedMessage.id).catch((e) => { + this.logger.error("Failed to release concurrency after validation failure", { + queueId, + messageId: storedMessage.id, + error: e instanceof Error ? e.message : String(e), + }); + }); + } return; }internal-packages/run-engine/src/batch-queue/index.ts (1)
664-700: Don’t cleanup Redis state if completion callback fails.
This matches an existing report: if the completion callback fails (e.g., DB update), deleting Redis progress makes the batch unrecoverable/stuck.if (this.completionCallback) { try { await this.completionCallback(result); } catch (error) { this.logger.error("Error in batch completion callback", { batchId, error: error instanceof Error ? error.message : String(error), }); + // Preserve Redis state for inspection/retry; do NOT cleanup on callback failure. + return; } } // Clean up Redis keys for this batch await this.completionTracker.cleanup(batchId);(If you still need eventual cleanup, consider a separate “reaper” job gated on DB state.)
🧹 Nitpick comments (6)
apps/webapp/app/runEngine/services/streamBatchItems.server.ts (2)
95-113: Index/order + payloadType handling is too loose for “in-order streaming”
- Line 105-113: only upper-bound check exists. If the contract expects “items are enqueued in order”, you should enforce
indexis an integer>= 0and (optionally) strictly increasing (or at least non-decreasing).- Line 116: deriving payload type from
item.options?.payloadTypeis brittle (and bypasses schema intent). Prefer an explicit field from the NDJSON schema (if present) or validate the option value is a string.Also applies to: 115-133
276-287: AsyncIterable wrapper should cancel the reader on early exit
If the consumer stops early (break/throw), releasing the lock (Line 285) doesn’t necessarily cancel the underlying stream. Considerawait reader.cancel()infinally(best-effort) beforereleaseLock().packages/trigger-sdk/src/v3/shared.ts (1)
613-733: Consider runtime validation for “stream inputs” (clearer errors thanfor awaitTypeError). Today, any non-array value that isn’t actuallyAsyncIterable/ReadableStreamwill fail later with a generic error. A small guard at the branch point would improve DX.Also applies to: 868-996, 1127-1250, 1387-1518
internal-packages/run-engine/src/batch-queue/types.ts (2)
22-40: Consider tighteningpayloadType/optionsshape (optional).
If the engine expects"application/json"vs"application/store"semantics, consider at least documenting/encoding those as a union (orz.enum([...]).catchall(...)) to reduce downstream ambiguity while staying permissive.
46-79:BatchMetavsInitializeBatchOptions: keep them intentionally in-sync.
These two structures mirror each other; a drift here will be painful. Consider a shared base type/schema or a single source-of-truth conversion helper.Also applies to: 133-164
packages/core/src/v3/apiClient/index.ts (1)
410-416: Harden retry option handling (avoidmaxAttempts!).
BecauseretryOptionsis built with spreads, it’s possible to end up withmaxAttempts: undefined(if a caller explicitly provides it asundefined). Prefer normalizing once (e.g.,const maxAttempts = retryOptions.maxAttempts ?? DEFAULT...) and removing the non-null assertion.Also applies to: 1602-1608, 1616-1673
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
references/hello-world/src/trigger/batches.tsis excluded by!references/**
📒 Files selected for processing (12)
apps/webapp/app/env.server.ts(3 hunks)apps/webapp/app/runEngine/concerns/batchLimits.server.ts(1 hunks)apps/webapp/app/runEngine/services/streamBatchItems.server.ts(1 hunks)apps/webapp/app/v3/runEngineHandlers.server.ts(2 hunks)internal-packages/run-engine/src/batch-queue/index.ts(1 hunks)internal-packages/run-engine/src/batch-queue/types.ts(1 hunks)packages/core/src/v3/apiClient/index.ts(6 hunks)packages/redis-worker/src/fair-queue/index.ts(1 hunks)packages/redis-worker/src/fair-queue/schedulers/weighted.ts(1 hunks)packages/redis-worker/src/fair-queue/tests/concurrency.test.ts(1 hunks)packages/redis-worker/src/fair-queue/types.ts(1 hunks)packages/trigger-sdk/src/v3/shared.ts(13 hunks)
🚧 Files skipped from review as they are similar to previous changes (3)
- packages/redis-worker/src/fair-queue/tests/concurrency.test.ts
- packages/redis-worker/src/fair-queue/schedulers/weighted.ts
- packages/redis-worker/src/fair-queue/types.ts
🧰 Additional context used
📓 Path-based instructions (7)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
**/*.{ts,tsx}: Use types over interfaces for TypeScript
Avoid using enums; prefer string unions or const objects instead
Files:
packages/core/src/v3/apiClient/index.tsapps/webapp/app/runEngine/concerns/batchLimits.server.tsinternal-packages/run-engine/src/batch-queue/types.tsapps/webapp/app/v3/runEngineHandlers.server.tsapps/webapp/app/env.server.tsapps/webapp/app/runEngine/services/streamBatchItems.server.tsinternal-packages/run-engine/src/batch-queue/index.tspackages/trigger-sdk/src/v3/shared.tspackages/redis-worker/src/fair-queue/index.ts
{packages/core,apps/webapp}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Use zod for validation in packages/core and apps/webapp
Files:
packages/core/src/v3/apiClient/index.tsapps/webapp/app/runEngine/concerns/batchLimits.server.tsapps/webapp/app/v3/runEngineHandlers.server.tsapps/webapp/app/env.server.tsapps/webapp/app/runEngine/services/streamBatchItems.server.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Use function declarations instead of default exports
Files:
packages/core/src/v3/apiClient/index.tsapps/webapp/app/runEngine/concerns/batchLimits.server.tsinternal-packages/run-engine/src/batch-queue/types.tsapps/webapp/app/v3/runEngineHandlers.server.tsapps/webapp/app/env.server.tsapps/webapp/app/runEngine/services/streamBatchItems.server.tsinternal-packages/run-engine/src/batch-queue/index.tspackages/trigger-sdk/src/v3/shared.tspackages/redis-worker/src/fair-queue/index.ts
**/*.{js,ts,jsx,tsx,json,md,css,scss}
📄 CodeRabbit inference engine (AGENTS.md)
Format code using Prettier
Files:
packages/core/src/v3/apiClient/index.tsapps/webapp/app/runEngine/concerns/batchLimits.server.tsinternal-packages/run-engine/src/batch-queue/types.tsapps/webapp/app/v3/runEngineHandlers.server.tsapps/webapp/app/env.server.tsapps/webapp/app/runEngine/services/streamBatchItems.server.tsinternal-packages/run-engine/src/batch-queue/index.tspackages/trigger-sdk/src/v3/shared.tspackages/redis-worker/src/fair-queue/index.ts
apps/webapp/app/**/*.{ts,tsx}
📄 CodeRabbit inference engine (.cursor/rules/webapp.mdc)
Access all environment variables through the
envexport ofenv.server.tsinstead of directly accessingprocess.envin the Trigger.dev webapp
Files:
apps/webapp/app/runEngine/concerns/batchLimits.server.tsapps/webapp/app/v3/runEngineHandlers.server.tsapps/webapp/app/env.server.tsapps/webapp/app/runEngine/services/streamBatchItems.server.ts
apps/webapp/**/*.{ts,tsx}
📄 CodeRabbit inference engine (.cursor/rules/webapp.mdc)
apps/webapp/**/*.{ts,tsx}: When importing from@trigger.dev/corein the webapp, use subpath exports from the package.json instead of importing from the root path
Follow the Remix 2.1.0 and Express server conventions when updating the main trigger.dev webapp
Files:
apps/webapp/app/runEngine/concerns/batchLimits.server.tsapps/webapp/app/v3/runEngineHandlers.server.tsapps/webapp/app/env.server.tsapps/webapp/app/runEngine/services/streamBatchItems.server.ts
packages/trigger-sdk/**/*.{ts,tsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
In the Trigger.dev SDK (packages/trigger-sdk), prefer isomorphic code like fetch and ReadableStream instead of Node.js-specific code
Files:
packages/trigger-sdk/src/v3/shared.ts
🧠 Learnings (29)
📓 Common learnings
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `tasks.batchTrigger()` to trigger multiple runs of a single task with different payloads
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.trigger()` to trigger multiple different tasks at once from backend code
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.triggerByTaskAndWait()` to batch trigger tasks by passing task instances and wait for results
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.triggerAndWait()` to batch trigger multiple different tasks and wait for results
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.triggerByTask()` to batch trigger tasks by passing task instances for static task sets
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `yourTask.batchTrigger()` to trigger multiple runs of a task from inside another task
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `runs.subscribeToBatch()` to subscribe to changes for all runs in a batch
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/webapp.mdc:0-0
Timestamp: 2025-11-27T16:26:58.661Z
Learning: Use `trigger.dev/redis-worker` for background job and worker system needs in the webapp and run engine
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.trigger()` to trigger multiple different tasks at once from backend code
Applied to files:
packages/core/src/v3/apiClient/index.tsinternal-packages/run-engine/src/batch-queue/types.tsapps/webapp/app/v3/runEngineHandlers.server.tspackages/trigger-sdk/src/v3/shared.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `tasks.batchTrigger()` to trigger multiple runs of a single task with different payloads
Applied to files:
packages/core/src/v3/apiClient/index.tsinternal-packages/run-engine/src/batch-queue/types.tsapps/webapp/app/v3/runEngineHandlers.server.tspackages/trigger-sdk/src/v3/shared.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.triggerAndWait()` to batch trigger multiple different tasks and wait for results
Applied to files:
packages/core/src/v3/apiClient/index.tsinternal-packages/run-engine/src/batch-queue/types.tsapps/webapp/app/v3/runEngineHandlers.server.tspackages/trigger-sdk/src/v3/shared.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `runs.subscribeToBatch()` to subscribe to changes for all runs in a batch
Applied to files:
packages/core/src/v3/apiClient/index.tsinternal-packages/run-engine/src/batch-queue/types.tsapps/webapp/app/v3/runEngineHandlers.server.tsinternal-packages/run-engine/src/batch-queue/index.tspackages/trigger-sdk/src/v3/shared.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.triggerByTaskAndWait()` to batch trigger tasks by passing task instances and wait for results
Applied to files:
packages/core/src/v3/apiClient/index.tsinternal-packages/run-engine/src/batch-queue/types.tsapps/webapp/app/v3/runEngineHandlers.server.tspackages/trigger-sdk/src/v3/shared.ts
📚 Learning: 2025-11-14T16:03:06.917Z
Learnt from: matt-aitken
Repo: triggerdotdev/trigger.dev PR: 2681
File: apps/webapp/app/services/platform.v3.server.ts:258-302
Timestamp: 2025-11-14T16:03:06.917Z
Learning: In `apps/webapp/app/services/platform.v3.server.ts`, the `getDefaultEnvironmentConcurrencyLimit` function intentionally throws an error (rather than falling back to org.maximumConcurrencyLimit) when the billing client returns undefined plan limits. This fail-fast behavior prevents users from receiving more concurrency than their plan entitles them to. The org.maximumConcurrencyLimit fallback is only for self-hosted deployments where no billing client exists.
Applied to files:
apps/webapp/app/runEngine/concerns/batchLimits.server.tsapps/webapp/app/env.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Limit task duration using the `maxDuration` property (in seconds)
Applied to files:
apps/webapp/app/runEngine/concerns/batchLimits.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `schemaTask()` from `trigger.dev/sdk/v3` with Zod schema for payload validation
Applied to files:
internal-packages/run-engine/src/batch-queue/types.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `batch.triggerByTask()` to batch trigger tasks by passing task instances for static task sets
Applied to files:
internal-packages/run-engine/src/batch-queue/types.tsapps/webapp/app/v3/runEngineHandlers.server.tspackages/trigger-sdk/src/v3/shared.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `yourTask.batchTrigger()` to trigger multiple runs of a task from inside another task
Applied to files:
apps/webapp/app/v3/runEngineHandlers.server.tspackages/trigger-sdk/src/v3/shared.ts
📚 Learning: 2025-11-27T16:26:58.661Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/webapp.mdc:0-0
Timestamp: 2025-11-27T16:26:58.661Z
Learning: Use the Run Engine 2.0 from `internal/run-engine` for new run lifecycle code in the webapp instead of the legacy run engine
Applied to files:
apps/webapp/app/v3/runEngineHandlers.server.ts
📚 Learning: 2025-09-03T14:34:41.781Z
Learnt from: myftija
Repo: triggerdotdev/trigger.dev PR: 2464
File: apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam.settings/route.tsx:357-371
Timestamp: 2025-09-03T14:34:41.781Z
Learning: When using Zod's safeParse, the .data property is undefined when parsing fails, but TypeScript may still complain about accessing .data without checking .success first. The suggested approach of checking .success before accessing .data improves type safety and code clarity.
Applied to files:
apps/webapp/app/v3/runEngineHandlers.server.ts
📚 Learning: 2025-11-27T16:26:37.432Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-11-27T16:26:37.432Z
Learning: Applies to packages/trigger-sdk/**/*.{ts,tsx} : In the Trigger.dev SDK (packages/trigger-sdk), prefer isomorphic code like fetch and ReadableStream instead of Node.js-specific code
Applied to files:
apps/webapp/app/v3/runEngineHandlers.server.tspackages/trigger-sdk/src/v3/shared.ts
📚 Learning: 2025-08-14T18:35:44.370Z
Learnt from: nicktrn
Repo: triggerdotdev/trigger.dev PR: 2390
File: apps/webapp/app/env.server.ts:764-765
Timestamp: 2025-08-14T18:35:44.370Z
Learning: The BoolEnv helper in apps/webapp/app/utils/boolEnv.ts uses z.preprocess with inconsistent default value types across the codebase - some usages pass boolean defaults (correct) while others pass string defaults (incorrect), leading to type confusion. The helper should enforce boolean-only defaults or have clearer documentation.
Applied to files:
apps/webapp/app/v3/runEngineHandlers.server.tsapps/webapp/app/env.server.ts
📚 Learning: 2025-10-08T11:48:12.327Z
Learnt from: nicktrn
Repo: triggerdotdev/trigger.dev PR: 2593
File: packages/core/src/v3/workers/warmStartClient.ts:168-170
Timestamp: 2025-10-08T11:48:12.327Z
Learning: The trigger.dev runners execute only in Node 21 and 22 environments, so modern Node.js APIs like AbortSignal.any (introduced in v20.3.0) are supported.
Applied to files:
apps/webapp/app/v3/runEngineHandlers.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Attach metadata to task runs using the metadata option when triggering, and access/update it inside runs using metadata functions
Applied to files:
apps/webapp/app/v3/runEngineHandlers.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use the `task()` function from `trigger.dev/sdk/v3` to define tasks with id and run properties
Applied to files:
apps/webapp/app/v3/runEngineHandlers.server.tspackages/trigger-sdk/src/v3/shared.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `yourTask.batchTriggerAndWait()` to batch trigger tasks and wait for all results from a parent task
Applied to files:
apps/webapp/app/v3/runEngineHandlers.server.tspackages/trigger-sdk/src/v3/shared.ts
📚 Learning: 2025-11-27T16:26:58.661Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/webapp.mdc:0-0
Timestamp: 2025-11-27T16:26:58.661Z
Learning: Applies to apps/webapp/app/**/*.{ts,tsx} : Access all environment variables through the `env` export of `env.server.ts` instead of directly accessing `process.env` in the Trigger.dev webapp
Applied to files:
apps/webapp/app/env.server.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Control concurrency using the `queue` property with `concurrencyLimit` option
Applied to files:
internal-packages/run-engine/src/batch-queue/index.tspackages/trigger-sdk/src/v3/shared.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `.withStreams()` to subscribe to realtime streams from task metadata in addition to run changes
Applied to files:
packages/trigger-sdk/src/v3/shared.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use metadata methods (set, del, replace, append, remove, increment, decrement, stream, flush) to update metadata during task execution
Applied to files:
packages/trigger-sdk/src/v3/shared.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Generate example payloads for tasks when possible
Applied to files:
packages/trigger-sdk/src/v3/shared.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `yourTask.trigger()` to trigger a task from inside another task with specified payload
Applied to files:
packages/trigger-sdk/src/v3/shared.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `tasks.trigger()` with type-only imports to trigger tasks from backend code without importing the task implementation
Applied to files:
packages/trigger-sdk/src/v3/shared.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `yourTask.triggerAndWait()` to trigger a task and wait for its result from a parent task
Applied to files:
packages/trigger-sdk/src/v3/shared.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `idempotencyKeys.create()` to create idempotency keys for preventing duplicate task executions
Applied to files:
packages/trigger-sdk/src/v3/shared.ts
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `idempotencyKeyTTL` option to define a time window during which duplicate triggers return the original run
Applied to files:
packages/trigger-sdk/src/v3/shared.ts
🧬 Code graph analysis (3)
apps/webapp/app/runEngine/concerns/batchLimits.server.ts (3)
apps/webapp/app/env.server.ts (1)
env(1289-1289)apps/webapp/app/services/rateLimiter.server.ts (3)
createRedisRateLimitClient(74-105)RateLimiter(21-72)Duration(17-17)apps/webapp/app/runEngine/concerns/batchGlobalRateLimiter.server.ts (1)
limit(27-33)
apps/webapp/app/runEngine/services/streamBatchItems.server.ts (4)
apps/webapp/app/v3/services/baseService.server.ts (1)
WithRunEngine(52-59)apps/webapp/app/runEngine/concerns/batchPayloads.server.ts (1)
BatchPayloadProcessor(28-164)packages/core/src/v3/isomorphic/friendlyId.ts (1)
BatchId(96-96)internal-packages/run-engine/src/batch-queue/types.ts (2)
BatchItem(22-40)BatchItem(41-41)
packages/trigger-sdk/src/v3/shared.ts (9)
packages/core/src/v3/schemas/api.ts (2)
BatchItemNDJSON(358-367)BatchItemNDJSON(369-369)packages/core/src/v3/utils/ioSerialization.ts (1)
stringifyIO(88-105)packages/core/src/v3/idempotencyKeys.ts (2)
makeIdempotencyKey(39-53)flattenIdempotencyKey(12-37)packages/core/src/v3/workers/index.ts (1)
getEnvVar(4-4)packages/core/src/v3/taskContext/index.ts (1)
ctx(26-28)packages/trigger-sdk/src/v3/batch.ts (1)
batch(18-24)packages/core/src/v3/runMetadata/index.ts (1)
stream(80-86)internal-packages/run-engine/src/index.ts (1)
BatchItem(16-16)packages/core/src/v3/types/schemas.ts (1)
SchemaParseFn(101-101)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (24)
- GitHub Check: Cursor Bugbot
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (7, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (6, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (4, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (8, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (5, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (8, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (1, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (2, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (3, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (7, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (3, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (5, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (2, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (6, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (1, 8)
- GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (4, 8)
- GitHub Check: units / packages / 🧪 Unit Tests: Packages (1, 1)
- GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - npm)
- GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - pnpm)
- GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - npm)
- GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - pnpm)
- GitHub Check: typecheck / typecheck
- GitHub Check: Analyze (javascript-typescript)
🔇 Additional comments (9)
apps/webapp/app/runEngine/concerns/batchLimits.server.ts (5)
9-16: LGTM!The BatchLimitsConfig schema is well-defined with appropriate defaults.
18-31: Good use of singleton pattern.The Redis client singleton addresses the previous concern about connection exhaustion from creating new clients on every call.
81-87: LGTM!The function provides a clean API for retrieving batch limits and rate limiter for an organization.
89-105: LGTM!The config resolution logic properly validates and falls back to defaults when needed.
111-123: LGTM!The error class provides all necessary information for constructing proper 429 responses.
apps/webapp/app/env.server.ts (2)
943-950: Defaults added for DRR settings.The previously flagged critical issue has been resolved—
BATCH_QUEUE_DRR_QUANTUMandBATCH_QUEUE_MAX_DEFICITnow have defaults (5 and 50). The optionalCONSUMER_COUNTandCONSUMER_INTERVAL_MSare safe since FairQueue provides defaults.
952-960: LGTM!Plan-based batch limits and concurrency settings are well-defined with sensible defaults for each tier.
packages/trigger-sdk/src/v3/shared.ts (2)
56-57: Type import forBatchItemNDJSONlooks right (keeps SDK isomorphic).
128-130: This concern does not apply—BatchTriggerErroris not part of the public API.
BatchTriggerErroris exported fromshared.tsbut never re-exported throughindex.ts, which is the sole public entry point (perpackage.jsonexports). As an internal-only export, the early re-export pattern does not affect consumers and poses no CJS transpilation risk. This same pattern is used consistently elsewhere in the file (e.g., line 124).Likely an incorrect or invalid review comment.
| constructor(protected readonly _prisma: PrismaClientOrTransaction = prisma) { | ||
| super({ prisma }); | ||
| this.payloadProcessor = new BatchPayloadProcessor(); | ||
| } |
There was a problem hiding this comment.
Constructor ignores injected Prisma client/transaction
Line 34-36: _prisma is accepted but super({ prisma }) always passes the global client, which can break transactional usage and any BaseService logic that relies on the injected client.
- constructor(protected readonly _prisma: PrismaClientOrTransaction = prisma) {
- super({ prisma });
+ constructor(protected readonly _prisma: PrismaClientOrTransaction = prisma) {
+ super({ prisma: _prisma });
this.payloadProcessor = new BatchPayloadProcessor();
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| constructor(protected readonly _prisma: PrismaClientOrTransaction = prisma) { | |
| super({ prisma }); | |
| this.payloadProcessor = new BatchPayloadProcessor(); | |
| } | |
| constructor(protected readonly _prisma: PrismaClientOrTransaction = prisma) { | |
| super({ prisma: _prisma }); | |
| this.payloadProcessor = new BatchPayloadProcessor(); | |
| } |
🤖 Prompt for AI Agents
In apps/webapp/app/runEngine/services/streamBatchItems.server.ts around lines 34
to 37, the constructor accepts an injected _prisma but always calls super({
prisma }) which uses the global client; change the super call to pass the
injected client (super({ prisma: _prisma })) so the BaseService and
transactional usage receive the provided PrismaClientOrTransaction, and ensure
subsequent code uses this._prisma where needed.
| // Convert friendly ID to internal ID | ||
| const batchId = BatchId.fromFriendlyId(batchFriendlyId); | ||
|
|
There was a problem hiding this comment.
Friendly ID parsing should be validated (avoid throwing non-400-ish errors)
Line 57: BatchId.fromFriendlyId() can throw for malformed IDs; currently that would likely surface as a 500 instead of a validation error.
- // Convert friendly ID to internal ID
- const batchId = BatchId.fromFriendlyId(batchFriendlyId);
+ // Convert friendly ID to internal ID
+ let batchId: string;
+ try {
+ batchId = BatchId.fromFriendlyId(batchFriendlyId);
+ } catch {
+ throw new ServiceValidationError(`Invalid batch ID: ${batchFriendlyId}`);
+ }Also applies to: 75-89
🤖 Prompt for AI Agents
In apps/webapp/app/runEngine/services/streamBatchItems.server.ts around lines
56-58 (and similarly for the block at 75-89), BatchId.fromFriendlyId(...) can
throw on malformed IDs and currently would surface as a 500; wrap each
fromFriendlyId call in a try/catch, validate the input, and convert parsing
failures into a clear validation error (e.g., throw or return a 400/BadRequest
with a descriptive message like "Invalid batchFriendlyId" including the
offending value), ensuring other errors continue to propagate unchanged.
| const response = await fetch(`${this.baseUrl}/api/v3/batches/${batchId}/items`, { | ||
| method: "POST", | ||
| headers, | ||
| body: forRequest, | ||
| // @ts-expect-error - duplex is required for streaming body but not in types | ||
| duplex: "half", | ||
| }); |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
Node.js fetch API ReadableStream request body duplex "half" support minimum version
💡 Result:
Minimum Node.js version: v18 (built-in fetch powered by undici). Node v18+ supports using a ReadableStream/async-iterable as the fetch request body and the duplex: "half" option (undici/Node fetch requires duplex: "half" for streaming request bodies). See Node v18 release notes and undici docs. [1][2][3]
Sources
- Node.js v18 release / fetch & Web Streams added. [1]
- undici README / docs: request.duplex = 'half' required for ReadableStream / AsyncIterables. [2]
- MDN / Chrome docs on fetch streaming requests and duplex: "half". [3]
🌐 Web query:
browser fetch API duplex option support request streaming body
💡 Result:
Short answer: Browsers support streaming request bodies only experimentally and mostly in Chromium-based browsers — you must pass duplex: "half" when using a ReadableStream as fetch body. Support is available in Chrome (and other Chromium) since ~v105 (over HTTP/2/3 + HTTPS); Safari has partial support quirks and Firefox historically lacked support. The feature is marked experimental (check compatibility before production). [1][2][3]
Key points:
- Use: fetch(url, { method:'POST', body: readableStream, duplex: 'half' }). [2]
- duplex: "half" is required for browser request streams (response is buffered until request finishes). [1][2]
- Experimental / limited browser support (Chromium-based OK; Safari/Firefox incomplete). Verify current compatibility for your target browsers. [1][2][3]
Sources: MDN (Request.duplex) [1], Chrome Developers article on streaming requests [2], ecosystem/compatibility notes (StackOverflow/npm docs) [3].
🏁 Script executed:
# Check package.json for Node.js version requirements
fd -type f -name "package.json" | head -5 | xargs grep -l "engines" -A 5Repository: triggerdotdev/trigger.dev
Length of output: 241
🏁 Script executed:
# Find and check package.json files
find . -name "package.json" -type f | grep -E "(^./package.json|packages/core/package.json)" | head -5Repository: triggerdotdev/trigger.dev
Length of output: 112
🏁 Script executed:
# Check the root and packages/core package.json for engines and browser/node targets
cat ./package.json | head -50Repository: triggerdotdev/trigger.dev
Length of output: 2451
🏁 Script executed:
# Check packages/core package.json
cat ./packages/core/package.json | head -80Repository: triggerdotdev/trigger.dev
Length of output: 2743
🏁 Script executed:
# Get the complete packages/core/package.json to check for engines specification
cat ./packages/core/package.json | grep -A 5 -B 5 "engines"Repository: triggerdotdev/trigger.dev
Length of output: 280
🏁 Script executed:
# Check if there's any documentation about Node version requirements
cat ./packages/core/package.jsonRepository: triggerdotdev/trigger.dev
Length of output: 18548
🏁 Script executed:
# Check the apiClient file to see where it's used and if it targets browsers
grep -r "apiClient" packages/core/src/v3 --include="*.ts" --include="*.tsx" -l | head -10Repository: triggerdotdev/trigger.dev
Length of output: 511
🏁 Script executed:
# Check the apiClient implementation to understand its usage and context
cat -n packages/core/src/v3/apiClient/index.ts | head -100Repository: triggerdotdev/trigger.dev
Length of output: 3694
🏁 Script executed:
# See the full context around line 433-439
cat -n packages/core/src/v3/apiClient/index.ts | sed -n '425,450p'Repository: triggerdotdev/trigger.dev
Length of output: 1254
🏁 Script executed:
# Check if apiClient is exported in the public API
grep -n "apiClient\|streamBatchItems" packages/core/src/v3/index.tsRepository: triggerdotdev/trigger.dev
Length of output: 380
🏁 Script executed:
# Check the v3/isomorphic export to see if it includes apiClient functionality
cat packages/core/src/v3/isomorphic/index.tsRepository: triggerdotdev/trigger.dev
Length of output: 290
🏁 Script executed:
# Check where streamBatchItems is used
rg "streamBatchItems" packages/core/src --type tsRepository: triggerdotdev/trigger.dev
Length of output: 595
Browser compatibility limitation: streaming request bodies with duplex: "half" lack universal support.
Node.js >=18.20.0 (per package.json engines) supports this pattern. However, browser support is experimental/limited: Chrome/Chromium v105+ supports it over HTTPS/HTTP/2/3, but Safari has quirks and Firefox historically lacks support. Since apiClient is publicly exported, SDK consumers using this in browser environments may encounter silent failures or cryptic errors, particularly on Safari/Firefox. Consider documenting this limitation or adding platform detection/fallback logic.
🤖 Prompt for AI Agents
In packages/core/src/v3/apiClient/index.ts around lines 433 to 439, the fetch
call uses the non-standard streaming option duplex: "half" which is not
universally supported in browsers; update the implementation to detect runtime
environment and avoid using duplex in unsupported environments (or throw a
clear, descriptive error): add a runtime check (e.g., Node vs browser via
process.versions.node or typeof window) and when in browsers that lack duplex
support either (1) fall back to a non-streaming request path (buffer the body
before send) or (2) throw a descriptive error that explains the limitation and
points to docs; also ensure tests/documentation are updated to note the
limitation and supported environments.
| const claimResult = await this.visibilityManager.claim<StoredMessage<z.infer<TPayloadSchema>>>( | ||
| queueId, | ||
| queueKey, | ||
| queueItemsKey, | ||
| loopId, | ||
| this.visibilityTimeoutMs | ||
| ); | ||
|
|
||
| if (!claimResult.claimed || !claimResult.message) { | ||
| // Queue is empty, update master queue | ||
| await this.redis.updateMasterQueueIfEmpty(masterQueueKey, queueKey, queueId); | ||
| return false; | ||
| } | ||
|
|
||
| const { message } = claimResult; | ||
|
|
||
| // Reserve concurrency slot | ||
| if (this.concurrencyManager) { | ||
| const reserved = await this.concurrencyManager.reserve(descriptor, message.messageId); | ||
| if (!reserved) { | ||
| // Release message back to queue | ||
| await this.visibilityManager.release(message.messageId, queueId, queueKey, queueItemsKey); | ||
| return false; | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
message.messageId / message.payload usage doesn’t match StoredMessage construction.
You construct StoredMessage with id (not messageId) (Line 263-283 / 365-385), but later reserve/release using message.messageId and call #processMessage(loopId, message.payload, ...) in direct mode. If VisibilityManager.claim<T>() truly returns T = StoredMessage, this is a correctness bug.
- const reserved = await this.concurrencyManager.reserve(descriptor, message.messageId);
+ const reserved = await this.concurrencyManager.reserve(descriptor, message.id);
@@
- const messageKey = `${message.messageId}:${queueId}`;
+ const messageKey = `${message.id}:${queueId}`;- await this.#processMessage(loopId, message.payload, queueId);
+ await this.#processMessage(loopId, message, queueId);If claim() intentionally returns a wrapper object, please rename types/variables to make that explicit and remove the claim<StoredMessage<...>> generic argument.
Also applies to: 1009-1067
| async function* readableStreamToAsyncIterable<T>(stream: ReadableStream<T>): AsyncIterable<T> { | ||
| const reader = stream.getReader(); | ||
| try { | ||
| while (true) { | ||
| const { done, value } = await reader.read(); | ||
| if (done) break; | ||
| yield value; | ||
| } | ||
| } finally { | ||
| reader.releaseLock(); | ||
| } | ||
| } | ||
|
|
||
| /** | ||
| * Normalize stream input to AsyncIterable | ||
| */ | ||
| function normalizeToAsyncIterable<T>( | ||
| input: AsyncIterable<T> | ReadableStream<T> | ||
| ): AsyncIterable<T> { | ||
| if (isReadableStream<T>(input)) { | ||
| return readableStreamToAsyncIterable(input); | ||
| } | ||
| return input; | ||
| } |
There was a problem hiding this comment.
ReadableStream conversion should cancel on early termination (avoid leaking the underlying stream). releaseLock() alone won’t necessarily stop upstream producers if the consumer stops early (common with errors/AbortSignal). Consider canceling in finally.
async function* readableStreamToAsyncIterable<T>(stream: ReadableStream<T>): AsyncIterable<T> {
const reader = stream.getReader();
try {
while (true) {
const { done, value } = await reader.read();
if (done) break;
yield value;
}
} finally {
+ try {
+ await reader.cancel();
+ } catch {
+ // ignore
+ }
reader.releaseLock();
}
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| async function* readableStreamToAsyncIterable<T>(stream: ReadableStream<T>): AsyncIterable<T> { | |
| const reader = stream.getReader(); | |
| try { | |
| while (true) { | |
| const { done, value } = await reader.read(); | |
| if (done) break; | |
| yield value; | |
| } | |
| } finally { | |
| reader.releaseLock(); | |
| } | |
| } | |
| /** | |
| * Normalize stream input to AsyncIterable | |
| */ | |
| function normalizeToAsyncIterable<T>( | |
| input: AsyncIterable<T> | ReadableStream<T> | |
| ): AsyncIterable<T> { | |
| if (isReadableStream<T>(input)) { | |
| return readableStreamToAsyncIterable(input); | |
| } | |
| return input; | |
| } | |
| async function* readableStreamToAsyncIterable<T>(stream: ReadableStream<T>): AsyncIterable<T> { | |
| const reader = stream.getReader(); | |
| try { | |
| while (true) { | |
| const { done, value } = await reader.read(); | |
| if (done) break; | |
| yield value; | |
| } | |
| } finally { | |
| try { | |
| await reader.cancel(); | |
| } catch { | |
| // If cancel fails, try to release the lock | |
| try { | |
| reader.releaseLock(); | |
| } catch { | |
| // ignore | |
| } | |
| } | |
| } | |
| } |
🤖 Prompt for AI Agents
In packages/trigger-sdk/src/v3/shared.ts around lines 1679 to 1702, the
readableStreamToAsyncIterable generator currently only calls
reader.releaseLock() in finally, which can leak the underlying ReadableStream
when the consumer terminates early; update the finally block to cancel the
stream (await stream.cancel(), ignoring any errors) before releasing the lock so
upstream producers are signaled to stop and resources are cleaned up.
New batch trigger system with larger payloads, streaming ingestion, larger batch sizes, and a fair processing system.
This PR introduces a new
FairQueueabstraction inspired by our ownRunQueuethat enables multi-tenant fair queueing with concurrency limits. The newBatchQueueis built on top of theFairQueue, and handles processing Batch triggers in a fair manner with per-environment concurrency limits defined per-org. Additionally, there is a global concurrency limit to prevent the BatchQueue system from creating too many runs too quickly, which can cause downstream issues.For this new BatchQueue system we have a completely new batch trigger creation and ingestion system. Previously this was a single endpoint with a single JSON body that defined details about the batch as well as all the items in the batch.
We're introducing a two-phase batch trigger ingestion system. In the first phase, the BatchTaskRun record is created (and possibly rate limited). The second phase is another endpoint that accepts an NDJSON body with each line being a single item/run with payload and options.
At ingestion time all items are added to a queue, in order, and then processed by the BatchQueue system.
New batch trigger rate limits
This PR implements a new batch trigger specific rate limit, configured on the
Organization.batchRateLimitConfigcolumn, and defaults using these environment variables:BATCH_RATE_LIMIT_REFILL_RATEdefaults to 10BATCH_RATE_LIMIT_REFILL_INTERVALthe duration interval, defaults to"10s"BATCH_RATE_LIMIT_MAXdefaults to 1200This rate limiter is scoped to the environment ID and controls how many runs can be submitted via batch triggers per interval. The SDK handles the retrying side.
Batch queue concurrency limits
The new column
Organization.batchQueueConcurrencyConfignow defines an org specificprocessingConcurrencyvalue, with a backup of the env varBATCH_CONCURRENCY_LIMIT_DEFAULTwhich defaults to 10. This controls how many batch queue items are processed concurrently per environment.There is also a global rate limit for the batch queue set via the
BATCH_QUEUE_GLOBAL_RATE_LIMITwhich defaults to being disabled. If set, the entire batch queue system won't process more thanBATCH_QUEUE_GLOBAL_RATE_LIMITitems per second. This allows controlling the maximum number of runs created per second via batch triggers.Batch trigger limits
STREAMING_BATCH_MAX_ITEMScontrols the maximum number of items in a single batchSTREAMING_BATCH_ITEM_MAXIMUM_SIZEcontrols the maximum size of each item in a batch