feat(devtools): declarative queue tuning + make the DLQ alarm capable of firing

[d-klotz]

Two related changes to integration-builder, both driven by a production incident in a Frigg app whose Attio integration queue reached 268k messages.

1. Definition.queue tuning knobs

SQS and queue-worker settings were hardcoded, so no app could override them: maxReceiveCount: 3, VisibilityTimeout: 1800, reservedConcurrency: 20.

Apps that needed different values patched them in post-deploy with aws lambda put-function-concurrency / aws sqs set-queue-attributes. That drifts from the CloudFormation template, and the drift is invisible until something causes CFN to update the resource — at which point it silently reverts.

We hit exactly that: a production app's template says ReservedConcurrentExecutions: 20 while the live function runs 50, and the same app's Aurora template says MaxCapacity: 1 while the cluster runs 5. Both survive only because CFN acts on template diffs, not on drift. The next change touching either resource reverts it.

This exposes an optional Definition.queue block. All knobs optional, all existing defaults preserved via ??, so apps that don't opt in render byte-identical templates (covered by a backward-compat test).

Knob	Maps to	Range	Default
queue.visibilityTimeout	SQS VisibilityTimeout	0..43200	1800
queue.messageRetentionPeriod	SQS MessageRetentionPeriod	60..1209600	345600
queue.maxReceiveCount	DLQ redrive maxReceiveCount	1..1000	3
queue.worker.batchSize	SQS event batchSize	1..10000	1
queue.worker.maximumBatchingWindow	SQS event batching window	0..300	unset
queue.worker.maximumConcurrency	ESM ScalingConfig.MaximumConcurrency	2..1000	unset
queue.worker.reservedConcurrency	Lambda ReservedConcurrency	0..1000	20
queue.worker.timeout	Lambda timeout (s)	1..900	900

Bounds-checked at template-generation time (consistent with aurora-builder's validation) so mistakes surface as clear errors rather than opaque CloudFormation rejections after packaging. This includes the cross-field rule that SQS rejects batchSize > 10 without a batching window — otherwise the template packages fine and fails at deploy.

maximumBatchingWindow / maximumConcurrency are emitted only when defined, keeping rendered templates stable for non-opt-in consumers. The serverless-framework input key is maximumBatchingWindow, not the longer CFN name maximumBatchingWindowInSeconds — osls and serverless@3+ reject the longer key.

External queues: queue-level knobs apply only to a queue this stack owns. When the queue is externally owned we must not mutate it, but queue.worker.* still applies — an asymmetry that would otherwise be silent. That path now warns which keys were ignored and runs the same bounds checks.

This half is a cherry-pick of a previously unmerged local commit, rebased onto next. One conflict resolved: next has since added an appDefinition parameter to createIntegrationQueue, so the signature is now (integrationName, result, appDefinition, queueConfig = {}) rather than either version alone.

2. The DLQ alarm could never fire

createDLQObservability generated this:

// CloudWatch Alarm: fires when any message lands in the DLQ
MetricName: 'ApproximateNumberOfMessagesVisible',
Statistic: 'Maximum',
Threshold: 500,

The comment states the intent correctly and the code does something else. It alarms on standing depth, and the same function attaches dlqProcessor as an SQS consumer (batchSize: 10) that drains that queue continuously. Depth never accumulates, so the threshold is unreachable.

To be fair to the original: AWS explicitly recommends ApproximateNumberOfMessagesVisible for DLQs. That guidance assumes the usual shape, where nothing consumes the DLQ. Frigg's does, which quietly invalidates it.

Observed in production: 146 messages landed in the DLQ in a single day while ApproximateNumberOfMessages read 0 throughout and no alarm fired. That is why several days of silent data loss in that app went unnoticed. Every Frigg app with a DLQ has the same blind spot.

Now two alarms, because a consumed DLQ needs both an arrival signal and a backlog signal:

DLQMessageAlarm — arrivals. NumberOfMessagesDeleted / Sum / > 0 / TreatMissingData: notBreaching.

Not NumberOfMessagesSent, which would be just as blind. Per the SQS docs: "Messages automatically moved to a DLQ due to processing failures are not captured by this metric." Redrive is the only way messages reach a DLQ. Confirmed in production — Sent read 0 for seven consecutive days while Received/Deleted tracked the real inflow of 4 → 55 → 146.

DLQBacklogAlarm — backlog. ApproximateNumberOfMessagesVisible / Minimum / > 0.

Deleted only increments while dlqProcessor is healthy; if it is throttled or erroring, messages pile up and the arrival alarm stays silent — the worst case to miss. Minimum rather than Maximum because dlqProcessor runs at concurrency 1 with batchSize: 10, so any larger burst leaves depth briefly above zero; Maximum would fire on every ordinary burst alongside DLQMessageAlarm and train people to ignore it. Minimum > 0 breaches only when the queue was never empty across the period — genuinely not draining.

Deliberately not included

Raising the maxReceiveCount default from 3 to 5. The production evidence supports it — DLQ inflow tracked a rate-limit storm exactly (4 → 55 → 146 → 22), the signature of retries exhausting during sustained 429s rather than a code defect. But that is policy, not mechanism: it would change behaviour for every existing Frigg app on upgrade. With the knob available, apps can set it themselves. This PR adds mechanism, fixes a bug, and changes no defaults.

Testing

integration-builder suites: 591 passing, 11 suites.

Whole packages/devtools/infrastructure tree, same environment, worktrees excluded:

	Suites failed	Tests failed	Passed	Total
next baseline	18	109	1,416	1,528
this branch	18	107	1,427	1,537

Same 18 pre-existing failures (health, networking, cloudformation-discovery, prisma-layer — all unrelated), no new ones.

integration-builder.js is prettier-clean. integration-builder.test.js and app-definition.js were already prettier-unclean on next; deliberately not reformatted, to keep the diff free of unrelated churn.

Known gaps, not addressed here

• No integer check — visibilityTimeout: 1.5 passes bounds and CFN rejects it later.
• No unknown-key detection — a maxRecieveCount typo is silently ignored, which is the same silent-no-op failure mode this PR fixes elsewhere. Worth a follow-up.
• queue.visibilityTimeout < worker.timeout causes in-flight redelivery and duplicate processing. Both are now user-settable independently, so this deserves a warning.
• Both alarms sit on the shared InternalErrorQueue, so there is no per-integration attribution. Pre-existing, unchanged.

[netlify]

✅ Deploy Preview for friggframework-org canceled.

Name	Link
🔨 Latest commit	7168888
🔍 Latest deploy log	https://app.netlify.com/projects/friggframework-org/deploys/6a6d1f1d6c38aa00084a8e35

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud