materialize-eventbridge: new connector#4359
Conversation
| case p.IsPrimaryKey: | ||
| constraint.Type = pm.Response_Validated_Constraint_LOCATION_REQUIRED | ||
| constraint.Reason = "Document keys must be included" |
There was a problem hiding this comment.
from @mdibaiee:
I believe keys should also be forbidden since we are doing the full document and there is no keying of the events at all?
Looking at google-pubsub it seems we have it as Optional there, but that’s because the key can be used to order the messages published, but here it seems we don’t use the key at all
I did try to switch primary key to forbidden, but the integration tests fail because the flow runtime requires key fields not be forbidden, so the best we can do is optional.
I've checked the other pattern-4 materializers:
- slack requires and uses keys
- pinecone requires and uses keys
- webhook requires, but does not use keys, only
RawJSON - google-pubsub makes keys optional, and uses
.PackedKey
Notably, this comment tags the pinecone logic:
We require collection keys be materialized because it seems pretty reasonable to
require they be included as metadata since the composite key is used as the basis for
the vector ID, and also to avoid complications from
estuary/flow#1057.
…atePrincipalPolicy
mdibaiee
left a comment
mdibaiee
left a comment
There was a problem hiding this comment.
LGTM % question regarding the check
| ResourceArns: []string{busARN}, | ||
| }) | ||
| if err != nil { | ||
| log.WithField("error", err).Warn("SimulatePrincipalPolicy failed; skipping write-permission check") |
2 participants
Description:
New
materialize-eventbridgeconnector — Pattern 4 (push-only, delta-updates) materialization to AWS EventBridge.Source+DetailType. The full materialized document is the eventDetailpayload.credentialsblock —AWSAccessKey(static keys) orAWSIAM(assume-role + STS, same pattern asmaterialize-dynamodb).Applyverifies the bus viaDescribeEventBus; at-least-once delivery; 256 KB per-entry limit enforced client-side.boilerplate.Materializer(the other Pattern-4 connectors hand-roll), so it runs on the standardbptestrig and produces the same snapshot files SQL materializers do.Closes #4262.
Workflow steps:
User picks region, bus name, and auth mode; adds bindings with
source+detail_type. Documents are published viaPutEventsbatched 10/call with boundederrgroupparallelism (storeConcurrency=8); partial-failure responses are classified per-entry — throttle codes retry with exponential backoff, permanent codes fail-fast.Documentation links affected:
User docs for
materialize-eventbridgewill follow in a separate doc PR.Notes for reviewers:
-eventbridge.test-all(resources documented inmaterialize-eventbridge/testdata/README.md— account076183946664, regionus-east-2).aws.EndpointResolverWithOptionsFuncfor consistency with four other AWS-touching connectors; deferred to a repo-wide migration.