Add connection validator for Amazon Kinesis [DBZ-9439]

[jpechane]

Migrated from DBZ-9439

This will require to:

• Provide an implementation of io.debezium.platform.environment.connection.ConnectionValidator interface for checking connection to Amazon Kinesis. 
• Add a new connection type to io.debezium.platform.data.model.ConnectionEntity.Type
• Add json schema to connection-schemas.json
• Add an integration test (if possible)

 

[Anurag-Yadav-Mekala]

Hi @jpechane , I’d like to work on this issue. Could you please assign it to me?

Thanks.

[Naros]

@Anurag-Yadav-Mekala assigned, please let us know if you have any questions or issues.

[jpechane]

@Anurag-Yadav-Mekala You can resume work from debezium/debezium-platform#88

[Anurag-Yadav-Mekala]

Hi,
While implementing AmazonKinesisConnectionValidator, I’m validating required fields such as region and streamName.

I wanted clarity regarding AWS credentials, should aws.accessKeyId and aws.secretAccessKey be treated as mandatory configuration fields for validation or should we rely on the AWS automatic credential lookup and only validate connectivity via DescribeStream?

I want to ensure the validator aligns with the intended deployment model of the platform (e.g, local minikube vs EKS with IAM roles).

Thanks!

[Naros]

Hi @Anurag-Yadav-Mekala, would it be possible to target both use cases?

[Anurag-Yadav-Mekala]

Hi,
I went through the documentation of Kinesis and DescrieStream API. From what I understand, calling this API already verifies multiple aspects which include authentication.

So explicitly validating aws.accessKeyId and aws.secretAccessKey in the validator might not be necessary. The AWS SDK’s default credential provider chain would handle credentials, while the DescribeStream call would naturally fail if authentication is incorrect.

Also, since local setups, deployments using IAM roles and other ways may provide credentials in different ways, avoiding explicit credential validation might keep the validator simpler and compatible with all use cases.

Apologies for the confusion in my earlier message.

Please let me know if this approach aligns with the intended design or if my understanding above is incorrect.

Thank you.

[jpechane]

@mfvitale Correct me if I am wrong but given the scenario (Debezium Platform) we'd need to have AWS credential lookup set both on Debezium Server side and on Debezium Platform server as well. I suppose this is not desirable so explicit validation might be better/necessary.

Am I right?

[mfvitale]

I suppose this is not desirable so explicit validation might be better/necessary.

I went through the documentation of Kinesis and DescrieStream API. From what I understand, calling this API already verifies multiple aspects which include authentication.

If this is true we can go directly with it, but I suppose that some credential should be passes in some way since looking to the Debezium Server sink doc, seems to me that the SDK has an automatic discovery mechanism for looking to the credentials.

So even if we find a way to validate the connection from the Platform directly using some sort of credentials then there is still the problem on the Platform itself. Wondering if we should remove the support for Amazon Kinesis for now. @jpechane WDYT?