Add support for ItemReader, ItemBatcher, ToleratedFailure* in Map states for both JSONpath and JSONata#18
Open
johnbenjaminmccarthy wants to merge 1 commit intobbc:mainfrom
Conversation
johnbenjaminmccarthy
force-pushed
the
map_state_updates
branch
from
fd0810d to
90bd155
Compare
This commit adds support for the following Map state configuration fields: - `ItemReader` specifying a resource or resources in S3 to read items from - `ItemBatcher` to batch map state executions - `ToleratedFailureCount` and `ToleratedFailurePercentage` on map states (and `-Path` variants for JSONpath). It also adds supports and checks for: - `MaxItems` on map states (and `MaxItemsPath` for JSONpath) - `ProcessorConfig.Mode` (`"INLINE"` or `"DISTRIBUTED"`) for validating whether `ItemReader` should be processed - `Label` for adding (more) accurate `context.Execution` information for distributed child map state executions All changes are implemented for both JSONpath and JSONata configuration of Map states. All changes are covered by new unit tests for map states, covering normal input, edge cases, and validating inputs for `ItemReader`, `ItemBatcher`, and `ToleratedFailure*` as well as `INLINE` v.s. `DISTRIBUTED`. This commit does not address the following additional functionalities of AWS map states which are still remaining: - `MaxConcurrency` - `MaxInputBytesPerBatch` (and `MaxInputBytesPerBatchPath` for JSONpath) - `ProcessorConfig.ExecutionType` for distributed map states. - `ResultWriter` - S3 object input types of `JSONL`, `CSV`, `MANIFEST`, or `PARQUET` in map states with `ItemReader` configured to use S3 `getObject` . - `Transformation === "LOAD_AND_FLATTEN"` functionality.
johnbenjaminmccarthy
force-pushed
the
map_state_updates
branch
from
90bd155 to
90a3a95
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit adds support for the following Map state configuration fields:
ItemReaderspecifying a resource or resources in S3 to read items fromItemBatcherto batch map state executionsToleratedFailureCountandToleratedFailurePercentageon map states (and-Pathvariants for JSONpath).It also adds supports and checks for:
MaxItemson map states (andMaxItemsPathfor JSONpath)ProcessorConfig.Mode("INLINE"or"DISTRIBUTED") for validating whetherItemReadershould be processedLabelfor adding (more) accuratecontext.Executioninformation for distributed child map state executionsAll changes are implemented for both JSONpath and JSONata configuration of Map states.
All changes are covered by new unit tests for map states, covering normal input, edge cases, and validating inputs for
ItemReader,ItemBatcher, andToleratedFailure*as well asINLINEv.s.DISTRIBUTED.This commit does not address the following additional functionalities of AWS map states which are still remaining:
MaxConcurrencyMaxInputBytesPerBatch(andMaxInputBytesPerBatchPathfor JSONpath)ProcessorConfig.ExecutionTypefor distributed map states.ResultWriterJSONL,CSV,MANIFEST, orPARQUETin map states withItemReaderconfigured to use S3getObject.Transformation === "LOAD_AND_FLATTEN"functionality.