[FEATURE] Detect AOSS generation from the control plane - #18
Open
noel-improv wants to merge 2 commits into
Open
noel-improv wants to merge 2 commits into
noel-improv wants to merge 2 commits into
Conversation
noel-improv
force-pushed
the
feat/aoss-generation-detection
branch
from
July 17, 2026 20:31
5539af9 to
c564a3d
Compare
noel-improv
marked this pull request as ready for review
July 17, 2026 20:37
acarbonetto
reviewed
Jul 17, 2026
acarbonetto
reviewed
Jul 17, 2026
| "aoss:BatchGetCollection", | ||
| "aoss:BatchGetCollectionGroup" | ||
| ], | ||
| "Resource": "*" |
There was a problem hiding this comment.
I think we should be able to point to the necessary resource instead of using *
Security will probably flag this as is.
There was a problem hiding this comment.
...unless the Action doesn't work against specific resources. In which case, we will have to keep the "*"
Owner
Author
There was a problem hiding this comment.
Checked the Service Authorization Reference, both aoss:BatchGetCollection and aoss:BatchGetCollectionGroup are management-plane actions with no resource type, they only work with Resource: "*". The collection group is also created implicitly by AOSS and isn't named in the template, so there's no ARN to target regardless.
noel-improv
force-pushed
the
feat/aoss-generation-detection
branch
2 times, most recently
from
July 17, 2026 23:14
419be40 to
2935582
Compare
acarbonetto
reviewed
Jul 20, 2026
acarbonetto
approved these changes
Jul 20, 2026
8 tasks
Resolve an AOSS collection's Classic/NextGen generation deterministically via BatchGetCollection/BatchGetCollectionGroup in index_exists(), ahead of the existing reactive error-string retry. Detection runs only for AOSS endpoints (is_sigv4_auth) with the generation unset; any failure -- missing IAM, a boto3 too old for the API, a non-AOSS host, or an unexpected response -- falls back to the reactive retry, which stays as the safety net. The boto3 floor is left at 1.40.61 rather than bumped to 1.43.17: the aiobotocore ceiling pulled in transitively still pins botocore below 1.40.62, so the API is detected at runtime and the path activates once that constraint is lifted. Grant aoss:BatchGetCollection and aoss:BatchGetCollectionGroup to the client role in the AOSS example templates so the detection has the permissions it needs.
Bound the control-plane probe with a short-timeout botocore Config so a deployment that reaches the AOSS data plane but not the opensearchserverless control-plane API fails fast into the reactive fallback instead of blocking on the default ~60s connect timeout for every index. Memoize the detected generation per (collection_id, region) so N indexes against the same collection cost two control-plane calls instead of 2N, which also avoids tripping ThrottlingException. Only successful resolutions are cached; a transient failure stays uncached so the next index retries. Broaden the AOSS host regex to accept the .cn partition suffix. Add tests for memoization, the uncached-transient-failure path, and .cn host parsing; add an autouse fixture to clear the cache between tests.
noel-improv
force-pushed
the
feat/aoss-generation-detection
branch
from
August 12, 2026 19:35
aeccb05 to
8762036
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
index_exists()picks the Classic or NextGen knn_vector mapping for an AOSS collection by trying Classic first and retrying with NextGen on a specific OpenSearch error string. This adds a deterministic check ahead of that retry: for AOSS endpoints with the generation unset, it reads the collection's generation from the control plane (BatchGetCollectionthenBatchGetCollectionGroup) and uses it directly. The reactive retry stays as the fallback whenever the control-plane check is unavailable.Changes
_parse_aoss_collection()to extract the collection id and region from an AOSS endpoint host._resolve_aoss_generation()to read thegenerationfield viabatch_get_collectionthenbatch_get_collection_group. Returns the generation, or None when it can't be determined.index_exists()ahead of the reactive retry. It runs only whenis_sigv4_authis true and the generation is unset; the retry stays armed until NextGen is confirmed.aoss:BatchGetCollectionandaoss:BatchGetCollectionGroupto the client role in the three AOSS example templates.Problem
Detection was coupled to an AWS error string, which could shift if the message changes. Reading the generation from the control plane removes that coupling where the two new IAM actions are granted. Raised in review of awslabs#391.
Related issue (if any): awslabs#399
Testing
pytest)21 unit tests cover the resolver and host parsing: NEXTGEN and CLASSIC honoured, no group falls back, AccessDenied / generic ClientError / old boto3 fall back, non-AOSS endpoints skip the call, explicit override skips detection. The 26 existing NextGen tests are unchanged. 78 OpenSearch unit tests pass locally.
Checklist
Dependency note
The boto3 floor stays at 1.40.61. Bumping to 1.43.17 (which
batch_get_collection_groupneeds) is a no-go: it reintroduces theaiobotocoreconflict resolved recently (aiobotocorepinsbotocore <1.40.62), so raising the floor breaks install. The API is detected at runtime instead — deterministic detection runs where boto3 is already >=1.43.17, and falls back to the existing reactive path everywhere else. No behavior change.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.