Skip to content

Transient anvilprod IT failure for managed_access_repository_files test #7957

@achave11-ucsc

Description

@achave11-ucsc

https://gitlab.explore.anvilproject.org/ucsc/azul/-/jobs/65093

2026-04-24 18:59:37,702    INFO MainThread test.integration_test: Beginning sub-test [managed_access_repository_files] {'catalog': 'anvil13-it'}
2026-04-24 18:59:37,703    INFO MainThread azul.oauth2: Making GET request to 'https://service.explore.anvilproject.org/index/files?catalog=anvil13-it&size=100&filters=%7B%22source_id%22%3A+%7B%22is%22%3A+%5B%22a23eea3c-e635-4f89-aaf9-610b872dbc9a%22%5D%7D%7D'
2026-04-24 18:59:37,703    INFO MainThread azul.oauth2: … without a request body
2026-04-24 18:59:37,704    INFO MainThread azul.oauth2: … with request headers [('x-goog-api-client', 'cred-type/sa'), ('authorization', 'REDACTED')]
2026-04-24 18:59:37,761    INFO MainThread azul.oauth2: Got 200 response after 0.058s from GET to https://service.explore.anvilproject.org/index/files?catalog=anvil13-it&size=100&filters=%7B%22source_id%22%3A+%7B%22is%22%3A+%5B%22a23eea3c-e635-4f89-aaf9-610b872dbc9a%22%5D%7D%7D
2026-04-24 18:59:37,761    INFO MainThread azul.oauth2: … with response headers HTTPHeaderDict({'Content-Type': 'application/json', 'Content-Length': '1482', 'Connection': 'keep-alive', 'Date': 'Fri, 24 Apr 2026 18:59:37 GMT', 'X-Amzn-Trace-Id': 'Root=1-69ebbd99-3b8fb42c082690da2c548923;Parent=1bbda066b1318cca;Sampled=0;Lineage=1:45061563:0', 'x-amzn-RequestId': '66a55c7c-5b52-407c-8154-a877abe548e7', 'Referrer-Policy': 'strict-origin-when-cross-origin', 'X-XSS-Protection': '1; mode=block', 'Access-Control-Allow-Origin': '*', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Access-Control-Allow-Headers': 'Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", 'x-amz-apigw-id': 'cVqQFEcvoAMEsmQ=', 'Cache-Control': 'no-store', 'X-Content-Type-Options': 'nosniff', 'X-Cache': 'Miss from cloudfront', 'Via': '1.1 0819e703158679947bab2d0bebe6285e.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'IAD61-P13', 'X-Amz-Cf-Id': '-JzTinVi_SrYSE19n1eIvh0htHjWLHswC_Ut9ZF6AOB9jaNuI1A6MQ=='})
2026-04-24 18:59:37,761    INFO MainThread azul.oauth2: … with a response body of length 1482 starting in b'{"hits":[],"pagination":{"count":0,"total":0,"size":100,"next":null,"previous":null,"pages":0,"sort":"files.file_id","order":"asc"},"termFacets":{"diagnoses.phenotype":{"terms":[],"total":0,"type":"terms"},"biosamples.disease":{"terms":[],"total":0,"type":"terms"},"diagnoses.disease":{"terms":[],"total":0,"type":"terms"},"diagnoses.phenopacket":{"terms":[],"total":0,"type":"terms"},"files.is_supplementary":{"terms":[],"total":0,"type":"terms"},"files.data_modality":{"terms":[],"total":0,"type":"terms"},"donors.reported_ethnicity":{"terms":[],"total":0,"type":"terms"},"datasets.consent_group":{"terms":[],"total":0,"type":"terms"},"activities.assay_type":{"terms":[],"total":0,"type":"terms"},"datasets.title":{"terms":[],"total":0,"type":"terms"},"biosamples.anatomical_site":{"terms":[],"total":0,"type":"terms"},"donors.organism_type":{"terms":[],"total":0,"type":"terms"},"datasets.data_use_permission":{"terms":[],"total":0,"type":"terms"},"donors.phenotypic_sex":{"terms":[],"total":0,"type":"terms"},"activit...'
2026-04-24 18:59:37,761    INFO MainThread test.integration_test: Failed sub-test [managed_access_repository_files] {'catalog': 'anvil13-it'}
  test_indexing (integration_test.IndexingIntegrationTest.test_indexing) [managed_access_repository_files] (catalog='anvil13-it') ... ERROR
…
======================================================================
ERROR: test_indexing (integration_test.IndexingIntegrationTest.test_indexing) [managed_access_repository_files] (catalog='anvil13-it')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builds/ucsc/azul/test/integration_test.py", line 402, in subTest
    yield
  File "/builds/ucsc/azul/test/integration_test.py", line 1462, in _test_managed_access
    files = self._test_managed_access_repository_files(catalog, ma_source)
  File "/builds/ucsc/azul/test/integration_test.py", line 1587, in _test_managed_access_repository_files
    file_url = furl(self.random.choice(sorted(managed_access_file_urls)))
                    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.14/random.py", line 353, in choice
    raise IndexError('Cannot choose from an empty sequence')
IndexError: Cannot choose from an empty sequence
----------------------------------------------------------------------
Ran 16 tests in 581.246s
FAILED (errors=1)
make: *** [Makefile:288: integration_test] Error 1
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: exit code 1

CloudWatch Logs Insights
region: us-east-1
log-group-names: /aws/lambda/azul-service-anvilprod
data-sources:
facets:
start-time: 2026-04-24T18:57:05.012Z
end-time: 2026-04-24T19:03:56.391Z
query-string:

fields @timestamp, @message
| filter @message like "5dccf342"
| sort @timestamp asc
| limit 10000

@timestamp @message
2026-04-24 18:59:37.726 START RequestId: 5dccf342-a6db-4743-98aa-a19513cf6aad Version: 29
2026-04-24 18:59:37.727 [INFO] 2026-04-24T18:59:37.727Z 5dccf342-a6db-4743-98aa-a19513cf6aad azul.chalice Received GET request for '/index/files', with {"query": {"catalog": "anvil13-it", "filters": "{"source_id": {"is": ["a23eea3c-e635-4f89-aaf9-610b872dbc9a"]}}", "size": "100"}, "headers": {"accept-encoding": "identity", "authorization": "Bearer ya29.REDACTED", "cloudfront-forwarded-proto": "https", "cloudfront-is-desktop-viewer": "true", "cloudfront-is-mobile-viewer": "false", "cloudfront-is-smarttv-viewer": "false", "cloudfront-is-tablet-viewer": "false", "cloudfront-viewer-asn": "14618", "cloudfront-viewer-country": "US", "host": "service.explore.anvilproject.org", "user-agent": "python-urllib3/2.6.3", "via": "1.1 0819e703158679947bab2d0bebe6285e.cloudfront.net (CloudFront)", "x-amz-cf-id": "-JzTinVi_SrYSE19n1eIvh0htHjWLHswC_Ut9ZF6AOB9jaNuI1A6MQ==", "x-amzn-trace-id": "Root=1-69ebbd99-3b8fb42c082690da2c548923", "x-forwarded-for": "35.168.152.160, 15.158.251.99", "x-forwarded-port": "443", "x-forwarded-proto": "https", "x-goog-api-client": "cred-type/sa"}}.
2026-04-24 18:59:37.728 [INFO] 2026-04-24T18:59:37.728Z 5dccf342-a6db-4743-98aa-a19513cf6aad azul.chalice Authenticated request as OAuth2(access_token='ya29.REDACTED')
2026-04-24 18:59:37.728 [INFO] 2026-04-24T18:59:37.728Z 5dccf342-a6db-4743-98aa-a19513cf6aad azul.chalice … without a request body
2026-04-24 18:59:37.729 [INFO] 2026-04-24T18:59:37.729Z 5dccf342-a6db-4743-98aa-a19513cf6aad azul.boto3 dynamodb.GetItem: Making POST request to https://dynamodb.us-east-1.amazonaws.com/
2026-04-24 18:59:37.729 [INFO] 2026-04-24T18:59:37.729Z 5dccf342-a6db-4743-98aa-a19513cf6aad azul.boto3 dynamodb.GetItem: … with a request body of length 1166 starting in b'{"TableName": "azul-sources_cache_by_auth-anvilprod", "Key": {"identity": {"S": "anvil13-it:ya29.REDACTED...'
2026-04-24 18:59:37.736 [INFO] 2026-04-24T18:59:37.736Z 5dccf342-a6db-4743-98aa-a19513cf6aad azul.boto3 dynamodb.GetItem: … with a response body of length 95653 starting in b'{"Item":{"expiration":{"N":"1777057437"},"sources":{"S":"[\"a691f69a-ce17-4c52-847f-6a020c35db18\", \"5d92f515-c1a0-4c48-b8c0-5b16014c8c32\", \"80e7abe5-630d-4c79-817b-0de41991b1d3\", \"375a67f8-da09-49e1-b0e8-e0ab7085ba50\", \"b52370d1-2142-4f47-adce-4755f5cb18a8\", \"d063c228-e96f-43e9-857d-f5b483570189\", \"3aeaeab7-b357-4ef1-8a64-a2074936fb6c\", \"6f321d22-6acb-4031-8a43-a1af76aa15af\", \"15d59914-1d05-438e-9cf4-c185d3824e23\", \"22feccf4-268c-4723-879b-64d21ae8628f\", \"afbe6b24-2cd3-490e-b4f5-2cbbfb96ad0f\", \"55d26ed5-f046-414f-8c8b-bdc941fa58aa\", \"a99d838e-e39b-424d-bf5a-2b4ee4e26f55\", \"75f5452d-ceca-402e-bfc4-759c8352f4da\", \"46fbcead-3689-4540-a29c-a5df56caf48d\", \"99f9f7aa-535c-44cb-b6f9-df8f479eb80a\", \"02137a6c-e68a-4ccb-b5aa-e1013d8e44da\", \"47d7f5c4-66af-4e3e-904c-5f82fd144499\", \"1ef77925-056b-408d-a8ff-446acb110957\", \"2a1375fc-a976-4327-829f-d0d0f6155cc5\", \"c713d6c5-d19e-4e5d-92ba-8de1b27881d8\", \"247b3216-ab1b-4d36-8fab-f6695c6481f9\", \"56d27a8d-c02d-4e17-a7e0-6db0219ffaea\...'
2026-04-24 18:59:37.736 [INFO] 2026-04-24T18:59:37.736Z 5dccf342-a6db-4743-98aa-a19513cf6aad azul.boto3 dynamodb.GetItem: Got 200 response
2026-04-24 18:59:37.741 [INFO] 2026-04-24T18:59:37.741Z 5dccf342-a6db-4743-98aa-a19513cf6aad opensearch Making POST request to https://vpc-azul-index-anvilprod-ggipah4skn2ftt47u4xgvydzqm.us-east-1.es.amazonaws.com:443/azul_v2_anvilprod_anvil13-it_files_aggregate/_search
2026-04-24 18:59:37.742 [INFO] 2026-04-24T18:59:37.741Z 5dccf342-a6db-4743-98aa-a19513cf6aad opensearch … with a request body of length 8221 starting in b'{"post_filter":{"bool":{"must":[{"constant_score":{"filter":{"terms":{"sources.id.keyword":["a23eea3c-e635-4f89-aaf9-610b872dbc9a"]}}}}]}},"aggs":{"source_id":{"filter":{"bool":{}},"aggs":{"myTerms":{"terms":{"field":"sources.id.keyword","size":99999},"meta":{"path":["sources","id"]}},"untagged":{"missing":{"field":"sources.id.keyword"}}}},"activities.activity_type":{"filter":{"bool":{"must":[{"constant_score":{"filter":{"terms":{"sources.id.keyword":["a23eea3c-e635-4f89-aaf9-610b872dbc9a"]}}}}]}},"aggs":{"myTerms":{"terms":{"field":"contents.activities.activity_type.keyword","size":99999},"meta":{"path":["contents","activities","activity_type"]}},"untagged":{"missing":{"field":"contents.activities.activity_type.keyword"}}}},"activities.assay_type":{"filter":{"bool":{"must":[{"constant_score":{"filter":{"terms":{"sources.id.keyword":["a23eea3c-e635-4f89-aaf9-610b872dbc9a"]}}}}]}},"aggs":{"myTerms":{"terms":{"field":"contents.activities.assay_type.keyword","size":99999},"meta":{"path":["contents","activitie...'
2026-04-24 18:59:37.753 [INFO] 2026-04-24T18:59:37.753Z 5dccf342-a6db-4743-98aa-a19513cf6aad opensearch … with a response body of length 4512 starting in '{"took":2,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":null,"hits":[]},"aggregations":{"diagnoses.phenotype":{"doc_count":0,"untagged":{"doc_count":0},"myTerms":{"meta":{"path":["contents","diagnoses","phenotype"]},"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[]}},"biosamples.disease":{"doc_count":0,"untagged":{"doc_count":0},"myTerms":{"meta":{"path":["contents","biosamples","disease"]},"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[]}},"diagnoses.disease":{"doc_count":0,"untagged":{"doc_count":0},"myTerms":{"meta":{"path":["contents","diagnoses","disease"]},"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[]}},"diagnoses.phenopacket":{"doc_count":0,"untagged":{"doc_count":0},"myTerms":{"meta":{"path":["contents","diagnoses","phenopacket"]},"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[]}},"files.is_supplementary":{"doc_count":0,"untagged":{"…'
2026-04-24 18:59:37.753 [INFO] 2026-04-24T18:59:37.753Z 5dccf342-a6db-4743-98aa-a19513cf6aad opensearch Got 200 response after 0.011s from POST to https://vpc-azul-index-anvilprod-ggipah4skn2ftt47u4xgvydzqm.us-east-1.es.amazonaws.com:443/azul_v2_anvilprod_anvil13-it_files_aggregate/_search
2026-04-24 18:59:37.754 [INFO] 2026-04-24T18:59:37.754Z 5dccf342-a6db-4743-98aa-a19513cf6aad azul.chalice … with a response body starting in {"hits": [], "pagination": {"count": 0, "total": 0, "size": 100, "next": null, "previous": null, "pages": 0, "sort": "files.file_id", "order": "asc"}, "termFacets": {"diagnoses.phenotype": {"terms": [], "total": 0, "type": "terms"}, "biosamples.disease": {"terms": [], "total": 0, "type": "terms"}, "diagnoses.disease": {"terms": [], "total": 0, "type": "terms"}, "diagnoses.phenopacket": {"terms": [], "total": 0, "type": "terms"}, "files.is_supplementary": {"terms": [], "total": 0, "type": "terms"}, "files.data_modality": {"terms": [], "total": 0, "type": "terms"}, "donors.reported_ethnicity": {"terms": [], "total": 0, "type": "terms"}, "datasets.consent_group": {"terms": [], "total": 0, "type": "terms"}, "activities.assay_type": {"terms": [], "total": 0, "type": "terms"}, "datasets.title": {"terms": [], "total": 0, "type": "terms"}, "biosamples.anatomical_site": {"terms": [], "total": 0, "type": "terms"}, "donors.organism_type": {"terms": [], "total": 0, "type": "terms"}, "datasets.data_use_permission": {"term
2026-04-24 18:59:37.754 [INFO] 2026-04-24T18:59:37.754Z 5dccf342-a6db-4743-98aa-a19513cf6aad azul.chalice Returning 200 response with headers {"headers": {"Access-Control-Allow-Origin": "*", "Access-Control-Allow-Headers": "Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key", "Content-Security-Policy": "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", "Referrer-Policy": "strict-origin-when-cross-origin", "Strict-Transport-Security": "max-age=63072000; includeSubDomains; preload", "X-Content-Type-Options": "nosniff", "X-Frame-Options": "DENY", "X-XSS-Protection": "1; mode=block", "Cache-Control": "no-store"}}.
2026-04-24 18:59:37.759 REPORT RequestId: 5dccf342-a6db-4743-98aa-a19513cf6aad Duration: 32.48 ms Billed Duration: 33 ms Memory Size: 768 MB Max Memory Used: 264 MB
2026-04-24 18:59:37.759 END RequestId: 5dccf342-a6db-4743-98aa-a19513cf6aad

Metadata

Metadata

Assignees

No one assigned

    Labels

    spike:3[process] Spike estimate of three pointstest[subject] Unit and integration test code

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions