Hangups from OpenSearch on `/index/summary` request result in 500 response

[achave11-ucsc]

CW Log Insights at the bottom.

Tracebacks:

[WARNING]	2026-04-29T16:29:56.820Z	a2734cbd-ea25-403c-88d2-fea4acc154d9	opensearch	Got no response after 0.057s from POST to https://vpc-azul-index-prod-mjluewlvxikvhillefxotw2tbm.us-east-1.es.amazonaws.com:443/azul_v2_prod_dcp59_projects_aggregate/_search
Traceback (most recent call last):
  File ""/opt/python/urllib3/connectionpool.py"", line 787, in urlopen
    response = self._make_request(
        conn,
    ...<10 lines>...
        **response_kw,
    )
  File ""/opt/python/urllib3/connectionpool.py"", line 534, in _make_request
    response = conn.getresponse()
  File ""/opt/python/urllib3/connection.py"", line 571, in getresponse
    httplib_response = super().getresponse()
  File ""/var/lang/lib/python3.14/http/client.py"", line 1430, in getresponse
    response.begin()
    ~~~~~~~~~~~~~~^^
  File ""/var/lang/lib/python3.14/http/client.py"", line 331, in begin
    version, status, reason = self._read_status()
                              ~~~~~~~~~~~~~~~~~^^
  File ""/var/lang/lib/python3.14/http/client.py"", line 300, in _read_status
    raise RemoteDisconnected(""Remote end closed connection without""
                             "" response"")
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ""/opt/python/opensearchpy/connection/http_urllib3.py"", line 280, in perform_request
    response = self.pool.urlopen(
        method, url, body, retries=Retry(False), headers=request_headers, **kw
    )
  File ""/var/task/azul/opensearch.py"", line 188, in urlopen
    return self._inner.urlopen(method, url, body, headers=request.headers, **kwargs)
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/opt/python/urllib3/connectionpool.py"", line 841, in urlopen
    retries = retries.increment(
        method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
    )
  File ""/opt/python/urllib3/util/retry.py"", line 465, in increment
    raise reraise(type(error), error, _stacktrace)
          ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/opt/python/urllib3/util/util.py"", line 38, in reraise
    raise value.with_traceback(tb)
  File ""/opt/python/urllib3/connectionpool.py"", line 787, in urlopen
    response = self._make_request(
        conn,
    ...<10 lines>...
        **response_kw,
    )
  File ""/opt/python/urllib3/connectionpool.py"", line 534, in _make_request
    response = conn.getresponse()
  File ""/opt/python/urllib3/connection.py"", line 571, in getresponse
    httplib_response = super().getresponse()
  File ""/var/lang/lib/python3.14/http/client.py"", line 1430, in getresponse
    response.begin()
    ~~~~~~~~~~~~~~^^
  File ""/var/lang/lib/python3.14/http/client.py"", line 331, in begin
    version, status, reason = self._read_status()
                              ~~~~~~~~~~~~~~~~~^^
  File ""/var/lang/lib/python3.14/http/client.py"", line 300, in _read_status
    raise RemoteDisconnected(""Remote end closed connection without""
                             "" response"")
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))"

[ERROR]	2026-04-29T16:29:57.441Z	a2734cbd-ea25-403c-88d2-fea4acc154d9	azul-service-prod	Caught exception for path /index/summary
Traceback (most recent call last):
  File ""/opt/python/urllib3/connectionpool.py"", line 787, in urlopen
    response = self._make_request(
        conn,
    ...<10 lines>...
        **response_kw,
    )
  File ""/opt/python/urllib3/connectionpool.py"", line 534, in _make_request
    response = conn.getresponse()
  File ""/opt/python/urllib3/connection.py"", line 571, in getresponse
    httplib_response = super().getresponse()
  File ""/var/lang/lib/python3.14/http/client.py"", line 1430, in getresponse
    response.begin()
    ~~~~~~~~~~~~~~^^
  File ""/var/lang/lib/python3.14/http/client.py"", line 331, in begin
    version, status, reason = self._read_status()
                              ~~~~~~~~~~~~~~~~~^^
  File ""/var/lang/lib/python3.14/http/client.py"", line 300, in _read_status
    raise RemoteDisconnected(""Remote end closed connection without""
                             "" response"")
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ""/opt/python/opensearchpy/connection/http_urllib3.py"", line 280, in perform_request
    response = self.pool.urlopen(
        method, url, body, retries=Retry(False), headers=request_headers, **kw
    )
  File ""/var/task/azul/opensearch.py"", line 188, in urlopen
    return self._inner.urlopen(method, url, body, headers=request.headers, **kwargs)
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/opt/python/urllib3/connectionpool.py"", line 841, in urlopen
    retries = retries.increment(
        method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
    )
  File ""/opt/python/urllib3/util/retry.py"", line 465, in increment
    raise reraise(type(error), error, _stacktrace)
          ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/opt/python/urllib3/util/util.py"", line 38, in reraise
    raise value.with_traceback(tb)
  File ""/opt/python/urllib3/connectionpool.py"", line 787, in urlopen
    response = self._make_request(
        conn,
    ...<10 lines>...
        **response_kw,
    )
  File ""/opt/python/urllib3/connectionpool.py"", line 534, in _make_request
    response = conn.getresponse()
  File ""/opt/python/urllib3/connection.py"", line 571, in getresponse
    httplib_response = super().getresponse()
  File ""/var/lang/lib/python3.14/http/client.py"", line 1430, in getresponse
    response.begin()
    ~~~~~~~~~~~~~~^^
  File ""/var/lang/lib/python3.14/http/client.py"", line 331, in begin
    version, status, reason = self._read_status()
                              ~~~~~~~~~~~~~~~~~^^
  File ""/var/lang/lib/python3.14/http/client.py"", line 300, in _read_status
    raise RemoteDisconnected(""Remote end closed connection without""
                             "" response"")
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ""/var/task/chalice/app.py"", line 1922, in _get_view_function_response
    response = view_function(**function_args)
  File ""/var/task/azul/service/index_controller.py"", line 346, in get_head_index_summary
    return self.summary()
           ~~~~~~~~~~~~^^
  File ""/var/task/azul/service/index_controller.py"", line 392, in summary
    response = self._service.summary(self.app.catalog, filters)
  File ""/var/task/azul/service/index_service.py"", line 259, in summary
    aggs = dict(executor.map(summary, aggs_by_authority))
  File ""/var/lang/lib/python3.14/concurrent/futures/_base.py"", line 639, in result_iterator
    yield _result_or_cancel(fs.pop())
          ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File ""/var/lang/lib/python3.14/concurrent/futures/_base.py"", line 311, in _result_or_cancel
    return fut.result(timeout)
           ~~~~~~~~~~^^^^^^^^^
  File ""/var/lang/lib/python3.14/concurrent/futures/_base.py"", line 443, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File ""/var/lang/lib/python3.14/concurrent/futures/_base.py"", line 395, in __get_result
    raise self._exception
  File ""/var/lang/lib/python3.14/concurrent/futures/thread.py"", line 86, in run
    result = ctx.run(self.task)
  File ""/var/lang/lib/python3.14/concurrent/futures/thread.py"", line 73, in run
    return fn(*args, **kwargs)
  File ""/var/task/azul/service/index_service.py"", line 254, in summary
    return entity_type, self._summary(catalog=catalog,
                        ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
                                      entity_type=entity_type,
                                      ^^^^^^^^^^^^^^^^^^^^^^^^
                                      filters=filters)
                                      ^^^^^^^^^^^^^^^^
  File ""/var/task/azul/service/index_service.py"", line 288, in _summary
    response = request.execute(ignore_cache=True)
  File ""/opt/python/opensearchpy/helpers/search.py"", line 759, in execute
    opensearch.search(
    ~~~~~~~~~~~~~~~~~^
        index=self._index, body=self.to_dict(), **self._params
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ),
    ^
  File ""/opt/python/opensearchpy/client/utils.py"", line 176, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File ""/opt/python/opensearchpy/client/__init__.py"", line 2431, in search
    return self.transport.perform_request(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        ""POST"",
        ^^^^^^^
    ...<3 lines>...
        body=body,
        ^^^^^^^^^^
    )
    ^
  File ""/opt/python/opensearchpy/transport.py"", line 455, in perform_request
    raise e
  File ""/opt/python/opensearchpy/transport.py"", line 418, in perform_request
    status, headers_response, data = connection.perform_request(
                                     ~~~~~~~~~~~~~~~~~~~~~~~~~~^
        method,
        ^^^^^^^
    ...<5 lines>...
        timeout=timeout,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File ""/var/task/azul/opensearch.py"", line 80, in perform_request
    return super().perform_request(method, url, params, body, timeout, ignore, headers)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/opt/python/opensearchpy/connection/http_urllib3.py"", line 295, in perform_request
    raise ConnectionError(""N/A"", str(e), e)
opensearchpy.exceptions.ConnectionError: ConnectionError(('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))) caused by: ProtocolError(('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')))

CW Insights

@timestamp	@message
2026-04-29 16:29:56.522	START RequestId: a2734cbd-ea25-403c-88d2-fea4acc154d9 Version: 26
2026-04-29 16:29:56.524	[INFO] 2026-04-29T16:29:56.523Z a2734cbd-ea25-403c-88d2-fea4acc154d9 azul.chalice Received GET request for '/index/summary', with {"query": {"catalog": "dcp59", "filters": "{\"projectId\":{\"is\":[\"63b5b6c1-bbcd-487d-8c2e-0095150c1ecd\"]}}"}, "headers": {"accept": "*/*", "accept-encoding": "gzip, deflate, br, zstd", "accept-language": "en-US,en;q=0.9", "cloudfront-forwarded-proto": "https", "cloudfront-is-desktop-viewer": "true", "cloudfront-is-mobile-viewer": "false", "cloudfront-is-smarttv-viewer": "false", "cloudfront-is-tablet-viewer": "false", "cloudfront-viewer-asn": "217", "cloudfront-viewer-country": "US", "host": "service.azul.data.humancellatlas.org", "origin": "https://explore.data.humancellatlas.org", "priority": "u=4", "referer": "https://explore.data.humancellatlas.org/", "sec-fetch-dest": "empty", "sec-fetch-mode": "cors", "sec-fetch-site": "same-site", "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:147.0) Gecko/20100101 Firefox/147.0", "via": "2.0 077041ddf56b8eb9505a28cba1464114.cloudfront.net (CloudFront)", "x-amz-cf-id": "zOpEqldBIDNWPSAFysROF-OD39_ToxhVVRemMseM7OHdHpJsr0uIgA==", "x-amzn-trace-id": "Root=1-69f23204-6c6ba0674ff6020e3d79d63b", "x-forwarded-for": "REDACTED, 18.68.0.81", "x-forwarded-port": "443", "x-forwarded-proto": "https"}}.
2026-04-29 16:29:56.524	[INFO] 2026-04-29T16:29:56.524Z a2734cbd-ea25-403c-88d2-fea4acc154d9 azul.chalice Did not authenticate request.
2026-04-29 16:29:56.524	[INFO] 2026-04-29T16:29:56.524Z a2734cbd-ea25-403c-88d2-fea4acc154d9 azul.chalice … without a request body
2026-04-29 16:29:56.685	[INFO] 2026-04-29T16:29:56.685Z a2734cbd-ea25-403c-88d2-fea4acc154d9 opensearch Making POST request to https://vpc-azul-index-prod-mjluewlvxikvhillefxotw2tbm.us-east-1.es.amazonaws.com:443/azul_v2_prod_dcp59_projects_aggregate/_search
2026-04-29 16:29:56.752	[INFO] 2026-04-29T16:29:56.752Z a2734cbd-ea25-403c-88d2-fea4acc154d9 opensearch … with a request body of length 17708 starting in b'{"query":{"bool":{"must":[{"constant_score":{"filter":{"terms":{"contents.projects.document_id.keyword":["63b5b6c1-bbcd-487d-8c2e-0095150c1ecd"]}}}}]}},"aggs":{"sourceId":{"filter":{"bool":{"must":[{"constant_score":{"filter":{"terms":{"contents.projects.document_id.keyword":["63b5b6c1-bbcd-487d-8c2e-0095150c1ecd"]}}}}]}},"aggs":{"myTerms":{"terms":{"field":"sources.id.keyword","size":99999},"meta":{"path":["sources","id"]}},"untagged":{"missing":{"field":"sources.id.keyword"}}}},"organ":{"filter":{"bool":{"must":[{"constant_score":{"filter":{"terms":{"contents.projects.document_id.keyword":["63b5b6c1-bbcd-487d-8c2e-0095150c1ecd"]}}}}]}},"aggs":{"myTerms":{"terms":{"field":"contents.samples.organ.keyword","size":99999},"meta":{"path":["contents","samples","organ"]}},"untagged":{"missing":{"field":"contents.samples.organ.keyword"}}}},"organPart":{"filter":{"bool":{"must":[{"constant_score":{"filter":{"terms":{"contents.projects.document_id.keyword":["63b5b6c1-bbcd-487d-8c2e-0095150c1ecd"]}}}}]}},"aggs":{"my...'
2026-04-29 16:29:57.002	[WARNING] 2026-04-29T16:29:56.820Z a2734cbd-ea25-403c-88d2-fea4acc154d9 opensearch Got no response after 0.057s from POST to https://vpc-azul-index-prod-mjluewlvxikvhillefxotw2tbm.us-east-1.es.amazonaws.com:443/azul_v2_prod_dcp59_projects_aggregate/_search Traceback (most recent call last): File "/opt/python/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( conn, ...<10 lines>... **response_kw, ) File "/opt/python/urllib3/connectionpool.py", line 534, in _make_request response = conn.getresponse() File "/opt/python/urllib3/connection.py", line 571, in getresponse httplib_response = super().getresponse() File "/var/lang/lib/python3.14/http/client.py", line 1430, in getresponse response.begin() ~~~~~~~~~~~~~~^^ File "/var/lang/lib/python3.14/http/client.py", line 331, in begin version, status, reason = self._read_status() ~~~~~~~~~~~~~~~~~^^ File "/var/lang/lib/python3.14/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" " response") http.client.RemoteDisconnected: Remote end closed connection without response During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/python/opensearchpy/connection/http_urllib3.py", line 280, in perform_request response = self.pool.urlopen( method, url, body, retries=Retry(False), headers=request_headers, **kw ) File "/var/task/azul/opensearch.py", line 188, in urlopen return self._inner.urlopen(method, url, body, headers=request.headers, **kwargs) ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/python/urllib3/connectionpool.py", line 841, in urlopen retries = retries.increment( method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] ) File "/opt/python/urllib3/util/retry.py", line 465, in increment raise reraise(type(error), error, _stacktrace) ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/python/urllib3/util/util.py", line 38, in reraise raise value.with_traceback(tb) File "/opt/python/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( conn, ...<10 lines>... **response_kw, ) File "/opt/python/urllib3/connectionpool.py", line 534, in _make_request response = conn.getresponse() File "/opt/python/urllib3/connection.py", line 571, in getresponse httplib_response = super().getresponse() File "/var/lang/lib/python3.14/http/client.py", line 1430, in getresponse response.begin() ~~~~~~~~~~~~~~^^ File "/var/lang/lib/python3.14/http/client.py", line 331, in begin version, status, reason = self._read_status() ~~~~~~~~~~~~~~~~~^^ File "/var/lang/lib/python3.14/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" " response") urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
2026-04-29 16:29:57.002	[WARNING] 2026-04-29T16:29:57.002Z a2734cbd-ea25-403c-88d2-fea4acc154d9 opensearch … without a response body
2026-04-29 16:29:57.042	[INFO] 2026-04-29T16:29:57.025Z a2734cbd-ea25-403c-88d2-fea4acc154d9 opensearch Making POST request to https://vpc-azul-index-prod-mjluewlvxikvhillefxotw2tbm.us-east-1.es.amazonaws.com:443/azul_v2_prod_dcp59_samples_aggregate/_search
2026-04-29 16:29:57.081	[INFO] 2026-04-29T16:29:57.081Z a2734cbd-ea25-403c-88d2-fea4acc154d9 opensearch … with a request body of length 837368 starting in b'{"query":{"bool":{"must":[{"constant_score":{"filter":{"terms":{"contents.projects.document_id.keyword":["63b5b6c1-bbcd-487d-8c2e-0095150c1ecd"]}}}},{"constant_score":{"filter":{"terms":{"sources.id.keyword":["009e77b1-362e-47e0-9ae7-a03a390a07ce","01d0e707-2021-4740-9b21-60c8ad24fc76","01e26559-f5a5-42f8-b601-c24ef42f4aba","020e1f1a-7314-44b5-a281-942a45269815","029f160a-b25f-44cd-ad7e-98ee3709135b","02bb75f3-e9c1-4ae4-bf99-ec8115cb1adf","02cb6c0a-5048-45ad-99aa-d03602f8453a","03447627-4e56-4ec5-a0c5-01b7eb71201c","039c615f-45e0-4661-9ce5-e65bbf391811","03d4e6b9-d1fe-4117-87cf-7933d2f4c859","04a6e8ab-5c9c-4c86-aacf-fdb96fc0e9be","055bcefd-9702-4426-afe9-2e2625ab7c96","05b2c591-58aa-425b-b7be-2b0a55c7b6e0","05e6fd2d-1d2b-487f-b00d-b0924aef5f99","05e95b36-cbd0-4ed9-9ed9-117e72726c25","06fea4ee-9ea4-4794-976b-226dfcd0beb7","07136895-d047-4cc4-92ba-d9e14abeb694","07396568-cc04-40d8-8f6c-00e3baafdd93","07413e37-0c3e-46a4-a92a-92a4f16425e7","07726113-9674-4330-b369-de61bc4577ed","07b0aa69-1687-45ae-812f-1390545...'
2026-04-29 16:29:57.120	[INFO] 2026-04-29T16:29:57.082Z a2734cbd-ea25-403c-88d2-fea4acc154d9 opensearch Making POST request to https://vpc-azul-index-prod-mjluewlvxikvhillefxotw2tbm.us-east-1.es.amazonaws.com:443/azul_v2_prod_dcp59_files_aggregate/_search
2026-04-29 16:29:57.121	[INFO] 2026-04-29T16:29:57.104Z a2734cbd-ea25-403c-88d2-fea4acc154d9 opensearch Making POST request to https://vpc-azul-index-prod-mjluewlvxikvhillefxotw2tbm.us-east-1.es.amazonaws.com:443/azul_v2_prod_dcp59_cell_suspensions_aggregate/_search
2026-04-29 16:29:57.121	[INFO] 2026-04-29T16:29:57.121Z a2734cbd-ea25-403c-88d2-fea4acc154d9 opensearch … with a request body of length 837133 starting in b'{"query":{"bool":{"must":[{"constant_score":{"filter":{"terms":{"contents.projects.document_id.keyword":["63b5b6c1-bbcd-487d-8c2e-0095150c1ecd"]}}}},{"constant_score":{"filter":{"terms":{"sources.id.keyword":["009e77b1-362e-47e0-9ae7-a03a390a07ce","01d0e707-2021-4740-9b21-60c8ad24fc76","01e26559-f5a5-42f8-b601-c24ef42f4aba","020e1f1a-7314-44b5-a281-942a45269815","029f160a-b25f-44cd-ad7e-98ee3709135b","02bb75f3-e9c1-4ae4-bf99-ec8115cb1adf","02cb6c0a-5048-45ad-99aa-d03602f8453a","03447627-4e56-4ec5-a0c5-01b7eb71201c","039c615f-45e0-4661-9ce5-e65bbf391811","03d4e6b9-d1fe-4117-87cf-7933d2f4c859","04a6e8ab-5c9c-4c86-aacf-fdb96fc0e9be","055bcefd-9702-4426-afe9-2e2625ab7c96","05b2c591-58aa-425b-b7be-2b0a55c7b6e0","05e6fd2d-1d2b-487f-b00d-b0924aef5f99","05e95b36-cbd0-4ed9-9ed9-117e72726c25","06fea4ee-9ea4-4794-976b-226dfcd0beb7","07136895-d047-4cc4-92ba-d9e14abeb694","07396568-cc04-40d8-8f6c-00e3baafdd93","07413e37-0c3e-46a4-a92a-92a4f16425e7","07726113-9674-4330-b369-de61bc4577ed","07b0aa69-1687-45ae-812f-1390545...'
2026-04-29 16:29:57.122	[INFO] 2026-04-29T16:29:57.121Z a2734cbd-ea25-403c-88d2-fea4acc154d9 opensearch … with a request body of length 836913 starting in b'{"query":{"bool":{"must":[{"constant_score":{"filter":{"terms":{"contents.projects.document_id.keyword":["63b5b6c1-bbcd-487d-8c2e-0095150c1ecd"]}}}},{"constant_score":{"filter":{"terms":{"sources.id.keyword":["009e77b1-362e-47e0-9ae7-a03a390a07ce","01d0e707-2021-4740-9b21-60c8ad24fc76","01e26559-f5a5-42f8-b601-c24ef42f4aba","020e1f1a-7314-44b5-a281-942a45269815","029f160a-b25f-44cd-ad7e-98ee3709135b","02bb75f3-e9c1-4ae4-bf99-ec8115cb1adf","02cb6c0a-5048-45ad-99aa-d03602f8453a","03447627-4e56-4ec5-a0c5-01b7eb71201c","039c615f-45e0-4661-9ce5-e65bbf391811","03d4e6b9-d1fe-4117-87cf-7933d2f4c859","04a6e8ab-5c9c-4c86-aacf-fdb96fc0e9be","055bcefd-9702-4426-afe9-2e2625ab7c96","05b2c591-58aa-425b-b7be-2b0a55c7b6e0","05e6fd2d-1d2b-487f-b00d-b0924aef5f99","05e95b36-cbd0-4ed9-9ed9-117e72726c25","06fea4ee-9ea4-4794-976b-226dfcd0beb7","07136895-d047-4cc4-92ba-d9e14abeb694","07396568-cc04-40d8-8f6c-00e3baafdd93","07413e37-0c3e-46a4-a92a-92a4f16425e7","07726113-9674-4330-b369-de61bc4577ed","07b0aa69-1687-45ae-812f-1390545...'
2026-04-29 16:29:57.346	[INFO] 2026-04-29T16:29:57.346Z a2734cbd-ea25-403c-88d2-fea4acc154d9 opensearch Got 200 response after 0.246s from POST to https://vpc-azul-index-prod-mjluewlvxikvhillefxotw2tbm.us-east-1.es.amazonaws.com:443/azul_v2_prod_dcp59_samples_aggregate/_search
2026-04-29 16:29:57.346	[INFO] 2026-04-29T16:29:57.346Z a2734cbd-ea25-403c-88d2-fea4acc154d9 opensearch … with a response body of length 10242 starting in '{"took":50,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":5,"relation":"eq"},"max_score":null,"hits":[]},"aggregations":{"organ":{"doc_count":5,"untagged":{"doc_count":0},"myTerms":{"meta":{"path":["contents","samples","organ"]},"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"umbilical cord","doc_count":5}]}},"sourceId":{"doc_count":5,"untagged":{"doc_count":0},"myTerms":{"meta":{"path":["sources","id"]},"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"f5b23f04-26cd-468b-9e2e-3ee76766b810","doc_count":5}]}},"sampleEntityType":{"doc_count":5,"untagged":{"doc_count":0},"myTerms":{"meta":{"path":["contents","samples","entity_type"]},"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"specimens","doc_count":5}]}},"dataUseRestriction":{"doc_count":5,"untagged":{"doc_count":0},"myTerms":{"meta":{"path":["contents","projects","data_use_restriction"]},"doc_count_error_upper_bound":0,"su…'
2026-04-29 16:29:57.351	[WARNING] 2026-04-29T16:29:57.351Z a2734cbd-ea25-403c-88d2-fea4acc154d9 urllib3.connectionpool Connection pool is full, discarding connection: vpc-azul-index-prod-mjluewlvxikvhillefxotw2tbm.us-east-1.es.amazonaws.com. Connection pool size: 1
2026-04-29 16:29:57.352	[INFO] 2026-04-29T16:29:57.352Z a2734cbd-ea25-403c-88d2-fea4acc154d9 opensearch … with a response body of length 10282 starting in '{"took":44,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":10,"relation":"eq"},"max_score":null,"hits":[]},"aggregations":{"organ":{"doc_count":10,"untagged":{"doc_count":0},"myTerms":{"meta":{"path":["contents","samples","organ"]},"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"umbilical cord","doc_count":10}]}},"sourceId":{"doc_count":10,"untagged":{"doc_count":0},"myTerms":{"meta":{"path":["sources","id"]},"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"f5b23f04-26cd-468b-9e2e-3ee76766b810","doc_count":10}]}},"sampleEntityType":{"doc_count":10,"untagged":{"doc_count":0},"myTerms":{"meta":{"path":["contents","samples","entity_type"]},"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"specimens","doc_count":10}]}},"dataUseRestriction":{"doc_count":10,"untagged":{"doc_count":0},"myTerms":{"meta":{"path":["contents","projects","data_use_restriction"]},"doc_count_error_upper_boun…'
2026-04-29 16:29:57.352	[INFO] 2026-04-29T16:29:57.352Z a2734cbd-ea25-403c-88d2-fea4acc154d9 opensearch Got 200 response after 0.230s from POST to https://vpc-azul-index-prod-mjluewlvxikvhillefxotw2tbm.us-east-1.es.amazonaws.com:443/azul_v2_prod_dcp59_cell_suspensions_aggregate/_search
2026-04-29 16:29:57.437	[INFO] 2026-04-29T16:29:57.437Z a2734cbd-ea25-403c-88d2-fea4acc154d9 opensearch Got 200 response after 0.315s from POST to https://vpc-azul-index-prod-mjluewlvxikvhillefxotw2tbm.us-east-1.es.amazonaws.com:443/azul_v2_prod_dcp59_files_aggregate/_search
2026-04-29 16:29:57.437	[INFO] 2026-04-29T16:29:57.437Z a2734cbd-ea25-403c-88d2-fea4acc154d9 opensearch … with a response body of length 10583 starting in '{"took":77,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":22,"relation":"eq"},"max_score":null,"hits":[]},"aggregations":{"organ":{"doc_count":22,"untagged":{"doc_count":2},"myTerms":{"meta":{"path":["contents","samples","organ"]},"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"umbilical cord","doc_count":20}]}},"sourceId":{"doc_count":22,"untagged":{"doc_count":0},"myTerms":{"meta":{"path":["sources","id"]},"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"f5b23f04-26cd-468b-9e2e-3ee76766b810","doc_count":22}]}},"sampleEntityType":{"doc_count":22,"untagged":{"doc_count":2},"myTerms":{"meta":{"path":["contents","samples","entity_type"]},"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"specimens","doc_count":20}]}},"dataUseRestriction":{"doc_count":22,"untagged":{"doc_count":0},"myTerms":{"meta":{"path":["contents","projects","data_use_restriction"]},"doc_count_error_upper_boun…'
2026-04-29 16:29:57.437	[WARNING] 2026-04-29T16:29:57.436Z a2734cbd-ea25-403c-88d2-fea4acc154d9 urllib3.connectionpool Connection pool is full, discarding connection: vpc-azul-index-prod-mjluewlvxikvhillefxotw2tbm.us-east-1.es.amazonaws.com. Connection pool size: 1
2026-04-29 16:29:57.462	[ERROR] 2026-04-29T16:29:57.441Z a2734cbd-ea25-403c-88d2-fea4acc154d9 azul-service-prod Caught exception for path /index/summary Traceback (most recent call last): File "/opt/python/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( conn, ...<10 lines>... **response_kw, ) File "/opt/python/urllib3/connectionpool.py", line 534, in _make_request response = conn.getresponse() File "/opt/python/urllib3/connection.py", line 571, in getresponse httplib_response = super().getresponse() File "/var/lang/lib/python3.14/http/client.py", line 1430, in getresponse response.begin() ~~~~~~~~~~~~~~^^ File "/var/lang/lib/python3.14/http/client.py", line 331, in begin version, status, reason = self._read_status() ~~~~~~~~~~~~~~~~~^^ File "/var/lang/lib/python3.14/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" " response") http.client.RemoteDisconnected: Remote end closed connection without response During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/python/opensearchpy/connection/http_urllib3.py", line 280, in perform_request response = self.pool.urlopen( method, url, body, retries=Retry(False), headers=request_headers, **kw ) File "/var/task/azul/opensearch.py", line 188, in urlopen return self._inner.urlopen(method, url, body, headers=request.headers, **kwargs) ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/python/urllib3/connectionpool.py", line 841, in urlopen retries = retries.increment( method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2] ) File "/opt/python/urllib3/util/retry.py", line 465, in increment raise reraise(type(error), error, _stacktrace) ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/python/urllib3/util/util.py", line 38, in reraise raise value.with_traceback(tb) File "/opt/python/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( conn, ...<10 lines>... **response_kw, ) File "/opt/python/urllib3/connectionpool.py", line 534, in _make_request response = conn.getresponse() File "/opt/python/urllib3/connection.py", line 571, in getresponse httplib_response = super().getresponse() File "/var/lang/lib/python3.14/http/client.py", line 1430, in getresponse response.begin() ~~~~~~~~~~~~~~^^ File "/var/lang/lib/python3.14/http/client.py", line 331, in begin version, status, reason = self._read_status() ~~~~~~~~~~~~~~~~~^^ File "/var/lang/lib/python3.14/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" " response") urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/task/chalice/app.py", line 1922, in _get_view_function_response response = view_function(**function_args) File "/var/task/azul/service/index_controller.py", line 346, in get_head_index_summary return self.summary() ~~~~~~~~~~~~^^ File "/var/task/azul/service/index_controller.py", line 392, in summary response = self._service.summary(self.app.catalog, filters) File "/var/task/azul/service/index_service.py", line 259, in summary aggs = dict(executor.map(summary, aggs_by_authority)) File "/var/lang/lib/python3.14/concurrent/futures/_base.py", line 639, in result_iterator yield _result_or_cancel(fs.pop()) ~~~~~~~~~~~~~~~~~^^^^^^^^^^ File "/var/lang/lib/python3.14/concurrent/futures/_base.py", line 311, in _result_or_cancel return fut.result(timeout) ~~~~~~~~~~^^^^^^^^^ File "/var/lang/lib/python3.14/concurrent/futures/_base.py", line 443, in result return self.__get_result() ~~~~~~~~~~~~~~~~~^^ File "/var/lang/lib/python3.14/concurrent/futures/_base.py", line 395, in __get_result raise self._exception File "/var/lang/lib/python3.14/concurrent/futures/thread.py", line 86, in run result = ctx.run(self.task) File "/var/lang/lib/python3.14/concurrent/futures/thread.py", line 73, in run return fn(*args, **kwargs) File "/var/task/azul/service/index_service.py", line 254, in summary return entity_type, self._summary(catalog=catalog, ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^ entity_type=entity_type, ^^^^^^^^^^^^^^^^^^^^^^^^ filters=filters) ^^^^^^^^^^^^^^^^ File "/var/task/azul/service/index_service.py", line 288, in _summary response = request.execute(ignore_cache=True) File "/opt/python/opensearchpy/helpers/search.py", line 759, in execute opensearch.search( ~~~~~~~~~~~~~~~~~^ index=self._index, body=self.to_dict(), **self._params ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ), ^ File "/opt/python/opensearchpy/client/utils.py", line 176, in _wrapped return func(*args, params=params, headers=headers, **kwargs) File "/opt/python/opensearchpy/client/__init__.py", line 2431, in search return self.transport.perform_request( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ "POST", ^^^^^^^ ...<3 lines>... body=body, ^^^^^^^^^^ ) ^ File "/opt/python/opensearchpy/transport.py", line 455, in perform_request raise e File "/opt/python/opensearchpy/transport.py", line 418, in perform_request status, headers_response, data = connection.perform_request( ~~~~~~~~~~~~~~~~~~~~~~~~~~^ method, ^^^^^^^ ...<5 lines>... timeout=timeout, ^^^^^^^^^^^^^^^^ ) ^ File "/var/task/azul/opensearch.py", line 80, in perform_request return super().perform_request(method, url, params, body, timeout, ignore, headers) ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/python/opensearchpy/connection/http_urllib3.py", line 295, in perform_request raise ConnectionError("N/A", str(e), e) opensearchpy.exceptions.ConnectionError: ConnectionError(('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))) caused by: ProtocolError(('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')))
2026-04-29 16:29:57.468	[INFO] 2026-04-29T16:29:57.468Z a2734cbd-ea25-403c-88d2-fea4acc154d9 azul.chalice … with a response body of length 133 being {"Code": "InternalServerError", "Message": "An internal server error occurred.", "RequestId": "a2734cbd-ea25-403c-88d2-fea4acc154d9"}
2026-04-29 16:29:57.468	[INFO] 2026-04-29T16:29:57.468Z a2734cbd-ea25-403c-88d2-fea4acc154d9 azul.chalice Returning 500 response with headers {"headers": {"Access-Control-Allow-Origin": "*", "Access-Control-Allow-Headers": "Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key", "Content-Security-Policy": "default-src 'self';img-src 'self' data:;script-src 'self';style-src 'self';frame-ancestors 'none';form-action 'self'", "Referrer-Policy": "strict-origin-when-cross-origin", "Strict-Transport-Security": "max-age=63072000; includeSubDomains; preload", "X-Content-Type-Options": "nosniff", "X-Frame-Options": "DENY", "X-XSS-Protection": "1; mode=block", "Cache-Control": "no-store"}}.
2026-04-29 16:29:57.482	END RequestId: a2734cbd-ea25-403c-88d2-fea4acc154d9
2026-04-29 16:29:57.482	REPORT RequestId: a2734cbd-ea25-403c-88d2-fea4acc154d9 Duration: 947.71 ms Billed Duration: 948 ms Memory Size: 512 MB Max Memory Used: 215 MB