Verbatim JSONL manifest failed to generate

[nadove-ucsc]

On hammerbox, using filters

{
    "source_id": {
        "is": [
            "a23eea3c-e635-4f89-aaf9-610b872dbc9a"
        ]
    },
    "files.is_supplementary": {
        "is": [
            true,
            false
        ]
    }
}

Resulted in

{
      "Code": "ChaliceViewError",
      "Message": "('Failed to generate manifest', 'FAILED', None)"
}

Caused by

{
  "errorMessage": "An error occurred (MalformedXML) when calling the CompleteMultipartUpload operation: The XML you provided was not well-formed or did not validate against our published schema",
  "errorType": "ClientError",
  "requestId": "a55ce9d0-080b-4cd2-91dc-6c7945008dd4",
  "stackTrace": [
    "  File \"/var/task/azul/chalice.py\", line 190, in patched_event_source_handler\n    return old_handler(self_, event, context)\n",
    "  File \"/var/task/chalice/app.py\", line 1761, in __call__\n    return self.handler(event_obj)\n",
    "  File \"/var/task/chalice/app.py\", line 1707, in __call__\n    return self._original_func(event.to_dict(), event.context)\n",
    "  File \"/var/task/azul/service/manifest_controller.py\", line 410, in generate_manifest\n    return self.generate(event)\n",
    "  File \"/var/task/azul/service/manifest_controller.py\", line 682, in generate\n    result = self._service.get_manifest(format=manifest_key.format,\n",
    "  File \"/var/task/azul/service/manifest_service.py\", line 637, in get_manifest\n    return self._generate_manifest(generator, manifest_key, partition)\n",
    "  File \"/var/task/azul/service/manifest_service.py\", line 646, in _generate_manifest\n    partition = generator.write(manifest_key, partition)\n",
    "  File \"/var/task/azul/service/manifest_service.py\", line 1376, in write\n    self.storage.complete_multipart_upload(object_key=object_key,\n",
    "  File \"/var/task/azul/service/storage_service.py\", line 273, in complete_multipart_upload\n    self._handle_overwrite(e, object_key)\n",
    "  File \"/var/task/azul/service/storage_service.py\", line 412, in _handle_overwrite\n    raise exception\n",
    "  File \"/var/task/azul/service/storage_service.py\", line 271, in complete_multipart_upload\n    self._s3.complete_multipart_upload(**request)\n",
    "  File \"/opt/python/botocore/client.py\", line 602, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n",
    "  File \"/opt/python/botocore/context.py\", line 123, in wrapper\n    return func(*args, **kwargs)\n",
    "  File \"/opt/python/botocore/client.py\", line 1078, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n"
  ]
}

I was expecting the manifest to be empty, so I suspect this was caused by trying to complete a multi-part upload with zero parts. That explanation remains speculative.

[achave11-ucsc]

@hannes-ucsc: "It looks like CompleteMultipartUpload want you to omit the Part property of the request body, instead of making it empty. Spike to verify this hypothesis."

[nadove-ucsc]

Verification failed. I get the same error when Parts is absent as when it is empty.

$ git diff
diff --git a/src/azul/service/storage_service.py b/src/azul/service/storage_service.py
index 2adab0adf..b067beb72 100644
--- a/src/azul/service/storage_service.py
+++ b/src/azul/service/storage_service.py
@@ -265,7 +265,7 @@ class StorageService:
             request = dict(Bucket=self.bucket_name,
                            Key=object_key,
                            UploadId=upload_id,
-                           MultipartUpload={'Parts': parts})
+                           MultipartUpload={'Parts': parts} if parts else {})
             if overwrite is False:
                 request['IfNoneMatch'] = '*'
             self._s3.complete_multipart_upload(**request)
$ cat /home/nadove/.config/JetBrains/PyCharm2026.1/scratches/scratch_7.py

from azul.service.storage_service import StorageService


def main():
    s = StorageService()
    key = '7969-TEST.txt'
    upload_id = s.create_multipart_upload(object_key=key)
    s.complete_multipart_upload(object_key=key, upload_id=upload_id, etags=[])


if __name__ == '__main__':
    main()
$ python3 /home/nadove/.config/JetBrains/PyCharm2026.1/scratches/scratch_7.py
Traceback (most recent call last):
    ...
botocore.exceptions.ClientError: An error occurred (MalformedXML) when calling the CompleteMultipartUpload operation: The XML you provided was not well-formed or did not validate against our published schema

[nadove-ucsc]

I think the correct way to interpret the statement

If you do not supply a valid Part with your request, the service sends back an HTTP 400 response.

is that the array of parts must be nonempty. That would imply that the only way to create a zero-size object via MPU is to upload one zero-size part before calling CompleteMultiPartUpload. This works:

$ cat /home/nadove/.config/JetBrains/PyCharm2026.1/scratches/scratch_7.py

from azul.service.storage_service import StorageService


def main():
    s = StorageService()
    key = '7969-TEST.txt'
    upload_id = s.create_multipart_upload(object_key=key)
    etag = s.upload_multipart_part(object_key=key, upload_id=upload_id, part_number=1, buffer=b'')
    s.complete_multipart_upload(object_key=key, upload_id=upload_id, etags=[etag])


if __name__ == '__main__':
    main()

https://us-east-1.console.aws.amazon.com/s3/object/edu-ucsc-gi-platform-anvil-dev-storage-nadove4.us-east-1?region=us-east-1&prefix=7969-TEST.txt

[hannes-ucsc]

For demo, attempt to reproduce.