Pilot 6552: update upload preupload logic to batchly create resumable_info.json

[colorzzr]

Summary

Previously, the preupload logic will generate resumable_info.json after all preupload finished. But if any batch raises error or gets interrupted, users will lose current upload completely. To resolve this, cli add a new field to keep track unregistered_item at local and update resumable_info.json per batch:

        manifest_json = {
            'project_code': self.project_code,
            'operator': self.operator,
            'zone': self.zone,
            'parent_folder_id': self.parent_folder_id,
            'current_folder_node': self.current_folder_node,
            'tags': self.tags,
            'registered_items': {file_object.item_id: file_object.to_dict() for file_object in registered_items},
            'unregistered_items': {file_object.local_path: file_object.to_dict() for file_object in unregistered_items},
            'attributes': self.attributes if self.attributes else {},
            'resumable_manifest_file': output_path,
        }

JIRA Issues

PILOT_6552

Type of Change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update
• Refactor or reformatting

Testing

Are there any new or updated tests to validate the changes?

• Yes
• No

Test Directions

add test cases to make sure outputs contains unregistered_items

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
__init__.py	0	0	100%
pilotcli.py	0	59	0%	5, 7–10, 12–18, 21–22, 24–25, 27–28, 30–34, 36–39, 41, 43–44, 46–47, 50, 53–57, 59, 61–64, 66–71, 74–80, 83–85
commands
__init__.py	0	0	100%
container_registry.py	0	19	59%	16, 22–24, 31–33, 41–44, 50–53, 62–65
dataset.py	0	9	89%	20, 38–39, 47, 82–83, 92, 126, 130
entry_point.py	0	11	85%	41, 47, 75, 98–100, 102–106
file.py	0	38	87%	48, 159–162, 166, 171, 174–176, 319–326, 328, 330, 344–350, 352, 381, 383, 387, 433, 436, 451–453, 459–460
folder.py	0	1	96%	23
project.py	0	2	95%	18, 61
user.py	0	9	82%	27, 44–45, 53–54, 62–63, 84–85
configs
__init__.py	0	0	100%
app_config.py	0	0	100%
config.py	0	0	100%
user_config.py	0	23	82%	35, 75, 84, 95, 128, 137–138, 141, 151, 157, 161, 165, 169, 173–174, 178, 182–183, 187, 191–192, 200, 204
utils.py	0	35	36%	13–14, 16–19, 22–23, 25, 27, 31–32, 34, 36–37, 40, 43, 45–47, 49–51, 54–57, 59, 67, 69–72, 74–75
models
__init__.py	0	0	100%
enums.py	0	0	100%
item.py	0	0	100%
service_meta_class.py	0	1	80%	9
singleton.py	0	0	100%
upload_form.py	0	4	33%	28, 40–42
resources
custom_error.py	0	0	100%
custom_help.py	0	0	100%
services
__init__.py	0	0	100%
services/clients
__init__.py	0	0	100%
base_auth_client.py	0	1	96%	54
base_client.py	0	2	95%	53, 104
services/container_registry_manager
container_registry_manager.py	0	108	17%	19–20, 23–26, 29–33, 36–44, 48–60, 62–64, 68–82, 84–86, 90–99, 101–103, 107–110, 125–136, 140–143, 147–162, 164, 166–169
services/crypto
__init__.py	0	0	100%
crypto.py	0	19	40%	35, 43–47, 49, 59–61, 69–75, 77–78
services/dataset_manager
dataset_detail.py	0	22	71%	38, 59, 66–71, 73–75, 77–86, 88
dataset_download.py	0	22	82%	53, 63–65, 74–76, 92, 94–95, 97–98, 100, 102–104, 112–115, 149, 164
dataset_list.py	0	8	81%	36–41, 43, 52
model.py	0	0	100%
services/file_manager
__init__.py	0	0	100%
file_list.py	0	12	85%	60–66, 86–88, 100, 102
file_manifests.py	0	80	31%	18–23, 43–47, 51–57, 59, 63–68, 70, 74–75, 79–83, 85, 88, 90–91, 93–94, 96–99, 104–110, 115–120, 123–129, 131–133, 136–142, 144–146, 148–151
file_tag.py	0	40	31%	23–24, 28–32, 34, 37–49, 51–53, 55–56, 60–62, 65–69, 71–74, 76–77
services/file_manager/file_download
__init__.py	0	0	100%
download_client.py	0	105	55%	52–61, 79–80, 122–126, 128–130, 142–143, 148–149, 153–162, 165–170, 180–181, 212, 217, 223–233, 235–236, 239–240, 245–246, 250–255, 259–260, 263–265, 267–278, 285, 290, 305, 309–311, 313–326, 328
model.py	0	0	100%
services/file_manager/file_metadata
__init__.py	0	0	100%
file_metadata_client.py	0	3	95%	98–100
folder_client.py	0	4	91%	50, 79–81
services/file_manager/file_move
__init__.py	0	0	100%
file_move_client.py	0	37	59%	60–64, 66, 74, 79–83, 86–94, 96, 98–103, 114, 116–120, 122, 172–173
services/file_manager/file_trash
__init__.py	0	0	100%
file_trash_client.py	0	2	96%	69, 90
utils.py	0	3	92%	39, 63, 70
services/file_manager/file_upload
__init__.py	0	0	100%
exception.py	0	0	100%
file_upload.py	0	26	87%	38–45, 103–105, 117, 142, 167–169, 198–199, 208, 213, 414–416, 425, 428, 432
models.py	0	2	96%	24, 44
upload_client.py	0	48	77%	106–109, 149, 221, 234–235, 237–251, 253, 255–259, 261–265, 267–268, 369–370, 372, 442, 448–449, 454–457, 459–460
upload_validator.py	0	11	76%	46–52, 55–56, 61, 63
services/logger_services
__init__.py	0	0	100%
debugging_log.py	0	0	100%
log_functions.py	0	0	100%
services/output_manager
__init__.py	0	0	100%
error_handler.py	0	0	100%
help_page.py	0	7	91%	13–17, 19, 21
message_handler.py	0	56	73%	24, 46, 51, 66–68, 73, 83, 88, 101, 106, 113, 118, 123, 127, 143, 178, 188, 197, 215, 237, 281–292, 303–308, 310–316, 327, 331–332, 336–337, 339–340, 344, 348, 352
services/project_manager
__init__.py	0	0	100%
project.py	0	6	84%	40, 46–47, 49–51
services/user_authentication
__init__.py	0	0	100%
decorator.py	0	2	93%	27, 30
token_manager.py	0	8	88%	42–46, 73–74, 91
user_login_logout.py	0	16	84%	29–30, 41, 126, 130–136, 138–140, 144–145
utils
__init__.py	0	0	100%
aggregated.py	0	26	84%	56, 66–68, 82–84, 116, 130–131, 144, 168–175, 177–178, 206, 214, 231–232, 270
TOTAL	3692	887	75%

[afmam]

Left a few comments

[afmam]

Thanks for making the changes! LGTM