Skip to content

Heartbeat not received when a transfer is in progress #131

@TAlonglong

Description

@TAlonglong

Describe the bug
When transfer large files or the connection is slow, heartbeats is not received during transfer.

To Reproduce

# Your code here

Expected behavior
I expect the heartbeat to be received during transfer and not the connection to be restarted due missed heartbeats when transfer in progress.

Actual results
Move-it-client logs:

[2022-10-03 05:47:57,500 DEBUG    trollmoves.client] Receiving (SUB) pytroll://XLBANDANTENNA/FENGYUN3D/RAWDATA file meos@<move-it-server> 2022-10-03T05:47:57.497649 v1.01 application/json {"
antenna": "XL-BAND", "sensor": ["mersi", "hiras", "vass"], "platform_name": "Fengyun-3D", "start_time": "2022-10-03T05:38:56", "end_time": "2022-10-03T05:47:57", "process_time": "20
22-10-03T05:38:56", "uid": "rawdata_FY3D_MPT_ch1_20221003T053856Z_20221003T054757Z_20221003T053856Z_IwmdPIGV3_1_25305.data", "type": "1", "orbit_number": "25305", "uri": "/data/rawd
ata1/rawdata_FY3D_MPT_ch1_20221003T053856Z_20221003T054757Z_20221003T053856Z_IwmdPIGV3_1_25305.data", "request_address": "<move-it-server>:9109"}
[2022-10-03 05:47:57,500 DEBUG    trollmoves.heartbeat_monitor] Resetting heartbeat alarm timer to 60.0 sec
[2022-10-03 05:47:57,501 INFO     trollmoves.client] Requesting: pytroll://XLBANDANTENNA/FENGYUN3D/RAWDATA push ubuntu@fengyun-a 2022-10-03T05:47:57.501200 v1.01 application/json {"
antenna": "XL-BAND", "sensor": ["mersi", "hiras", "vass"], "platform_name": "Fengyun-3D", "start_time": "2022-10-03T05:38:56", "end_time": "2022-10-03T05:47:57", "process_time": "20
22-10-03T05:38:56", "uid": "rawdata_FY3D_MPT_ch1_20221003T053856Z_20221003T054757Z_20221003T053856Z_IwmdPIGV3_1_25305.data", "type": "1", "orbit_number": "25305", "uri": "/data/rawd
ata1/rawdata_FY3D_MPT_ch1_20221003T053856Z_20221003T054757Z_20221003T053856Z_IwmdPIGV3_1_25305.data", "request_address": "<move-it-server>:9109", "destination": "scp://fengyun-a/data/p
ytroll/fengyun3/rawdata"}
[2022-10-03 05:47:57,502 DEBUG    trollmoves.client] Send and recv timeout is 240.00 seconds
[2022-10-03 05:48:57,603 DEBUG    trollmoves.heartbeat_monitor] Missing heartbeat alarm!
[2022-10-03 05:49:29,711 DEBUG    posttroll.publisher] Publish heartbeat (min_interval is 30.0 sec)
[2022-10-03 05:49:29,712 DEBUG    posttroll.publisher] Publish heartbeat (min_interval is 30.0 sec)
[2022-10-03 05:49:29,712 DEBUG    posttroll.publisher] Publish heartbeat (min_interval is 30.0 sec)
[2022-10-03 05:49:29,809 DEBUG    trollmoves.client] Receiving (SUB) pytroll://heartbeat/move_it_server beat eumetcast@sater6 2022-10-03T05:49:29.808737 v1.01 application/json {"min_interval": 30}
[2022-10-03 05:49:29,810 DEBUG    trollmoves.heartbeat_monitor] Resetting heartbeat alarm timer to 900.0 sec
[2022-10-03 05:49:57,078 DEBUG    trollmoves.client] Receiving (SUB) pytroll://heartbeat/move_it_server beat ubuntu@fengyun-a 2022-10-03T05:49:57.076502 v1.01 application/json {"min_interval": 30}
[2022-10-03 05:49:57,078 DEBUG    trollmoves.heartbeat_monitor] Resetting heartbeat alarm timer to 900.0 sec
[2022-10-03 05:49:57,654 DEBUG    trollmoves.heartbeat_monitor] Missing heartbeat alarm!
[2022-10-03 05:49:57,820 DEBUG    trollmoves.client] Receiving (SUB) pytroll://heartbeat/move_it_server beat eumetcast@sater5 2022-10-03T05:49:57.818825 v1.01 application/json {"min_interval": 30}
[2022-10-03 05:49:57,821 DEBUG    trollmoves.heartbeat_monitor] Resetting heartbeat alarm timer to 900.0 sec
[2022-10-03 05:49:59,743 DEBUG    posttroll.publisher] Publish heartbeat (min_interval is 30.0 sec)
[2022-10-03 05:49:59,744 DEBUG    posttroll.publisher] Publish heartbeat (min_interval is 30.0 sec)
[2022-10-03 05:49:59,745 DEBUG    posttroll.publisher] Publish heartbeat (min_interval is 30.0 sec)
[2022-10-03 05:49:59,839 DEBUG    trollmoves.client] Receiving (SUB) pytroll://heartbeat/move_it_server beat eumetcast@sater6 2022-10-03T05:49:59.839160 v1.01 application/json {"min_interval": 30}
[2022-10-03 05:49:59,840 DEBUG    trollmoves.heartbeat_monitor] Resetting heartbeat alarm timer to 900.0 sec
[2022-10-03 05:50:18,015 DEBUG    trollmoves.client] Receiving (REQ) pytroll://XLBANDANTENNA/FENGYUN3D/RAWDATA file meos@<move-it-server> 2022-10-03T05:50:17.936793 v1.01 application/json {"antenna": "XL-BAND", "sensor": ["mersi", "hiras", "vass"], "platform_name": "Fengyun-3D", "start_time": "2022-10-03T05:38:56", "end_time": "2022-10-03T05:47:57", "process_time": "2022-10-03T05:38:56", "uid": "rawdata_FY3D_MPT_ch1_20221003T053856Z_20221003T054757Z_20221003T053856Z_IwmdPIGV3_1_25305.data", "type": "1", "orbit_number": "25305", "uri": "/data/rawdata1/rawdata_FY3D_MPT_ch1_20221003T053856Z_20221003T054757Z_20221003T053856Z_IwmdPIGV3_1_25305.data", "request_address": "<move-it-server>:9109", "destination": "scp://fengyun-a/data/pytroll/fengyun3/rawdata"}
[2022-10-03 05:50:18,015 DEBUG    trollmoves.client] Server done sending file
[2022-10-03 05:50:18,015 DEBUG    trollmoves.client] Add rawdata_FY3D_MPT_ch1_20221003T053856Z_20221003T054757Z_20221003T053856Z_IwmdPIGV3_1_25305.data to file cache
[2022-10-03 05:50:18,016 DEBUG    trollmoves.client] Sending a public 'ack' of completed transfer: pytroll://XLBANDANTENNA/FENGYUN3D/RAWDATA ack ubuntu@fengyun-a 2022-10-03T05:50:18.016318 v1.01 application/json {"antenna": "XL-BAND", "sensor": ["mersi", "hiras", "vass"], "platform_name": "Fengyun-3D", "start_time": "2022-10-03T05:38:56", "end_time": "2022-10-03T05:47:57", "process_time": "2022-10-03T05:38:56", "uid": "rawdata_FY3D_MPT_ch1_20221003T053856Z_20221003T054757Z_20221003T053856Z_IwmdPIGV3_1_25305.data", "type": "1", "orbit_number": "25305", "uri": "/data/rawdata1/rawdata_FY3D_MPT_ch1_20221003T053856Z_20221003T054757Z_20221003T053856Z_IwmdPIGV3_1_25305.data", "request_address": "<move-it-server>:9109"}
[2022-10-03 05:50:18,023 DEBUG    trollmoves.client] publishing pytroll://XLBANDANTENNA/FENGYUN3D/RAWDATA file ubuntu@fengyun-a 2022-10-03T05:50:18.017102 v1.01 application/json {"antenna": "XL-BAND", "sensor": ["mersi", "hiras", "vass"], "platform_name": "fengyun 3d", "start_time": "2022-10-03T05:38:56", "end_time": "2022-10-03T05:47:57", "process_time": "2022-10-03T05:38:56", "uid": "rawdata_FY3D_MPT_ch1_20221003T053856Z_20221003T054757Z_20221003T053856Z_IwmdPIGV3_1_25305.data", "type": "1", "orbit_number": "25305", "uri": "ssh://fengyun-a/data/pytroll/fengyun3/rawdata/rawdata_FY3D_MPT_ch1_20221003T053856Z_20221003T054757Z_20221003T053856Z_IwmdPIGV3_1_25305.data", "origin": "<move-it-server>:9109"}
[2022-10-03 05:50:18,023 DEBUG    trollmoves.client] Remove uid 20f5d7994f0033bd9db078f826867045: []
[2022-10-03 05:50:18,024 WARNING  trollmoves.client] Missing a heartbeat, restarting the subscriber to dict_values(['tcp://<move-it-server>:9019']).
[2022-10-03 05:50:19,026 DEBUG    trollmoves.client] Exiting listener tcp://<move-it-server>:9019
[2022-10-03 05:50:19,026 DEBUG    trollmoves.client] Starting listener tcp://<move-it-server>:9019
[2022-10-03 05:50:19,026 INFO     trollmoves.client] Subscribing to tcp://<move-it-server>:9019 with topics ['/XLBANDANTENNA/FENGYUN3D/RAWDATA', '/heartbeat/move_it_server', '/heartbeat/move_it_RAWDATA-FENGYUN3D']
[2022-10-03 05:50:19,027 INFO     posttroll.subscriber] Subscriber adding address tcp://<move-it-server>:9019 with topics ['pytroll://XLBANDANTENNA/FENGYUN3D/RAWDATA', 'pytroll://heartbeat/move_it_server', 'pytroll://heartbeat/move_it_RAWDATA-FENGYUN3D']

Environment Info:

  • OS: Linux
  • Trollmoves Version: 0.12.0

Additional context
Add any other context about the problem here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions