Skip to content

[k8s]: Correct role assignment and termination #770

Open
wanyunSu wants to merge 4 commits intodevelopfrom
wanyunSu/k8s-tree
Open

[k8s]: Correct role assignment and termination #770
wanyunSu wants to merge 4 commits intodevelopfrom
wanyunSu/k8s-tree

Conversation

@wanyunSu
Copy link
Contributor

@wanyunSu wanyunSu commented Feb 11, 2026 •
edited
Loading

Description

Address issue #768 .

Correctly assign role labels to the pod.
Performs an ordered shutdown of matched processes by their role labels: unknown → application → segment-controller → root-controller → infrastructure-applications. Segment-controllers are further split by tree-id depth, shutting down the deepest first

Type of change

  • Documentation (non-breaking change that adds or improves the documentation)
  • New feature (non-breaking change which adds functionality)
  • Optimization (non-breaking, back-end change that speeds up the code)
  • Bug fix (non-breaking change which fixes an issue)
  • Breaking change (whatever its nature)

Key checklist

  • All tests pass (eg. python -m pytest)
  • Pre-commit hooks run successfully (eg. pre-commit run --all-files)

Further checks

  • Code is commented, particularly in hard-to-understand areas
  • Tests added or an issue has been opened to tackle that in the future.
    (Indicate issue here: # (issue))

@wanyunSu wanyunSu self-assigned this Feb 11, 2026
@wanyunSu wanyunSu mentioned this pull request Feb 11, 2026
Open
@MRiganSUSX MRiganSUSX self-requested a review February 24, 2026 18:09
This was referenced Mar 12, 2026
Merged
Open
Base automatically changed from wanyunSu/k8s-refactor to develop March 15, 2026 17:19
@PawelPlesniak
Copy link
Collaborator

PawelPlesniak commented Mar 15, 2026

Initial tests demonstrate that this PR does not achieve the desired functionality. I have run the nestedConfig session using SSH:

((dbt) ) pplesnia@np04-srv-029 ~/nightlyDev/260315/pythoncode/drunc (wanyunSu/k8s-tree) $ drunc-unified-shell ssh-standalone config/tests/nestedConfig.data.xml test-config pawel
[2026/03/15 18:11:34 UTC] INFO       shell.py:180                             drunc.unified_shell                                Setting up to use the process manager with configuration ssh-standalone and configuration id "test-config" from oksconflibs:config/tests/nestedConfig.data.xml
[2026/03/15 18:11:34 UTC] INFO       shell.py:202                             drunc.unified_shell                                Starting process manager
[2026/03/15 18:11:35 UTC] INFO       process_manager.py:109                   drunc.process_manager                              process_manager communicating through address 10.73.136.71:40737
[2026/03/15 18:11:35 UTC] INFO       shell.py:539                             drunc.unified_shell                                unified_shell ready with process_manager and controller commands
drunc-unified-shell > boot
[2026/03/15 18:11:36 UTC] INFO       process_manager_driver.py:104            drunc.process_manager_driver                       Booting session pawel
[2026/03/15 18:11:36 UTC] INFO       process_manager_driver.py:481            drunc.process_manager_driver                       Configuration did not require modifications.
[2026/03/15 18:11:36 UTC] INFO       ssh_process_manager.py:368               drunc.process_manager.SSH_SHELL_process_manager    Booted 'local-connection-server' from session 'pawel' with UUID be7bd2d5-89c1-489e-9a54-1e132503b65f
[2026/03/15 18:11:37 UTC] INFO       ssh_process_manager.py:368               drunc.process_manager.SSH_SHELL_process_manager    Booted 'top-segment-controller' from session 'pawel' with UUID 27f4d25c-4738-4816-a51c-45b6372ecc11
[2026/03/15 18:11:37 UTC] INFO       ssh_process_manager.py:368               drunc.process_manager.SSH_SHELL_process_manager    Booted 'nested-segment-controller' from session 'pawel' with UUID b28f37de-e5e3-4953-8a3b-2bdccf4c1e0d
[2026/03/15 18:11:37 UTC] INFO       ssh_process_manager.py:368               drunc.process_manager.SSH_SHELL_process_manager    Booted 'bottom-segment-1-controller' from session 'pawel' with UUID b57cc572-61c4-47d9-b8c5-e34d4062802b
[2026/03/15 18:11:37 UTC] INFO       ssh_process_manager.py:368               drunc.process_manager.SSH_SHELL_process_manager    Booted 'bottom-segment-1-application' from session 'pawel' with UUID 9ac1bab6-f1c4-43dd-9ace-583a6fee67a3
[2026/03/15 18:11:38 UTC] INFO       ssh_process_manager.py:368               drunc.process_manager.SSH_SHELL_process_manager    Booted 'bottom-segment-2-controller' from session 'pawel' with UUID e6347575-3b0c-449f-9e21-f0cc57d280a2
[2026/03/15 18:11:38 UTC] INFO       ssh_process_manager.py:368               drunc.process_manager.SSH_SHELL_process_manager    Booted 'bottom-segment-2-application' from session 'pawel' with UUID 8272a6d8-4984-4e73-bb37-e2d949b9c0ff
[2026/03/15 18:11:38 UTC] INFO       ssh_process_manager.py:368               drunc.process_manager.SSH_SHELL_process_manager    Booted 'nested-segment-application' from session 'pawel' with UUID 0b3409bf-d482-461c-ba57-a40526e8ce98
  Looking for top-segment-controller on the connectivity service... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 0:00:00
⠋ Trying to talk to the root controller... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -:--:-- 0:00:00
                                                        pawel status                                                        
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name                               ┃ Info ┃ State   ┃ Substate ┃ In error ┃ Included ┃ Endpoint                          ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ top-segment-controller             │      │ initial │ initial  │ No       │ Yes      │ grpc://np04-srv-029.cern.ch:30006 │
│   nested-segment-controller        │      │ initial │ initial  │ No       │ Yes      │ grpc://np04-srv-029.cern.ch:39941 │
│     bottom-segment-1-controller    │      │ initial │ initial  │ No       │ Yes      │ grpc://np04-srv-029.cern.ch:34353 │
│       bottom-segment-1-application │      │ initial │ idle     │ No       │ Yes      │ rest://np04-srv-029.cern.ch:49795 │
│     bottom-segment-2-controller    │      │ initial │ initial  │ No       │ Yes      │ grpc://np04-srv-029.cern.ch:39845 │
│       bottom-segment-2-application │      │ initial │ idle     │ No       │ Yes      │ rest://np04-srv-029.cern.ch:56107 │
│     nested-segment-application     │      │ initial │ idle     │ No       │ Yes      │ rest://np04-srv-029.cern.ch:40599 │
└────────────────────────────────────┴──────┴─────────┴──────────┴──────────┴──────────┴───────────────────────────────────┘
Waiting on tree initialisation... ╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   2% 0:01:09
[2026/03/15 18:11:41 UTC] INFO       commands.py:94                           drunc.unified_shell.boot                           Booted successfully
drunc-unified-shell > terminate
[2026/03/15 18:11:42 UTC] INFO       ssh_process_manager.py:203               drunc.process_manager.SSH_SHELL_process_manager    Terminating
[2026/03/15 18:11:42 UTC] INFO       ssh_process_manager.py:206               drunc.process_manager.SSH_SHELL_process_manager    Killing all the known processes before exiting
[2026/03/15 18:11:42 UTC] INFO       ssh_process_lifetime_manager_shell.py:56 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Terminating role 'unknown' from provided UUIDs ---
[2026/03/15 18:11:42 UTC] INFO       ssh_process_lifetime_manager_shell.py:50 drunc.drunc.processes.ssh_process_lifetime_manager Killing 2 process(es) with role 'unknown' from 8 candidates
[2026/03/15 18:11:53 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 9ac1bab6-f1c4-43dd-9ace-583a6fee67a3 (PID 3266397) did not terminate after SIGQUIT signal.
[2026/03/15 18:11:53 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 8272a6d8-4984-4e73-bb37-e2d949b9c0ff (PID 3266732) did not terminate after SIGQUIT signal.
[2026/03/15 18:11:53 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'bottom-segment-2-application' (session: 'pawel', user: 'pplesnia') process exited with exit code 137
[2026/03/15 18:11:53 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'bottom-segment-1-application' (session: 'pawel', user: 'pplesnia') process exited with exit code 137
[2026/03/15 18:11:54 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 8272a6d8-4984-4e73-bb37-e2d949b9c0ff (PID 3266732) terminated forcibly following SIGKILL signal.
[2026/03/15 18:11:54 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 9ac1bab6-f1c4-43dd-9ace-583a6fee67a3 (PID 3266397) terminated forcibly following SIGKILL signal.
[2026/03/15 18:11:54 UTC] INFO       ssh_process_lifetime_manager_shell.py:57 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Role 'unknown' complete ---
[2026/03/15 18:11:54 UTC] INFO       ssh_process_lifetime_manager_shell.py:56 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Terminating role 'application' from provided UUIDs ---
[2026/03/15 18:11:54 UTC] INFO       ssh_process_lifetime_manager_shell.py:50 drunc.drunc.processes.ssh_process_lifetime_manager Killing 3 process(es) with role 'application' from 8 candidates
[2026/03/15 18:11:55 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'bottom-segment-2-controller' (session: 'pawel', user: 'pplesnia') process exited with exit code 0
[2026/03/15 18:11:55 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process e6347575-3b0c-449f-9e21-f0cc57d280a2 (PID 3266578) terminated gracefully following SIGQUIT signal.
[2026/03/15 18:11:57 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process b57cc572-61c4-47d9-b8c5-e34d4062802b (PID 3266326) terminated gracefully following SIGQUIT signal.
[2026/03/15 18:11:57 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'bottom-segment-1-controller' (session: 'pawel', user: 'pplesnia') process exited with exit code 0
[2026/03/15 18:12:05 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 0b3409bf-d482-461c-ba57-a40526e8ce98 (PID 3266867) did not terminate after SIGQUIT signal.
[2026/03/15 18:12:05 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'nested-segment-application' (session: 'pawel', user: 'pplesnia') process exited with exit code 137
[2026/03/15 18:12:06 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 0b3409bf-d482-461c-ba57-a40526e8ce98 (PID 3266867) terminated forcibly following SIGKILL signal.
[2026/03/15 18:12:06 UTC] INFO       ssh_process_lifetime_manager_shell.py:57 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Role 'application' complete ---
[2026/03/15 18:12:06 UTC] INFO       ssh_process_lifetime_manager_shell.py:56 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Terminating role 'segment-controller' from provided UUIDs ---
[2026/03/15 18:12:06 UTC] INFO       ssh_process_lifetime_manager_shell.py:50 drunc.drunc.processes.ssh_process_lifetime_manager Killing 1 process(es) with role 'segment-controller' from 8 candidates
[2026/03/15 18:12:09 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'nested-segment-controller' (session: 'pawel', user: 'pplesnia') process exited with exit code 0
[2026/03/15 18:12:09 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process b28f37de-e5e3-4953-8a3b-2bdccf4c1e0d (PID 3266157) terminated gracefully following SIGQUIT signal.
[2026/03/15 18:12:10 UTC] INFO       ssh_process_lifetime_manager_shell.py:57 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Role 'segment-controller' complete ---
[2026/03/15 18:12:10 UTC] INFO       ssh_process_lifetime_manager_shell.py:56 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Terminating role 'root-controller' from provided UUIDs ---
[2026/03/15 18:12:10 UTC] INFO       ssh_process_lifetime_manager_shell.py:50 drunc.drunc.processes.ssh_process_lifetime_manager Killing 1 process(es) with role 'root-controller' from 8 candidates
[2026/03/15 18:12:11 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process 27f4d25c-4738-4816-a51c-45b6372ecc11 (PID 3266056) terminated gracefully following SIGQUIT signal.
[2026/03/15 18:12:11 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'top-segment-controller' (session: 'pawel', user: 'pplesnia') process exited with exit code 0
[2026/03/15 18:12:11 UTC] INFO       ssh_process_lifetime_manager_shell.py:57 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Role 'root-controller' complete ---
[2026/03/15 18:12:11 UTC] INFO       ssh_process_lifetime_manager_shell.py:56 drunc.drunc.processes.ssh_process_lifetime_manager --- Shutdown stage: Terminating role 'infrastructure-applications' from provided UUIDs ---
[2026/03/15 18:12:11 UTC] INFO       ssh_process_lifetime_manager_shell.py:58 drunc.drunc.processes.ssh_process_lifetime_manager Fallback: Killing 1 process(es) without role metadata asynchronously
[2026/03/15 18:12:12 UTC] INFO       ssh_process_lifetime_manager_shell.py:10 drunc.drunc.processes.ssh_process_lifetime_manager Remote process be7bd2d5-89c1-489e-9a54-1e132503b65f (PID 3265778) terminated gracefully following SIGQUIT signal.
[2026/03/15 18:12:12 UTC] INFO       ssh_process_manager.py:305               drunc.process_manager.SSH_SHELL_process_manager    Process 'local-connection-server' (session: 'pawel', user: 'pplesnia') process exited with exit code 0
                                                        Terminated process                                                        
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┓
┃ session ┃ friendly name                      ┃ user     ┃ host      ┃ uuid                                 ┃ alive ┃ exit-code ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━┩
│ pawel   │ local-connection-server            │ pplesnia │ localhost │ be7bd2d5-89c1-489e-9a54-1e132503b65f │ False │ 0         │
│ pawel   │ top-segment-controller             │ pplesnia │ localhost │ 27f4d25c-4738-4816-a51c-45b6372ecc11 │ False │ 0         │
│ pawel   │   nested-segment-controller        │ pplesnia │ localhost │ b28f37de-e5e3-4953-8a3b-2bdccf4c1e0d │ False │ 0         │
│ pawel   │     bottom-segment-1-controller    │ pplesnia │ localhost │ b57cc572-61c4-47d9-b8c5-e34d4062802b │ False │ 0         │
│ pawel   │     nested-segment-application     │ pplesnia │ localhost │ 0b3409bf-d482-461c-ba57-a40526e8ce98 │ False │ 137       │
│ pawel   │       bottom-segment-1-application │ pplesnia │ localhost │ 9ac1bab6-f1c4-43dd-9ace-583a6fee67a3 │ False │ 137       │
│ pawel   │     bottom-segment-2-controller    │ pplesnia │ localhost │ e6347575-3b0c-449f-9e21-f0cc57d280a2 │ False │ 0         │
│ pawel   │       bottom-segment-2-application │ pplesnia │ localhost │ 8272a6d8-4984-4e73-bb37-e2d949b9c0ff │ False │ 137       │
└─────────┴────────────────────────────────────┴──────────┴───────────┴──────────────────────────────────────┴───────┴───────────┘

And with k8s (skipping many of the logs that do not contribute towards the output)

((dbt) ) pplesnia@np04-srv-029 ~/nightlyDev/260315/pythoncode/drunc (wanyunSu/k8s-tree) $ drunc-unified-shell k8s config/tests/nestedConfig.data.xml test-config pawel
[2026/03/15 18:15:01 UTC] INFO       shell.py:180                             drunc.unified_shell                                Setting up to use the process manager with configuration k8s and configuration id "test-config" from oksconflibs:config/tests/nestedConfig.data.xml
[2026/03/15 18:15:02 UTC] INFO       shell.py:202                             drunc.unified_shell                                Starting process manager
[2026/03/15 18:15:02 UTC] INFO       k8s_process_manager.py:70                drunc.process_manager.k8s-process-manager          K8sPodWatcherThread started
[2026/03/15 18:15:02 UTC] INFO       k8s_process_manager.py:271               drunc.process_manager.k8s-process-manager          No active namespace created by drunc
[2026/03/15 18:15:02 UTC] INFO       process_manager.py:109                   drunc.process_manager                              process_manager communicating through address 10.73.136.71:45345
[2026/03/15 18:15:02 UTC] INFO       shell.py:539                             drunc.unified_shell                                unified_shell ready with process_manager and controller commands
drunc-unified-shell > boot
[2026/03/15 18:15:03 UTC] INFO       process_manager_driver.py:104            drunc.process_manager_driver                       Booting session pawel
[2026/03/15 18:15:03 UTC] INFO       process_manager_driver.py:481            drunc.process_manager_driver                       Configuration did not require modifications.
[2026/03/15 18:15:04 UTC] INFO       k8s_process_manager.py:1048              drunc.process_manager.k8s-process-manager          Assigning labels for 'local-connection-server': role=infrastructure-applications, tree-id=1
[2026/03/15 18:15:04 UTC] INFO       k8s_process_manager.py:605               drunc.process_manager.k8s-process-manager          Creating "pawel" namespace.
[2026/03/15 18:15:04 UTC] INFO       k8s_process_manager.py:623               drunc.process_manager.k8s-process-manager          Namespace 'pawel' is Active and ready.
[2026/03/15 18:15:04 UTC] INFO       k8s_process_manager.py:438               drunc.process_manager.k8s-process-manager          Added label "creator.drunc.daq:K8sProcessManager" to namespace "pawel"
[2026/03/15 18:15:04 UTC] INFO       k8s_process_manager.py:1760              drunc.process_manager.k8s-process-manager          Extracted gunicorn port 30005 from environment variable 'CONNECTION_PORT'.
[2026/03/15 18:15:04 UTC] INFO       k8s_process_manager.py:948               drunc.process_manager.k8s-process-manager          Adding 'log-mount' for directory: '/nfs/home/pplesnia/nightlyDev/260315/pythoncode/drunc'
[2026/03/15 18:15:04 UTC] INFO       k8s_process_manager.py:1643              drunc.process_manager.k8s-process-manager          Disabling hostNetwork for 'local-connection-server' to avoid port conflicts with NodePort service
[2026/03/15 18:15:04 UTC] INFO       k8s_process_manager.py:1300              drunc.process_manager.k8s-process-manager          Resolved localhost to 'np04-srv-029' for node selection
[2026/03/15 18:15:04 UTC] INFO       k8s_process_manager.py:570               drunc.process_manager.k8s-process-manager          Host 'np04-srv-029' verified and available
[2026/03/15 18:15:04 UTC] INFO       k8s_process_manager.py:1307              drunc.process_manager.k8s-process-manager          Pod 'local-connection-server' will be scheduled on node 'np04-srv-029' (from boot request)
[2026/03/15 18:15:04 UTC] INFO       k8s_process_manager.py:1507              drunc.process_manager.k8s-process-manager          Creating pod "pawel.local-connection-server"
[2026/03/15 18:15:04 UTC] INFO       k8s_process_manager.py:810               drunc.process_manager.k8s-process-manager          Created NodePort service "pawel.local-connection-server" on port 30005 (NodePort: 30005 for external access)
[2026/03/15 18:15:04 UTC] INFO       k8s_process_manager.py:427               drunc.process_manager.k8s-process-manager          Added label "uuid.drunc.daq:d46e44f5-67b2-41c6-ac13-d94f816a2b04" to pod "pawel.local-connection-server"
[2026/03/15 18:15:04 UTC] INFO       k8s_process_manager.py:2173              drunc.process_manager.k8s-process-manager          "pawel.local-connection-server":d46e44f5-67b2-41c6-ac13-d94f816a2b04 boot request sent.
[2026/03/15 18:15:04 UTC] INFO       k8s_process_manager.py:2077              drunc.process_manager.k8s-process-manager          Waiting for LCS 'local-connection-server' to be fully ready...
[2026/03/15 18:15:04 UTC] INFO       k8s_process_manager.py:1937              drunc.process_manager.k8s-process-manager          Stage 1: Waiting for 'local-connection-server' pod to be Running and Ready...
[2026/03/15 18:15:06 UTC] INFO       k8s_process_manager.py:1957              drunc.process_manager.k8s-process-manager          Stage 1: Pod 'local-connection-server' is API Ready on node np04-srv-029.
[2026/03/15 18:15:06 UTC] INFO       k8s_process_manager.py:1989              drunc.process_manager.k8s-process-manager          Stage 2: Waiting for NodePort http://np04-srv-029:30005 to be reachable...
[2026/03/15 18:15:06 UTC] INFO       k8s_process_manager.py:1995              drunc.process_manager.k8s-process-manager          Stage 2: NodePort http://np04-srv-029:30005 is now active.
[2026/03/15 18:15:06 UTC] INFO       k8s_process_manager.py:2099              drunc.process_manager.k8s-process-manager          Connection server 'local-connection-server' is fully ready.
[2026/03/15 18:15:06 UTC] INFO       k8s_process_manager.py:1048              drunc.process_manager.k8s-process-manager          Assigning labels for 'top-segment-controller': role=root-controller, tree-id=0
[2026/03/15 18:15:06 UTC] INFO       k8s_process_manager.py:948               drunc.process_manager.k8s-process-manager          Adding 'log-mount' for directory: '/nfs/home/pplesnia/nightlyDev/260315/pythoncode/drunc'
[2026/03/15 18:15:06 UTC] INFO       k8s_process_manager.py:1790              drunc.process_manager.k8s-process-manager          Extracted drunc-controller gRPC port 30006 from command.
[2026/03/15 18:15:06 UTC] INFO       k8s_process_manager.py:1643              drunc.process_manager.k8s-process-manager          Disabling hostNetwork for 'top-segment-controller' to avoid port conflicts with NodePort service
[2026/03/15 18:15:06 UTC] INFO       k8s_process_manager.py:1300              drunc.process_manager.k8s-process-manager          Resolved localhost to 'np04-srv-029' for node selection
[2026/03/15 18:15:06 UTC] INFO       k8s_process_manager.py:1307              drunc.process_manager.k8s-process-manager          Pod 'top-segment-controller' will be scheduled on node 'np04-srv-029' (from boot request)
[2026/03/15 18:15:06 UTC] INFO       k8s_process_manager.py:1350              drunc.process_manager.k8s-process-manager          Pod 'top-segment-controller' will resolve localhost to connection server IP 10.97.167.82
[2026/03/15 18:15:06 UTC] INFO       k8s_process_manager.py:1507              drunc.process_manager.k8s-process-manager          Creating pod "pawel.top-segment-controller"
[2026/03/15 18:15:06 UTC] INFO       k8s_process_manager.py:1567              drunc.process_manager.k8s-process-manager          'top-segment-controller' is the root controller, checking for NodePort service.
[2026/03/15 18:15:06 UTC] INFO       k8s_process_manager.py:1790              drunc.process_manager.k8s-process-manager          Extracted drunc-controller gRPC port 30006 from command.
[2026/03/15 18:15:06 UTC] INFO       k8s_process_manager.py:1574              drunc.process_manager.k8s-process-manager          Extracted port 30006 for 'top-segment-controller' NodePort.
[2026/03/15 18:15:06 UTC] INFO       k8s_process_manager.py:810               drunc.process_manager.k8s-process-manager          Created NodePort service "pawel.top-segment-controller" on port 30006 (NodePort: 30006 for external access)
[2026/03/15 18:15:06 UTC] INFO       k8s_process_manager.py:427               drunc.process_manager.k8s-process-manager          Added label "uuid.drunc.daq:9fe995d5-ddd9-46b1-a9aa-4329b42697c3" to pod "pawel.top-segment-controller"
[2026/03/15 18:15:06 UTC] INFO       k8s_process_manager.py:2173              drunc.process_manager.k8s-process-manager          "pawel.top-segment-controller":9fe995d5-ddd9-46b1-a9aa-4329b42697c3 boot request sent.
[2026/03/15 18:15:06 UTC] INFO       k8s_process_manager.py:2120              drunc.process_manager.k8s-process-manager          Waiting for controller 'top-segment-controller' (NodePort) to become ready...
[2026/03/15 18:15:06 UTC] INFO       k8s_process_manager.py:1790              drunc.process_manager.k8s-process-manager          Extracted drunc-controller gRPC port 30006 from command.
[2026/03/15 18:15:06 UTC] INFO       k8s_process_manager.py:1937              drunc.process_manager.k8s-process-manager          Stage 1: Waiting for 'top-segment-controller' pod to be Running and Ready...
[2026/03/15 18:15:08 UTC] INFO       k8s_process_manager.py:1957              drunc.process_manager.k8s-process-manager          Stage 1: Pod 'top-segment-controller' is API Ready on node np04-srv-029.
[2026/03/15 18:15:08 UTC] INFO       k8s_process_manager.py:2026              drunc.process_manager.k8s-process-manager          Stage 2: Waiting for NodePort np04-srv-029:30006 to be reachable...
[2026/03/15 18:15:08 UTC] INFO       k8s_process_manager.py:2038              drunc.process_manager.k8s-process-manager          Stage 2: NodePort np04-srv-029:30006 is active (TCP connect success).
[2026/03/15 18:15:08 UTC] INFO       k8s_process_manager.py:2140              drunc.process_manager.k8s-process-manager          Drunc controller 'top-segment-controller' is fully ready.
[2026/03/15 18:15:08 UTC] INFO       k8s_process_manager.py:1048              drunc.process_manager.k8s-process-manager          Assigning labels for 'nested-segment-controller': role=segment-controller, tree-id=0.0
[2026/03/15 18:15:08 UTC] INFO       k8s_process_manager.py:948               drunc.process_manager.k8s-process-manager          Adding 'log-mount' for directory: '/nfs/home/pplesnia/nightlyDev/260315/pythoncode/drunc'
[2026/03/15 18:15:08 UTC] INFO       k8s_process_manager.py:1300              drunc.process_manager.k8s-process-manager          Resolved localhost to 'np04-srv-029' for node selection
[2026/03/15 18:15:08 UTC] INFO       k8s_process_manager.py:1307              drunc.process_manager.k8s-process-manager          Pod 'nested-segment-controller' will be scheduled on node 'np04-srv-029' (from boot request)
[2026/03/15 18:15:08 UTC] INFO       k8s_process_manager.py:1350              drunc.process_manager.k8s-process-manager          Pod 'nested-segment-controller' will resolve localhost to connection server IP 10.97.167.82
[2026/03/15 18:15:08 UTC] INFO       k8s_process_manager.py:1507              drunc.process_manager.k8s-process-manager          Creating pod "pawel.nested-segment-controller"
[2026/03/15 18:15:08 UTC] INFO       k8s_process_manager.py:751               drunc.process_manager.k8s-process-manager          Created headless service "pawel.nested-segment-controller"
[2026/03/15 18:15:09 UTC] INFO       k8s_process_manager.py:427               drunc.process_manager.k8s-process-manager          Added label "uuid.drunc.daq:d0b2d692-849a-4029-ae01-a1cf0e4d5c33" to pod "pawel.nested-segment-controller"
[2026/03/15 18:15:09 UTC] INFO       k8s_process_manager.py:2173              drunc.process_manager.k8s-process-manager          "pawel.nested-segment-controller":d0b2d692-849a-4029-ae01-a1cf0e4d5c33 boot request sent.
[2026/03/15 18:15:09 UTC] INFO       k8s_process_manager.py:1048              drunc.process_manager.k8s-process-manager          Assigning labels for 'bottom-segment-1-controller': role=application, tree-id=0.0.0
[2026/03/15 18:15:09 UTC] INFO       k8s_process_manager.py:948               drunc.process_manager.k8s-process-manager          Adding 'log-mount' for directory: '/nfs/home/pplesnia/nightlyDev/260315/pythoncode/drunc'
[2026/03/15 18:15:09 UTC] INFO       k8s_process_manager.py:1300              drunc.process_manager.k8s-process-manager          Resolved localhost to 'np04-srv-029' for node selection
[2026/03/15 18:15:09 UTC] INFO       k8s_process_manager.py:1307              drunc.process_manager.k8s-process-manager          Pod 'bottom-segment-1-controller' will be scheduled on node 'np04-srv-029' (from boot request)
[2026/03/15 18:15:09 UTC] INFO       k8s_process_manager.py:1350              drunc.process_manager.k8s-process-manager          Pod 'bottom-segment-1-controller' will resolve localhost to connection server IP 10.97.167.82
[2026/03/15 18:15:09 UTC] INFO       k8s_process_manager.py:1507              drunc.process_manager.k8s-process-manager          Creating pod "pawel.bottom-segment-1-controller"
[2026/03/15 18:15:09 UTC] INFO       k8s_process_manager.py:751               drunc.process_manager.k8s-process-manager          Created headless service "pawel.bottom-segment-1-controller"
[2026/03/15 18:15:09 UTC] INFO       k8s_process_manager.py:427               drunc.process_manager.k8s-process-manager          Added label "uuid.drunc.daq:a007de18-b398-4de3-bbb4-63e1fe381581" to pod "pawel.bottom-segment-1-controller"
[2026/03/15 18:15:09 UTC] INFO       k8s_process_manager.py:2173              drunc.process_manager.k8s-process-manager          "pawel.bottom-segment-1-controller":a007de18-b398-4de3-bbb4-63e1fe381581 boot request sent.
[2026/03/15 18:15:09 UTC] INFO       k8s_process_manager.py:1048              drunc.process_manager.k8s-process-manager          Assigning labels for 'bottom-segment-1-application': role=unknown, tree-id=0.0.0.0
[2026/03/15 18:15:09 UTC] INFO       k8s_process_manager.py:948               drunc.process_manager.k8s-process-manager          Adding 'log-mount' for directory: '/nfs/home/pplesnia/nightlyDev/260315/pythoncode/drunc'
[2026/03/15 18:15:09 UTC] INFO       k8s_process_manager.py:1300              drunc.process_manager.k8s-process-manager          Resolved localhost to 'np04-srv-029' for node selection
[2026/03/15 18:15:09 UTC] INFO       k8s_process_manager.py:1307              drunc.process_manager.k8s-process-manager          Pod 'bottom-segment-1-application' will be scheduled on node 'np04-srv-029' (from boot request)
[2026/03/15 18:15:10 UTC] INFO       k8s_process_manager.py:1350              drunc.process_manager.k8s-process-manager          Pod 'bottom-segment-1-application' will resolve localhost to connection server IP 10.97.167.82
[2026/03/15 18:15:10 UTC] INFO       k8s_process_manager.py:1507              drunc.process_manager.k8s-process-manager          Creating pod "pawel.bottom-segment-1-application"
[2026/03/15 18:15:10 UTC] INFO       k8s_process_manager.py:751               drunc.process_manager.k8s-process-manager          Created headless service "pawel.bottom-segment-1-application"
[2026/03/15 18:15:10 UTC] INFO       k8s_process_manager.py:427               drunc.process_manager.k8s-process-manager          Added label "uuid.drunc.daq:421fb89c-e6ba-4fdc-9ebf-88b03154158a" to pod "pawel.bottom-segment-1-application"
[2026/03/15 18:15:10 UTC] INFO       k8s_process_manager.py:2173              drunc.process_manager.k8s-process-manager          "pawel.bottom-segment-1-application":421fb89c-e6ba-4fdc-9ebf-88b03154158a boot request sent.
[2026/03/15 18:15:10 UTC] INFO       k8s_process_manager.py:1048              drunc.process_manager.k8s-process-manager          Assigning labels for 'bottom-segment-2-controller': role=application, tree-id=0.0.1
[2026/03/15 18:15:10 UTC] INFO       k8s_process_manager.py:948               drunc.process_manager.k8s-process-manager          Adding 'log-mount' for directory: '/nfs/home/pplesnia/nightlyDev/260315/pythoncode/drunc'
[2026/03/15 18:15:10 UTC] INFO       k8s_process_manager.py:1300              drunc.process_manager.k8s-process-manager          Resolved localhost to 'np04-srv-029' for node selection
[2026/03/15 18:15:10 UTC] INFO       k8s_process_manager.py:1307              drunc.process_manager.k8s-process-manager          Pod 'bottom-segment-2-controller' will be scheduled on node 'np04-srv-029' (from boot request)
[2026/03/15 18:15:10 UTC] INFO       k8s_process_manager.py:1350              drunc.process_manager.k8s-process-manager          Pod 'bottom-segment-2-controller' will resolve localhost to connection server IP 10.97.167.82
[2026/03/15 18:15:13 UTC] INFO       k8s_process_manager.py:1507              drunc.process_manager.k8s-process-manager          Creating pod "pawel.bottom-segment-2-controller"
[2026/03/15 18:15:14 UTC] INFO       k8s_process_manager.py:751               drunc.process_manager.k8s-process-manager          Created headless service "pawel.bottom-segment-2-controller"
[2026/03/15 18:15:15 UTC] INFO       k8s_process_manager.py:427               drunc.process_manager.k8s-process-manager          Added label "uuid.drunc.daq:a067f90b-e216-453a-8b0e-2472acb582a6" to pod "pawel.bottom-segment-2-controller"
[2026/03/15 18:15:15 UTC] CRITICAL   process_manager.py:246                   drunc.process_manager.k8s-process-manager          Process bottom-segment-1-controller has died with a return code 0
[2026/03/15 18:15:15 UTC] INFO       k8s_process_manager.py:2173              drunc.process_manager.k8s-process-manager          "pawel.bottom-segment-2-controller":a067f90b-e216-453a-8b0e-2472acb582a6 boot request sent.
[2026/03/15 18:15:15 UTC] CRITICAL   process_manager.py:246                   drunc.process_manager.k8s-process-manager          Process bottom-segment-1-application has died with a return code 0
[2026/03/15 18:15:15 UTC] CRITICAL   process_manager.py:246                   drunc.process_manager.k8s-process-manager          Process bottom-segment-2-controller has died with a return code 0
[2026/03/15 18:15:15 UTC] INFO       k8s_process_manager.py:1048              drunc.process_manager.k8s-process-manager          Assigning labels for 'bottom-segment-2-application': role=unknown, tree-id=0.0.1.0
[2026/03/15 18:15:15 UTC] INFO       k8s_process_manager.py:948               drunc.process_manager.k8s-process-manager          Adding 'log-mount' for directory: '/nfs/home/pplesnia/nightlyDev/260315/pythoncode/drunc'
[2026/03/15 18:15:15 UTC] INFO       k8s_process_manager.py:1300              drunc.process_manager.k8s-process-manager          Resolved localhost to 'np04-srv-029' for node selection
[2026/03/15 18:15:15 UTC] INFO       k8s_process_manager.py:1307              drunc.process_manager.k8s-process-manager          Pod 'bottom-segment-2-application' will be scheduled on node 'np04-srv-029' (from boot request)
[2026/03/15 18:15:15 UTC] INFO       k8s_process_manager.py:1350              drunc.process_manager.k8s-process-manager          Pod 'bottom-segment-2-application' will resolve localhost to connection server IP 10.97.167.82
[2026/03/15 18:15:15 UTC] INFO       k8s_process_manager.py:1507              drunc.process_manager.k8s-process-manager          Creating pod "pawel.bottom-segment-2-application"
[2026/03/15 18:15:16 UTC] INFO       k8s_process_manager.py:751               drunc.process_manager.k8s-process-manager          Created headless service "pawel.bottom-segment-2-application"
[2026/03/15 18:15:18 UTC] INFO       k8s_process_manager.py:427               drunc.process_manager.k8s-process-manager          Added label "uuid.drunc.daq:94565cdc-47a1-4244-bf23-1e4e140c91bb" to pod "pawel.bottom-segment-2-application"
[2026/03/15 18:15:18 UTC] INFO       k8s_process_manager.py:2173              drunc.process_manager.k8s-process-manager          "pawel.bottom-segment-2-application":94565cdc-47a1-4244-bf23-1e4e140c91bb boot request sent.
[2026/03/15 18:15:18 UTC] INFO       k8s_process_manager.py:1048              drunc.process_manager.k8s-process-manager          Assigning labels for 'nested-segment-application': role=application, tree-id=0.0.0
[2026/03/15 18:15:18 UTC] INFO       k8s_process_manager.py:948               drunc.process_manager.k8s-process-manager          Adding 'log-mount' for directory: '/nfs/home/pplesnia/nightlyDev/260315/pythoncode/drunc'
[2026/03/15 18:15:18 UTC] INFO       k8s_process_manager.py:1300              drunc.process_manager.k8s-process-manager          Resolved localhost to 'np04-srv-029' for node selection
[2026/03/15 18:15:18 UTC] INFO       k8s_process_manager.py:1307              drunc.process_manager.k8s-process-manager          Pod 'nested-segment-application' will be scheduled on node 'np04-srv-029' (from boot request)
[2026/03/15 18:15:18 UTC] INFO       k8s_process_manager.py:1350              drunc.process_manager.k8s-process-manager          Pod 'nested-segment-application' will resolve localhost to connection server IP 10.97.167.82
[2026/03/15 18:15:20 UTC] INFO       k8s_process_manager.py:1507              drunc.process_manager.k8s-process-manager          Creating pod "pawel.nested-segment-application"
[2026/03/15 18:15:21 UTC] INFO       k8s_process_manager.py:751               drunc.process_manager.k8s-process-manager          Created headless service "pawel.nested-segment-application"
[2026/03/15 18:15:22 UTC] INFO       k8s_process_manager.py:427               drunc.process_manager.k8s-process-manager          Added label "uuid.drunc.daq:002ba72d-e993-4664-beba-1d2677efe9eb" to pod "pawel.nested-segment-application"
[2026/03/15 18:15:22 UTC] INFO       k8s_process_manager.py:2173              drunc.process_manager.k8s-process-manager          "pawel.nested-segment-application":002ba72d-e993-4664-beba-1d2677efe9eb boot request sent.
  Looking for top-segment-controller on the connectivity service... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 0:00:00
⠋ Trying to talk to the root controller... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -:--:-- 0:00:00
                                                      pawel status                                                       
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name                            ┃ Info ┃ State   ┃ Substate ┃ In error ┃ Included ┃ Endpoint                          ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ top-segment-controller          │      │ initial │ initial  │ No       │ Yes      │ grpc://10.244.12.156:30006        │
│   nested-segment-controller     │      │ initial │ initial  │ No       │ Yes      │ grpc://np04-srv-029.cern.ch:35459 │
│     bottom-segment-1-controller │      │ initial │ initial  │ No       │ Yes      │ grpc://np04-srv-029.cern.ch:45633 │
│     bottom-segment-2-controller │      │ initial │ initial  │ No       │ Yes      │ grpc://np04-srv-029.cern.ch:45481 │
└─────────────────────────────────┴──────┴─────────┴──────────┴──────────┴──────────┴───────────────────────────────────┘
Waiting on tree initialisation... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━  80% 0:00:14
[2026/03/15 18:16:20 UTC] INFO       commands.py:94                           drunc.unified_shell.boot                           Booted successfully
drunc-unified-shell > ps
                                                          Processes running                                                          
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┓
┃ session ┃ friendly name                      ┃ user     ┃ host         ┃ uuid                                 ┃ alive ┃ exit-code ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━┩
│ pawel   │ local-connection-server            │ pplesnia │ np04-srv-029 │ d46e44f5-67b2-41c6-ac13-d94f816a2b04 │ True  │ 0         │
│ pawel   │ top-segment-controller             │ pplesnia │ np04-srv-029 │ 9fe995d5-ddd9-46b1-a9aa-4329b42697c3 │ True  │ 0         │
│ pawel   │   nested-segment-controller        │ pplesnia │ np04-srv-029 │ d0b2d692-849a-4029-ae01-a1cf0e4d5c33 │ True  │ 0         │
│ pawel   │     bottom-segment-1-controller    │ pplesnia │ np04-srv-029 │ a007de18-b398-4de3-bbb4-63e1fe381581 │ True  │ 0         │
│ pawel   │     nested-segment-application     │ pplesnia │ np04-srv-029 │ 002ba72d-e993-4664-beba-1d2677efe9eb │ True  │ 0         │
│ pawel   │       bottom-segment-1-application │ pplesnia │ np04-srv-029 │ 421fb89c-e6ba-4fdc-9ebf-88b03154158a │ True  │ 0         │
│ pawel   │     bottom-segment-2-controller    │ pplesnia │ np04-srv-029 │ a067f90b-e216-453a-8b0e-2472acb582a6 │ True  │ 0         │
│ pawel   │       bottom-segment-2-application │ pplesnia │ np04-srv-029 │ 94565cdc-47a1-4244-bf23-1e4e140c91bb │ True  │ 0         │
└─────────┴────────────────────────────────────┴──────────┴──────────────┴──────────────────────────────────────┴───────┴───────────┘
drunc-unified-shell > terminate
[2026/03/15 18:16:33 UTC] INFO       k8s_process_manager.py:2536              drunc.process_manager.k8s-process-manager          Terminating all known K8s processes.
[2026/03/15 18:16:33 UTC] INFO       k8s_process_manager.py:2414              drunc.process_manager.k8s-process-manager          Starting staged termination for 8 pod(s)...
[2026/03/15 18:16:33 UTC] INFO       k8s_process_manager.py:2485              drunc.process_manager.k8s-process-manager          --- Termination Step: Shutting down role 'unknown' (2 pod(s)) ---
[2026/03/15 18:16:33 UTC] INFO       k8s_process_manager.py:2427              drunc.process_manager.k8s-process-manager          Gracefully terminating 2 pod(s)...
[2026/03/15 18:16:33 UTC] INFO       k8s_process_manager.py:2436              drunc.process_manager.k8s-process-manager          Killing pod "pawel/bottom-segment-1-application" (UUID 421fb89c-e6ba-4fdc-9ebf-88b03154158a)
[2026/03/15 18:16:33 UTC] INFO       k8s_process_manager.py:2436              drunc.process_manager.k8s-process-manager          Killing pod "pawel/bottom-segment-2-application" (UUID 94565cdc-47a1-4244-bf23-1e4e140c91bb)
[2026/03/15 18:17:33 UTC] ERROR      process_manager_driver.py:831            drunc.process_manager_driver                       [DEADLINE_EXCEEDED] No message
Traceback (most recent call last):
  File "/nfs/home/pplesnia/nightlyDev/260315/.venv/lib/python3.12/site-packages/drunc/process_manager/process_manager_driver.py", line 827, in terminate
    response = self.stub.terminate(request, timeout=timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nfs/home/pplesnia/nightlyDev/260315/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1181, in __call__
    return _end_unary_response_blocking(state, call, False, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nfs/home/pplesnia/nightlyDev/260315/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1009, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.DEADLINE_EXCEEDED
        details = "Deadline Exceeded"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_status:4, grpc_message:"Deadline Exceeded"}"
>

The above exception was the direct cause of the following exception:

drunc.utils.grpc_utils.ServerTimeout: Deadline Exceeded
drunc-unified-shell > [2026/03/15 18:18:33 UTC] WARNING    k8s_process_manager.py:2449              drunc.process_manager.k8s-process-manager          Timeout. Remaining: {'94565cdc-47a1-4244-bf23-1e4e140c91bb', '421fb89c-e6ba-4fdc-9ebf-88b03154158a'}
[2026/03/15 18:18:33 UTC] INFO       k8s_process_manager.py:2489              drunc.process_manager.k8s-process-manager          --- Termination Step: Role 'unknown' complete ---
[2026/03/15 18:18:33 UTC] INFO       k8s_process_manager.py:2485              drunc.process_manager.k8s-process-manager          --- Termination Step: Shutting down role 'application' (3 pod(s)) ---
[2026/03/15 18:18:33 UTC] INFO       k8s_process_manager.py:2427              drunc.process_manager.k8s-process-manager          Gracefully terminating 3 pod(s)...
[2026/03/15 18:18:33 UTC] INFO       k8s_process_manager.py:2436              drunc.process_manager.k8s-process-manager          Killing pod "pawel/bottom-segment-1-controller" (UUID a007de18-b398-4de3-bbb4-63e1fe381581)
[2026/03/15 18:18:33 UTC] INFO       k8s_process_manager.py:2436              drunc.process_manager.k8s-process-manager          Killing pod "pawel/bottom-segment-2-controller" (UUID a067f90b-e216-453a-8b0e-2472acb582a6)
[2026/03/15 18:18:34 UTC] INFO       k8s_process_manager.py:2436              drunc.process_manager.k8s-process-manager          Killing pod "pawel/nested-segment-application" (UUID 002ba72d-e993-4664-beba-1d2677efe9eb)
[2026/03/15 18:18:34 UTC] INFO       k8s_process_manager.py:349               drunc.process_manager.k8s-process-manager          Pod 'bottom-segment-1-application' (session: 'pawel', user: 'pplesnia', uuid: 421fb89c-e6ba-4fdc-9ebf-88b03154158a) terminated with exit code 137. Reason: Error
[2026/03/15 18:18:34 UTC] INFO       k8s_process_manager.py:349               drunc.process_manager.k8s-process-manager          Pod 'bottom-segment-2-application' (session: 'pawel', user: 'pplesnia', uuid: 94565cdc-47a1-4244-bf23-1e4e140c91bb) terminated with exit code 137. Reason: Error
[2026/03/15 18:18:34 UTC] INFO       k8s_process_manager.py:349               drunc.process_manager.k8s-process-manager          Pod 'bottom-segment-1-controller' (session: 'pawel', user: 'pplesnia', uuid: a007de18-b398-4de3-bbb4-63e1fe381581) terminated with exit code 0. Reason: GracefulShutdown
[2026/03/15 18:18:37 UTC] INFO       k8s_process_manager.py:349               drunc.process_manager.k8s-process-manager          Pod 'bottom-segment-2-controller' (session: 'pawel', user: 'pplesnia', uuid: a067f90b-e216-453a-8b0e-2472acb582a6) terminated with exit code 0. Reason: GracefulShutdown
[2026/03/15 18:20:34 UTC] WARNING    k8s_process_manager.py:2449              drunc.process_manager.k8s-process-manager          Timeout. Remaining: {'002ba72d-e993-4664-beba-1d2677efe9eb'}
[2026/03/15 18:20:34 UTC] INFO       k8s_process_manager.py:2489              drunc.process_manager.k8s-process-manager          --- Termination Step: Role 'application' complete ---
[2026/03/15 18:20:34 UTC] INFO       k8s_process_manager.py:2485              drunc.process_manager.k8s-process-manager          --- Termination Step: Shutting down role 'segment-controller' (1 pod(s)) ---
[2026/03/15 18:20:34 UTC] INFO       k8s_process_manager.py:2427              drunc.process_manager.k8s-process-manager          Gracefully terminating 1 pod(s)...
[2026/03/15 18:20:34 UTC] INFO       k8s_process_manager.py:2436              drunc.process_manager.k8s-process-manager          Killing pod "pawel/nested-segment-controller" (UUID d0b2d692-849a-4029-ae01-a1cf0e4d5c33)
[2026/03/15 18:20:34 UTC] INFO       k8s_process_manager.py:349               drunc.process_manager.k8s-process-manager          Pod 'nested-segment-application' (session: 'pawel', user: 'pplesnia', uuid: 002ba72d-e993-4664-beba-1d2677efe9eb) terminated with exit code 137. Reason: Error
[2026/03/15 18:20:36 UTC] INFO       k8s_process_manager.py:349               drunc.process_manager.k8s-process-manager          Pod 'nested-segment-controller' (session: 'pawel', user: 'pplesnia', uuid: d0b2d692-849a-4029-ae01-a1cf0e4d5c33) terminated with exit code 0. Reason: GracefulShutdown
[2026/03/15 18:20:36 UTC] INFO       k8s_process_manager.py:2489              drunc.process_manager.k8s-process-manager          --- Termination Step: Role 'segment-controller' complete ---
[2026/03/15 18:20:36 UTC] INFO       k8s_process_manager.py:2485              drunc.process_manager.k8s-process-manager          --- Termination Step: Shutting down role 'root-controller' (1 pod(s)) ---
[2026/03/15 18:20:36 UTC] INFO       k8s_process_manager.py:2427              drunc.process_manager.k8s-process-manager          Gracefully terminating 1 pod(s)...
[2026/03/15 18:20:36 UTC] INFO       k8s_process_manager.py:2436              drunc.process_manager.k8s-process-manager          Killing pod "pawel/top-segment-controller" (UUID 9fe995d5-ddd9-46b1-a9aa-4329b42697c3)
[2026/03/15 18:20:37 UTC] INFO       k8s_process_manager.py:349               drunc.process_manager.k8s-process-manager          Pod 'top-segment-controller' (session: 'pawel', user: 'pplesnia', uuid: 9fe995d5-ddd9-46b1-a9aa-4329b42697c3) terminated with exit code 0. Reason: GracefulShutdown
[2026/03/15 18:20:37 UTC] INFO       k8s_process_manager.py:2489              drunc.process_manager.k8s-process-manager          --- Termination Step: Role 'root-controller' complete ---
[2026/03/15 18:20:37 UTC] INFO       k8s_process_manager.py:2485              drunc.process_manager.k8s-process-manager          --- Termination Step: Shutting down role 'infrastructure-applications' (1 pod(s)) ---
[2026/03/15 18:20:37 UTC] INFO       k8s_process_manager.py:2427              drunc.process_manager.k8s-process-manager          Gracefully terminating 1 pod(s)...
[2026/03/15 18:20:37 UTC] INFO       k8s_process_manager.py:2436              drunc.process_manager.k8s-process-manager          Killing pod "pawel/local-connection-server" (UUID d46e44f5-67b2-41c6-ac13-d94f816a2b04)
[2026/03/15 18:20:38 UTC] INFO       k8s_process_manager.py:349               drunc.process_manager.k8s-process-manager          Pod 'local-connection-server' (session: 'pawel', user: 'pplesnia', uuid: d46e44f5-67b2-41c6-ac13-d94f816a2b04) terminated with exit code 0. Reason: GracefulShutdown
[2026/03/15 18:20:38 UTC] INFO       k8s_process_manager.py:2489              drunc.process_manager.k8s-process-manager          --- Termination Step: Role 'infrastructure-applications' complete ---
[2026/03/15 18:20:38 UTC] INFO       k8s_process_manager.py:2511              drunc.process_manager.k8s-process-manager          All tracked processes terminated. Cleaning up managed namespace...
[2026/03/15 18:20:38 UTC] INFO       k8s_process_manager.py:2516              drunc.process_manager.k8s-process-manager          Session "pawel" is empty, deleting namespace.

And in the bootom-segment-X-application logs, I see

ConnectionError: HTTPConnectionPool(host='10.73.136.71', port=53103): Max 
retries exceeded with url: / (Caused by 
NewConnectionError('<urllib3.connection.HTTPConnection object at 
0x7f1bb0520d40>: Failed to establish a new connection: [Errno 111] Connection 
refused'))

I am unsure if this is from the fake app or from this PR, but these apps are still getting marked as unknown apps, so I am not proceeding with integration tests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@MRiganSUSX MRiganSUSX Awaiting requested review from MRiganSUSX

At least 1 approving review is required to merge this pull request.

Assignees

@wanyunSu wanyunSu

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@wanyunSu @PawelPlesniak