Module execution failed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out - (orchestrator:keynesian_beauty_contest)

[enricorotundo]

SDK logs:

naptha-sdk-py3.12➜  naptha-sdk git:(main) time naptha run orchestrator:keynesian_beauty_contest -p "num_agents=10" --agent_nodes "localhost,localhost" 
2025-02-17 13:14:18,205 - naptha_sdk.client.node - INFO - Node URL: http://localhost:7001
2025-02-17 13:14:18,205 - naptha_sdk.inference - INFO - Node URL: http://localhost:7001
2025-02-17 13:14:18,205 - naptha_sdk.client.hub - INFO - Hub URL: ws://localhost:3001/rpc
Parsed parameters: {'num_agents': '10'}
2025-02-17 13:14:18,225 - naptha_sdk.client.hub - INFO - Attempting authentication for user: enricorotundo1
2025-02-17 13:14:18,254 - naptha_sdk.client.hub - INFO - Successfully authenticated user: enricorotundo1
...
pending orchestrator keynesian_beauty_contest
running orchestrator keynesian_beauty_contest
error orchestrator keynesian_beauty_contest
Error in _run_module_async: Module execution failed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
naptha run orchestrator:keynesian_beauty_contest -p "num_agents=10"    0.54s user 0.13s system 4% cpu 13.605 total

Node logs:

INFO:     127.0.0.1:56457 - "POST /orchestrator/run HTTP/1.1" 200 OK
2025-02-17 13:14:25 - node.server.http_server - INFO - Checking orchestrator run
2025-02-17 13:14:25 - node.server.http_server - INFO - Found orchestrator status: pending
2025-02-17 13:14:25 - node.server.http_server - INFO - Response: None
INFO:     127.0.0.1:56472 - "POST /orchestrator/check HTTP/1.1" 200 OK
2025-02-17 13:14:28 - node.server.http_server - INFO - Checking orchestrator run
2025-02-17 13:14:28 - node.server.http_server - INFO - Found orchestrator status: running
2025-02-17 13:14:28 - node.server.http_server - INFO - Response: None
INFO:     127.0.0.1:56522 - "POST /orchestrator/check HTTP/1.1" 200 OK
2025-02-17 13:14:31 - node.server.http_server - INFO - Checking orchestrator run
2025-02-17 13:14:31 - node.server.http_server - INFO - Found orchestrator status: error
2025-02-17 13:14:31 - node.server.http_server - INFO - Orchestrator run failed. Returning error: Error in _run_module_async: Module execution failed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
2025-02-17 13:14:31 - node.server.http_server - INFO - Response: Error in _run_module_async: Module execution failed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
INFO:     127.0.0.1:56528 - "POST /orchestrator/check HTTP/1.1" 200 OK

Celery logs:

[2025-02-17 16:38:06,153: WARNING/ForkPoolWorker-64] 2025-02-17 16:38:06,153 - naptha_sdk.client.node - ERROR - Error running agent: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
[2025-02-17 16:38:06,153: ERROR/ForkPoolWorker-64] Error running agent: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
[2025-02-17 16:38:06,155: WARNING/ForkPoolWorker-64] 2025-02-17 16:38:06,154 - naptha_sdk.client.node - ERROR - Full traceback: NoneType: None
[2025-02-17 16:38:06,154: ERROR/ForkPoolWorker-64] Full traceback: NoneType: None

[2025-02-17 16:38:06,156: ERROR/ForkPoolWorker-64] Module import paths: ['/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest', '/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/.venv/lib/python3.12/site-packages']
[2025-02-17 16:38:06,156: ERROR/ForkPoolWorker-64] Current working directory: /Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest
[2025-02-17 16:38:06,157: ERROR/ForkPoolWorker-64] Module directory contents: [PosixPath('/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/pyproject.toml'), PosixPath('/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/tests'), PosixPath('/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/keynesian_beauty_contest'), PosixPath('/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/README.md'), PosixPath('/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/.gitignore'), PosixPath('/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/.venv'), PosixPath('/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/.gitattributes'), PosixPath('/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/poetry.lock'), PosixPath('/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/.git')]
[2025-02-17 16:38:06,158: ERROR/ForkPoolWorker-64] Error running orchestrator: Module execution failed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
[2025-02-17 16:38:06,160: ERROR/ForkPoolWorker-64] Traceback: Traceback (most recent call last):
  File "/Users/enricorotundo/git/naptha/node/node/worker/package_worker.py", line 258, in load_and_run
    result = await run_func(module_run=module_run_dict)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/keynesian_beauty_contest/run.py", line 65, in run
    results = await beauty_contest.run_beauty_contest(module_run, *args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/keynesian_beauty_contest/run.py", line 49, in run_beauty_contest
    results = await asyncio.gather(*tasks)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/multiagent_chat/.venv/lib/python3.12/site-packages/naptha_sdk/modules/agent.py", line 17, in run
    agent_run = await node.run_module(module_type="agent", run_input=module_run_input)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/multiagent_chat/.venv/lib/python3.12/site-packages/naptha_sdk/client/node.py", line 98, in run_module
    return await self.run_module_ws(module_type, run_input)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/multiagent_chat/.venv/lib/python3.12/site-packages/naptha_sdk/client/node.py", line 123, in run_module_ws
    raise Exception(response['message'])
Exception: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/enricorotundo/git/naptha/node/node/worker/package_worker.py", line 329, in start_run
    response = await loader.load_and_run(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/worker/package_worker.py", line 268, in load_and_run
    raise RuntimeError(f"Module execution failed: {str(e)}") from e
RuntimeError: Module execution failed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more

[2025-02-17 16:38:06,160: ERROR/ForkPoolWorker-64] Error in _run_module_async: Module execution failed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
[2025-02-17 16:38:06,162: ERROR/ForkPoolWorker-64] Traceback: Traceback (most recent call last):
  File "/Users/enricorotundo/git/naptha/node/node/worker/package_worker.py", line 258, in load_and_run
    result = await run_func(module_run=module_run_dict)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/keynesian_beauty_contest/run.py", line 65, in run
    results = await beauty_contest.run_beauty_contest(module_run, *args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/keynesian_beauty_contest/keynesian_beauty_contest/run.py", line 49, in run_beauty_contest
    results = await asyncio.gather(*tasks)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/multiagent_chat/.venv/lib/python3.12/site-packages/naptha_sdk/modules/agent.py", line 17, in run
    agent_run = await node.run_module(module_type="agent", run_input=module_run_input)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/multiagent_chat/.venv/lib/python3.12/site-packages/naptha_sdk/client/node.py", line 98, in run_module
    return await self.run_module_ws(module_type, run_input)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/storage/hub/modules/multiagent_chat/.venv/lib/python3.12/site-packages/naptha_sdk/client/node.py", line 123, in run_module_ws
    raise Exception(response['message'])
Exception: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/enricorotundo/git/naptha/node/node/worker/package_worker.py", line 128, in _run_module_async
    await module_run_engine.start_run()
  File "/Users/enricorotundo/git/naptha/node/node/worker/package_worker.py", line 329, in start_run
    response = await loader.load_and_run(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/enricorotundo/git/naptha/node/node/worker/package_worker.py", line 268, in load_and_run
    raise RuntimeError(f"Module execution failed: {str(e)}") from e
RuntimeError: Module execution failed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more

[enricorotundo]

Got the error above with another orchestrator as well:

naptha run orchestrator:multiagent_chat -p "prompt='i would like to count up to ten, one number at a time. ill start. one.'" --agent_nodes "localhost,localhost" --kb_nodes "localhost"