Repository navigation
knative-operator goes to Error during remove event #91
Description
During application removal, knative operator enters error state with juju status:
knative-operator/0* error idle 10.1.208.139 hook failed: "remove"
and debug logs of:
2023-03-07 20:51:34 INFO juju-log The OpenTelemetry Collector may not be deployed yet.This may be temporary or due to a missing otel-collector relation.
2023-03-07 20:51:34 INFO juju.worker.uniter.operation runhook.go:146 ran "update-status" hook (via hook dispatching script: dispatch)
2023-03-07 20:52:01 INFO juju-log The OpenTelemetry Collector may not be deployed yet.This may be temporary or due to a missing otel-collector relation.
2023-03-07 20:52:02 INFO juju-log Rendering manifests
2023-03-07 20:52:05 INFO juju-log Reconcile completed successfully
2023-03-07 20:52:05 INFO juju.worker.uniter.operation runhook.go:146 ran "config-changed" hook (via hook dispatching script: dispatch)
2023-03-07 20:52:06 INFO juju-log The OpenTelemetry Collector may not be deployed yet.This may be temporary or due to a missing otel-collector relation.
2023-03-07 20:52:06 INFO juju-log Rendering manifests
2023-03-07 20:52:08 INFO juju-log Reconcile completed successfully
2023-03-07 20:52:09 INFO juju.worker.uniter.operation runhook.go:146 ran "config-changed" hook (via hook dispatching script: dispatch)
2023-03-07 20:53:13 INFO juju.worker.caasunitterminationworker worker.go:82 terminating due to SIGTERM
2023-03-07 20:53:13 WARNING juju.worker.uniter.operation leader.go:123 we should run a leader-deposed hook here, but we can't yet
2023-03-07 20:53:14 INFO juju-log The OpenTelemetry Collector may not be deployed yet.This may be temporary or due to a missing otel-collector relation.
2023-03-07 20:53:14 INFO juju.worker.uniter.operation runhook.go:146 ran "stop" hook (via hook dispatching script: dispatch)
^[[A^[[B2023-03-07 20:53:15 INFO juju-log The OpenTelemetry Collector may not be deployed yet.This may be temporary or due to a missing otel-collector relation.
2023-03-07 20:53:15 INFO juju-log Rendering manifests
2023-03-07 20:53:16 INFO juju-log Rendering manifests
2023-03-07 20:53:17 WARNING juju-log Log string greater than 131071. Splitting into multiple chunks:
2023-03-07 20:53:17 WARNING juju-log Failed to delete resources: ... TRUNCATED ... with: deployments.apps "otel-collector" not found
2023-03-07 20:53:17 ERROR juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/lightkube/core/generic_client.py", line 188, in raise_for_status
resp.raise_for_status()
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/httpx/_models.py", line 745, in raise_for_status
raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '404 Not Found' for url 'https://10.152.183.1/apis/apps/v1/namespaces/kubeflow/deployments/otel-collector'
For more information check: https://httpstatuses.com/404
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./src/charm.py", line 287, in <module>
main(KnativeOperatorCharm)
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/ops/main.py", line 438, in main
_emit_charm_event(charm, dispatcher.event_name)
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/ops/main.py", line 150, in _emit_charm_event
event_to_emit.emit(*args, **kwargs)
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/ops/framework.py", line 355, in emit
framework._emit(event) # noqa
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/ops/framework.py", line 856, in _emit
self._reemit(event_path)
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/ops/framework.py", line 931, in _reemit
custom_handler(event)
File "./src/charm.py", line 282, in _on_remove
raise e
File "./src/charm.py", line 277, in _on_remove
delete_many(
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/charmed_kubeflow_chisme/lightkube/batch/_many.py", line 97, in delete_many
client.delete(res=obj.__class__, name=obj.metadata.name, namespace=namespace)
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/lightkube/core/client.py", line 73, in delete
return self._client.request("delete", res=res, name=name, namespace=namespace)
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/lightkube/core/generic_client.py", line 245, in request
return self.handle_response(method, resp, br)
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/lightkube/core/generic_client.py", line 196, in handle_response
self.raise_for_status(resp)
File "/var/lib/juju/agents/unit-knative-operator-0/charm/venv/lightkube/core/generic_client.py", line 190, in raise_for_status
raise transform_exception(e)
lightkube.core.exceptions.ApiError: deployments.apps "otel-collector" not found
2023-03-07 20:53:17 ERROR juju.worker.uniter.operation runhook.go:140 hook "remove" (via hook dispatching script: dispatch) failed: exit status 1
Activity
dariofaccin commentedon Sep 2, 2026
This issue is still happening, although for different reason: