Skip to content

Collect component stdout logs with a filelog receiver #1741

Description

Every ate component writes structured JSON logs to stdout: ateapi, atelet, atecontroller, atenet, ateom, podcertcontroller. Today kubectl logs is the only way to read them, so there's no way to search across components or across time.

Came out of the discussion on #1683.

What we need

  • An optional filelog receiver over /var/log/pods in the kind manifests.
  • That means a DaemonSet collector. The current one is a Deployment with no hostPath.
  • k8sattributes on the logs pipeline, so records get pod and node. Associate on k8s.pod.uid from the path.

Gotchas

  • Set severity parsing explicitly.
  • trace_id, span_id and trace_flags are already top level lowercase hex, so trace_parser maps them with no extra work.
  • ateom wraps actor output with the ate.* keys in a nested group, and the key name differs by environment: labels on kind, logging.googleapis.com/labels on GKE. Handle both or we silently lose every actor on one of them.
  • ateapi also sends its two lifecycle events over OTLP ((feat): Actor lifecycle events over otlp #1658), so those would arrive twice. Either leave ate-system out of the include globs, or drop records whose scope is github.com/agent-substrate/substrate/internal/actorevent.

cc Jeff Luo (@JeffLuoo)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions