https://boristane.com/blog/observability-wide-events-101/

[utterances-bot]

Observability wide events 101 | Boris Tane

What are wide events, why and how should you implement them?

https://boristane.com/blog/observability-wide-events-101/

[ganapatib]

Thanks for the great article explaining the value of wide events. Do you have any pointers or resources for the commit markers? I couldn't quite visualise how that would look like. Each file is marked with the commit of last change?

[boristane]

Hey @ganapatib, thanks for your comment!

Markers are typically a feature of your observability tooling, separate from how applications are instrumented. The Baselime CLI for example supports creating markers, including from ci/cd pipelines.

A few other vendors provide very similar capabilities :)

[khoovirajsingh]

Wow! This is a fantastic article to learn about wide events with great visuals and examples, thanks for sharing. I personally learnt a lot for it!

[basardaldal]

When you say, "...you definitely should replace all your application metrics with wide events", do you mean "do not manage counters, gauges etc in your code, instead calculate those by running queries over your wide events data"? I assume it is like, for example, if I want to keep a metric like "requestsReceived", it is obvious I can get this by doing a count of the rows that represent wide events that are printed when a request is received.
Assuming this is the idea, I would like to ask if this is still valid if sampling is applied? There is a chance that sampled data will not always give the real metric value, even it is based a -pre-determined ratio - or am I wrong in that?

[boristane]

@basardaldal most large scale systems will use sampling, there are ways to ensure the computations you get from your wide events are not far off the "real" value, even with sampling. every major provider should cover this for you

[rdasan]

What about the Size of the wide event / log? If you have too many dependecies and calls in a request scope, would'nt that result in a really huge sized event log. some log ingestion Platorms have limits on log size

[arjunnayak]

@rdasan That's a problem indeed–especially because stream providers have limits on message sizes so your logs get dropped. 1MB is the limit for AWS kinesis and the default for kafka afaik.

To prevent the log from getting dropped, you'd want to implement a check/truncate in your logging library or collector (like fluent-bit). I'm curious how large scale orgs are preventing/raising this unbounded context issue early, before it hits production.

[LilianBoulard]

Your articles and Brandur's (Stripe) are great resources to learn about this topic, thanks!
Something that's been on my mind: how do you deal with data naturally structured as an array? For instance in the example above - which stores everything as key-value pairs - there's a single DB query and single cache operation, but in practice anything non-trivial will have multiple such ops. What are your insights regarding that? Storing lists in the wide event, creating multiple wide events, or something else?

[theogravity]

If you are looking for wide event support in JavaScript / TypeScript, we just added support in the LogLayer logging library with dedicated APIs for this use-case:

https://loglayer.dev/guides/wide-events.html

LogLayer plugs into various logging libraries like pino and winston, along with cloud services like datadog; if you use the wide events feature from LogLayer, those logging libraries / cloud services also gets support for them as well.