Add instrumentation / telemetry

[liamwh]

Hi there,

Thank you for making this! It's really a great project, well done, and well documented. I've enjoyed checking it out!

One thing I notice that Rusve is clearly missing to make it production ready, which is absolutely essential in a microservices architecture, is telemetry. For example, the observability to follow a single request across services with a trace is very important to understanding the behaviour of the app.

To achieve this, the code would need to be instrumented with an SDK, as well as having a tool to send the traces to, like Grafana Tempo.

If you're interested in adding this to the repo, then I have some resources for you:

There are docker-compose examples for Tempo here. Although I've recently set this up myself in an example repo which you might find more helpful here.

To instrument your Rust code, I can share this which might help get you started:

use opentelemetry_otlp::WithExportConfig;
use tracing_bunyan_formatter::{BunyanFormattingLayer, JsonStorageLayer};
use tracing_subscriber::prelude::__tracing_subscriber_SubscriberExt;

const OTEL_EXPORTER_OTLP_ENDPOINT_ENV_VAR: &str = "OTEL_EXPORTER_OTLP_ENDPOINT";
const OTEL_EXPORTER_OTLP_ENDPOINT_DEFAULT: &str = "http://localhost:4317";

const OBSERVABILITY_SERVICE_NAME_ENV_VAR: &str = "OBSERVABILITY_SERVICE_NAME";
const OBSERVABILITY_SERVICE_NAME_DEFAULT: &str = "rusve-xyz-service";

#[tracing::instrument]
pub async fn configure_observability() -> std::result::Result<(), crate::error::Error> {
    let otel_exporter_endpoint =
        dotenvy::var(OTEL_EXPORTER_OTLP_ENDPOINT_ENV_VAR).unwrap_or_else(|_| {
            tracing::warn!(
                "{} Env var not set, using default",
                OTEL_EXPORTER_OTLP_ENDPOINT_ENV_VAR
            );
            OTEL_EXPORTER_OTLP_ENDPOINT_DEFAULT.to_string()
        });

    let observability_service_name = dotenvy::var(OBSERVABILITY_SERVICE_NAME_ENV_VAR)
        .unwrap_or_else(|_| OBSERVABILITY_SERVICE_NAME_DEFAULT.to_string());

    let tracer = opentelemetry_otlp::new_pipeline()
        .tracing()
        .with_exporter(
            opentelemetry_otlp::new_exporter()
                .tonic()
                .with_endpoint(otel_exporter_endpoint),
        )
        .with_trace_config(opentelemetry::sdk::trace::config().with_resource(
            opentelemetry::sdk::Resource::new(vec![opentelemetry::KeyValue::new(
                "service.name",
                observability_service_name.clone(),
            )]),
        ))
        .install_batch(opentelemetry::runtime::Tokio)?;

    // Create a tracing layer with the configured tracer
    let telemetry_layer = tracing_opentelemetry::layer().with_tracer(tracer);

    let filter = tracing_subscriber::EnvFilter::from_default_env();

    cfg_if::cfg_if! {
    if #[cfg(feature="bunyan")] {
            // Create a new formatting layer to print bunyan formatted logs to stdout, pipe into bunyan to view
            let formatting_layer = BunyanFormattingLayer::new(observability_service_name, std::io::stdout);
            let subscriber = tracing_subscriber::Registry::default()
                .with(filter)
                .with(telemetry_layer)
                .with(JsonStorageLayer)
                .with(formatting_layer);
    } else {
            let subscriber = tracing_subscriber::Registry::default()
            .with_filter(filter),
            .with_writer(std::io::stdout)
            .with(telemetry_layer);
        }
    }

    // Use the tracing subscriber `Registry`, or any other subscriber
    // that impls `LookupSpan`

    Ok(tracing::subscriber::set_global_default(subscriber)?)
}

[mpiorowski]

Not gonna lie, i am not an expert in Rust, just started learning it a few months ago, still proud that the code use min uwrap / cloning. Love it.

So all the help like Yours are even more welcome, best way to learn how things should be done.

Will look into this, thanks again @liamwh ;)

Fell free to open pr if You want / have time.

[liamwh]

No problem @mpiorowski . 👍 Well done with the Rust code!

You will need to implement instrumentation not only in Rust but also in the Go and SvelteKit code, because you'll want each request to have it's own trace ID. This will let you follow the request across services when using a trace observability tool like Jaeger or Tempo. Each service should adopt a consistent HTTP header for propagating the trace ID, popular options include "X-Request-ID" or "X-Correlation-ID".

I would love to make a PR but don't expect I will find the time, however hope to help you make rusve even better with this issue / guidance.