From fa22b073dc2870537ebff05b4f4bf83b19885b54 Mon Sep 17 00:00:00 2001 From: Gnani Rahul Nutakki Date: Sat, 18 Jul 2026 12:40:35 -0500 Subject: [PATCH] feat(observability): count bounded auth outcomes Expose exactly two preinitialized authentication outcome series at the local verifier boundary while retaining legacy refusal counting and refusal-only audit and log sinks. GSTACK-Checkpoint: 2026-07-18/e10-auth-outcome-counter#1 Signed-off-by: Gnani Rahul Nutakki --- README.md | 22 +++--- charts/sith-hub/README.md | 8 ++ docs/EPICS.md | 20 ++++- docs/SITH-NOTION.md | 2 +- internal/auditdelivery/process.go | 2 +- internal/auditdelivery/process_test.go | 14 ++++ internal/hubserver/auth.go | 7 +- internal/hubserver/auth_observability.go | 15 ++-- internal/hubserver/auth_observability_test.go | 38 +++++++++- internal/hubserver/console.go | 1 + internal/hubserver/console_test.go | 18 ++++- internal/observability/auth.go | 2 +- internal/observability/auth_test.go | 1 + internal/observability/metrics.go | 27 +++++-- internal/observability/metrics_test.go | 6 ++ .../2026-07-18-e10-auth-outcome-counter.md | 73 +++++++++++++++++++ 16 files changed, 219 insertions(+), 37 deletions(-) create mode 100644 sessions/2026-07-18-e10-auth-outcome-counter.md diff --git a/README.md b/README.md index 6164f1c..0d765c7 100644 --- a/README.md +++ b/README.md @@ -311,16 +311,18 @@ listener, Service, exporter, queue, persistence, remote telemetry, request metad data, or raw payload retention. The drop counter is scrapeable only when the same optional loopback metrics endpoint above is enabled. -The same already-sanitized middleware refusal also increments the preinitialized, unlabeled -`sith_auth_refusals_total` counter. Runtime fanout delivers independently to the process audit child -and metric observer; either observer's panic is isolated and cannot suppress the other or alter the -uniform HTTP 401 response. The counter spans the bearer API and browser-session console middleware -that already emit this one closed event. It contains no credential mode, failure reason, tenant, -workspace, actor, principal, token, IP, path, request, trace, or correlation label. It does not -cover successful authentication, OIDC provider exchange/callback failures, authorization denials, -or every future authentication mode. Without a success or attempt denominator it is not a ratio, -brute-force detector, alert threshold, SLO, error budget, page, or complete security-monitoring -control, and it adds no new scrape or storage path. +The same already-sanitized bearer and browser-session boundaries increment exactly two +preinitialized `sith_auth_attempts_total{outcome="accepted|refused"}` series. `accepted` means the +local verifier succeeded and is emitted before workspace authorization; `refused` covers the +existing uniform authentication rejection paths. Every refusal also increments the legacy +unlabeled `sith_auth_refusals_total` counter exactly once. Runtime fanout is panic-isolated per +destination: metrics consume both outcomes, while the process audit child and structured-log +adapter remain refusal-only. No credential mode, failure reason, tenant, workspace, actor, +principal, token, IP, path, method, request, trace, or correlation value becomes a label or record. +These counters do not cover OIDC provider exchange/callback failures, authorization denials, +handler outcomes, or every future authentication mode. They are raw substrate, not a configured +ratio, brute-force detector, alert threshold, SLO, error budget, page, or complete security control, +and they add no new scrape or storage path. Every referenced key, certificate, or CA file must be a read-only regular file from a deployment mount. The runtime obtains its Kubernetes identity only with in-cluster configuration; it has no diff --git a/charts/sith-hub/README.md b/charts/sith-hub/README.md index b5dd657..d163911 100644 --- a/charts/sith-hub/README.md +++ b/charts/sith-hub/README.md @@ -66,6 +66,14 @@ the fixed record without delaying the governed response and increments the unlab scrape that process-wide loss signal; the chart still renders no related Service port, ingress, sidecar, queue, exporter, or remote telemetry path. +The same optional endpoint exposes two fixed, preinitialized +`sith_auth_attempts_total{outcome="accepted|refused"}` series. A successful local verifier decision +counts as `accepted` before workspace authorization; every authentication rejection counts as +`refused` and also increments the legacy unlabeled `sith_auth_refusals_total` counter. The audit +child remains refusal-only. These counters contain no tenant, workspace, identity, credential, +request, network, error, trace, or authorization labels and do not add a Service, exporter, +persistence, remote write, alert, SLO, or cloud resource. + Validate supplied values before applying anything: ```bash diff --git a/docs/EPICS.md b/docs/EPICS.md index bd974dc..ba8d05d 100644 --- a/docs/EPICS.md +++ b/docs/EPICS.md @@ -2496,7 +2496,7 @@ and alerting · F10.5 crown-jewel hardening. ### F10.1 — Metrics **What it is.** Metrics about Sith's own health and behavior: control-plane liveness, federation -freshness, intent throughput, sanitized authentication-refusal counts, and future derived rates +freshness, intent throughput, bounded sanitized authentication-outcome counts, and future derived rates where trustworthy denominators exist, abstention rates, and PDP latency. **How it works.** @@ -2560,9 +2560,21 @@ Runtime fanout independently reaches the existing process audit observer and the observer panics cannot suppress a later destination or alter the uniform HTTP 401 response. The counter carries no reason, credential mode, tenant, workspace, actor, principal, token, IP, path, request, trace, or correlation label. It does not count successful authentication, OIDC provider -exchange/callback failures, authorization denials, or every future authentication mode. Without a -success or attempt denominator it is not a ratio, brute-force detector, alert threshold, SLO, -error budget, page, or complete security-monitoring control. +exchange/callback failures, authorization denials, or every future authentication mode. The legacy +counter alone is not a denominator and remains compatible with existing scrapes. + +**Implementation note (F10.1h).** Each completed local bearer-token or browser-session verifier +decision increments one of exactly two preinitialized +`sith_auth_attempts_total{outcome="accepted|refused"}` series. `accepted` is emitted immediately +after verifier success and before workspace authorization; a later forbidden authorization is +therefore not misclassified as failed authentication. Every `refused` outcome also increments the +legacy unlabeled refusal counter exactly once. Metrics consume both outcomes, while the process +audit observer and structured-log adapter remain refusal-only and accepted observations cannot +write a datagram, log, or delivery-drop count. The outcome label is closed and carries no +credential mode, reason, tenant, workspace, actor, principal, token, IP, path, method, request, +trace, correlation, authorization, or handler-result dimension. The counters exclude provider +exchange/callback failures and define no ratio, alert, brute-force detector, SLO, error budget, +page, listener, exporter, persistence, remote write, or cloud resource. ### F10.2 — Distributed tracing diff --git a/docs/SITH-NOTION.md b/docs/SITH-NOTION.md index 95e3856..f59df3a 100644 --- a/docs/SITH-NOTION.md +++ b/docs/SITH-NOTION.md @@ -2539,7 +2539,7 @@ and alerting · F10.5 crown-jewel hardening. ### F10.1 — Metrics **What it is.** Metrics about Sith's own health and behavior: control-plane liveness, federation -freshness, intent throughput, sanitized authentication-refusal counts, and future derived rates +freshness, intent throughput, bounded sanitized authentication-outcome counts, and future derived rates where trustworthy denominators exist, abstention rates, and PDP latency. **How it works.** diff --git a/internal/auditdelivery/process.go b/internal/auditdelivery/process.go index 2688b5d..a84f527 100644 --- a/internal/auditdelivery/process.go +++ b/internal/auditdelivery/process.go @@ -159,7 +159,7 @@ func (observer *ProcessObserver) ObserveAuth(event hubserver.AuthEvent) { } func encodeAuthRefusal(event hubserver.AuthEvent) []byte { - if event.Validate() != nil { + if event.Validate() != nil || event.Outcome != hubserver.AuthOutcomeRefused { return nil } return []byte{authRecordVersion, authRecordRefused} diff --git a/internal/auditdelivery/process_test.go b/internal/auditdelivery/process_test.go index 85cdbd1..97f410d 100644 --- a/internal/auditdelivery/process_test.go +++ b/internal/auditdelivery/process_test.go @@ -29,6 +29,7 @@ func TestProcessObserverDeliversOnlyFixedRecordAndReapsChild(t *testing.T) { defer writer.Close() observer := newTestProcessObserver(t, writer, nil) + observer.ObserveAuth(hubserver.AuthEvent{Outcome: hubserver.AuthOutcomeAccepted}) observer.ObserveAuth(hubserver.AuthEvent{Outcome: "token=secret"}) observer.ObserveAuth(hubserver.AuthEvent{Outcome: hubserver.AuthOutcomeRefused}) line := make(chan string, 1) @@ -54,6 +55,19 @@ func TestProcessObserverDeliversOnlyFixedRecordAndReapsChild(t *testing.T) { } } +func TestProcessObserverNeverDeliversOrDropsAcceptedAuthentication(t *testing.T) { + var drops atomic.Uint64 + observer := &ProcessObserver{drops: dropObserverFunc(func() { drops.Add(1) })} + observer.ObserveAuth(hubserver.AuthEvent{Outcome: hubserver.AuthOutcomeAccepted}) + if drops.Load() != 0 { + t.Fatalf("accepted authentication delivery drops = %d, want 0", drops.Load()) + } + observer.ObserveAuth(hubserver.AuthEvent{Outcome: hubserver.AuthOutcomeRefused}) + if drops.Load() != 1 { + t.Fatalf("refused authentication delivery drops = %d, want 1", drops.Load()) + } +} + func TestProcessObserverDropsFullDatagramBufferWithoutBlocking(t *testing.T) { parent, child := socketPair(t) defer child.Close() diff --git a/internal/hubserver/auth.go b/internal/hubserver/auth.go index 67d91e7..8fb5201 100644 --- a/internal/hubserver/auth.go +++ b/internal/hubserver/auth.go @@ -27,9 +27,9 @@ func Authenticate(verifier Verifier, next http.Handler) (http.Handler, error) { return AuthenticateWithObserver(verifier, nil, next) } -// AuthenticateWithObserver constructs authentication middleware with one passive refusal -// observer. The observer is never given request metadata, credentials, verifier errors, or caller -// correlation values, and cannot alter the uniform unauthorized response. +// AuthenticateWithObserver constructs authentication middleware with one passive outcome observer. +// The observer is never given request metadata, credentials, verifier errors, or caller correlation +// values, and cannot alter the uniform unauthorized response or successful handler path. func AuthenticateWithObserver(verifier Verifier, observer AuthObserver, next http.Handler) (http.Handler, error) { if verifier == nil { return nil, fmt.Errorf("construct authentication middleware: verifier is required") @@ -53,6 +53,7 @@ func AuthenticateWithObserver(verifier Verifier, observer AuthObserver, next htt refuseAuthentication(observer, response) return } + ObserveAuth(observer, AuthEvent{Outcome: AuthOutcomeAccepted}) ctx := context.WithValue(cloned.Context(), principalContextKey{}, principal) next.ServeHTTP(response, cloned.WithContext(ctx)) }), nil diff --git a/internal/hubserver/auth_observability.go b/internal/hubserver/auth_observability.go index ed415b7..c048e21 100644 --- a/internal/hubserver/auth_observability.go +++ b/internal/hubserver/auth_observability.go @@ -4,12 +4,17 @@ package hubserver import "fmt" -// AuthOutcome is the closed self-observability result of one pre-principal -// authentication attempt. It intentionally does not distinguish credential failure modes. +// AuthOutcome is the closed self-observability result of one completed authentication verifier +// decision. It intentionally does not distinguish credential modes or failure reasons. type AuthOutcome string -// AuthOutcomeRefused is emitted for every request the bearer-token middleware rejects. -const AuthOutcomeRefused AuthOutcome = "refused" +const ( + // AuthOutcomeAccepted is emitted after a bearer token or browser session verifies successfully, + // before any workspace authorization decision. + AuthOutcomeAccepted AuthOutcome = "accepted" + // AuthOutcomeRefused is emitted for every request the authentication boundary rejects. + AuthOutcomeRefused AuthOutcome = "refused" +) // AuthEvent is one passive, sanitized authentication observation. It deliberately has no // request, credential, verifier-error, principal, path, network, or correlation fields: none are @@ -20,7 +25,7 @@ type AuthEvent struct { // Validate rejects unsupported outcome values before an observer can emit them. func (event AuthEvent) Validate() error { - if event.Outcome != AuthOutcomeRefused { + if event.Outcome != AuthOutcomeAccepted && event.Outcome != AuthOutcomeRefused { return fmt.Errorf("authentication event outcome is unsupported") } return nil diff --git a/internal/hubserver/auth_observability_test.go b/internal/hubserver/auth_observability_test.go index 1c2f4cb..a9e21ce 100644 --- a/internal/hubserver/auth_observability_test.go +++ b/internal/hubserver/auth_observability_test.go @@ -62,7 +62,7 @@ func TestAuthenticateWithObserverRecordsOnlyUniformRefusals(t *testing.T) { } } -func TestAuthenticateWithObserverIsSilentAfterValidAuthentication(t *testing.T) { +func TestAuthenticateWithObserverRecordsAcceptedAfterValidAuthentication(t *testing.T) { now := time.Date(2026, 7, 14, 13, 0, 0, 0, time.UTC) publicKey, privateKey := hubTestKeyPair() verifier, err := hubauth.NewJWTVerifier(hubauth.JWTConfig{ @@ -82,11 +82,24 @@ func TestAuthenticateWithObserverIsSilentAfterValidAuthentication(t *testing.T) request.Header.Set("Authorization", "Bearer "+signHubTestToken(t, hubValidClaims(now), privateKey)) response := httptest.NewRecorder() handler.ServeHTTP(response, request) - if response.Code != http.StatusNoContent || len(events) != 0 { + if response.Code != http.StatusNoContent || len(events) != 1 || events[0] != (AuthEvent{Outcome: AuthOutcomeAccepted}) { t.Fatalf("status = %d events = %#v", response.Code, events) } } +func TestAuthEventValidationUsesOnlyClosedOutcomes(t *testing.T) { + for _, outcome := range []AuthOutcome{AuthOutcomeAccepted, AuthOutcomeRefused} { + if err := (AuthEvent{Outcome: outcome}).Validate(); err != nil { + t.Fatalf("Validate(%q) = %v", outcome, err) + } + } + for _, outcome := range []AuthOutcome{"", "token=secret", "forbidden"} { + if err := (AuthEvent{Outcome: outcome}).Validate(); err == nil { + t.Fatalf("Validate(%q) accepted an unsupported outcome", outcome) + } + } +} + func TestObserveAuthRejectsUnsafeEventsAndContainsObserverPanics(t *testing.T) { called := false ObserveAuth(AuthObserverFunc(func(AuthEvent) { called = true }), AuthEvent{Outcome: "token=secret"}) @@ -95,6 +108,7 @@ func TestObserveAuthRejectsUnsafeEventsAndContainsObserverPanics(t *testing.T) { } ObserveAuth(AuthObserverFunc(func(AuthEvent) { panic("observer fault") }), AuthEvent{Outcome: AuthOutcomeRefused}) + ObserveAuth(AuthObserverFunc(func(AuthEvent) { panic("observer fault") }), AuthEvent{Outcome: AuthOutcomeAccepted}) handler, err := AuthenticateWithObserver(authVerifierFunc(func(context.Context, string) (tenancy.Principal, error) { return tenancy.Principal{}, errors.New("invalid") @@ -109,6 +123,22 @@ func TestObserveAuthRejectsUnsafeEventsAndContainsObserverPanics(t *testing.T) { if response.Code != http.StatusUnauthorized || response.Body.String() != "{\"error\":\"unauthorized\"}\n" { t.Fatalf("status = %d, body = %q", response.Code, response.Body.String()) } + + successful, err := AuthenticateWithObserver(authVerifierFunc(func(context.Context, string) (tenancy.Principal, error) { + return tenancy.Principal{}, nil + }), AuthObserverFunc(func(AuthEvent) { panic("observer fault") }), http.HandlerFunc(func(response http.ResponseWriter, _ *http.Request) { + response.WriteHeader(http.StatusNoContent) + })) + if err != nil { + t.Fatal(err) + } + successRequest := httptest.NewRequest(http.MethodGet, "https://hub.sith.test/api", nil) + successRequest.Header.Set("Authorization", "Bearer valid") + successResponse := httptest.NewRecorder() + successful.ServeHTTP(successResponse, successRequest) + if successResponse.Code != http.StatusNoContent { + t.Fatalf("successful authentication with panicking observer status = %d", successResponse.Code) + } } func TestAuthObserverFanoutIsolatesEachRequiredDestination(t *testing.T) { @@ -126,7 +156,9 @@ func TestAuthObserverFanoutIsolatesEachRequiredDestination(t *testing.T) { observers[1] = AuthObserverFunc(func(AuthEvent) { t.Fatal("fanout retained caller-owned slice") }) ObserveAuth(fanout, AuthEvent{Outcome: AuthOutcomeRefused}) - if len(deliveries) != 2 || deliveries[0] != "first" || deliveries[1] != "second" { + ObserveAuth(fanout, AuthEvent{Outcome: AuthOutcomeAccepted}) + if len(deliveries) != 4 || deliveries[0] != "first" || deliveries[1] != "second" || + deliveries[2] != "first" || deliveries[3] != "second" { t.Fatalf("fanout deliveries = %#v, want independently isolated order", deliveries) } diff --git a/internal/hubserver/console.go b/internal/hubserver/console.go index f2b3560..75f00dd 100644 --- a/internal/hubserver/console.go +++ b/internal/hubserver/console.go @@ -408,6 +408,7 @@ func (handler *ConsoleHandler) authorize( refuseAuthentication(handler.authObserver, response) return tenancy.Scope{}, "", false } + ObserveAuth(handler.authObserver, AuthEvent{Outcome: AuthOutcomeAccepted}) scope, err := principal.Scope(workspaceID) if err != nil { writeConsoleError(response, http.StatusForbidden, "forbidden") diff --git a/internal/hubserver/console_test.go b/internal/hubserver/console_test.go index f58bd7e..dce3659 100644 --- a/internal/hubserver/console_test.go +++ b/internal/hubserver/console_test.go @@ -174,7 +174,7 @@ func TestConsoleRoutesResolveWorkspaceThroughServeMux(t *testing.T) { func TestConsoleHandlerForwardsUniformAuthRefusalsToConfiguredObserver(t *testing.T) { now := time.Date(2026, 7, 18, 8, 0, 0, 0, time.UTC) - verifier, _ := fleetTestVerifier(t, now) + verifier, privateKey := fleetTestVerifier(t, now) var events []AuthEvent handler, err := NewConsoleHandler(ConsoleHandlerConfig{ Verifier: verifier, AuthObserver: AuthObserverFunc(func(event AuthEvent) { @@ -202,8 +202,20 @@ func TestConsoleHandlerForwardsUniformAuthRefusalsToConfiguredObserver(t *testin t.Fatalf("session %q status/body = %d/%q", session, response.Code, response.Body.String()) } } - if len(events) != 2 || events[0] != (AuthEvent{Outcome: AuthOutcomeRefused}) || events[1] != events[0] { - t.Fatalf("console authentication events = %#v, want two uniform refusals", events) + foreign := consoleTestRequest( + http.MethodGet, + "/v1/workspaces/workspace-b/console", + "workspace-b", + signHubTestToken(t, hubValidClaims(now), privateKey), + ) + foreignResponse := httptest.NewRecorder() + handler.ServePage(foreignResponse, foreign) + if foreignResponse.Code != http.StatusForbidden || foreignResponse.Body.String() != "{\"error\":\"forbidden\"}\n" { + t.Fatalf("foreign workspace status/body = %d/%q", foreignResponse.Code, foreignResponse.Body.String()) + } + if len(events) != 3 || events[0] != (AuthEvent{Outcome: AuthOutcomeRefused}) || events[1] != events[0] || + events[2] != (AuthEvent{Outcome: AuthOutcomeAccepted}) { + t.Fatalf("console authentication events = %#v, want two refusals then accepted before authorization", events) } } diff --git a/internal/observability/auth.go b/internal/observability/auth.go index 676b360..70ff2b6 100644 --- a/internal/observability/auth.go +++ b/internal/observability/auth.go @@ -26,7 +26,7 @@ type slogAuthObserver struct { } func (observer slogAuthObserver) ObserveAuth(event hubserver.AuthEvent) { - if observer.logger == nil || event.Validate() != nil { + if observer.logger == nil || event.Validate() != nil || event.Outcome != hubserver.AuthOutcomeRefused { return } observer.logger.Warn( diff --git a/internal/observability/auth_test.go b/internal/observability/auth_test.go index f0a0814..dae8a94 100644 --- a/internal/observability/auth_test.go +++ b/internal/observability/auth_test.go @@ -19,6 +19,7 @@ func TestSlogAuthObserverEmitsOnlyValidatedFixedFields(t *testing.T) { t.Fatal(err) } observer.ObserveAuth(hubserver.AuthEvent{Outcome: hubserver.AuthOutcomeRefused}) + observer.ObserveAuth(hubserver.AuthEvent{Outcome: hubserver.AuthOutcomeAccepted}) observer.ObserveAuth(hubserver.AuthEvent{Outcome: "token=secret"}) lines := strings.Split(strings.TrimSpace(output.String()), "\n") diff --git a/internal/observability/metrics.go b/internal/observability/metrics.go index 8e38bdf..6075055 100644 --- a/internal/observability/metrics.go +++ b/internal/observability/metrics.go @@ -47,6 +47,7 @@ type Metrics struct { fleetReadFreshness *prometheus.CounterVec readinessChecks *prometheus.CounterVec readinessDuration *prometheus.HistogramVec + authAttempts *prometheus.CounterVec authRefusals prometheus.Counter authDeliveryDrops prometheus.Counter } @@ -113,6 +114,10 @@ func New(config Config) (*Metrics, error) { Namespace: "sith", Subsystem: "hub", Name: "readiness_check_duration_seconds", Help: "Duration of completed Sith Hub database readiness checks by closed outcome.", }, []string{"outcome"}), + authAttempts: prometheus.NewCounterVec(prometheus.CounterOpts{ + Namespace: "sith", Subsystem: "auth", Name: "attempts_total", + Help: "Total completed Sith authentication verifier decisions by closed outcome.", + }, []string{"outcome"}), authRefusals: prometheus.NewCounter(prometheus.CounterOpts{ Namespace: "sith", Subsystem: "auth", Name: "refusals_total", Help: "Total authentication requests refused by Sith's sanitized middleware boundary.", @@ -131,7 +136,7 @@ func New(config Config) (*Metrics, error) { }) buildInfo.Set(1) - registered := make([]prometheus.Collector, 0, 15) + registered := make([]prometheus.Collector, 0, 16) for _, collector := range []struct { name string collector prometheus.Collector @@ -149,6 +154,7 @@ func New(config Config) (*Metrics, error) { {name: "fleet read freshness", collector: metrics.fleetReadFreshness}, {name: "Hub readiness checks", collector: metrics.readinessChecks}, {name: "Hub readiness duration", collector: metrics.readinessDuration}, + {name: "authentication attempts", collector: metrics.authAttempts}, {name: "authentication refusals", collector: metrics.authRefusals}, {name: "authentication-refusal delivery drops", collector: metrics.authDeliveryDrops}, } { @@ -190,6 +196,12 @@ func New(config Config) (*Metrics, error) { metrics.readinessChecks.WithLabelValues(string(outcome)) metrics.readinessDuration.WithLabelValues(string(outcome)) } + for _, outcome := range []hubserver.AuthOutcome{ + hubserver.AuthOutcomeAccepted, + hubserver.AuthOutcomeRefused, + } { + metrics.authAttempts.WithLabelValues(string(outcome)) + } return metrics, nil } @@ -260,14 +272,17 @@ func (metrics *Metrics) ObserveReadiness(outcome hubserver.ReadinessOutcome, dur metrics.readinessDuration.WithLabelValues(string(outcome)).Observe(normalizedDuration(duration)) } -// ObserveAuth records one already-sanitized middleware refusal. The counter is unlabeled because -// no principal, workspace, correlation, request, credential mode, or failure reason is trusted at -// this boundary. Invalid events are discarded rather than fabricating an observation. +// ObserveAuth records one already-sanitized verifier decision with a closed outcome. Refusals also +// increment the legacy unlabeled counter. No principal, workspace, correlation, request, credential +// mode, or failure reason is exposed. Invalid events are discarded rather than creating a series. func (metrics *Metrics) ObserveAuth(event hubserver.AuthEvent) { - if metrics == nil || metrics.authRefusals == nil || event.Validate() != nil { + if metrics == nil || metrics.authAttempts == nil || event.Validate() != nil { return } - metrics.authRefusals.Inc() + metrics.authAttempts.WithLabelValues(string(event.Outcome)).Inc() + if event.Outcome == hubserver.AuthOutcomeRefused && metrics.authRefusals != nil { + metrics.authRefusals.Inc() + } } // ObserveAuthRefusalDeliveryDrop records one bounded process-local delivery drop. It carries no diff --git a/internal/observability/metrics_test.go b/internal/observability/metrics_test.go index f39418f..97c7632 100644 --- a/internal/observability/metrics_test.go +++ b/internal/observability/metrics_test.go @@ -58,6 +58,7 @@ func TestMetricsExposeOnlyBoundedSelfObservability(t *testing.T) { metrics.ObserveReadiness(hubserver.ReadinessOutcomeReady, 10*time.Millisecond) metrics.ObserveReadiness(hubserver.ReadinessOutcomeUnavailable, -time.Second) metrics.ObserveReadiness(hubserver.ReadinessOutcome("database endpoint secret"), time.Second) + metrics.ObserveAuth(hubserver.AuthEvent{Outcome: hubserver.AuthOutcomeAccepted}) metrics.ObserveAuth(hubserver.AuthEvent{Outcome: hubserver.AuthOutcomeRefused}) metrics.ObserveAuth(hubserver.AuthEvent{Outcome: "token=secret"}) metrics.ObserveAuthRefusalDeliveryDrop() @@ -93,6 +94,8 @@ func TestMetricsExposeOnlyBoundedSelfObservability(t *testing.T) { `sith_hub_readiness_checks_total{outcome="unavailable"} 1`, `sith_hub_readiness_check_duration_seconds_count{outcome="ready"} 1`, `sith_hub_readiness_check_duration_seconds_count{outcome="unavailable"} 1`, + `sith_auth_attempts_total{outcome="accepted"} 1`, + `sith_auth_attempts_total{outcome="refused"} 1`, "sith_auth_refusals_total 1", "sith_auth_refusal_delivery_drops_total 1", `verb="fleet.read"`, @@ -149,6 +152,8 @@ func TestMetricsUseIndependentRegistriesAndNormalizeBuildLabels(t *testing.T) { `sith_hub_readiness_checks_total{outcome="unavailable"} 0`, `sith_hub_readiness_check_duration_seconds_count{outcome="ready"} 0`, `sith_hub_readiness_check_duration_seconds_count{outcome="unavailable"} 0`, + `sith_auth_attempts_total{outcome="accepted"} 0`, + `sith_auth_attempts_total{outcome="refused"} 0`, `sith_auth_refusals_total 0`, } { if !strings.Contains(body, preinitialized) { @@ -205,6 +210,7 @@ func assertSithMetricLabels(t *testing.T, metrics *Metrics) { "sith_federation_fleet_read_freshness_total": {"outcome": true}, "sith_hub_readiness_checks_total": {"outcome": true}, "sith_hub_readiness_check_duration_seconds": {"outcome": true}, + "sith_auth_attempts_total": {"outcome": true}, "sith_auth_refusals_total": {}, "sith_auth_refusal_delivery_drops_total": {}, } diff --git a/sessions/2026-07-18-e10-auth-outcome-counter.md b/sessions/2026-07-18-e10-auth-outcome-counter.md new file mode 100644 index 0000000..7f48a72 --- /dev/null +++ b/sessions/2026-07-18-e10-auth-outcome-counter.md @@ -0,0 +1,73 @@ +# E10 F10.1h bounded authentication outcomes + +**Builder:** Gnani Rahul · **Branch:** `gnanirahulnutakki/feat/e10-auth-outcome-counter` + +**Slice:** E10 F10.1h / [#272](https://github.com/ArdurAI/sith/issues/272) · **Status:** local gates clean; landing pending + +**Base:** `origin/dev` at `89cffa5cb6f4e51c8c5f4ef9410f323ee044f493` + +## [G] Goal + +Add the smallest trustworthy authentication-attempt denominator for Sith's existing local bearer +and browser-session verifier boundaries without expanding the audit or privacy surface. + +## [S] Scope and decision + +- `sith_auth_attempts_total{outcome="accepted|refused"}` has exactly two preinitialized series. +- `accepted` is emitted immediately after local verifier success, before workspace authorization or + the protected handler. A valid credential forbidden from a workspace is still accepted authn. +- `refused` is emitted exactly once on each existing uniform HTTP 401 path and also increments the + legacy unlabeled `sith_auth_refusals_total` counter exactly once. +- The process-supervised audit observer and slog adapter remain refusal-only. Accepted events cannot + log, write a datagram, increment a delivery-drop counter, or start child work. +- The event and metric contain no credential, reason, tenant, workspace, identity, token, IP, path, + method, request, error, trace, correlation, authorization, or handler-result dimension. + +## [A] Analysis and nonclaims + +- Authentication success and workspace authorization are separate decisions. Emitting accepted + before `Principal.Scope` preserves that boundary and avoids a misleading denominator. +- A single bounded `outcome` label follows Prometheus guidance to expose one logical counter family + whose known series are initialized at startup. +- This does not cover provider exchange/callback failures, authorization denials, handler outcomes, + or every future authentication mode. +- This slice publishes counters only. It does not define a ratio, threshold, brute-force detector, + alert, SLO, error budget, page, listener, Service, exporter, persistence, remote write, retention, + or cloud resource. +- Runtime cost is one fixed counter increment per completed verifier decision. Existing scraper and + time-series retention costs remain operator-owned. + +## [T] Verification plan + +- Unit tests: closed event outcomes, accepted/refused increments, zero-at-start series, invalid-event + silence, legacy refusal compatibility, and forbidden-label inspection. +- Boundary tests: bearer success, browser success before authorization, all existing refusal paths, + and observer panic isolation preserve governed HTTP behavior. +- Refusal-sink tests: accepted events produce no log, datagram, or delivery-drop increment. +- Gates: focused and race suites, full CI, forced-RLS/isolation fuzz, release/Helm/OCI/two-cluster + Kind validation, complete-diff CodeRabbit review, signed DCO/GSTACK commit, exact-head hosted CI + and CodeQL, empty review/security queues, merge, and exact post-merge `dev` proof. + +## Sources + +- [Prometheus instrumentation](https://prometheus.io/docs/practices/instrumentation/) +- [Prometheus metric naming](https://prometheus.io/docs/practices/naming/) + +## [C] Local verification checkpoint + +- Focused package tests and focused race tests pass for `internal/hubserver`, + `internal/observability`, `internal/auditdelivery`, and `internal/hubruntime`. +- `make ci` passes: formatting, golangci-lint with zero issues, vet, `govulncheck` with no + reachable vulnerabilities, repository-wide race/coverage, safety policies, eight portable alert + rules, latency guard, tagged binary e2e, and build. `internal/observability` coverage is 94.7%. +- `make e2e-isolation` passes PostgreSQL 18.4 forced-RLS suites at 76.2% `hubdb` coverage plus both + 50,000-execution cross-workspace fuzz campaigns. +- `make release-check` passes two reproducible Darwin/Linux amd64/arm64 builds, SPDX SBOMs, + checksums, Homebrew metadata, and the release-derived two-platform OCI layout. +- Pinned Helm 4.2.3 and standalone two-platform OCI contract gates pass. +- The Kubernetes v1.36.1 real two-cluster Kind gate passes in 238.997 seconds. Independent cleanup + checks find no Kind clusters and no Sith/Kind test containers afterward. +- README review is complete and documents the exact verifier-before-authorization boundary, legacy + counter compatibility, refusal-only sinks, privacy exclusions, and nonclaims. +- Repeated complete 16-file CodeRabbit reviews have zero findings. The final checkpointed + secret-signature scan found zero candidates.