Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 12 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -311,16 +311,18 @@ listener, Service, exporter, queue, persistence, remote telemetry, request metad
data, or raw payload retention. The drop counter is scrapeable only when the same optional loopback
metrics endpoint above is enabled.

The same already-sanitized middleware refusal also increments the preinitialized, unlabeled
`sith_auth_refusals_total` counter. Runtime fanout delivers independently to the process audit child
and metric observer; either observer's panic is isolated and cannot suppress the other or alter the
uniform HTTP 401 response. The counter spans the bearer API and browser-session console middleware
that already emit this one closed event. It contains no credential mode, failure reason, tenant,
workspace, actor, principal, token, IP, path, request, trace, or correlation label. It does not
cover successful authentication, OIDC provider exchange/callback failures, authorization denials,
or every future authentication mode. Without a success or attempt denominator it is not a ratio,
brute-force detector, alert threshold, SLO, error budget, page, or complete security-monitoring
control, and it adds no new scrape or storage path.
The same already-sanitized bearer and browser-session boundaries increment exactly two
preinitialized `sith_auth_attempts_total{outcome="accepted|refused"}` series. `accepted` means the
local verifier succeeded and is emitted before workspace authorization; `refused` covers the
existing uniform authentication rejection paths. Every refusal also increments the legacy
unlabeled `sith_auth_refusals_total` counter exactly once. Runtime fanout is panic-isolated per
destination: metrics consume both outcomes, while the process audit child and structured-log
adapter remain refusal-only. No credential mode, failure reason, tenant, workspace, actor,
principal, token, IP, path, method, request, trace, or correlation value becomes a label or record.
These counters do not cover OIDC provider exchange/callback failures, authorization denials,
handler outcomes, or every future authentication mode. They are raw substrate, not a configured
ratio, brute-force detector, alert threshold, SLO, error budget, page, or complete security control,
and they add no new scrape or storage path.

Every referenced key, certificate, or CA file must be a read-only regular file from a deployment
mount. The runtime obtains its Kubernetes identity only with in-cluster configuration; it has no
Expand Down
8 changes: 8 additions & 0 deletions charts/sith-hub/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,14 @@ the fixed record without delaying the governed response and increments the unlab
scrape that process-wide loss signal; the chart still renders no related Service port, ingress,
sidecar, queue, exporter, or remote telemetry path.

The same optional endpoint exposes two fixed, preinitialized
`sith_auth_attempts_total{outcome="accepted|refused"}` series. A successful local verifier decision
counts as `accepted` before workspace authorization; every authentication rejection counts as
`refused` and also increments the legacy unlabeled `sith_auth_refusals_total` counter. The audit
child remains refusal-only. These counters contain no tenant, workspace, identity, credential,
request, network, error, trace, or authorization labels and do not add a Service, exporter,
persistence, remote write, alert, SLO, or cloud resource.

Validate supplied values before applying anything:

```bash
Expand Down
20 changes: 16 additions & 4 deletions docs/EPICS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2496,7 +2496,7 @@ and alerting · F10.5 crown-jewel hardening.
### F10.1 — Metrics

**What it is.** Metrics about Sith's own health and behavior: control-plane liveness, federation
freshness, intent throughput, sanitized authentication-refusal counts, and future derived rates
freshness, intent throughput, bounded sanitized authentication-outcome counts, and future derived rates
where trustworthy denominators exist, abstention rates, and PDP latency.

**How it works.**
Expand Down Expand Up @@ -2560,9 +2560,21 @@ Runtime fanout independently reaches the existing process audit observer and the
observer panics cannot suppress a later destination or alter the uniform HTTP 401 response. The
counter carries no reason, credential mode, tenant, workspace, actor, principal, token, IP, path,
request, trace, or correlation label. It does not count successful authentication, OIDC provider
exchange/callback failures, authorization denials, or every future authentication mode. Without a
success or attempt denominator it is not a ratio, brute-force detector, alert threshold, SLO,
error budget, page, or complete security-monitoring control.
exchange/callback failures, authorization denials, or every future authentication mode. The legacy
counter alone is not a denominator and remains compatible with existing scrapes.

**Implementation note (F10.1h).** Each completed local bearer-token or browser-session verifier
decision increments one of exactly two preinitialized
`sith_auth_attempts_total{outcome="accepted|refused"}` series. `accepted` is emitted immediately
after verifier success and before workspace authorization; a later forbidden authorization is
therefore not misclassified as failed authentication. Every `refused` outcome also increments the
legacy unlabeled refusal counter exactly once. Metrics consume both outcomes, while the process
audit observer and structured-log adapter remain refusal-only and accepted observations cannot
write a datagram, log, or delivery-drop count. The outcome label is closed and carries no
credential mode, reason, tenant, workspace, actor, principal, token, IP, path, method, request,
trace, correlation, authorization, or handler-result dimension. The counters exclude provider
exchange/callback failures and define no ratio, alert, brute-force detector, SLO, error budget,
page, listener, exporter, persistence, remote write, or cloud resource.

### F10.2 — Distributed tracing

Expand Down
2 changes: 1 addition & 1 deletion docs/SITH-NOTION.md
Original file line number Diff line number Diff line change
Expand Up @@ -2539,7 +2539,7 @@ and alerting · F10.5 crown-jewel hardening.
### F10.1 — Metrics

**What it is.** Metrics about Sith's own health and behavior: control-plane liveness, federation
freshness, intent throughput, sanitized authentication-refusal counts, and future derived rates
freshness, intent throughput, bounded sanitized authentication-outcome counts, and future derived rates
where trustworthy denominators exist, abstention rates, and PDP latency.

**How it works.**
Expand Down
2 changes: 1 addition & 1 deletion internal/auditdelivery/process.go
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ func (observer *ProcessObserver) ObserveAuth(event hubserver.AuthEvent) {
}

func encodeAuthRefusal(event hubserver.AuthEvent) []byte {
if event.Validate() != nil {
if event.Validate() != nil || event.Outcome != hubserver.AuthOutcomeRefused {
return nil
}
return []byte{authRecordVersion, authRecordRefused}
Expand Down
14 changes: 14 additions & 0 deletions internal/auditdelivery/process_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ func TestProcessObserverDeliversOnlyFixedRecordAndReapsChild(t *testing.T) {
defer writer.Close()
observer := newTestProcessObserver(t, writer, nil)

observer.ObserveAuth(hubserver.AuthEvent{Outcome: hubserver.AuthOutcomeAccepted})
observer.ObserveAuth(hubserver.AuthEvent{Outcome: "token=secret"})
observer.ObserveAuth(hubserver.AuthEvent{Outcome: hubserver.AuthOutcomeRefused})
line := make(chan string, 1)
Expand All @@ -54,6 +55,19 @@ func TestProcessObserverDeliversOnlyFixedRecordAndReapsChild(t *testing.T) {
}
}

func TestProcessObserverNeverDeliversOrDropsAcceptedAuthentication(t *testing.T) {
var drops atomic.Uint64
observer := &ProcessObserver{drops: dropObserverFunc(func() { drops.Add(1) })}
observer.ObserveAuth(hubserver.AuthEvent{Outcome: hubserver.AuthOutcomeAccepted})
if drops.Load() != 0 {
t.Fatalf("accepted authentication delivery drops = %d, want 0", drops.Load())
}
observer.ObserveAuth(hubserver.AuthEvent{Outcome: hubserver.AuthOutcomeRefused})
if drops.Load() != 1 {
t.Fatalf("refused authentication delivery drops = %d, want 1", drops.Load())
}
}

func TestProcessObserverDropsFullDatagramBufferWithoutBlocking(t *testing.T) {
parent, child := socketPair(t)
defer child.Close()
Expand Down
7 changes: 4 additions & 3 deletions internal/hubserver/auth.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@ func Authenticate(verifier Verifier, next http.Handler) (http.Handler, error) {
return AuthenticateWithObserver(verifier, nil, next)
}

// AuthenticateWithObserver constructs authentication middleware with one passive refusal
// observer. The observer is never given request metadata, credentials, verifier errors, or caller
// correlation values, and cannot alter the uniform unauthorized response.
// AuthenticateWithObserver constructs authentication middleware with one passive outcome observer.
// The observer is never given request metadata, credentials, verifier errors, or caller correlation
// values, and cannot alter the uniform unauthorized response or successful handler path.
func AuthenticateWithObserver(verifier Verifier, observer AuthObserver, next http.Handler) (http.Handler, error) {
if verifier == nil {
return nil, fmt.Errorf("construct authentication middleware: verifier is required")
Expand All @@ -53,6 +53,7 @@ func AuthenticateWithObserver(verifier Verifier, observer AuthObserver, next htt
refuseAuthentication(observer, response)
return
}
ObserveAuth(observer, AuthEvent{Outcome: AuthOutcomeAccepted})
ctx := context.WithValue(cloned.Context(), principalContextKey{}, principal)
next.ServeHTTP(response, cloned.WithContext(ctx))
}), nil
Expand Down
15 changes: 10 additions & 5 deletions internal/hubserver/auth_observability.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,17 @@ package hubserver

import "fmt"

// AuthOutcome is the closed self-observability result of one pre-principal
// authentication attempt. It intentionally does not distinguish credential failure modes.
// AuthOutcome is the closed self-observability result of one completed authentication verifier
// decision. It intentionally does not distinguish credential modes or failure reasons.
type AuthOutcome string

// AuthOutcomeRefused is emitted for every request the bearer-token middleware rejects.
const AuthOutcomeRefused AuthOutcome = "refused"
const (
// AuthOutcomeAccepted is emitted after a bearer token or browser session verifies successfully,
// before any workspace authorization decision.
AuthOutcomeAccepted AuthOutcome = "accepted"
// AuthOutcomeRefused is emitted for every request the authentication boundary rejects.
AuthOutcomeRefused AuthOutcome = "refused"
)

// AuthEvent is one passive, sanitized authentication observation. It deliberately has no
// request, credential, verifier-error, principal, path, network, or correlation fields: none are
Expand All @@ -20,7 +25,7 @@ type AuthEvent struct {

// Validate rejects unsupported outcome values before an observer can emit them.
func (event AuthEvent) Validate() error {
if event.Outcome != AuthOutcomeRefused {
if event.Outcome != AuthOutcomeAccepted && event.Outcome != AuthOutcomeRefused {
return fmt.Errorf("authentication event outcome is unsupported")
}
return nil
Expand Down
38 changes: 35 additions & 3 deletions internal/hubserver/auth_observability_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ func TestAuthenticateWithObserverRecordsOnlyUniformRefusals(t *testing.T) {
}
}

func TestAuthenticateWithObserverIsSilentAfterValidAuthentication(t *testing.T) {
func TestAuthenticateWithObserverRecordsAcceptedAfterValidAuthentication(t *testing.T) {
now := time.Date(2026, 7, 14, 13, 0, 0, 0, time.UTC)
publicKey, privateKey := hubTestKeyPair()
verifier, err := hubauth.NewJWTVerifier(hubauth.JWTConfig{
Expand All @@ -82,11 +82,24 @@ func TestAuthenticateWithObserverIsSilentAfterValidAuthentication(t *testing.T)
request.Header.Set("Authorization", "Bearer "+signHubTestToken(t, hubValidClaims(now), privateKey))
response := httptest.NewRecorder()
handler.ServeHTTP(response, request)
if response.Code != http.StatusNoContent || len(events) != 0 {
if response.Code != http.StatusNoContent || len(events) != 1 || events[0] != (AuthEvent{Outcome: AuthOutcomeAccepted}) {
t.Fatalf("status = %d events = %#v", response.Code, events)
}
}

func TestAuthEventValidationUsesOnlyClosedOutcomes(t *testing.T) {
for _, outcome := range []AuthOutcome{AuthOutcomeAccepted, AuthOutcomeRefused} {
if err := (AuthEvent{Outcome: outcome}).Validate(); err != nil {
t.Fatalf("Validate(%q) = %v", outcome, err)
}
}
for _, outcome := range []AuthOutcome{"", "token=secret", "forbidden"} {
if err := (AuthEvent{Outcome: outcome}).Validate(); err == nil {
t.Fatalf("Validate(%q) accepted an unsupported outcome", outcome)
}
}
}

func TestObserveAuthRejectsUnsafeEventsAndContainsObserverPanics(t *testing.T) {
called := false
ObserveAuth(AuthObserverFunc(func(AuthEvent) { called = true }), AuthEvent{Outcome: "token=secret"})
Expand All @@ -95,6 +108,7 @@ func TestObserveAuthRejectsUnsafeEventsAndContainsObserverPanics(t *testing.T) {
}

ObserveAuth(AuthObserverFunc(func(AuthEvent) { panic("observer fault") }), AuthEvent{Outcome: AuthOutcomeRefused})
ObserveAuth(AuthObserverFunc(func(AuthEvent) { panic("observer fault") }), AuthEvent{Outcome: AuthOutcomeAccepted})

handler, err := AuthenticateWithObserver(authVerifierFunc(func(context.Context, string) (tenancy.Principal, error) {
return tenancy.Principal{}, errors.New("invalid")
Expand All @@ -109,6 +123,22 @@ func TestObserveAuthRejectsUnsafeEventsAndContainsObserverPanics(t *testing.T) {
if response.Code != http.StatusUnauthorized || response.Body.String() != "{\"error\":\"unauthorized\"}\n" {
t.Fatalf("status = %d, body = %q", response.Code, response.Body.String())
}

successful, err := AuthenticateWithObserver(authVerifierFunc(func(context.Context, string) (tenancy.Principal, error) {
return tenancy.Principal{}, nil
}), AuthObserverFunc(func(AuthEvent) { panic("observer fault") }), http.HandlerFunc(func(response http.ResponseWriter, _ *http.Request) {
response.WriteHeader(http.StatusNoContent)
}))
if err != nil {
t.Fatal(err)
}
successRequest := httptest.NewRequest(http.MethodGet, "https://hub.sith.test/api", nil)
successRequest.Header.Set("Authorization", "Bearer valid")
successResponse := httptest.NewRecorder()
successful.ServeHTTP(successResponse, successRequest)
if successResponse.Code != http.StatusNoContent {
t.Fatalf("successful authentication with panicking observer status = %d", successResponse.Code)
}
}

func TestAuthObserverFanoutIsolatesEachRequiredDestination(t *testing.T) {
Expand All @@ -126,7 +156,9 @@ func TestAuthObserverFanoutIsolatesEachRequiredDestination(t *testing.T) {
observers[1] = AuthObserverFunc(func(AuthEvent) { t.Fatal("fanout retained caller-owned slice") })

ObserveAuth(fanout, AuthEvent{Outcome: AuthOutcomeRefused})
if len(deliveries) != 2 || deliveries[0] != "first" || deliveries[1] != "second" {
ObserveAuth(fanout, AuthEvent{Outcome: AuthOutcomeAccepted})
if len(deliveries) != 4 || deliveries[0] != "first" || deliveries[1] != "second" ||
deliveries[2] != "first" || deliveries[3] != "second" {
t.Fatalf("fanout deliveries = %#v, want independently isolated order", deliveries)
}

Expand Down
1 change: 1 addition & 0 deletions internal/hubserver/console.go
Original file line number Diff line number Diff line change
Expand Up @@ -408,6 +408,7 @@ func (handler *ConsoleHandler) authorize(
refuseAuthentication(handler.authObserver, response)
return tenancy.Scope{}, "", false
}
ObserveAuth(handler.authObserver, AuthEvent{Outcome: AuthOutcomeAccepted})
scope, err := principal.Scope(workspaceID)
if err != nil {
writeConsoleError(response, http.StatusForbidden, "forbidden")
Expand Down
18 changes: 15 additions & 3 deletions internal/hubserver/console_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ func TestConsoleRoutesResolveWorkspaceThroughServeMux(t *testing.T) {

func TestConsoleHandlerForwardsUniformAuthRefusalsToConfiguredObserver(t *testing.T) {
now := time.Date(2026, 7, 18, 8, 0, 0, 0, time.UTC)
verifier, _ := fleetTestVerifier(t, now)
verifier, privateKey := fleetTestVerifier(t, now)
var events []AuthEvent
handler, err := NewConsoleHandler(ConsoleHandlerConfig{
Verifier: verifier, AuthObserver: AuthObserverFunc(func(event AuthEvent) {
Expand Down Expand Up @@ -202,8 +202,20 @@ func TestConsoleHandlerForwardsUniformAuthRefusalsToConfiguredObserver(t *testin
t.Fatalf("session %q status/body = %d/%q", session, response.Code, response.Body.String())
}
}
if len(events) != 2 || events[0] != (AuthEvent{Outcome: AuthOutcomeRefused}) || events[1] != events[0] {
t.Fatalf("console authentication events = %#v, want two uniform refusals", events)
foreign := consoleTestRequest(
http.MethodGet,
"/v1/workspaces/workspace-b/console",
"workspace-b",
signHubTestToken(t, hubValidClaims(now), privateKey),
)
foreignResponse := httptest.NewRecorder()
handler.ServePage(foreignResponse, foreign)
if foreignResponse.Code != http.StatusForbidden || foreignResponse.Body.String() != "{\"error\":\"forbidden\"}\n" {
t.Fatalf("foreign workspace status/body = %d/%q", foreignResponse.Code, foreignResponse.Body.String())
}
if len(events) != 3 || events[0] != (AuthEvent{Outcome: AuthOutcomeRefused}) || events[1] != events[0] ||
events[2] != (AuthEvent{Outcome: AuthOutcomeAccepted}) {
t.Fatalf("console authentication events = %#v, want two refusals then accepted before authorization", events)
}
}

Expand Down
2 changes: 1 addition & 1 deletion internal/observability/auth.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ type slogAuthObserver struct {
}

func (observer slogAuthObserver) ObserveAuth(event hubserver.AuthEvent) {
if observer.logger == nil || event.Validate() != nil {
if observer.logger == nil || event.Validate() != nil || event.Outcome != hubserver.AuthOutcomeRefused {
return
}
observer.logger.Warn(
Expand Down
1 change: 1 addition & 0 deletions internal/observability/auth_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ func TestSlogAuthObserverEmitsOnlyValidatedFixedFields(t *testing.T) {
t.Fatal(err)
}
observer.ObserveAuth(hubserver.AuthEvent{Outcome: hubserver.AuthOutcomeRefused})
observer.ObserveAuth(hubserver.AuthEvent{Outcome: hubserver.AuthOutcomeAccepted})
observer.ObserveAuth(hubserver.AuthEvent{Outcome: "token=secret"})

lines := strings.Split(strings.TrimSpace(output.String()), "\n")
Expand Down
Loading