Fix ranking ingestion, preserve Warden mode, and schedule weekly rollover

docs/battlelogs.md

@@ -65,6 +65,8 @@ Reads target tables and the requested player's current Town Hall from `basic_pla
 
 Ranked and Legend rows retain the normalized share code directly. The hot writer inserts each distinct code into `army_compositions`, keyed only by `share_code`; there is no army hash or parser-version identity. The daily Legend closeout reuses those compositions for family and item aggregation.
 
+Hero mode segments such as `h2m1p16e5_41` are accepted and preserved in canonical share codes. Mode is not a composition column and does not participate in army-family similarity; `m0` and `m1` can remain distinct exact codes in the same family. The earlier rejection came from Tracking's share-code validator, not clashy.go.
+
 The battle identity is `(player_tag, battle_time)`. Deploy this writer only with the final DevKit migration 017 contract. Existing raw rows remain intact.
 
 The scheduled closeout rebuilds completed Legend-day and Ranked-season aggregates from raw attack perspectives. It groups newly observed Legend armies into immutable direct-anchor families and writes daily family outcomes. Farming and Ranked/Legend raw rows retain one year.

docs/leaderboards.md

@@ -2,11 +2,11 @@
 
 ## What this workload is for
 
-Leaderboards fetch current global/location rankings and maintain the typed current/history tables and lightweight API cache. It is documented separately because it makes many paging and replacement decisions even though it runs inside `scheduled`.
+Scheduled ranking jobs maintain official current/history tables. The adjacent leaderboard loop refreshes Legend I players and publishes SQL materialized snapshots for player Town Hall and league boards. No player or clan leaderboard is published to Valkey.
 
 ## When it runs
 
-The current leaderboard loop runs every `leaderboards.interval_seconds`. Ordinary boards share `scheduled.requests_per_second`; the live Legend I player refresh has its own fixed 20-request-per-second limiter so its contract is independent of the general scheduled budget.
+The live Legend I loop runs every `leaderboards.interval_seconds` with its own fixed 20-request-per-second limiter. A separate maintenance loop checks every minute and refreshes player Town Hall/league materialized views once six hours have elapsed since the last successful refresh. It populates uninitialized views on its first run. Ordinary official ranking fetches still share `scheduled.requests_per_second`.
 
 ## Targets and endpoints
 
@@ -25,7 +25,11 @@ Load locations
   -> append/replace completed historical snapshot
   -> fetch every stored Legend I player at 20 requests per second
   -> atomically replace legend_rankings_current
-  -> refresh API cache metadata
+  -> publish SQL-backed live Legend leaderboard
+
+basic_player
+  -> every six hours: refresh player_townhall_leaderboards / player_league_leaderboards
+  -> API reads up to 500 rows per board, without rebuilding ranks per request
 ```
 
 ## Data written
@@ -36,6 +40,11 @@ Load locations
 - `legend_rankings_current`, atomically replaced only after the full Legend I refresh.
 - `ranked_league_group_members`.
 - Changed `basic_player` facts learned from ranked players.
+- `player_townhall_leaderboards`: ranked tiers only, ordered by league descending, trophies descending, then tag.
+- `player_league_leaderboards`: exact ranked tier, ordered by trophies descending, then tag.
+- `tracking_scheduled_jobs`: durable refresh completion and weekly reset progress.
+
+The SQL snapshots include ranked players with zero trophies. Unranked and unknown leagues are excluded. Snapshots use existing `basic_player` observations; there is no separate Town Hall/league candidate-fetch pass. Clan display metadata is still joined from `basic_clan` by the API.
 
 Leaderboard writes to `basic_player` are accepted because volume is bounded and the shared SQL upsert writes only changed columns.
 
@@ -52,6 +61,20 @@ Leaderboard writes to `basic_player` are accepted because volume is bounded and
 
 A board is replaced only with a complete accepted result. API errors are recorded per request. Availability pauses the entire fetch path, and a later scheduled pass retries naturally.
 
+Recognized "rankings not found for location" errors on non-global boards are counted and skipped without replacing existing data. Global 404s and unrelated errors remain failures. Maintenance errors have their own `leaderboards.maintenance` readiness entry and do not terminate the Legend refresh loop.
+
+## Weekly Ranked trophy reset
+
+On Monday between 16:30 and 17:00 UTC only, maintenance resets nonzero `basic_player.trophies` for league IDs 105000000 through 105000035, including Unranked. Legend I (105000036) and unknown leagues are untouched. Monday's Ranked closeout must have recorded successful completion first.
+
+Each transaction changes at most 5,000 players and saves its last tag. A process restart resumes that cursor inside the same safe window. Completed jobs never repeat for that Monday; a missed or partial job is not caught up outside the window. Completion forces both player snapshots to refresh, regardless of the six-hour timer. No historical trophy records are changed, no Legend season reset is performed, and normal profile writers remain free to record later observations.
+
+The maintenance session holds a PostgreSQL advisory lock to prevent overlapping scheduled processes from running it simultaneously. There is deliberately no cross-writer stale-response coordination.
+
+## Rollout order
+
+Apply DevKit migration 020, deploy this Tracking change, verify both player views are populated, then deploy the API snapshot reader. The views are created WITH NO DATA so migration itself does not perform the population query. Deploying the API before the first refresh will fail those board reads. No production migration or deployment is performed by this PR.
+
 ## What it deliberately does not do
 
 - It does not produce live events.

docs/scheduled.md

@@ -6,12 +6,12 @@
 
 ## When it runs
 
-The main scheduled cycle uses `scheduled.interval_seconds`. Current player/clan leaderboards run alongside it at the leaderboard interval. League closeout processes the latest eligible shifted Legend day on startup, then runs daily at 05:12 UTC; Monday's closeout also discovers and finalizes completed Ranked seasons from the IDs returned by player profiles and matching league-history entries.
+The main scheduled cycle uses `scheduled.interval_seconds`. The Legend player refresh runs alongside it. League closeout runs daily at 05:12 UTC, not on startup; Monday's closeout also discovers and finalizes completed Ranked seasons from player profiles and matching league-history entries, then records completion for the guarded weekly trophy reset.
 
 ## Work owned here
 
 - Current and historical player/clan leaderboard snapshots.
-- Legend history completion/backfill for completed seasons.
+- Forward-only Legend history completion for recent completed seasons. Already stored season IDs are skipped before parsing; routine discovery never backfills beyond a 35-day lookback or the newest stored completion, whichever is later. Older repairs require an explicit separate operation.
 - Ranked league member snapshots and season tier aggregates.
 - Legend daily hit-rate, usage, and immutable army-family aggregates.
 - Scheduled broad statistics and date-bound maintenance already implemented in `scripts/scheduled.go`.
@@ -44,7 +44,9 @@ Army-family matching compares each exact army directly with immutable anchors. T
 
 ## Events and Valkey
 
-Scheduled statistics normally write SQL/cache snapshots and do not emit live Discord events. Leaderboard cache output is consumed by API reads. This process does not use the event stream as a job queue.
+Scheduled statistics write SQL snapshots and do not emit live Discord events. Player and clan leaderboards are not published to Valkey. This process does not use the event stream as a job queue.
+
+Official player history retains valid ranking facts when optional clan metadata is incomplete, leaving all three clan snapshot fields absent rather than borrowing today's clan details. Upstream `previousRank=-1` means no previous rank and is stored as SQL NULL; other negative ranks remain invalid. Current clan ranking writes use only the existing five schema columns, without `updated_at`.
 
 ## Configuration
 

scripts/battlelogs.go

@@ -1110,19 +1110,26 @@ func validateArmyHeroSection(payload string) error {
 		}
 		petSeen := false
 		equipmentSeen := false
+		modeSeen := false
 		for rest != "" {
 			marker := rest[0]
 			if marker == '_' {
 				if !equipmentSeen {
 					return fmt.Errorf("equipment continuation without equipment in %q", part)
 				}
-			} else if marker != 'p' && marker != 'e' {
+			} else if marker != 'p' && marker != 'e' && marker != 'm' {
 				return fmt.Errorf("unknown hero item marker %q in %q", marker, part)
 			}
 			value, next := leadingInt(rest[1:])
 			if value < 0 {
 				return fmt.Errorf("missing hero item id in %q", part)
 			}
+			if marker == 'm' {
+				if modeSeen {
+					return fmt.Errorf("multiple hero modes in %q", part)
+				}
+				modeSeen = true
+			}
 			if marker == 'p' {
 				if petSeen {
 					return fmt.Errorf("multiple pets in %q", part)
@@ -1298,6 +1305,7 @@ func normalizeArmyShareCode(link string) string {
 
 type armyHeroLoadout struct {
 	HeroID    int
+	Mode      *int // Preserved in share codes, intentionally absent from composition/family fields.
 	PetID     int
 	Equipment []int
 }
@@ -1315,6 +1323,14 @@ func parseArmyHeroLoadouts(payload string) []armyHeroLoadout {
 		loadout := armyHeroLoadout{HeroID: heroID, PetID: -1}
 		for rest != "" {
 			marker := rest[0]
+			if marker == 'm' {
+				value, next := leadingInt(rest[1:])
+				if value >= 0 {
+					loadout.Mode = &value
+				}
+				rest = next
+				continue
+			}
 			if marker != 'p' && marker != 'e' {
 				if marker == '_' {
 					rest = rest[1:]
@@ -1364,6 +1380,9 @@ func encodeArmyHeroSection(heroes []armyHeroLoadout) string {
 	parts := make([]string, 0, len(heroes))
 	for _, hero := range heroes {
 		part := strconv.Itoa(hero.HeroID)
+		if hero.Mode != nil {
+			part += "m" + strconv.Itoa(*hero.Mode)
+		}
 		if hero.PetID >= 0 {
 			part += "p" + strconv.Itoa(hero.PetID)
 		}

scripts/leaderboards.go

@@ -3,7 +3,6 @@ package scripts
 import (
 	"context"
 	"database/sql"
-	"encoding/json"
 	"errors"
 	"fmt"
 	"sort"
@@ -18,7 +17,6 @@ import (
 	clashy "github.com/clashkinginc/clashy.go"
 	"github.com/jackc/pgx/v5/pgconn"
 	"github.com/jackc/pgx/v5/pgxpool"
-	valkey "github.com/valkey-io/valkey-go"
 )
 
 const (
@@ -33,13 +31,6 @@ const (
 	leaderboardMaterializedViewCount          = 5
 )
 
-var leaderboardCacheScript = valkey.NewLuaScript(`
-	for i = 1, #KEYS do
-		redis.call('SET', KEYS[i], ARGV[i])
-	end
-	return #KEYS
-`)
-
 var leaderboardMaterializedViewRefreshQueries = [...]string{
 	`REFRESH MATERIALIZED VIEW CONCURRENTLY clan_leaderboards`,
 	`REFRESH MATERIALIZED VIEW war_league_counts`,
@@ -137,8 +128,7 @@ type leaderboardCandidate struct {
 }
 
 type timescaleLeaderboardStore struct {
-	pool   *pgxpool.Pool
-	valkey valkey.Client
+	pool *pgxpool.Pool
 }
 
 type leaderboardCacheSet struct {
@@ -208,6 +198,13 @@ func (d *leaderboardsDomain) Run(ctx context.Context, app *platform.App) error {
 	d.store = store
 	d.refreshMaterializedViews = store.RefreshMaterializedViews
 	defer store.Close()
+	maintenanceCtx, stopMaintenance := context.WithCancel(ctx)
+	maintenanceDone := make(chan struct{})
+	go func() {
+		defer close(maintenanceDone)
+		store.runPlayerLeaderboardMaintenance(maintenanceCtx, app)
+	}()
+	defer func() { stopMaintenance(); <-maintenanceDone }()
 
 	interval := time.Duration(app.Config.LeaderboardIntervalSeconds) * time.Second
 	for {
@@ -247,9 +244,6 @@ func validateLeaderboardsConfig(cfg platform.Config) error {
 	if cfg.TimescaleURL == "" {
 		return errors.New("TIMESCALE_* connection variables are required for leaderboards")
 	}
-	if cfg.ValkeyAddr == "" {
-		return errors.New("valkey_addr is required for leaderboards")
-	}
 	return nil
 }
 
@@ -258,7 +252,7 @@ func (d *leaderboardsDomain) openStore(ctx context.Context, app *platform.App) (
 	if err != nil {
 		return nil, err
 	}
-	return &timescaleLeaderboardStore{pool: pool, valkey: app.Valkey}, nil
+	return &timescaleLeaderboardStore{pool: pool}, nil
 }
 
 func (d *leaderboardsDomain) runCycle(ctx context.Context, app *platform.App) error {
@@ -268,59 +262,7 @@ func (d *leaderboardsDomain) runCycle(ctx context.Context, app *platform.App) er
 	if d.legendLimiter == nil {
 		return errors.New("20 RPS Legend request limiter is required for leaderboards")
 	}
-	legendPlayers, err := d.refreshLegendRankings(ctx, app)
-	if err != nil {
-		return err
-	}
-	candidates, err := d.store.LoadCandidates(ctx, d.limit)
-	if err != nil {
-		return err
-	}
-	tags := leaderboardCandidateTags(candidates)
-	if len(tags) == 0 {
-		cache := buildLeaderboardCache(candidates, nil, nil, time.Now().UTC(), d.limit, d.nullAssetURL)
-		return d.store.CacheBoards(ctx, cache)
-	}
-	leagues, err := d.fetchLeagues(ctx, app, d.limiter)
-	if err != nil {
-		return err
-	}
-	legendByTag := make(map[string]leaderboardPlayerRow, len(legendPlayers))
-	for _, player := range legendPlayers {
-		legendByTag[player.Tag] = player
-	}
-	standardTags := make([]string, 0, len(tags))
-	for _, tag := range tags {
-		if _, refreshedAsLegend := legendByTag[tag]; !refreshedAsLegend {
-			standardTags = append(standardTags, tag)
-		}
-	}
-	players, deletedTags, err := d.fetchPlayers(ctx, app, d.limiter, standardTags, leagues)
-	if err != nil {
-		return err
-	}
-	for _, tag := range tags {
-		if player, ok := legendByTag[tag]; ok {
-			players = append(players, player)
-		}
-	}
-	basicPlayers := leaderboardBasicPlayerRows(players)
-	deleted, err := d.store.DeletePlayers(ctx, deletedTags)
-	if err != nil {
-		return err
-	}
-	if err := d.store.UpdatePlayers(ctx, basicPlayers); err != nil {
-		return err
-	}
-	clans, err := d.store.LoadClanMetadata(ctx, leaderboardPlayerClanTags(players))
-	if err != nil {
-		return err
-	}
-	cache := buildLeaderboardCache(candidates, players, clans, time.Now().UTC(), d.limit, d.nullAssetURL)
-	err = d.store.CacheBoards(ctx, cache)
-	if err == nil {
-		app.Stats.RecordWrite(leaderboardsDomainName, len(basicPlayers)+deleted+len(cache.Boards)+1)
-	}
+	_, err := d.refreshLegendRankings(ctx, app)
 	return err
 }
 
@@ -789,26 +731,6 @@ func (s *timescaleLeaderboardStore) LoadClanMetadata(ctx context.Context, clanTa
 	return out, rows.Err()
 }
 
-func (s *timescaleLeaderboardStore) CacheBoards(ctx context.Context, cache leaderboardCacheSet) error {
-	rawIndex, err := json.Marshal(cache)
-	if err != nil {
-		return err
-	}
-	keys := make([]string, 0, len(cache.Boards)+1)
-	values := make([]string, 0, len(cache.Boards)+1)
-	for _, board := range cache.Boards {
-		raw, err := json.Marshal(board)
-		if err != nil {
-			return err
-		}
-		keys = append(keys, leaderboardCacheKey(board.Kind, board.Key))
-		values = append(values, string(raw))
-	}
-	keys = append(keys, leaderboardIndexKey)
-	values = append(values, string(rawIndex))
-	return leaderboardCacheScript.Exec(ctx, s.valkey, keys, values).Error()
-}
-
 func (s *timescaleLeaderboardStore) RefreshMaterializedViews(ctx context.Context) error {
 	for i, query := range leaderboardMaterializedViewRefreshQueries {
 		if _, err := s.pool.Exec(ctx, query); err != nil {