Skip to content

Retrain heuristic weights (2026-08-21) - #13

Open
github-actions[bot] wants to merge 1 commit into
mainfrom
retrain
Open

Retrain heuristic weights (2026-08-21)#13
github-actions[bot] wants to merge 1 commit into
mainfrom
retrain

Conversation

@github-actions

@github-actions github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown

Automated re-fit. Review the weight diff below, then update the golden tests
in HeuristicArenaDataSourceTests with the printed values before merging.

Golden gate (run in the retrain job — a GITHUB_TOKEN PR does not trigger build):
FAIL — goldens must be updated from the trainer output before merging

training dir: D:\a\HdtArenaHelper\HdtArenaHelper\HdtArenaHelper.Training
live fetch; snapshotting payloads to D:\a\HdtArenaHelper\HdtArenaHelper\HdtArenaHelper.Training\.snapshot
fetching HSReplay arena card stats...
HSReplay pooled (card,class) rows: 2187
rows: 2187 (single source)
features: 32 fitted, 18 dropped
  dropped: kw_charge(n=11), kw_colossal(n=11), kw_combo(n=21), kw_echo(n=6), kw_forge(n=3), kw_freeze(n=23), kw_magnetic(n=14), kw_outcast(n=18), kw_poisonous(n=35), kw_spellpower(n=19), kw_stealth(n=10), kw_windfury(n=8), tx_armor(n=28), tx_destroy_minion(n=29), tx_discover(redundant), tx_restore_amt(n=18), tx_silence(n=17), tx_transform(n=32)
penalty BEFORE normalizing weights: alpha=10 vs mean(diag)=75867  -> shrinkage 0.000132, df_eff=31.0/32
CV: 5-fold grouped by card (1087 cards, 2187 rows), 5 repeats, 11 alphas

     alpha   withinClassRho       se
      0.03           0.2132   0.0035
       0.1           0.2132   0.0035
       0.3           0.2134   0.0035
         1           0.2138   0.0035
         3           0.2145   0.0032
        10           0.2174   0.0026
        30           0.2192   0.0021
       100           0.2207   0.0018
       300           0.2163   0.0020
      1000           0.2014   0.0033
      3000           0.1775   0.0028
CV best alpha=100 (rho=0.2207); 1-SE rule picks alpha=100 (rho=0.2207, threshold 0.2189)
penalty AFTER: alpha=100 vs mean(diag)=2381  -> shrinkage 0.040306, df_eff=28.2/32

holdout evaluation (pooled Spearman on held-out rows, alpha=100):
  random cards (baseline)            n= 108  rho= 0.2863  (se 0.0288)
  lowest-games decile                n= 108  rho= 0.1390  (up to 197 games)
  -> covariate-shift gap vs baseline: -0.1473
  target spread / model error:
      thin  sd(y)=6.904  MAE=5.871
      rest  sd(y)=5.044  MAE=3.272
  calibration slope (truth regressed on prediction):
      thin decile  0.8108   random holdout  0.8427  (se 0.1509)
      -> measured ModelOnlyShrink = 0.9622 (thin/random; set the runtime constant from THIS, not from a correlation ratio)
  leave-out set CORE                 n= 207  rho= 0.1878
  leave-out set TIME_TRAVEL          n= 168  rho= 0.3785
  leave-out set BATTLE_OF_THE_BANDS  n= 158  rho= 0.2990
  leave-out set WHIZBANGS_WORKSHOP   n= 153  rho= 0.2461

fitting ridge (alpha=100) on 2187 rows, 32 features...
display anchor (pool median raw): -1.12, robust sigma: 1.3521 -> 15 pts per robust SD
wrote D:\a\HdtArenaHelper\HdtArenaHelper\HdtArenaHelper.Training\arena_weights.generated.json

feature                  committed   generated     diff
intercept                   +1.09      +0.60    -0.49
anchor_median_raw           -0.92      -1.12    -0.20  <-- differs
attack                      -0.15      -0.20    -0.05  <-- differs
cost                        +0.15      +0.00    -0.15  <-- differs
has_tribe                   +0.12      +0.50    +0.38  <-- differs
health                      -0.18      -0.20    -0.02  <-- differs
is_hero                     +0.81      +4.04    +3.23  <-- differs
is_loc                      -0.88      +0.12    +1.00  <-- differs
is_minion                   +2.61      +1.52    -1.09  <-- differs
is_neutral                  -1.94      -2.00    -0.06  <-- differs
is_spell                    -2.75      -1.72    +1.03  <-- differs
is_weapon                   -0.94      -0.24    +0.70  <-- differs
kw_battlecry                +0.70      +0.72    +0.02  <-- differs
kw_deathrattle              -0.41      -0.81    -0.40  <-- differs
kw_discover                 +0.86      +0.73    -0.13  <-- differs
kw_divine_shield            +0.10      -0.05    -0.15  <-- differs
kw_lifesteal                +0.84      +0.69    -0.15  <-- differs
kw_reborn                   +1.83      +1.16    -0.67  <-- differs
kw_rush                     +1.05      +1.02    -0.03  <-- differs
kw_secret                   -0.19      -1.20    -1.01  <-- differs
kw_taunt                    -0.10      +0.06    +0.16  <-- differs
kw_tradeable                +0.71      +1.07    +0.36  <-- differs
stat_per_mana               -2.40      -1.33    +1.07  <-- differs
statline                    +0.34      +0.18    -0.16  <-- differs
tx_aoe                      +0.51      +0.52    +0.01  <-- differs
tx_damage_amt               +0.08      +0.13    +0.05  <-- differs
tx_dmg_per_mana             +0.00      -0.28    -0.28  <-- differs
tx_draw                     -0.31      -0.07    +0.24  <-- differs
tx_gain_card                -0.87      -0.65    +0.22  <-- differs
tx_mana_cheat               +1.05      +0.61    -0.44  <-- differs
tx_persistent               +0.15      +0.48    +0.33  <-- differs
tx_random                   +0.32      +0.23    -0.09  <-- differs
tx_summon                   +0.95      +1.13    +0.18  <-- differs
weapon_value                -0.20      -0.35    -0.15  <-- differs

max abs weight diff: 3.23
weights differing by >= 0.01: 33
GATE: generated weights are close but not identical (see rows above).

golden scores for HeuristicArenaDataSourceTests.Goldens
(paste the VALUES over the literals on adopt; keep the trailing comments):
  { HSCard.PlatedBeetle, 24.63 },
  { HSCard.ElvenArcher, 50.44 },
  { HSCard.DefenderOfArgus, 40.95 },
  { HSCard.FieryWarAxe, 51.44 },
  { HSCard.Fireball, 54.93 },
  { HSCard.ColdlightOracle, 46.89 },
  { HSCard.DireFrenzy, 50.00 },
  { HSCard.NorthshireCleric, 54.10 },
  { HSCard.DarkIronDwarf, 28.39 },
  { HSCard.Deathwing, 0.00 },

bootstrapping 300 card-resampled refits...

pick instability under resampling, by how clearly the model named a winner:
  winner leads by >  0.0 pts:  27.84% of those picks flip
  winner leads by >  1.0 pts:  25.73% of those picks flip
  winner leads by >  3.0 pts:  21.64% of those picks flip
  winner leads by >  5.0 pts:  18.25% of those picks flip
  winner leads by > 10.0 pts:  11.20% of those picks flip

feature                    weight       se    |w|/se   signCons
attack                      -0.20     0.09       2.2       0.98
cost                        +0.04     0.08       0.4       0.68  <-- within noise
has_tribe                   +0.50     0.25       2.0       0.98  <-- within noise
health                      -0.20     0.09       2.3       0.99
is_hero                     +4.04     3.01       1.3       0.79  <-- within noise
is_loc                      +0.12     0.96       0.1       0.57  <-- within noise
is_minion                   +1.52     0.71       2.1       1.00
is_neutral                  -2.00     0.29       7.0       1.00
is_spell                    -1.72     0.67       2.6       1.00
is_weapon                   -0.24     1.10       0.2       0.59  <-- within noise
kw_battlecry                +0.72     0.29       2.5       1.00
kw_deathrattle              -0.81     0.31       2.7       1.00
kw_discover                 +0.73     0.29       2.5       0.99
kw_divine_shield            -0.05     0.56       0.1       0.53  <-- within noise
kw_lifesteal                +0.69     0.45       1.5       0.93  <-- within noise
kw_reborn                   +1.16     1.03       1.1       0.87  <-- within noise
kw_rush                     +1.02     0.36       2.8       1.00
kw_secret                   -1.20     0.56       2.1       0.98
kw_taunt                    +0.06     0.35       0.2       0.55  <-- within noise
kw_tradeable                +1.07     0.54       2.0       0.97  <-- within noise
stat_per_mana               -1.33     1.06       1.3       0.85  <-- within noise
statline                    +0.18     0.07       2.6       0.99
tx_aoe                      +0.52     0.36       1.4       0.94  <-- within noise
tx_damage_amt               +0.13     0.12       1.1       0.87  <-- within noise
tx_dmg_per_mana             -0.28     0.46       0.6       0.68  <-- within noise
tx_draw                     -0.07     0.37       0.2       0.52  <-- within noise
tx_gain_card                -0.65     0.34       1.9       0.97  <-- within noise
tx_mana_cheat               +0.61     0.40       1.5       0.93  <-- within noise
tx_persistent               +0.48     0.32       1.5       0.92  <-- within noise
tx_random                   +0.23     0.28       0.8       0.78  <-- within noise
tx_summon                   +1.13     0.29       3.9       1.00
weapon_value                -0.35     0.26       1.3       0.98  <-- within noise
release-gate noise floor (p95 of the bootstrap null): 27.84% of picks flip on resampling alone
wrote D:\a\HdtArenaHelper\HdtArenaHelper\HdtArenaHelper.Training\metrics.json
pick flips vs committed: 20.07% of same-class triples (resampling noise floor 27.84%); per-card score volatility p95 16.7 pts
MATERIAL: this refit changes more recommendations than noise would - worth a review.

@github-actions github-actions Bot changed the title Retrain heuristic weights (2026-08-14) Retrain heuristic weights (2026-08-21) Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants