docs(refresh): SOTA-2026 handoff — SonarQube 26.7 empty project list

refresh/sonarqube-267-empty-projects-2026-09-12/CHECKLIST.md

@@ -0,0 +1,18 @@
+# CHECKLIST — resume path
+
+[ ] Read SITREP.md -> CONTEXT.md -> FILES.md -> NEXT-STEPS.md -> GIT.md (this folder).
+[ ] Detect contention: `git -C /mnt/hdd/home/bruno/gitops worktree list` (a concurrent session may hold the tree).
+[ ] Re-derive evidence (NEXT-STEPS step 0, read-only) to confirm 26.7 server + 26.5.0 plugin still in place.
+[ ] Run the DECISIVE experiment (NEXT-STEPS step 1) — disable =web javaagent via Git, NOT kubectl.
+      [ ] coordinate the SQ pod restart with Bruno first (team-wide brief outage)
+      [ ] edit values.yaml (+ bump ?v=3 -> ?v=4 on appset-helm.yaml line 123)
+      [ ] validate render (helm template / argocd diff) before merge
+[ ] Confirm search_projects returns non-empty components -> root cause proven.
+[ ] Choose the permanent fix (NEXT-STEPS step 2) with Bruno; implement in values.yaml; PR; merge; verify live.
+[ ] Clean up: `./claim.sh --release sonarqube-267-empty-projects-2026-09-12` once merged/abandoned.
+
+RULES
+- Git-push only; never kubectl patch/apply/delete on ArgoCD-managed resources.
+- Never `git add -A`/`checkout`/`stash` in the shared tree while another worktree is active.
+- Bump the valueURL `?v=` on EVERY values.yaml change or ArgoCD keeps the old render.
+- No secret values in any handoff (Gap-Auth is a header name; Vault paths are references — fine).

refresh/sonarqube-267-empty-projects-2026-09-12/CONTEXT.md

@@ -0,0 +1,49 @@
+# CONTEXT — SonarQube 26.7 empty project list
+
+## Source of truth: helm/releases/sonarqube/values.yaml
+Version pins (all point at plugin/webapp 26.5.0 while the chart ships server 26.7.0):
+- line 5   : init container downloads `.../releases/download/26.5.0/sonarqube-webapp.zip` -> mounted over /opt/sonarqube/web
+- line 40  : plugins.install `.../26.5.0/sonarqube-community-branch-plugin-26.5.0.jar`
+- line 53  : sonar.ce.javaAdditionalOpts  `-javaagent:.../sonarqube-community-branch-plugin-26.5.0.jar=ce`
+- line 55  : sonar.web.javaAdditionalOpts `-javaagent:.../sonarqube-community-branch-plugin-26.5.0.jar=web`
+
+## AppSet wiring (cache-buster trap)
+bootstrap/appset-helm.yaml:
+- line 121 : chart version 2026.4.1
+- line 123 : valueURL https://raw.githubusercontent.com/ebpro/gitops/main/helm/releases/sonarqube/values.yaml?v=3
+  -> ArgoCD caches raw valueFiles aggressively. Any values.yaml change MUST bump `?v=3` -> `?v=4`
+     or ArgoCD keeps rendering the OLD values (see AGENTS.md "ArgoCD ignores remote valueFile changes").
+
+## Search internals (why 26.5.0 plugin breaks 26.7 search)
+SonarQube 26.7 resolves components by document UUID and stores per-project auth in `auth_<uuid>`
+shadow docs. The community branch plugin (javaagent + patched webapp) hooks the web/ce search path;
+a 26.5.0 agent predates the 26.7 UUID/shadow-doc model, so `search_projects` returns the right
+`total` but an empty `components[]`.
+
+## Upstream plugin status
+- Latest release: 26.5.0 (no 26.7.0).
+- PR #1280 (mc1arke/sonarqube-community-branch-plugin): open, unmerged, unreviewed, Snyk green.
+  Head f47c931debe54f0ead79801546937ce03acf137f — only known 26.7-compatible code path.
+
+## Access
+- Admin API inside pod: header `Gap-Auth: admin` (SSO break-glass path, see AGENTS.md).
+- ES: `http://localhost:9001/` from within sonarqube-sonarqube-0.
+- DB: psql via CNPG pod `sonarqube-db-1` container `postgres`, db `sonardb`.
+
+## ES index shape — evidence AGAINST "reindex will fix it" (verified 2026-09-12)
+Read-only probe of `components` index inside sonarqube-sonarqube-0 (`http://localhost:9001/components/_mapping` / `_count`):
+- `_count` = 30 (5/5 shards successful, 0 failed → GREEN).
+- mapping `properties` = auth_allowAnyone, auth_groupIds, auth_userIds, indexType, join_components, key, name, qualifier, uuid.
+- `_source: enabled:false` (normal for SonarQube; it reads doc-values).
+The presence of `auth_*` + `join_components` fields is the **26.7 auth-shadow-doc model** — a
+stale/26.5-shaped or corrupted index would NOT carry them. So the index is already correct-shape
+and 1:1 consistent with the 15 DB projects. A reindex rebuilds the SAME 30 docs from the SAME DB
+-> same hydration outcome. **Conclusion: refreshing the ES index will NOT fix the empty list; the
+failure is downstream of ES (Web resolving the 15 UUIDs it already has into 0 objects).**
+
+## No cheap/safe reindex lever in 26.7 (verified 2026-09-12)
+- `GET /api/webservices/list` (top-level key `webServices`) exposes NO full DB->ES reindex action;
+  the only `reindex` key is a narrow per-project ISSUE reindex, not components/projects.
+- Therefore "reindex everything" = delete the ES index dir on the PVC + restart = downtime + risk to
+  branch-plugin structures. STRICTLY worse than the reversible `=web` javaagent test (NEXT-STEPS 1).
+- internal_properties has no reindex/search/es key (queried; 0 rows) — there is no pending trigger.

refresh/sonarqube-267-empty-projects-2026-09-12/FILES.md

@@ -0,0 +1,20 @@
+# FILES — touched / relevant to this job
+
+## Repo files that carry the version pins (the actual fix surface)
+- `helm/releases/sonarqube/values.yaml`
+  - line 5  : webapp.zip 26.5.0 download URL (init container)
+  - line 40 : plugin jar 26.5.0 install URL
+  - line 53 : `sonar.ce.javaAdditionalOpts`  =javaagent ...26.5.0.jar=ce
+  - line 55 : `sonar.web.javaAdditionalOpts` =javaagent ...26.5.0.jar=web
+- `bootstrap/appset-helm.yaml`
+  - line 121: chart version 2026.4.1 (server 26.7.0.124771)
+  - line 123: valueURL ...?v=3  -> bump to ?v=4 whenever values.yaml changes (cache-buster)
+
+## In-cluster artifacts (read-only inspection, NOT git-managed)
+- `/opt/sonarqube/extensions/plugins/sonarqube-community-branch-plugin-26.5.0.jar` (live pod)
+- `/opt/sonarqube/web/**` (overridden by the 26.5.0 webapp.zip init container)
+- `/opt/sonarqube/logs/` : web.log, ce.log, es.log, access.log (observed clean)
+
+## NOT part of this job (leave for owner sessions)
+- `arm-test.yaml`, `helm/releases/pact-broker/values.yaml.backup`,
+  `kubernetes/keycloak-realm/platform-realm-setup-job.yaml.bak` — unrelated stray files.

refresh/sonarqube-267-empty-projects-2026-09-12/GIT.md

@@ -0,0 +1,34 @@
+# Git preservation
+
+## Where this is saved
+- Dedicated NON-main branch (inert to ArgoCD):
+  - repo:   git@github.com:ebpro/gitops.git
+  - branch: preservation/sonarqube-267-empty-projects-2026-09-12
+  - path:   refresh/sonarqube-267-empty-projects-2026-09-12/
+  - base:   origin/main @ 01039ae (created via isolated `git worktree`, shared checkout untouched)
+- Discoverable copy (unversioned): /home/bruno/REFRESH/sonarqube-267-empty-projects-2026-09-12/
+- Claimed via the shared /home/bruno/REFRESH/claim.sh; a row was auto-appended to /home/bruno/REFRESH/INDEX.md.
+
+## Retrieve it (fresh session)
+```
+git -C /mnt/hdd/home/bruno/gitops fetch origin
+git -C /mnt/hdd/home/bruno/gitops show \
+  origin/preservation/sonarqube-267-empty-projects-2026-09-12:refresh/sonarqube-267-empty-projects-2026-09-12/SITREP.md
+# or check the whole folder out to a scratch worktree:
+git -C /mnt/hdd/home/bruno/gitops worktree add /tmp/sq-handoff preservation/sonarqube-267-empty-projects-2026-09-12
+```
+
+## Why a worktree, not main
+At handoff time a CONCURRENT session held a registered worktree
+(`/tmp/wt-open-webui-sso-remediation-2026-09-12`) on the same repo. Per
+/home/bruno/REFRESH/CONCURRENCY.md, the shared primary tree is a race condition — so this job was
+claimed on its own branch/worktree and will publish by PR, never by pushing to main directly.
+
+## Redaction status
+- No plaintext secrets/tokens. `Gap-Auth` is a header NAME; the admin break-glass value is not stored.
+- jdbcSecretName / account.adminPasswordSecretName are Kubernetes Secret NAMES, not values.
+- Vault paths are references only.
+
+## NOT committed (deliberately excluded — foreign to this job)
+- arm-test.yaml, helm/releases/pact-broker/values.yaml.backup,
+  kubernetes/keycloak-realm/platform-realm-setup-job.yaml.bak  -> left for their owner session.

refresh/sonarqube-267-empty-projects-2026-09-12/NEXT-STEPS.md

@@ -0,0 +1,50 @@
+# NEXT-STEPS
+
+## 0. Re-derive the evidence (read-only, from inside the pod)
+```
+# API symptoms
+kubectl -n sonarqube exec sonarqube-sonarqube-0 -c sonarqube -- sh -c \
+ 'curl -s -H "Gap-Auth: admin" "http://localhost:9000/api/components/search_projects?ps=50"'
+kubectl -n sonarqube exec sonarqube-sonarqube-0 -c sonarqube -- sh -c \
+ 'curl -s -H "Gap-Auth: admin" "http://localhost:9000/api/projects/search?ps=50"'
+# ES doc counts (15 real + 15 auth_ shadows expected)
+kubectl -n sonarqube exec sonarqube-sonarqube-0 -c sonarqube -- sh -c \
+ 'curl -s "http://localhost:9001/_cat/indices?v" ; \
+  curl -s "http://localhost:9001/components/_count"'
+# DB truth (15 TRK, private=f)
+kubectl -n sonarqube exec sonarqube-db-1 -c postgres -- \
+ psql -U postgres -d sonardb -c "SELECT count(*), qualifier, private FROM projects GROUP BY qualifier, private;"
+```
+
+## 1. DECISIVE EXPERIMENT (the whole point) — disable the 26.5.0 web-layer agent
+Prove the version-skew hypothesis: render the app WITHOUT the `=web` javaagent (and ideally WITHOUT
+the 26.5.0 webapp.zip override) and re-check `search_projects`. If projects come back -> confirmed.
+
+Two ways, both via Git (NEVER kubectl patch on the live resource):
+  A) Minimal test: temporarily comment out line 55 (`sonar.web.javaAdditionalOpts` =web agent) in
+     helm/releases/sonarqube/values.yaml, bump `?v=3`->`?v=4` on appset-helm.yaml line 123, commit to
+     the preservation branch, and validate render (`argocd app diff`/`helm template`) before merging.
+  B) Cleaner test: also drop the webapp.zip init container (line 5) + its /opt/sonarqube/web mount
+     so the stock 26.7 webapp is used.
+This restarts the SQ pod (brief outage for the team) -> coordinate before merging to main.
+
+## 2. If confirmed, pick the fix
+  - Preferred: pin the SERVER down to match the plugin. Force image 26.5.x in values.yaml
+    (chart default is 26.7 — set `image.tag` explicitly) so plugin 26.5.0 + webapp 26.5.0 align.
+    Lowest risk, keeps branch-plugin features. Remember ?v bump.
+  - Or: upgrade to a 26.7-compatible plugin — none released; only PR #1280 (head f47c931...) is a
+    candidate. Build/track that fork, then bump lines 5/40/53/55 together and ?v bump.
+  - Or: drop the community branch plugin entirely (lose branch features) and use stock 26.7.
+  Decide with Bruno based on whether branch analysis is needed.
+
+## 3. Deferred (from pre-compaction session)
+  - Finish reading the javaagent source in the plugin clone (`/tmp/opencode/sqcb-plugin` — EPHEMERAL,
+    may be gone; re-clone `mc1arke/sonarqube-community-branch-plugin` if needed) to pinpoint the exact
+    26.7 search-path hook. Optional; step 1 gives the faster proof.
+
+## 1b. Do NOT bother with an ES reindex (settled 2026-09-12)
+The tempting "DB good / index corrupt" theory was checked and REJECTED as a fix path. See CONTEXT.md
+"ES index shape". Short version: components index is already 26.7-shaped (auth_*/join_components, 30
+docs = 15 real + 15 auth shadow, GREEN, matches DB 1:1) and there is NO safe reindex API in 26.7 —
+a rebuild would only wipe ES + restart and reproduce the same content. Go straight to step 1
+(disable the `=web` javaagent via Git + ?v bump + restart); it is the reversible, diagnostic test.

refresh/sonarqube-267-empty-projects-2026-09-12/SITREP.md

@@ -0,0 +1,40 @@
+# SITREP — SonarQube 26.7 empty project list
+
+## Symptom
+UI project list is empty. API confirms the backend returns zero resolvable projects even though
+15 projects exist in the DB and in Elasticsearch.
+
+- `GET /api/components/search_projects?ps=50` -> HTTP 200, `paging.total: 15`, but `components: []`
+- `GET /api/projects/search?ps=50`            -> `paging.total: 0`
+- Reproduced twice (plain + UI-style with facets).
+
+## Environment
+- SonarQube **Community Build v26.7.0.124771**, ns `sonarqube`, pod `sonarqube-sonarqube-0` (container `sonarqube`).
+- Community Branch Plugin **26.5.0** installed as a javaagent on BOTH ce and web.
+- DB: CNPG pod `sonarqube-db-1` (container `postgres`), DB `sonardb`. Admin via header `Gap-Auth: admin`.
+- Embedded Elasticsearch 8.19.16 at `http://localhost:9001/` (inside the SQ pod).
+
+## Evidence gathered (read-only)
+- DB `projects`: 15 rows, all `qualifier=TRK`, all `private=f`.
+- ES `components` index: 30 docs = 15 real + 15 `auth_<uuid>` shadow docs; `_source` disabled; index GREEN.
+- DB project UUIDs match the 15 real ES component doc ids **1:1**.
+- Web/CE/access logs clean — no WARN/ERROR, no 500s.
+- So ES itself is healthy and the failure is **downstream of ES**: Web receives 15 UUIDs but resolves 0 project objects.
+
+## Leading hypothesis
+Plugin/webapp **26.5.0** is incompatible with server **26.7.0**'s search internals:
+26.7 changed component/project search to resolve documents by UUID with `auth_` shadow docs;
+the 26.5.0 branch-plugin javaagent patches the web layer and the 26.5.0 webapp.zip is mounted over
+`/opt/sonarqube/web`, so search calls the plugin's old resolution path and returns empty.
+No 26.7.0 plugin release exists upstream (latest is 26.5.0). PR #1280 in the plugin repo is the
+only known 26.7-compatible work and is unmerged/unreviewed.
+
+## Ruled out
+- DB/ES inconsistency (they match 1:1).
+- Index corruption / red indices (GREEN, docs present).
+- Auth/permissions (admin header path, all projects private=f).
+- A transient (reproduced twice, logs clean).
+
+## Not yet done (see NEXT-STEPS)
+- Definitive proof: start Web WITHOUT the `=web` javaagent / WITHOUT the 26.5.0 webapp override and
+  re-check `search_projects`. This is the decisive experiment but requires a GitOps change + pod restart.

refresh/sonarqube-267-empty-projects-2026-09-12/START-SOTA-2026.md

@@ -0,0 +1,49 @@
+# SOTA 2026 start prompt — copy/paste into a fresh opencode session
+
+---
+ROLE: You are diagnosing/fixing a SonarQube GitOps issue on a K3s + ArgoCD + CNPG + Vault platform.
+Evidence-first. Read-only cluster/ES/DB inspection is allowed; direct cluster mutation is NOT —
+all fixes flow through git (ArgoCD auto-syncs main with prune+selfHeal).
+
+CONTEXT FOLDER (read in order):
+  /home/bruno/REFRESH/sonarqube-267-empty-projects-2026-09-12/
+  1. SITREP.md  2. CONTEXT.md  3. FILES.md  4. NEXT-STEPS.md  5. CHECKLIST.md  6. GIT.md
+
+WORKING DIR: /mnt/hdd/home/bruno/gitops   remote: git@github.com:ebpro/gitops.git
+
+PROBLEM: SonarQube 26.7.0.124771 shows an EMPTY project list. API `search_projects` returns
+`total:15` but `components:[]`; `projects/search` returns `total:0`. DB + ES are consistent (15 TRK
+projects, UUIDs match 1:1), indices GREEN, logs clean -> failure is downstream of ES.
+
+LEADING ROOT CAUSE (unconfirmed): version skew — Community Branch Plugin + webapp are pinned to
+**26.5.0** (values.yaml lines 5/40/53/55) while the chart (2026.4.1, appset-helm.yaml line 121)
+ships server **26.7.0**. The 26.5.0 javaagent/webapp predates 26.7's UUID + `auth_` shadow-doc search
+model, so Web resolves 0 objects. No 26.7 plugin release exists; PR #1280 (head f47c931) is the only
+26.7-compatible candidate (unmerged/unreviewed).
+
+DO FIRST:
+  git -C /mnt/hdd/home/bruno/gitops worktree list          # a concurrent session may hold the tree
+  git -C /mnt/hdd/home/bruno/gitops fetch origin
+  # Claim an isolated worktree + branch (do NOT switch the shared tree / never add -A / stash):
+  cd /home/bruno/REFRESH && ./claim.sh sonarqube-267-empty-projects-2026-09-12 origin/main
+  cd /tmp/wt-sonarqube-267-empty-projects-2026-09-12
+
+EXECUTE (NEXT-STEPS.md):
+  1. Re-derive the symptom (read-only curl via Gap-Auth:admin + ES/_count + psql count).
+  2. DECISIVE TEST: render the app WITHOUT the `=web` javaagent (line 55) and WITHOUT the 26.5.0
+     webapp override (line 5) via a Git change + bump appset-helm.yaml line 123 `?v=3`->`?v=4`.
+     Validate with helm template / argocd diff BEFORE merging. Confirm search_projects is non-empty.
+     -> this restarts the SQ pod; COORDINATE the brief outage with Bruno first.
+  3. Pick the permanent fix (align server down to 26.5.x, OR adopt a 26.7-compatible plugin fork,
+     OR drop the plugin) with Bruno; implement in values.yaml; open a PR; merge; verify live.
+
+HARD RULES
+  - Git-push only. Never kubectl patch/apply/delete/exec-write on ArgoCD-owned resources.
+  - Isolate with a git worktree; never checkout/add -A/stash the shared primary tree.
+  - Bump the valueURL `?v=` on every values.yaml change or ArgoCD keeps the stale render.
+  - Publish the fix by PR to main (merge is human-coordinated); keep the handoff on preservation/* branch.
+  - Redact: no secret VALUES in any file; secret NAMES and Vault paths are fine.
+  - If this job references screenshots/attachments: they exceeded the provider size limit and were
+    dropped — re-send smaller/fewer, or re-derive from the read-only commands above.
+
+CLEANUP when done: cd /home/bruno/REFRESH && ./claim.sh --release sonarqube-267-empty-projects-2026-09-12

refresh/sonarqube-267-empty-projects-2026-09-12/WORKDIR.txt

@@ -0,0 +1,29 @@
+JOB: SonarQube 26.7 — empty project list (search_projects total:15 / components:[]) diagnosis
+SESSION: opencode SOTA 2026 handoff
+CREATED: 2026-09-12 (this session)
+
+=== FINISH THE JOB IN ===
+  cd /mnt/hdd/home/bruno/gitops      (== /home/bruno/gitops, same repo/inode)
+  remote: git@github.com:ebpro/gitops.git
+  ArgoCD auto-syncs main (prune+selfHeal). Read-only cluster/ES/DB inspection allowed;
+  NO direct kubectl mutation — all fixes flow through git.
+
+  FIRST, detect partial work from earlier aborted delegations (shared-tree warning):
+    git -C /mnt/hdd/home/bruno/gitops worktree list        # a concurrent session may hold the tree
+    git -C /mnt/hdd/home/bruno/gitops fetch origin
+    git -C /mnt/hdd/home/bruno/gitops log --oneline origin/main -5
+  Do NOT `checkout`/`add -A`/`stash` in the shared primary tree if another worktree is active —
+  use an isolated worktree (see GIT.md / CONCURRENCY.md).
+
+=== THIS HANDOFF COPY (discoverable) ===
+  /home/bruno/REFRESH/sonarqube-267-empty-projects-2026-09-12/
+
+=== GIT-PRESERVED COPY ===
+  repo: ebpro/gitops  branch: preservation/sonarqube-267-empty-projects-2026-09-12
+  path: refresh/sonarqube-267-empty-projects-2026-09-12/   (see GIT.md)
+
+=== MEDIA ATTACHMENT NOTICE ===
+  The originating request included image/file attachments that exceeded the provider size limit
+  and were dropped from context. If the investigation references a screenshot or exported file,
+  it is NOT available here — re-send smaller or fewer files, or re-derive the evidence with the
+  read-only commands in NEXT-STEPS.md.