Configure Horizon and OAP by environment variable, mount a file only on request

[wu-sheng]

Follow-up to #194.

Horizon's image ships a complete /app/horizon.yaml in which all 62 fields are ${HORIZON_*:default} placeholders. The chart was mounting a ConfigMap over it on every install — and that mount replaces the file rather than merging with it. So every token the chart didn't carry over was gone, and the matching HORIZON_* variable stopped working, silently.

The practical consequence: HORIZON_AUTH_LOCAL_USERS supplied from a Secret did nothing. That is the first thing an operator tries.

What changes

The chart no longer mounts anything by default. It sets what it can compute as plain environment variables and leaves the image's file intact, so every other field stays settable:

variable	when
HORIZON_SERVER_PORT	always, from ui.service.internalPort
HORIZON_OAP_QUERY_URL	always
HORIZON_OAP_ADMIN_URL	when oap.ports.admin is set
HORIZON_OAP_ZIPKIN_URL	when oap.ports.zipkin-query is set
HORIZON_PUBLIC_URL	when an ingress with hosts is configured

Structured settings take JSON in a single variable, so a Secret covers users, LDAP, SSO and the RBAC table with no file anywhere:

ui:
  envFromSecret: horizon-secrets   # holds HORIZON_AUTH_LOCAL_USERS

ui.config remains as the third mechanism, for pinning a field regardless of the environment. It is now opt-in: no ConfigMap is created and nothing is mounted until it is set. When it is set, the chart merges its computed values in as tokens, so OAP stays reachable.

OAP gains the same shape. It had oap.env — a plain map that cannot express valueFrom — and no envFrom at all, so a storage credential could only be a literal in values or a file mount. Adds oap.extraEnv (a list, so entries can carry valueFrom) and oap.envFromSecret, both applied to the Deployment and the init Job, which needs the same storage credentials.

oap.config stays as it was, and the docs now say why the two components differ: OAP reads real files from /skywalking/config for log4j2.xml and the OAL/MAL rules, so a ConfigMap is the only way to supply those. Horizon doesn't, so it doesn't need one.

Falls out for free

Hot reload starts working — a subPath mount is never updated in place by Kubernetes, so Horizon's file watcher could never fire — and there is no longer a checksum/config annotation forcing a pod recreate on every config change.

E2E

The suite's login moves from ui.config to HORIZON_AUTH_LOCAL_USERS, so CI exercises the path the documentation recommends rather than the escape hatch.

A sixth cell, e2e-config-override.yaml, covers the ConfigMap path for both components — otherwise the opt-in mount would ship untested, and it is the riskier path. It asserts the mounts exist, that Horizon read templates.mode from the mounted file, that OAP's log4j2.xml landed at /skywalking/config, and — the one that matters — that OAP is still reachable while a file is mounted, which is what breaks if the chart ever stops merging its computed values back in. Deliberately cheap: BanyanDB standalone, no istio, no bookinfo, no traffic.

One bug caught in review, worth recording

The e2e login initially used a folded scalar whose continuation line was indented one space deeper than the first. In YAML a more-indented line inside a folded scalar is literal, so the newline survived into the env var; substituted into the image's users: ${HORIZON_AUTH_LOCAL_USERS:[]} at indent 4 it produced an under-indented flow sequence.

Horizon parses with the npm yaml package, which rejects that, and the boot path doesn't catch the error — so the UI would have crash-looped and every cell failed. PyYAML accepts the same document, which is how it passed a first check. Verified against the real parser now.

Docs

configure.md rebuilt around the three mechanisms; logins.md shows a Secret alone configuring logins; oap-configuration.md documents the new OAP values; the values reference carries the new rows.

Several pages asserted the opposite of current behaviour rather than merely lagging it — "HORIZON_TRUST_PROXY is inert under this chart", "the HORIZON_SESSION_COOKIE_SECURE env var will not do it". Both work now, and those pages led operators down a dead end. Corrected in ui-service-and-ingress.md, tls.md and ui-and-login.md.

Also corrects the values.yaml claim that /data holds the login audit — Horizon 1.0.0's audit backend is Postgres or nothing; /data holds the wire debug log.

Verification

helm lint clean; all three storage backends render; the ui.config path renders with the OAP tokens merged; menu paths resolve with no orphans; no broken links; workflows and all six e2e configs parse.