jesse/blocked clusters

api/v1/adxcluster_types.go

@@ -82,6 +82,32 @@ type ADXClusterFederationSpec struct {
 	//+kubebuilder:validation:Pattern=^(\d+h)?(\d+m)?(\d+s)?$
 	// If role is "Federated", specifies the ADX cluster's heartbeat table TTL.
 	HeartbeatTTL *string `json:"heartbeatTTL,omitempty"`
+
+	//+kubebuilder:validation:Optional
+	// Specifies partition clusters that should be excluded from federation macros.
+	BlockedClusters *ADXClusterFederationBlockedClustersSpec `json:"blockedClusters,omitempty"`
+}
+
+type ADXClusterFederationBlockedClustersSpec struct {
+	//+kubebuilder:validation:Optional
+	// Static list of partition cluster endpoints to exclude from macro generation. Values are case-insensitive and trimmed of trailing slashes before comparison.
+	Static []string `json:"static,omitempty"`
+
+	//+kubebuilder:validation:Optional
+	// Optional Kusto function that returns additional endpoints to block. The function should return a tabular result with a string column named "ClusterEndpoint" (or "Endpoint").
+	KustoFunction *ADXClusterFederationBlockedClustersFunctionSpec `json:"kustoFunction,omitempty"`
+}
+
+type ADXClusterFederationBlockedClustersFunctionSpec struct {
+	//+kubebuilder:validation:Required
+	//+kubebuilder:validation:Pattern=^[a-zA-Z0-9_]+$
+	// Database containing the Kusto function.
+	Database string `json:"database"`
+
+	//+kubebuilder:validation:Required
+	//+kubebuilder:validation:Pattern=^[A-Za-z_][A-Za-z0-9_]*$
+	// Name of the Kusto function returning blocked cluster endpoints.
+	Name string `json:"name"`
 }
 
 type ADXClusterFederatedClusterSpec struct {

docs/adxcluster-controller.md

@@ -76,6 +76,19 @@ Federation mode: `Partition` (local storage + heartbeat to hub) or `Federated` (
 - `heartbeatDatabase`, `heartbeatTable`: where partitions write
 - `heartbeatTTL`: freshness window (default `1h`)
 
+#### Blocked partition list (Federated hubs)
+`spec.federation.blockedClusters` lets hub clusters exclude quarantined or misbehaving partitions from macro generation
+without editing the partition CRDs. The controller merges two sources into a normalized block list (lower-cased, trimmed
+of whitespace and trailing slashes):
+
+- `blockedClusters.static`: literal list of partition endpoints to suppress.
+- `blockedClusters.kustoFunction`: optional break-glass function looked up in the specified `database`/`name`. The
+  function must return a tabular result with a string column named `ClusterEndpoint` (preferred) or `Endpoint`. Missing
+  functions are treated as warnings; other errors halt the reconciliation so ops teams can inspect the failure.
+
+After fetching heartbeats the controller filters any partitions whose endpoints match the block list, logs how many
+entries were removed vs. matched, and proceeds with macro creation using the remaining schema.
+
 ### `criteriaExpression`
 Optional CEL expression against operator cluster labels. Empty/missing = `true`. Errors/false → reconciliation blocked. Example: `labels["geo"] == "eu" && labels.has("tier")` keeps the object scoped to European, tiered operators. See the [CEL language spec](https://opensource.google/projects/cel) for expression syntax.
 
@@ -91,7 +104,8 @@ Authentication: `DefaultAzureCredential` for local cluster; switches to `managed
 Failures to push heartbeats surface in the operator logs (`adxcluster-controller` logger) alongside the CRD name; reconcile once connectivity or permissions are restored.
 
 ### Federated Hubs  
-Every 10min: query heartbeat table (`WHERE Timestamp > ago(heartbeatTTL)`) → extract partition schemas → ensure OTLP hub tables exist → generate cross-cluster functions.
+Every 10min: query heartbeat table (`WHERE Timestamp > ago(heartbeatTTL)`) → extract partition schemas → optionally
+filter out any endpoints listed in `spec.federation.blockedClusters` (static values plus any returned by the Kusto function) → ensure OTLP hub tables exist → generate cross-cluster functions. Filtering happens before schema aggregation so blocked partitions contribute neither tables nor functions.
 
 Heartbeat table schema: `Timestamp:datetime, ClusterEndpoint:string, Schema:dynamic, PartitionMetadata:dynamic`
 
@@ -184,8 +198,18 @@ spec:
     heartbeatDatabase: "FleetDiscovery"
     heartbeatTable: "Heartbeats"
     heartbeatTTL: "2h"
+    blockedClusters:
+      static:
+        - "https://rogue-partition.kusto.windows.net"
+      kustoFunction:
+        database: FleetDiscovery
+        name: GetBlockedPartitions
 ```
 
+The static list handles known quarantined clusters, while the `GetBlockedPartitions()` function can emit emergent
+endpoints discovered by external tooling. The controller merges both sources and keeps deduplicated, normalized entries
+for filtering during macro reconciliation.
+
 ### Gate Reconciliation with Criteria
 Restrict reconciliation to operators that carry matching labels.
 ```yaml

docs/crds.md

@@ -51,6 +51,33 @@ spec:
       geo: "EU"
       location: "westeurope"
 ```
+
+**Federated hub example with blocked clusters (static + dynamic):**
+```yaml
+apiVersion: adx-mon.azure.com/v1
+kind: ADXCluster
+metadata:
+  name: global-hub
+spec:
+  clusterName: global-hub
+  endpoint: "https://global-hub.kusto.windows.net"
+  role: Federated
+  federation:
+    heartbeatDatabase: "FleetDiscovery"
+    heartbeatTable: "Heartbeats"
+    heartbeatTTL: "2h"
+    blockedClusters:
+      static:
+        - "https://legacy-partition.kusto.windows.net"
+        - "https://quarantined-partition.kusto.windows.net/"  # trailing slashes ignored
+      kustoFunction:
+        database: FleetDiscovery
+        name: GetBlockedPartitions
+```
+The controller normalizes each endpoint (lower case and no trailing slash) and merges the static list with the results of
+`GetBlockedPartitions()` before generating macros. Any partition whose heartbeat endpoint matches the merged list is
+excluded from schema fan-out.
+
 **Key Fields:**
 - `clusterName`: Name for the ADX cluster.
 - `endpoint`: Existing ADX cluster URI (omit to provision new). When set, the controller mirrors this value into status
@@ -61,7 +88,34 @@ spec:
   `resourceGroup`, `location`, `skuName`, and `tier` explicitly—the controller no longer auto-detects or mutates these
   values.
 - `role`: `Partition` (default) or `Federated` for multi-cluster.
-- `federation`: Federation/partitioning config for multi-cluster.
+- `federation`: Federation/partitioning config for multi-cluster. Federated hubs can optionally specify
+  `federation.blockedClusters` to exclude rogue or quarantined partitions from macro generation. The block list accepts a
+  static array of endpoints as well as a break-glass Kusto function (returning `ClusterEndpoint` or `Endpoint` columns)
+  whose results are normalized (trimmed, case-insensitive, trailing slashes removed) before filtering the partition
+  schema set.
+
+### Creating a blocked cluster function in Kusto
+The optional `blockedClusters.kustoFunction` expects a function that returns a table containing either a
+`ClusterEndpoint` or `Endpoint` string column. You can create one in the heartbeat database with a command like:
+
+```kusto
+.create-or-alter function with (docstring="Return partitions blocked from federation", folder="FleetSafety") GetBlockedPartitions()
+{
+  let manualOverrides = datatable(ClusterEndpoint:string)
+  [
+    "https://legacy-partition.kusto.windows.net",
+    "https://maintenance-partition.kusto.windows.net"
+  ];
+  FleetSafetyBlockedPartitions
+  | project ClusterEndpoint = tostring(ClusterEndpoint)
+  | union manualOverrides
+  | distinct ClusterEndpoint
+}
+```
+
+- `FleetSafetyBlockedPartitions` can be any table you manage (for example, populated via Azure Monitor alerts or manual entries).
+- The function may also emit an `Endpoint` column; the controller checks both and trims whitespace before use.
+- Missing functions are logged as warnings, while other Kusto errors block reconciliation so operators can investigate.
 
 **Status highlights:**
 - `status.endpoint`: Observed Kusto endpoint used by dependent components (mirrors `spec.endpoint` in BYO mode).

kustomize/bases/adxclusters_crd.yaml

@@ -101,6 +101,37 @@ spec:
                 description: Supports cluster partitioning. Only relevant if Role
                   is set.
                 properties:
+                  blockedClusters:
+                    description: Specifies partition clusters that should be excluded
+                      from federation macros.
+                    properties:
+                      kustoFunction:
+                        description: Optional Kusto function that returns additional
+                          endpoints to block. The function should return a tabular
+                          result with a string column named "ClusterEndpoint" (or
+                          "Endpoint").
+                        properties:
+                          database:
+                            description: Database containing the break-glass function.
+                            pattern: ^[a-zA-Z0-9_]+$
+                            type: string
+                          name:
+                            description: Name of the Kusto function returning blocked
+                              cluster endpoints.
+                            pattern: ^[A-Za-z_][A-Za-z0-9_]*$
+                            type: string
+                        required:
+                        - database
+                        - name
+                        type: object
+                      static:
+                        description: Static list of partition cluster endpoints to
+                          exclude from macro generation. Values are case-insensitive
+                          and trimmed of trailing slashes before comparison.
+                        items:
+                          type: string
+                        type: array
+                    type: object
                   federatedTargets:
                     description: If role is "Partition", specifies the Federated cluster(s)
                       details for heartbeating.