Support scaling all clusters in an instance based on the metrics of the "primary" cluster

[srwilson]

I work for Sift and we have possibly a unique use case for our Bigtable instances. For each of our instances we have 2 clusters. At a given time one instance is "primary" and serves all live requests (configured via application profiles). The "secondary" instance is mostly idle so cpu usage is low. We want to keep the secondary the same size as the primary instance so that we can switch live traffic to the secondary instance.

This use case does not seem to be supported in the current implementation here. It seems scaling decisions are made independently for each cluster based on the metrics for that cluster.

Our plan is to fork the repo and make a few tweaks. In AutoscaleJob we lookup what is the primary cluster for that instance (by getting app profile data) and use that cluster as the source for metrics for the scaling algorithms. This will work for us but probably couldn't be contributed upstream in its current form.

Is there interest in a feature like this being contributed? Have others expressed interested in functionality similar to this?

cc @rravi-sift

[d4nilos]

This is an interesting use case, answering your question, I believe that this is not a functionality that was requested before (at least not exactly this).

Thinking about it, is this the best approach for you? I mean, does these switches happen too often?
If not, wouldn't it make sense to have a service that requests the secondary cluster to scale up the number of nodes, to the same as the primary, only when it's going to be used? If you do that, the autoscaler would be able to handle the traffic changes after the secondary assumes the handling of the traffic.

You could use the endpoint explained in here to notify the autoscaler that it need extra nodes for a short period of time, to prepare the secondary cluster to assume the load: https://github.com/spotify/bigtable-autoscaler/blob/b0efec1e02310e2dcf3c53048c6528f01b0ebaa0/README.md#does-it-handle-sudden-load-spikes-for-instance-dataflow-jobs-readingwriting-batch-data

[srwilson]

Failing over the secondary does not happen frequently, but when it does happen we need to be able to failover very quickly. Adding a manual step to scale up the secondary cluster, then wait for tablets to be moved to the new nodes would delay our recovery. Also just switching traffic to the secondary and waiting for autoscale to happen would mean very poor performance for an unacceptable length of time.

If we were to put together a PR that adds a field in BigtableCluster for app profile for primary cluster. If this is set, the scaling logic will use that primary cluster from the app profile to scale that cluster. Would you be interested in accepting that or should we continue with our forked repo?