Proposal for feature: Releases

[Datron]

Proposal

Customers are using experiments to stagger a single variant vs control experiment till 50% and then stabilizing the change if they like the outcome(s) of the experiment. A requested feature from users is to be able to stagger the experimental variant to 100%.

Could we re-use the concept of experiments for this?

• Releases are added as an experiment_type like default and delete_overrides
• The actions of create, start, pause, conclude, revert, discard and ramp are the same
• The corresponding states for these actions are also the same

Though experiments and releases seem to map well, it would benefit to see them as two separate features because

• Releases and experiments compare variant and control differently, experiments do an equal population control comparison whereas releases would set a baseline or threshold and an outcome and check if the variant moves towards the outcome set
• The bucketing algorithm for both would be different
• Users may want to delete_overrides with a Release or perform other experiment actions through a release
• The interaction a release has with other releases would be different than experiments

Table structure

The APIs and table structure of releases would very similar as experiments

CREATE TABLE IF NOT EXISTS releases (
    id bigint PRIMARY KEY,
    description TEXT NOT NULL,
    change_reason TEXT NOT NULL,
    created_at timestamp with time zone DEFAULT CURRENT_TIMESTAMP NOT NULL,
    created_by text NOT NULL,
    last_modified timestamp with time zone DEFAULT now() NOT NULL,
    started_at TIMESTAMP,
    started_by TEXT,
    name text NOT NULL,
    override_keys {replaceme}.not_null_text [] NOT NULL,
    status public.release_status_type NOT NULL,
    traffic_percentage integer NOT NULL,
    context json NOT NULL,
    variants json NOT NULL,
    last_modified_by text DEFAULT 'Null'::text NOT NULL,
    chosen_variant text,
    metrics JSON DEFAULT '{"enabled": false}'::json NOT NULL,
   CONSTRAINT experiments_traffic_percentage_check CHECK ((traffic_percentage >= 0))
);

[Datron]

Things to discuss:

Releases interaction

Lets take an example, suppose the following releases exist

[[overrides]]
_context_ = { os = "android", client_id="juspay"}
a = 1
b = 2

[[overrides]]
_context_ = { os = "android"}
a = 2
c= 2

Do the releases interact the same way as experiments where we allow overrides overlap or a more generic or specific context?

Any other cases where a release would lead to an ambiguous overrides

[mahatoankitkumar]

@Datron i guess you meant this?

[[overrides]]
_context_ = { os = "android"}
a = 2
c= 2