Skip to content

Add new 100qps-40klogs staging list#37

Merged
rgdd merged 1 commit into
mainfrom
add-higher-perf-staging-list
Mar 26, 2026
Merged

Add new 100qps-40klogs staging list#37
rgdd merged 1 commit into
mainfrom
add-higher-perf-staging-list

Conversation

@rgdd
Copy link
Copy Markdown
Collaborator

@rgdd rgdd commented Mar 26, 2026

We use non-overlapping lists, asking witness operators to configure the union of all lists they can accommodate (starting from lower performance profiles and moving upwards). As maintainers we allocate logs to the most appropriate lists based on performance profile, so that the list in question doesn't fill up too quickly (i.e., to ensure that it is a good spending of witness resources when adding a log to a list).

The alternative would have been to have overlapping lists, e.g., where the 100qps list has a 10qps list as a sublist; and the 10qps list might in turn have a 1qps list as a sublist. A participating witness would then choose the highest performance profile they can accommodate, as opposed to having to configure the union of the three different lists.

Both should work and it's mostly a matter of documentation. But a nice property of non-overlapping lists is that they compose well as unions. No questions about committed capacity given a list of lists.

The property of "any witness which commits to reading list X is compatible with my log" is also nice for non-overlapping lists. The equivalent of this property with overlapping lists would have been "any witness which commits to reading list X or higher is compatible with my witness", which seems slightly more complex to explain and maintain.

We might want a higher performance profile in the future, but we're starting with 100qps-40klogs now in staging to see how it is received.

These additions were co-authored with Al, thanks!

We use non-overlapping lists, asking witness operators to configure the
union of all lists they can accommodate (starting from lower performance
profiles and moving upwards).  As maintainers we allocate logs to the
most appropriate lists based on performance profile, so that the list in
question doesn't fill up too quickly (i.e., to ensure that it is a good
spending of witness resources when adding a log to a list).

The alternative would have been to have overlapping lists, e.g., where
the 100qps list has a 10qps list as a sublist; and the 10qps list might
in turn have a 1qps list as a sublist.  A participating witness would
then choose the highest performance profile they can accommodate, as
opposed to having to configure the union of the three different lists.

Both should work and it's mostly a matter of documentation.  But a
nice property of non-overlapping lists is that they compose well as
unions.  No questions about committed capacity given a list of lists.

The property of "any witness which commits to reading list X is
compatible with my log" is also nice for non-overlapping lists.  The
equivalent of this property with overlapping lists would have been "any
witness which commits to reading list X or higher is compatible with my
witness", which seems slightly more complex to explain and maintain.

We might want a higher performance profile in the future, but we're
starting with 100qps-40klogs now in staging to see how it is received.

These additions were co-authored with Al, thanks!
@rgdd rgdd merged commit d5e5dec into main Mar 26, 2026
4 checks passed
@AlCutter AlCutter deleted the add-higher-perf-staging-list branch March 27, 2026 10:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants