the appender gate is about batching, not about a disk - #9
Merged
Merged
Conversation
The test that says an appender beats INSERT asked for twenty times and got eighteen on a CI runner, where a commit costs twenty five milliseconds of a disk shared with everybody else and the appender's single commit is most of what it spends. The same comparison is 150 times on a laptop, and it rises with the row count either way, because one commit is one commit however many rows it carries. Five is the gate now, which is the number that says the rows were batched rather than committed one at a time, and is not a number a slow disk can take away. Raising the row count instead would buy a wider margin at the price of seconds of INSERT per run, since that half is three seconds at two hundred rows here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test_appending_beats_inserting_by_the_margin_that_makes_it_worth_havingfailed on main: it asks for twenty times and got eighteen, on a runner where 200INSERTstatements took 524 ms and the appender took 28. Nothing regressed. A commit on a shared CI disk costs about twenty five milliseconds, the appender spends one of them, and that single commit is most of its 28 ms, so the ratio there is eighteen where it is 150 on a laptop.The gate is five now. What the test is for is that rows go in batched rather than a commit at a time, and five says that on any disk, where twenty says it only on a disk nobody else is using. The number that would have made twenty comfortable is a bigger row count, and that half costs three seconds of
INSERTat two hundred rows on this machine already, which is not worth buying a margin with.Local:
ruff check,ruff format --check, andpytest tests/test_appender.py, green.