[Due for payment 2026-04-19] [Onyx Audit] Performance improvements to SQLiteProvider #80243

Activity

1. fabioh8010 commented on Jan 22, 2026

   

   fabioh8010

   on Jan 22, 2026

   ContributorAuthor

   More actions

   I will work on this one starting tomorrow.
2. mountiny

   assigned

   fabioh8010

   on Jan 22, 2026
3. mountiny

   added this to [#whatsnext] #qualityon Jan 22, 2026
4. mountiny

   moved this to HIGH in [#whatsnext] #qualityon Jan 22, 2026
5. mountiny

   added

   DailyKSv2KSv2

   Task

   on Jan 22, 2026
6. fabioh8010 commented on Jan 23, 2026

   

   fabioh8010

   on Jan 23, 2026

   ContributorAuthor

   More actions

   Updates:

   • Started checking my old commit, and will begin measurements again with baseline vs my changes but without the small batch condition logic in multiGet.
7. fabioh8010 commented on Jan 26, 2026

   

   fabioh8010

   on Jan 26, 2026

   ContributorAuthor

   More actions

   Updates:

   • Started to get the baseline measurements and ended up taking a closer look at a old PR of mine that attempts to fix the performance metrics. I think I was finally able to understand it and get something workable (at least for mobile), so I might be able to ship this improvement too.
   • With the performance metrics fixed it will be easier to get these metrics, which I use a complicated way currently.
   • I'm using my playground branch to measure.
8. mountiny

   added a parent issue on Jan 26, 2026

   • [Tracking] Onyx Improvements #80355
9. fabioh8010 commented on Jan 29, 2026

   

   fabioh8010

   on Jan 29, 2026

   ContributorAuthor

   More actions

   Updates:

   I was able to finish measurements today on both baseline (main) and solution branch.

   These are the improvements:

   • multiGet optimization (chunked / temp-table join): Reduce query parsing and variable binding overhead from very long IN (?, ?, ...) clauses, and improve scalability for large batches. The initial solution was to perform the current IN clauses logic on datasets below 100 records, and the new temp-table solution on datasets above it.
   • Removing json() wrappers from SQL statements: Eliminate redundant json() calls around parameters, which parse and re-encode JSON strings inside SQLite.

   Operation	Records	Before (ms)	After (ms)	Difference (%)
   multiGet	100	14.24	18.66	+31.02% (worse than current logic)
   	500	32.24	32.53	+0.90% (break-even point)
   	1000	46.40	41.20	−11.20%
   	10000	407.03	308.08	−24.31%
   multiMerge	100	12.37	11.39	−7.92%
   	1000	78.80	68.45	−13.13%
   	10000	1884.70	1006.36	−46.60%
   multiSet	100	7.51	7.25	−3.46%
   	1000	59.13	52.58	−11.09%
   	10000	553.57	460.33	−16.84%

   For multiGet it looks like the temp-table solution is worse than current logic below 500 records, but it start to be more performant above 500 records, so I adjusted my solution to use the 500 threshold.

   I will perform some testing with real scenarios around the app in a heavy account to see if I catch bugs, if everything is okay we can move with a proposal.
10. melvin-bot commented on Feb 3, 2026

    

    melvin-botbot

    on Feb 3, 2026 – with Melvin Bot

    More actions

    @fabioh8010 Eep! 4 days overdue now. Issues have feelings too...
11. melvin-bot

    added

    Overdue

    on Feb 3, 2026
12. fabioh8010 commented on Feb 6, 2026

    

    fabioh8010

    on Feb 6, 2026

    ContributorAuthor

    More actions

    Updates:

    • Posted proposal for internal review
13. melvin-bot

    removed

    Overdue

    on Feb 6, 2026
14. fabioh8010 commented on Feb 9, 2026

    

    fabioh8010

    on Feb 9, 2026

    ContributorAuthor

    More actions

    Updates:

    • Posted proposal for external review -> https://expensify.slack.com/archives/C05LX9D6E07/p1770627746422309
15. 24 remaining items

    Load more

16. suneox commented on Apr 17, 2026

    

    suneox

    on Apr 17, 2026

    Contributor

    More actions

    @fabioh8010 I reviewed this PR, so Can I be eligible to receive payment for this one? Thanks!
17. fabioh8010 commented on Apr 17, 2026

    

    fabioh8010

    on Apr 17, 2026

    ContributorAuthor

    More actions

    cc @mountiny
18. mountiny

    reopened this on Apr 19, 2026
19. mountiny

    added

    Awaiting PaymentAuto-added when associated PR is deployed to productionAuto-added when associated PR is deployed to production

    on Apr 19, 2026
20. melvin-bot

    assigned

    mallenexpensify

    on Apr 19, 2026
21. melvin-bot commented on Apr 19, 2026

    

    melvin-botbot

    on Apr 19, 2026 – with Melvin Bot

    More actions

    Triggered auto assignment to @mallenexpensify (Awaiting Payment)
22. mountiny

    changed the title [-][Onyx Audit] Performance improvements to SQLiteProvider[/-] [+][Due for payment 2026-04-19] [Onyx Audit] Performance improvements to SQLiteProvider[/+] on Apr 19, 2026
23. mountiny

    added

    DailyKSv2KSv2

    and removed

    WeeklyKSv2KSv2

    ReviewingHas a PR in reviewHas a PR in review

    on Apr 19, 2026
24. melvin-bot commented on Apr 19, 2026

    

    melvin-botbot

    on Apr 19, 2026 – with Melvin Bot

    More actions

    Skipping the payment summary for this issue since all the assignees are employees or vendors. If this is incorrect, please manually add the payment summary SO.
25. mountiny

    assigned

    suneox

    on Apr 19, 2026
26. mallenexpensify commented on Apr 20, 2026

    

    mallenexpensify

    on Apr 20, 2026

    Contributor

    More actions

    Payment Summary

    Contributor+: @suneox due $125 via NewDot

    $125 cuz no tests. I know some onyx reviews lately were $75 but we didn't standardize on which ones. If we want to do so, I need more deets)

    👍React with 👍1Reacted by Cong Pham
27. mallenexpensify

    closed this as completedon Apr 20, 2026
28. garrettmknight commented on Apr 27, 2026

    

    garrettmknight

    on Apr 27, 2026

    Contributor

    More actions

    $125 approved for @suneox