Skip to content

Rewrite data package caching#38

Merged
N00byKing merged 5 commits into
N00byKing:mainfrom
KScl:datapkg_cache_2
Mar 22, 2026
Merged

Rewrite data package caching#38
N00byKing merged 5 commits into
N00byKing:mainfrom
KScl:datapkg_cache_2

Conversation

@KScl

@KScl KScl commented Mar 18, 2026

Copy link
Copy Markdown
Contributor

Rewrites the data package caching system so that it stores game's data packages as separate files in a .datapkg-cache subfolder.

  • We now fetch multiple games' data packages from the server simultaneously, to take advantage of websocket compression that we're now guaranteed to be using. Less requests to the server also means less time waiting for server replies, which does a wonder in speeding this process up.
  • We only load data packages for games that are actually in our current multiworld, instead of everything in the cache. We also don't leave any resident Json::Values in memory after parsing them and adding them to our location and item maps. Both of these likely ease memory concerns.
  • Because we're writing individual files per game instead of one giant cache file, writing them out becomes significantly quicker and isn't affected by how many other games we've cached previously.
  • Previously, if the data package cache got corrupted, which could happen if the process exited mid-write (game exited, crashed, etc.), this caused whatever game using APCpp to crash when attempting to read it. It also meant that there was nothing recoverable from the cache, so we had to redownload every data package; nightmarish for huge asyncs given how long downloading took previously. It now gracefully recovers from this and just redownloads the one game's data from the server if it's been corrupted.

The upshot of this? The async I'm a part of with 131 unique games now only takes about 7 or 8 seconds to start up with a completely empty data package cache, when it took over a minute to do so previously. (Obviously, YMMV; network latency affects this, but this should be even faster on archipelago.gg because we're sending a third of the requests.)

Do note that std::filesystem is used to manage data package file paths and create the directory if it needs to be done. This shouldn't be a problem because APCpp already requires C++17 anyway, but just in case I figured I'd mention it.

@KScl

KScl commented Mar 18, 2026

Copy link
Copy Markdown
Contributor Author

Windows's wide strings got me again, damn it. One moment...

@N00byKing N00byKing left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reasoning for separate files sounds fair to me. Using std::filesystems is also good.

Just some comments, will try to get to testing in the weekend

Comment thread Archipelago.cpp Outdated
Comment thread Archipelago.cpp Outdated
std::min must be wrapped in parenthesis to prevent a macro from taking over
and breaking it on Windows. It also wasn't too happy with size_ts, so we go
back to using ints.
Comment thread Archipelago.cpp Outdated
Comment thread Archipelago.cpp Outdated
Comment thread Archipelago.cpp Outdated
Comment thread Archipelago.cpp Outdated
@N00byKing

Copy link
Copy Markdown
Owner

Thanks!

@N00byKing N00byKing merged commit be711f6 into N00byKing:main Mar 22, 2026
4 checks passed
@KScl KScl deleted the datapkg_cache_2 branch March 22, 2026 16:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants