Come say hi in the SIBR discord! Feedback is welcome in #econ-spreadsheets (projects role) or #:moneybag:-economy (research role). Currently maintained by Frostbird#9525
In short, this repo processes data from the Blaseball.com/database/ REST API and turns it into snack-relevant snacks. These stats inform people how best to build their snack packs and who to idol. To this end, it's important that the data provided is accurate, comprehensive, prompt, and stable.
Stability: This repository is deployed on the SIBR server as a docker container. This provides excellent stability. The files Dockerfile and requirements.txt are required for the container to build properly. The repository is also set up to rebuild and update the container after every push to the master branch.
Prompt: Being deployed on a server means that updates can be processed 24/7, which is one piece of being prompt. The other piece is having the updates as soon as they are needed. The file snack_waiter.py listens to the streamData SSE endpoint for the end of each day's games, siestas, and elections. On deployment of the Docker container, run_waiter.sh is executed which starts the Waiter. When the relevent events happen, Waiter triggers update_all.py which then runs all of the modules that actually pull and process the needed data.
Comprehensive: All snacks have their income calculated by these many scripts (except popcorn... maybe to come later!). This requires getting stats about all hitters, pitchers, and weathers from each day of blaseball. Fortunately, The Game Band have given the community the API to get this data in a lightweight, programmatic way.
Accurate: Accuracy is a constantly-moving goalpost due to the rapidly-changing nature of blaseball. New mechanics are added all the time that change the calculations for optimizing one's earnings. Stat calculations are best done in a general, extensible way rather than shoehorning in exceptions for specific players or teams, as the specific exceptions could change many times throughout a season. For very specific details, see the comments inside each of the processing modules. A rough overview of each module is provided below.
statsheets.py - This is the main workhorse. Each day, this module crawls through the each game's statsheets to get each team's statsheets to get each player's statsheets. The player statsheets are then split into pitchers vs. hitters, various statsheets are discarded for a myriad of reasons, and duplicate statsheets are merged. Then, each player's stats for the day are saved to a SQLite database to be processed further.
hitterstats.py - This takes the stats generated by statsheets.py and cooks it into a more useful form to use in the spreadsheets. Some of these stats require using the entire seasons' worth of data, such as H/PA or PA/G, while others require the most recent game, such as lineup_size. In addition, this script determines whether a hitter is eligible to earn money and whether they have a payout multiplier. Since the main goal of the economy spreadsheets is to make it clear which idols are optimal for various snack packs, these last two details are extremely important additions. Once the data is in a final form, it is posted to the All Hitters spreadsheet via a google service account and the gspread module. The Hitting Leaders and Hitting Future Income spreadsheets depend entirely on the data in the All Hitters sheet. Before the upload, the data is saved into a table in the sqlite database, but it is completely overwritten every time this module is run. The table serves little function other than to provide an easy means of sorting players by the desired field (Team, in this case).
pitcherstats.py - This module is analogous to hitterstats.py for pitchers. It takes the stats generated by statsheets.py and cooks it into a more useful form to use in the spreadsheets. Basically all pitchers stats are calculated at the day level in statsheets.py, though. Mainly, this script determines whether a hitter is eligible to earn money and whether they have a payout multiplier. Once the data is in a final form, it is posted to the All Pitchers spreadsheet via a google service account and the gspread module. The Pitching Leaders and Tomorrow's Pitchers spreadsheets depend almost entirely on the All Pitchers sheet. The data is saved into a table in the sqlite database, but it completely overwritten every time this module is run. The table serves little function other than to provide an easy means of sorting players by the desired field (Team, in this case).
tomorrowpitchers.py - Pitchers do not pitch every game. During the season, only 24 pitchers will be the starting pitchers for each day, and this number shrinks in the playoffs. Users with pitching snacks often swap pitching idols to maximize their profits, so it's important to inform these users of their choices for the next game. This script pulls all of the pitchers for the upcoming games to put on the Tomorrow's Pitchers spreadsheet. The stats associated with these pitchers don't need to be calculated; they are simply pulled from the All Pitchers sheet. This module does add a few interesting stats relevant only to tomorrow's games, though, like the website-calculated odds of winning and whether this player could be faxed out of the game.
weathersnacks.py - Weather snacks are relatively simple to accommodate. Currently this module simply counts the occurances of each snack getting a payout and puts it into the Weather Snacks sheet. This is done with simple feed queries, so no game-by-game processing is needed! Each snack has its own quirks that slightly affect the difference between activations and payouts. In the future, it would be nice to expand this module and spreadsheet to show number of games/innings per weather type. Also, it might be wise to add all weather types to the spreadsheet to prepare for new weather snacks.
Server Deployment - This code is deployed on the SIBR server and can be monitored with Portainer. The Elder Servorcerers of SIBR can help if a problem arises related to hosting to code. A working commit will automatically update the docker container, so even someone without access to the Portainer can update this repository if they have commit access. With access to the Portainer, waiter.log can be checked to see the last time the update pipeline was run. The Portainer log can also be useful to see error messages, which is especially useful if a bad commit is pushed or if the container stops for some reason.
Additionally, https://snacks.sibr.dev redirects to the snacks spreadsheet, though this redirect must be updated over time.
Updating for new seasons - It is quite simple to update this repository for each season. The spreadsheet ID for the new season must be added to update_all.py in the spreadsheet_ids dictionary. This dictionary is also present in hitterstats.py, pitcherstats.py, tomorrowpitchers.py, and weathersnacks.py, but these only need updated if a maintainer wants to run a specific module on its own. It is advised to keep these update to date as well, though.
The redirect can be changed on the SIBR server in /srv/docker/nginx-general/snacks/index.html. The old page may get stuck in your browser's cache. Some workarounds are: (1) Clear your cache, (2) Add a '?' to the end of the url
Troubleshooting/What not to do
- Firstly, never try to run updates while games are in progress. This has a chance of throwing errors. It's best to test and deploy new commits after the day's games are finished.
- Secondly, due to the lack of record of incinerations, if games are not processed before the next one starts, the database could calculate incorrect lineup sizes, which can have significant impact on the
Hitting Future Incomespreadsheet. For the same reason, it is very unadvised to delete the database file and start fresh during or after a season. That should only be done as a last resort. As a final caveat to this, runningupdate_all.pylocally to test the pipeline before commiting will probably put some incorrect data into the spreadsheets. If this is done, it is best to then manually run the SIBR server instance ofupdate_all.pyvia Portainer to correct this. - Third, bad commits will occasionally break the Docker container. To fix this, first fix the problems with the commit, test it locally, and push a fix. Once the commit has cleared the github actions and are sitting on Docker Hub, wait a few minutes to see if the Docker container comes back to life. If it does not, contact a SIBR Servorcerer and politely ask them to recreate the container. Remember that Waiter will not trigger an update while games are in progress, so if it is recreated during this period, wait until the games are finished to see if the fix was successful.