Reduce repository size by cleaning deleted large files from git history

[bendichter]

The repository currently clones at ~250 MB due to large binary files that were deleted but remain in git history.

Largest deleted blobs still in history:

Size	File
63 MB	data/A2929-200711/A2929-200711.spk.1
46 MB	data/A2929-200711/A2929-200711.eeg
35 MB	docs/examples/nwb-files/Mouse32-140822.nwb
23 MB	data/A2929-200711/A2929-200711_1_analogin.dat
22 MB	data/A2929-200711/A2929-200711.spk.2
17 MB	data/A2929-200711/A2929-200711.fet.1
14 MB	data/A2929-200711/A2929-200711_1.csv
14 MB	.doctrees/environment.pickle
6 MB	data/A2929-200711/A2929-200711.fet.2
2 MB	docs/banner_logo.png
2 MB	docs/examples/nwb-files/sub-4_ses-4_ecephys+image.nwb

Suggested approach:

1. Use git filter-repo to purge these deleted files from history
2. All collaborators would need to re-clone after the history rewrite

This would reduce clone size from ~250 MB to ~50 MB.

[bendichter]

Here's the exact command to run (requires git-filter-repo):

git filter-repo \
  --path data/A2929-200711/ \
  --path docs/examples/nwb-files/ \
  --path docs/banner_logo.png \
  --path .doctrees/ \
  --invert-paths

After running this, all collaborators would need to re-clone.

[gviejo]

That is definitely doable.

[billbrod]

Hey Ben, popping in here to ask if you have experience using git filter-repo on a library like this. I'm also planning on doing this for plenoptic (which has an exceedingly large repo due to the history of jupyter notebooks that we no longer use), but am wary of messing with the git history. I have used git filter-repo on personal projects, but that feels lower stakes. Have you used filter-repo on a library with multiple contributors and, if so, were there any difficulties, or is the only issue making sure that all contributors re-clone (and, presumably, making sure permissions are set so they cannot force push from their old copies, which would undo the git surgery done by filter-repo).

[bendichter]

@billbrod my experience is similar- I've never used filter-repo on a large project, only a personal one. I'm sure it will create some friction for users that have less experience with the nuances of git. IMO it's worth it, but it's definitely a judgement call worth considering the pros and cons.

[billbrod]

As an alternative to git surgery: if the main problem is that contributors will be cloning a large repo, we could update contributor guidelines to give them instructions on how to use git clone --filter. I haven't played around with this yet, but it looks like git filter --filter=blob:limit=<size> is what we want, as it "will filter out all blobs of size at least <size>"

[h-mayorquin]

We have a similar problem in SpikeInterface. Tagging @alejoe91 and @samuelgarcia here as we might be able to learn from what you decide.

I think is as @billbrod mentioned, it can be done if there is few people who can push force to main/dev (the ideal number is zero anyway). With that safeguard in place, PRs that are open will break and contributors will have to re-clone.

I personally think it is totally worth it. Doing it this way pushes the burden to contributors instead of users who clone the repo which I think is best.