Implement snapshotting in the Raft layer

[awhvish]

Currently the raft log grows indefinitely, which is a serious memory management issue and unfeasible. A node that was offline for hours must receive potentially millions of log entries over the network via AppendEntries RPCs. Persisted state in raft_state_*.gob grows without limit.
Implement snapshotting mechanism, as per the Raft paper - Section 7 to solve this issue.
In short,

• Periodically snapshot the KV state machine at a committed index
• Truncate log entries before that index
• Add InstallSnapshot RPC for leaders to send snapshots to lagging followers instead of replaying the entire log

[DNahar74]

Hello,
I was reading up on raft recently and I'd love to work on this issue, also any tips you have on how I could proceed would be helpful

[awhvish]

Hi @DNahar74, so basically when the raft log gets too big, given a threshold - you would have to serialize the entire KV state to disk as a "snapshot", then truncate all log entries before that point.

You have to create a InstallSnapshot RPC, which should contain specifically these two

       LastIncludedIndex int
       LastIncludedTerm  int

referenced in the figure 13 of the Raft paper and store the serialized data in,
Data []byte

After truncation, rf.log[0] is no longer index 0. Use actualIndex = lastIncludedIndex + position for all log access.

In raft/appendEntries.go inside sendHeartBeats(), check: if nextIndex[peer] <= lastIncludedIndex and call InstallSnapshot instead of AppendEntries

This is a pretty complex task, given you have already read the raft paper this should be a good issue for you. Take your time understanding the codebase and how the raft internals are working and then submit a PR.

[awhvish]

#13

[DNahar74]

Hi, I was checking out the codebase and I think I'll take a few more days before I can raise a PR. Also, the issue that you've added here: "an incremental WAL for raft".
Are you working on it yourself or would you like me to work on it, because I feel it will probably be an extension to the snapshotting system that I am trying to implement here.

[awhvish]

I am already working on it, and will merge by today. Try to work on top of that, the incremental WAL should be checked for over a threshold size and will truncate. Rest the serialization of the kv-store remains the same.

Edit: d2d0e61 added raft-log.

[awhvish]

@DNahar74 are you still working on this?

[DNahar74]

Yes, I am I'll try to raise a PR by Tuesday