Fix #943: don't try to take 2 snapshots at a time

[dgnorton]

The TakeSnapshot() function in _vendor/raft/server.go was failing at s.stateMachine.Save() and exiting without resetting s.pendingShapshot = nil. That left the raft server in an invalid state.

s.stateMachine.Save() was failing with the error message "gob: encodeReflectValue: nil element", which comes from the Save() function in cluster/cluster_configuration.go. This is a separate issue.

[otoolep]

Interesting. This PR:

goraft/raft#242

is pretty close to the same thing.

[otoolep]

Why this change?

[dgnorton]

Without the lock, influxdb attempts to take multiple snapshots at the same time. If you comment those lines out, rebuild influxdb, and then run it & influxdb_stress (described above) at the same time...you'll see the messages in the log.

[otoolep]

OK, got it, I was thinking that might be it from the title of the PR. I guess I might have put the lock inside TakeSnapshot(). This is kind of a matter of taste though -- depends on one thinks about TakeSnapshot().

[dgnorton]

@otoolep it was related. There were actually two spots that needed to be reset to nil.

[jvshahid]

Do you know why this gob: encodeReflectValue: nil element is happening ?

[dgnorton]

@jvshahid no but I think it's a separate issue.

[jvshahid]

Agree, I was just wondering why it's failing to save the state in the first place. I'll merge this in a bit.