Skip to content

Fix #943: don't try to take 2 snapshots at a time #1081

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dgnorton wants to merge 1 commit into from
Closed

Fix #943: don't try to take 2 snapshots at a time #1081

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions _vendor/raft/server.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1203,6 +1203,7 @@ func (s *server) TakeSnapshot() error {

state, err := s.stateMachine.Save()
if err != nil {
s.pendingSnapshot = nil
return err
}

Expand Down Expand Up @@ -1237,6 +1238,7 @@ func (s *server) saveSnapshot() error {

// Write snapshot to disk.
if err := s.pendingSnapshot.save(); err != nil {
s.pendingSnapshot = nil
return err
}

Expand Down
13 changes: 13 additions & 0 deletions _vendor/raft/snapshot_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package raft

import (
"errors"
"fmt"
"io/ioutil"
"os"
Expand Down Expand Up @@ -121,6 +122,18 @@ func TestSnapshotRequest(t *testing.T) {
})
}

func TestTakeSnapshotWhenStateMachineSaveFails(t *testing.T) {
runServerWithMockStateMachine(Leader, func(s Server, m *mock.Mock) {
m.On("Save").Return([]byte("foo"), errors.New("test error message"))

s.Do(&testCommand1{})
err := s.TakeSnapshot()
assert.Equal(t, err.Error(), "test error message")
assert.Nil(t, s.(*server).pendingSnapshot)
s.Stop()
})
}

func runServerWithMockStateMachine(state string, fn func(s Server, m *mock.Mock)) {
var m mockStateMachine
s := newTestServer("1", &testTransporter{})
Expand Down
2 changes: 2 additions & 0 deletions coordinator/raft_server.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -331,6 +331,8 @@ const (
)

func (s *RaftServer) ForceLogCompaction() error {
s.mutex.Lock()
Copy link
Copy Markdown
Contributor

@otoolep otoolep Oct 30, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this change?

Copy link
Copy Markdown
Contributor Author

@dgnorton dgnorton Oct 30, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without the lock, influxdb attempts to take multiple snapshots at the same time. If you comment those lines out, rebuild influxdb, and then run it & influxdb_stress (described above) at the same time...you'll see the messages in the log.

Copy link
Copy Markdown
Contributor

@otoolep otoolep Oct 30, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, got it, I was thinking that might be it from the title of the PR. I guess I might have put the lock inside TakeSnapshot(). This is kind of a matter of taste though -- depends on one thinks about TakeSnapshot().

defer s.mutex.Unlock()
err := s.raftServer.TakeSnapshot()
if err != nil {
log.Error("Cannot take snapshot: %s", err)
Expand Down