DRAFT + WIP [DPE-5661] patch stateful set to ensure pod isn't removed prematurely #348
DRAFT + WIP [DPE-5661] patch stateful set to ensure pod isn't removed prematurely #348MiaAltieri wants to merge 9 commits into
Conversation
MiaAltieri
force-pushed
the
DPE-5661-safe-drainage
branch
from
2a9b2bf to
2a9415f
Compare
MiaAltieri
force-pushed
the
DPE-5661-safe-drainage
branch
from
2a9415f to
9c88b59
Compare
MiaAltieri
force-pushed
the
DPE-5661-safe-drainage
branch
from
0af8c24 to
fc992f0
Compare
MiaAltieri
changed the title
| # updating the termination grace period is only useful for shards, whose sudden removal | ||
| # can result in data-loss | ||
| if not self.is_role(Config.Role.SHARD): | ||
| return |
There was a problem hiding this comment.
im unsure about this bit
Gu1nness
left a comment
Gu1nness
left a comment
There was a problem hiding this comment.
Overall it should be working.
I would try to leave it to run and play with it to see how stable it stays in the time
| from lightkube import Client | ||
| from lightkube.resources.apps_v1 import StatefulSet | ||
| from lightkube.types import PatchType |
There was a problem hiding this comment.
Need to import lightkube in pyproject.toml
| "spec": { | ||
| "template": { | ||
| "spec": {"terminationGracePeriodSeconds": ONE_YEAR}, | ||
| "metadata": {"annotations": {"force-update": str(int(time.time()))}}, |
There was a problem hiding this comment.
What it the force-update annotation useful for here?
| patch_data = { | ||
| "spec": { | ||
| "template": { | ||
| "spec": {"terminationGracePeriodSeconds": ONE_YEAR}, |
There was a problem hiding this comment.
Isn't one year a bit too much?
It means that your pod can take up to 1 year to be removed.
I would keep it lower, up to a day?
There was a problem hiding this comment.
I chose a ridiculously long amount on purpose to ensure that shard drainage would have enough time in worst case scenarios.
i.e. if there are jumbo-chunks that we cannot automatically drain (because the user should decide the new shard key) - then that will require user intervention and I would like to give them more than enough time to do so.
| name=self.app.name, | ||
| namespace=self.model.name, | ||
| obj=patch_data, | ||
| patch_type=PatchType.MERGE, |
There was a problem hiding this comment.
I think MERGE or STRATEGIC don't change anything here so I would not specify it and keep the default value of STRATEGIC, but it's a nitpick.
Gu1nness
left a comment
Gu1nness
left a comment
There was a problem hiding this comment.
Overall it should be working.
I would try to leave it to run and play with it to see how stable it stays in the time
MiaAltieri
changed the title
MiaAltieri
force-pushed
the
DPE-5661-safe-drainage
branch
2 times, most recently
from
81d2589 to
d51f6a2
Compare
MiaAltieri
force-pushed
the
DPE-5661-safe-drainage
branch
from
d51f6a2 to
f726e56
Compare
2 participants
Issue
Solution