PVC sizing issues and non-optional log volumes for HA

[mwashley2]

I've recently started using memgraph in my organization and we have started low in terms of sizing our PV/PVCs using the SA/HA charts with the idea in mind to grow as needed. I've found that when I do need to increase the PVC size which is normally able to be done without downtime, I get errors to the effect that the statefulset cannot be patched because volumeClaimTemplates within a statefulset are immutable.

My work-around has been to manually patch the pvc to increase it, then delete the statefulset with --cascade=orphan, then I can deploy the updated chart. This will not be ideal once we get to production as privileges will be more limited.

Is it possible to update the chart to manage PVCs separately from the statefulset.

As a separate question to the above (happy to open a separate issue if needed), we will be using flags to send logs to --also-log-to-stderr and "--log-file=" as we'll be shipping logs elsewhere so ideally I'd like to optionally not specify log-related storage and not have those volumes created or mounted. I think a similar if statement to what is used for the coredump volume would suit this. Or is this volume perhaps required at some other level and simply must exist?

[as51340]

Hi @mwashley2!

For logs in HA, you are right, we are missing option to optionally use log volumes, we will add that.

For increasing PVC size, I guess you are right, we would have to create separate PVC resources and then just link it to StatefulSet. I will research more but I think this is a good feature.

[as51340]

AFAICT some deployments will still require pod restarts (not all PVC types support online volume expansion). Before we start doing anything here, can you

1. check that your storage class will support volume expansion?
2. Your CSI driver will support volume expansion without pod restarts?
3. Your underlying volume (disk type) will support online resize?

[as51340]

Note that also in some situations doing:

kubectl patch pvc memgraph-db-lib-storage-memgraph-db-0 -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'
kubectl patch pvc memgraph-db-log-storage-memgraph-db-0 -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'

should work too

[as51340]

@mwashley2 Could you check if this would work for you for log volumes? #234

[mwashley2]

@as51340 apologies I didn't see this until now.. I'm in AWS using ebs-gp3 storage class with volume expansion supported. Other technologies like elastic we run have tolerated expansion of the storage without any pod restarts needed using the same storageclass.

Currently solution for expanding drives is to patch the pvc as you described, then delete the statefulset with the orphan flag, so the pod doesn't go with it, then deploy the updated helm template/profile with no other changes, at which time the new statefulset picks up the existing pod

And thank you, #234 is exactly what I wanted there!

[as51340]

Okay then, I will start working on this improvement and ping you if I will have any questions

[as51340]

Hi @mwashley2! I went researching this a bit more and this is actually quite a tricky thing! First, pretty much every chart I found online uses the same approach as we do and leaves the operational burden on the users to do the patch by themselves. Second, there is a long-standing issue/wish from K8s community to implement this upgrade support directly in K8s kubernetes/enhancements#4651. It seems that is going to end eventually in, the question is only when. Third, elastic is special in that case because they are using operator-based pattern so the operator does automatically what you would have to do manually in our case. Now, if we are going to go with operator-based solution, we should definitely think how to do it, do we go full in with operators and completely replace charts, or do we keep existing architecture and somehow optionally add smaller operators in

[as51340]

if this is what you really need, we should sync on the meet and include more people in

[mwashley2]

Thanks Andi, yes ultimately we would like to transition to the operator once it is battle-ready though I see it has been a while since it has been updated and considered WIP. That being the case I've documented steps on my end for expanding disk in the current setup for now