store alloy logs agent position file on hostpath

[samidbb]

This pull request updates the Alloy logs DaemonSet configuration to use a persistent storage path and mounts for log data, improving data durability and aligning with best practices for stateful components. The changes ensure that Alloy's log storage is backed by a hostPath volume at /srv/alloy/data instead of the previous ephemeral /tmp/alloy path.

Persistent storage improvements:

• Changed Alloy's storage path from /tmp/alloy to /srv/alloy/data in the DaemonSet arguments for more durable log data storage.
• Added a new volume mount for /srv/alloy/data using the alloy-storage-logs volume in the container spec.
• Defined a new hostPath volume named alloy-storage-logs at /srv/alloy/data with type DirectoryOrCreate to ensure the directory exists on the host.

Helm release configuration updates:

• Updated release.yaml to add storagePath and mount definitions for /srv/alloy/data, and included the corresponding hostPath volume under volumes.extra.

[SEQUOIIA]

Hmm, thoughts on doing it via hostPath instead? The data is only relevant to the node the pod is on, if it's a different node it's different logs, thus attaching a pvc with a WAL/positions from a different node doesn't make sense since it can't access those logs. With the hostPath the WAL/positions are stored with the node the logs are on anyway and we avoid creating pvcs/pvs.

[wcarlsen]

Hmm, thoughts on doing it via hostPath instead? The data is only relevant to the node the pod is on, if it's a different node it's different logs, thus attaching a pvc with a WAL/positions from a different node doesn't make sense since it can't access those logs. With the hostPath the WAL/positions are stored with the node the logs are on anyway and we avoid creating pvcs/pvs.

If possible I think this could make a lot of sense.

[samidbb]

Hmm, thoughts on doing it via hostPath instead? The data is only relevant to the node the pod is on, if it's a different node it's different logs, thus attaching a pvc with a WAL/positions from a different node doesn't make sense since it can't access those logs. With the hostPath the WAL/positions are stored with the node the logs are on anyway and we avoid creating pvcs/pvs

it makes sense for me as well. I will try to use the hostpath instead

[SEQUOIIA]

for using hostPath instead. The suggested path is the one that currently has the migrated WAL/position data in production.