Version probe fails with read-only root filesystem because clickhouse local requires writable /tmp

[RakeshThirunahari]

The version-probe Job fails when readOnlyRootFilesystem is enabled, which is required by our Gatekeeper/Azure Policy.

Policy error:

admission webhook "validation.gatekeeper.sh" denied the request:
[azurepolicy-k8sazurev3readonlyrootfilesyst-805b702bad9f5d2dabcb]
Readonly root filesystem is required for container.
pod: 'clickhouse-cluster-dev-clickhouse-version-probe-6e525b6a-bwh6f',
container: 'version-probe'.

When readOnlyRootFilesystem is set to true, the Pod is admitted but the probe fails with:

std::exception. Code: 1001, type: std::__1::filesystem::filesystem_error, e.what() = filesystem error: in create_directories: Read-only file system ["/tmp/clickhouse-local-cb58997b-d7ff-46f4-9568-d8dc5bbf1ec3"]

Stack trace:

0. std::system_error::system_error(std::error_code, String const&) @ 0x0000000023549757
1. std::filesystem::filesystem_error::filesystem_error[abi:fqe220101](String const&, std::filesystem::path const&, std::error_code) @ 0x0000000016647e2a
2. void std::filesystem::__throw_filesystem_error[abi:fqe220101]<String&, std::filesystem::path const&, std::error_code const&>(String&, std::filesystem::path const&, std::error_code const&) @ 0x00000000234fe76d
3. std::filesystem::detail::ErrorHandler::report(std::error_code const&) const @ 0x0000000023501762
4. std::filesystem::__create_directories(std::filesystem::path const&, std::error_code*) @ 0x00000000235028a2
5. DB::LocalServer::tryInitPath() @ 0x00000000168bc03f
6. DB::LocalServer::processConfig() @ 0x00000000168c33c6
7. DB::LocalServer::main(std::vector<String, std::allocator> const&) @ 0x00000000168c034c
8. Poco::Util::Application::run() @ 0x0000000022289bb1
9. mainEntryClickHouseLocal(int, char**) @ 0x00000000168d5a59
10. main @ 0x000000000f82c3e0
11. __pow_finite @ 0x0000000000029d90
12. __libc_start_main @ 0x0000000000029e40
13. _start @ 0x000000000894962e

Cannot print extra info for Poco::Exception (version 26.4.5.143 (official build))

Background:

In v0.0.4, commit 273e164, the version probe used:

sh -c "/usr/bin/clickhouse --version > /dev/termination-log 2>&1"

The current implementation uses clickhouse local, which creates temporary directories under /tmp.

The current versionProbeTemplate CRD supports configuring the security context but does not support configuring Pod volumes or container volume mounts. Therefore, users cannot mount a writable emptyDir volume at /tmp while keeping the root filesystem read-only.

Expected solution:

The generated version-probe Pod should contain:

securityContext:
readOnlyRootFilesystem: true

volumeMounts:

• name: tmp
  mountPath: /tmp

volumes:

• name: tmp
  emptyDir: {}

The preferred solution is for the operator to automatically add a writable emptyDir volume mounted at /tmp for the version-probe container while preserving readOnlyRootFilesystem: true.