Support storing data (and indexes) on tiered storage

[eolivelli]

FEATURE REQUEST

1. Please describe the feature you are requesting.

Currently it is possible to store data and indexes on disks or on the bookies.

Another possibility is to store the data and index pages on a cheaper object storage, like S3.
This would work really well for databases used almost for writes.

Of course we need to locally cache data downloaded from s3, in order to not download it too often.

In this mode we could work with only 1 active server per tablespace, like we do in diskless mode. This way we keep only one copy of the data on s3.

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

Should have

3. Provide any additional detail on your proposed use case for this feature.

4. If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task:

• sub-task1-issue-number: sub-task1 discription here,
• sub-task2-issue-number: sub-task2 discription