Auto-Configure Metrics Indices with Data Streams and Rollup Jobs

[medcl]

Goal: Improve performance and storage efficiency for metrics by automatically configuring Easysearch when the Console starts.

Current Problem:
The .infini_metrics index is created as a standard index. This is not optimal for time-series data and will lead to slow searches and high storage costs as data grows.

Proposed Solution:
On first run, the INFINI Console should automatically configure its connected Easysearch instance with smart defaults for the .infini_metrics data.

Key Changes:

1. Create a Data Stream named infini_metrics

   • Why: Data streams automatically handle time-series data (create new indices, manage rollovers).
   • How: Console creates an index template that points the .infini_metrics alias to a new data stream.

2. Create a Rollup Job

   • Why: To downsample old data, saving huge amounts of storage and making historical searches very fast.
   • How: A job will summarize detailed data into hourly aggregates (e.g., avg, max, min) and store them in a .infini_metrics_rollup index.

3. Add a Lifecycle Policy (ILM)

   • Why: To automatically manage the data lifecycle.
   • How: A policy will:
     • Keep raw data for 3 days (for detailed analysis).
     • Then, delete the raw data (after the rollup job has saved the aggregated results).

What the User Sees:

• Nothing! The Console works exactly the same, but searches are faster and storage use is much lower.

Technical Tasks:

• Console checks on startup if the required Data Stream template exists in Easysearch.
• If not, it automatically creates the Index Template, ILM Policy, and Rollup Job.
• Console search queries combine the active Data Stream and the Rollup index.