Proposal to collect additional metrics needed for cost monitoring

[simanadler]

What type of PR is this?
/kind feature

The purpose of this PR is to obtain feedback on the proposed approach.

What this PR does / why we need it:
Phase 1 of inference costing monitoring enabled an enterprise to monitor costs at the model/infrastructure level and token levels. (Implemented via OpenCost additions and llm-d deployment guide)

This proposal extends those capabilities so that the enterprise can also monitor cost by whom (requestor, tenant, workload) the models are being used.

Related to the approved cost proposal.

[ArshVermaGit]

This proposal is well thought out. Extending the existing cost monitoring to include requester, tenant, and workload level metrics seems like a natural next step. The motivation is clearly documented, and the references to the related work provide good context. LGTM.

[ronenkat]

I suggest that instead of adding plugin, the IPP response processor will produce a special OTEL usage log.
Not for distributed tracing, but rather for usage accumulation.
The usage log will be used by 3rd party SW for processing to address usage analysis and reporting.
How: OTEL supports custom headers, so IPP can be configured with a mapping between body (or HTPP) headers and OTEL custom headers (e.g., tenant, user, ...)
Since OTEL records can be omitted directly from the response processor, there is no need to background processing. All request headers are available at the time of response processing.
Note: llm-d code does not need to be aware of concepts like user and tenant, it can be abstracted by deciding what to log, and can be define in a well-lit path documentation as guides.

[simanadler]

Thanks for your thoughts @ronenkat! Some questions/comments:

I suggest that instead of adding plugin, the IPP response processor will produce a special OTEL usage log. Not for distributed tracing, but rather for usage accumulation. The usage log will be used by 3rd party SW for processing to address usage analysis and reporting.

This is included in my proposal, described as tier 2 in addition to generating prometheus metrics.

How: OTEL supports custom headers, so IPP can be configured with a mapping between body (or HTPP) headers and OTEL custom headers (e.g., tenant, user, ...) Since OTEL records can be omitted directly from the response processor, there is no need to background processing. All request headers are available at the time of response processing. Note: llm-d code does not need to be aware of concepts like user and tenant, it can be abstracted by deciding what to log, and can be define in a well-lit path documentation as guides.

I looked into this approach and summarized my findings in the section about why traces do bridge the attribution gap. I would really appreciate it if you can provide detailed feedback there.

[davidbreitgand]

@simanadler , @ronenkat , PR #269 : produces a DEBUG-gated log of token consumption and cost per request per model and also at the distribution level per model.

The per token pricing information is obtained by the IPP plugin model-config-datasource from the model configuration file.

Phase 1 of inference cost tracking offers per model per token costs.

If actual per token costs of inference are scraped and the model configuration file is updated with this information, then the cost calculations per model in the requestcostmetadata extractor plugin will be the the costs of inference, which requestcostmetadata can attribute to additional tenant, workload, etc. using body and HTTP headers information. Of course, can be implemented through syncing via configmap.

Hence, my question: maybe we do not need a second plugin, but rather extend the requestcostmetadata extractor plugin to cater for attributing tokens consumption and costs to tenants and workloads using appropriate HTTP headers?

Logging at the DEBUG level can be reconfigured to logging at the INFO level.

I believe that having just one plugin that owns all issues related to cost attribution might be a cleaner option.

What do you think?

[davidbreitgand]

Suggested change

per-model token-count metrics but carries no identity labels — it has no concept of tenant, workload, or requestor. IPP's existing `model-cost-extractor` produces a cost distribution *per model* based on static data from a config file. **None of these can attribute consumption to an identity.**

per-model token-count metrics but carries no identity labels — it has no concept of tenant, workload, or requestor. IPP's existing [`model-cost-extractor`](https://github.com/llm-d/llm-d-inference-payload-processor/blob/main/pkg/framework/plugins/datalayer/requestcostmetadata/README.md) produces a cost distribution *per model* based on the token usage counts collected from responses and static pricing data from a config file. **None of these attributes consumption to an identity.**

[ArshVermaGit]

Nice proposal. The additional metrics make the cost monitoring approach more complete, and the motivation is clearly documented. It builds naturally on the existing design and should help improve cost visibility across different workloads. LGTM.

[github-actions]

This PR is marked as stale after 21d of inactivity. After an additional 14d of inactivity (7d to become rotten, then 7d more), it will be closed. To prevent this PR from being closed, add a comment or remove the lifecycle/stale label.