feat: streaming api logs in the admin web ui

[chet]

Description

I was having a bunch of issues the other day when I was trying to look at carbide-api logs. It was annoying, and I was like this shouldn't be so difficult just to check carbide-api logs. Seemed to make sense to just add a Logs section to the web UI where I can stream API logs, so this does that.

My plan is to actually have support for:

• carbide-api logs (which is what this does).
• scout logs (via WebSockets using the ScoutStream thing we have).
• nico-dpu-agent logs (via WebSockets, also using a similar approach to ScoutStream).

Tested via my local-dev environment. Screen capture attached showing it in action!

api_web_ui_log_stream.mov

Signed-off-by: Chet Nichols III chetn@nvidia.com

Type of Change

• Add - New feature or capability
• Change - Changes in existing functionality
• Fix - Bug fixes
• Remove - Removed features or deprecated functionality
• Internal - Internal changes (refactoring, tests, docs, etc.)

Related Issues (Optional)

Breaking Changes

• This PR contains breaking changes

Testing

• Unit tests added/updated
• Integration tests added/updated
• Manual testing performed
• No testing required (docs, internal refactor, etc.)

Additional Notes

[Matthias247]

It looks cool. But I personally wouldn't feel like there should be product code doing any of this. I'd probably extend the UI to fetch logs from where they are stored (e.g. Loki) - but obviously it depends on the log storage.
Or just link to remote log search. I think we have that capability already.

[poroh]

It looks cool. But I personally wouldn't feel like there should be product code doing any of this. I'd probably extend the UI to fetch logs from where they are stored (e.g. Loki) - but obviously it depends on the log storage. Or just link to remote log search. I think we have that capability already.

Maybe under config flag? I see that it can be useful for debug purposes at minimum.

Concern in comments

[chet]

It looks cool. But I personally wouldn't feel like there should be product code doing any of this. I'd probably extend the UI to fetch logs from where they are stored (e.g. Loki) - but obviously it depends on the log storage. Or just link to remote log search. I think we have that capability already.

Maybe under config flag? I see that it can be useful for debug purposes at minimum.

Yeah we could flag it? I was just having problems with Loki and kubectl, and I was like omg today is NOT my day, and I just wanted to see the API logs, lol.

[ajf]

It looks cool. But I personally wouldn't feel like there should be product code doing any of this. I'd probably extend the UI to fetch logs from where they are stored (e.g. Loki) - but obviously it depends on the log storage.
Or just link to remote log search. I think we have that capability already.

I'd say the reverse. Basically that we shouldn't presume Loki or any log / metrics platform exists. We don't want to force a deployment of a particular metrics so this log view will work. I hear people saying "what about vault and Postgres" - well we're moving off vault and you can run different databases without losing anything; a metrics platform there's generally only one of and people are very passionate about them.

[krish-nvidia]

This is slick! Individual managed hosts and explored endpoints pages have a log button in the top right corner that just links to loki today (filtered on that object) but maybe it can point to this new view instead with the filter pre-populated? I'm happy to take that on as a follow-up PR later :)

[Matthias247]

@ajf @krish-nvidia One major difference is that this tooling won't show old logs. Which doesn't make it too useful for debugging anything that happened in the past. There's some chance that a feature request will be coming in afterwards that also asks for these logs - and then there's probably a decision to be made whether any log storage service (like Loki) should just always directly be bundled as part of NICo.

But if people think that this tool adds enough value on its own I'm fine adding it. At least in the current state its not too much code.

[ajf]

and then there's probably a decision to be made whether any log storage service (like Loki) should just always directly be bundled as part of NICo.

The decision would obviously be that we wouldn't ship Loki - since it doesn't really make sense to ship a log analysis platform with a host lifecycle manager. Shipping in Otel format to some collector seems like the right choice? Then someone can just do whatever they want with the logs. NICo isn't an appliance.

[Matthias247]

OTEL unfortunately doesn't help with this use-case, since it doesn't allow to query logs in any form. It's just an interface to stream telemetry when its created.

If the UI needs to have a builtin way to query past logs, then a more concrete interface to a log storage service is required.

[ajf]

If the UI needs to have a builtin way to query past logs, then a more concrete interface to a log storage service is required.

Your call Chet, but If I was looking at logs I'd expect it to be historical and current. Not just current + future.

It Was useful to you, it may be useful to others.