Inspectables V2

[matthew-humphrey]

Background

Prior to open sourcing the project, the Kanagawa compiler supported a feature called "Inspectables". With this feature the user would decorate static variables or memories in their Kanagawa code with an attribute and a name. The compiler would then insert logic to introspect this variable/memory at runtime. The "inspectables" were connected together in daisy-chain fashion, and an interface on the generated top-level module would provide a way to retrieve these values individually.

This feature was removed before OSS for a couple of reasons:

1. Its interface had become cumbersome and "hackish" when it was refactored to support the compiler emitting multiple modules.
2. It's original design was to be used over a high latency, single-threaded interface (non-pipelined PCIe MMIO), and therefore the underlying inspectables protocol was not conducive to retrieving the values in a pipelined fashion. In addition, because memory inspectables worked by stealing unused cycles, it was possible for a memory to block retrieval.

Core Features

We would like to have a new version of inspectables. This version should support the following features:

1. Allow the inspectables interface for multiple Kanagawa compiler-generated modules to be daisy-chained together, and for the controller retrieving inspectables to be able to determine the identity of each module on this chain and how many inspectables it contains.
2. (Needed for number 1 above) Some concept of an identifier for a module on an inspectables chain. Ideally we would also support a version, but at a minimum a hash value should also be provided. Note that identifier and version will need to be provided by the user in the source code.
3. Ability to do an efficient pipelined retrieval of NON-MEMORY inspectables. With this an external controller (which is outside of the scope of inspectables) would send a command telling Kanagawa generated modules to start sending their inspectables. By skipping memories, there are no issues with stalls (or deadlocks) due to free memory read cycles not being available.
4. Ability to retrieve memory values, with some reasonable (configurable?) timeout if memory values can not be retrieved due to unavailable free read cycles.
5. The equivalent of the "control state" inspectable feature - a compiler option that will automatically add "inspectables" for pipeline control state.

Optional Features

1. One of the challenges with the old inspectables is that the metadata for them was not baked in the hardware and was contained in a file generated by the compiler. It would be nice if there was an option to have the compiler include some reasonable, baseline metatadata in the hardware, and a mechanism to retrieve that.
2. A mechanism to allow Kanagawa code to retrieve inspectables (self-inspection)

Out of Scope

1. Any facility to expose Inspectables over PCIe or other I/O interface.
2. Writing (Inspectables will remain read-only)

[teqdruid]

The way this feature is proposed, is there a difference between this and an easy-to-use telemetry feature? (Other than prod/debug intent.)

It's my understanding that the old, internal inspectables feature is that it would also expose some Kanagawa-internal state machines and perhaps some other internally generated stuff.

[alandy]

The ControlState Inspectables are the most critical for debugging use. Was that an intentional omission here, or do you expect that for the purpose of this discussion those are included and no different from the variable Inspectables?

[matthew-humphrey]

The ControlState Inspectables are the most critical for debugging use. Was that an intentional omission here, or do you expect that for the purpose of this discussion those are included and no different from the variable Inspectables?

Not intentional, that should be included as well. I have added it.

[xmaams]

Writing (Inspectables will remain read-only)

Any chance to reconsider this? We have found it very cumbersome to implement configuration registers in Kanagawa, and a writeable inspectable seems like a potential solution. The benefits of Kanagawa (auto-generation of the chain and metadata) synergize well.

[teqdruid]

Writing (Inspectables will remain read-only)

Any chance to reconsider this? We have found it very cumbersome to implement configuration registers in Kanagawa, and a writeable inspectable seems like a potential solution. The benefits of Kanagawa (auto-generation of the chain and metadata) synergize well.

I don't believe this to be the correct mechanism for setting configuration registers -- it would be an abuse of something intended for debug. (It would go away when/if you compiled without inspectables.)

[xmaams]

I don't believe this to be the correct mechanism for setting configuration registers -- it would be an abuse of something intended for debug. (It would go away when/if you compiled without inspectables.)

Good point, I didn’t consider how it could be disabled so easily.

[blakepelton]

It would be good to detail how exported classes work with this feature. If export class A contains two instances of export class B, can someone query the inspectable ports on A to get the inspectable data for the two instances of B?

Similarly, if export class A instances two instances of an external class B, but the RTL for B was actually generated by Kanagawa, will the inspectable be hooked up?

[matthew-humphrey]

My opinion: everything on the chain should be queryable from any interface into the chain.

Regarding the external class, it seems there would need to be some syntax to tell Kanagawa that the external class has an inspectable interface to be wired up or a way to manually wire it up via Kanagawa code. So maybe initially we should skip that feature.

[teqdruid]

It may be a good idea to expose the inspectables interface just as a Kanagawa function. One that just happens to be automatically generated by the compiler. Then the instantiating code would be able to decide how to wire it up (and expose it to the host).

WDYT?

[matthew-humphrey]

It may be a good idea to expose the inspectables interface just as a Kanagawa function. One that just happens to be automatically generated by the compiler. Then the instantiating code would be able to decide how to wire it up (and expose it to the host).

This idea has some merit. It would make doing thing like capturing inspectables into a buffer and DMA-ing it easier. However, there's some benefit to it being outside the Kanagawa code, for example to diagnose lock-ups (via control state). And it's always possible to connect the compiler-generated interface into Kanagawa - look at the PDS "self inspection" feature in the Microsoft internal repo.

[teqdruid]

It would be up to the instantiating code to ensure there are no lockups in the inspectables query path. That should be imminently possible as long as it's just being plumbed through. External code (e.g. SystemVerilog) would be required to create an always-accessible inspectables accessor method to diagnose lockups in the top-level Kanagawa code.

I think it would be possible to create a "safe" top level in Kanagawa (certainly in some RTL) that passed through everything except for the inspectables to the "real"/"functional" top level. It could then expose the inspectables in such a way so as to ensure that lock-ups in the "functional" top level could always be queried. (We can discuss ways to do that for internal systems offline, but there are several I can think of.) This way of accessing the inspectables would obviously not preclude the possibility of maintaining the functionality you implemented internally in your PDS system. I think it would just allow additional functionality.

As an additional benefit, it would mitigate the physical design issue. By having a "top-level" instantiating module (Kanagawa, SystemVerilog, PyCDE, ...) decide how to wire it up, it wouldn't necessarily have to be a ring network. Or whatever structure the Kanagawa compiler decides to use to implement it internally. (This is clearly already possible if the instantiating "top-level" is RTL.)

Not clear to me if it's a good idea to allow classes to call their own inspectables methods. Like you mentioned that could easily lead to not being able to inspect things in the case of a lock-up and we don't want to provide users with a footgun. I suspect it would be a good idea to only expose these methods to the instantiating code. (E.g. the Kanagawa code which instantiates an extern class.)

[teqdruid]

I'd like to revive this discussion. At a minimum, we need the ability to look at the control state in a single Kanagawa export. For the moment, I think we don't have to deal with Kanagawa instantiating exported modules.

The simplest way I can think of to accomplish this functionality is to use ESI. Specifically, ESI services. Wherever you have a control state, just add an ESI client service request. It'll be plumbed through to the top level (creating a shit-ton of top-level ESI ports in the process) automatically and by default. Then it's ESI's problem and I can take care of it from there in the BSP. Hell, it could even hook up to the existing telemetry service to make it report out over MMIO. (This approach makes it vulnerable to lockups though.) We'd have to disambiguate client IDs by having instantiations identified with AppIDs.

[blakepelton]

When you say control state in a single Kanagawa export, you mean just the control state to the basic block which is the first basic block in a public method of an exported class? Or do you mean the control states associated with all basic blocks?

[teqdruid]

I mean whatever control state that's being connected to the inspectables chain currently. I don't really know much about it since I've never used it. I've never had to debug a stuck Kanagawa system.

@alandy / @xmaams what are you actually looking for in terms of debug capabilities from inspectables WRT control state?