Clarify IOFENCE synchronization for I/O MPT Checker configuration updates

[Abhinav Agarwal (abhinavagarwal07)]

Modifying a previously valid SDCL rule, or the supervisor domain configurations it references, is only guaranteed to be observed by the I/O MPT Checker within a bounded but UNSPECIFIED time. IOFENCE already provides the explicit observation guarantee, but the text only says software "may use" it, so nothing requires the fence before a request that depends on the new configuration is allowed through.

This makes it a requirement. The MPTINVAL ordering requirement is unchanged.

[Abhinav Agarwal (abhinavagarwal07)]

Bit of context on why I went with "must" here instead of leaving it at "may use".

The bullet a few paragraphs up (chapter6.adoc L106-L110) already does this for MPT entries:

I/O MPT Checker is not required to immediately observe the software updates to an MPT entry. Software must use the MPTINVAL operation outlined in <> to invalidate any previous copies of that entry [...]

IOFENCE is the same situation - bounded but unspecified observation time - but there it's only "may use", so nothing says when a config change is actually in effect for requests that come in after it.

That said, chapter3.adoc L126-L128 says it "may be necessary" to execute MFENCE.PA around an mmpt write, and satp/SFENCE.VMA is worded the same way, so "may" isn't out of place. What pushed me the other way is that those are about a hart's own translations after the write, where software controls the ordering. An SDCL change has to hold against device requests that show up on their own. That felt closer to hgatp.MODE / HFENCE.GVMA:

If hgatp.MODE is changed for a given VMID, an HFENCE.GVMA with rs1=x0 (and rs2 set to either x0 or the VMID) must be executed to order subsequent guest translations with the MODE change - even if the old MODE or new MODE is Bare.

If you'd rather keep it consistent with the mmpt wording, the weaker form still covers most of what I was after.

[Ved Shanbhogue (ved-rivos)]

I do not think the MPTINVAL requirement is equivalent to this case.

For an SDCL or supervisor-domain configuration update, the specification already requires the I/O MPT Checker to observe the update within a bounded but UNSPECIFIED time. An IOFENCE provides software with an architecturally identifiable synchronization point when one is required. Software may otherwise tolerate the transition period, or batch multiple configuration updates before issuing a single IOFENCE.

The proposed wording adds a new requirement that software issue an IOFENCE before permitting a dependent request. The existing architecture does not require software to establish such a synchronization point merely because it modifies a configuration. In addition, “whose handling depends on them” is ambiguous, since every request matching the modified configuration could be said to depend on it.

MPT entries are different. An implementation may retain cached MPT information until an applicable MPTINVAL is performed; there is no corresponding bounded-time guarantee for automatic observation of an MPT update. Consequently, MPTINVAL is required when software needs to synchronize an MPT update with the I/O MPT Checker. The statement that software “must use” MPTINVAL specifies the mechanism that must be used to invalidate previous cached copies and perform that synchronization. It does not require software to perform an MPTINVAL after every MPT update.

The existing text already defines when software can rely on a configuration update: successful completion of an IOFENCE guarantees that all preceding configuration updates have been observed. It need not additionally require software to use that synchronization mechanism.

[Abhinav Agarwal (abhinavagarwal07)]

Fair point on the distinction. That's the difference, and "whose handling depends on them" was doing too much work anyway. I'll drop it.

One thing I'm still unsure about in the required sequence when PPN changes. Say SDID S uses root A, where address P is allowed, and software moves it to root B, where P is denied. Assume B's MPT was fully initialized and globally visible beforehand.

1. SET_SDCFG_ENTRY moves S from A to B and completes; per your model B needn't have been observed yet.
2. MPTINVAL with PPNV=0, SDIDV=1, SDID=S completes.
3. Before B is observed, a device request classified to S accesses P. If A is still the configuration in use, the checker may check against A and cache information permitting P.
4. IOFENCE completes, so B is now observed.
5. A new request to P arrives.

What makes the information from step 3 ineligible for step 5? The cache rule allows information valid since the most recent applicable MPTINVAL, and I can't find text retiring information obtained under a superseded SDCFG, or giving IOFENCE that effect.

The IOMMU guidelines invalidate after the update is globally visible, which is what made me look. I'm not assuming the mechanisms should match.

If something does exclude it, I'm happy to repoint the PR at making that explicit, or just close it. My first instinct was to swap MPTINVAL and IOFENCE, but MPTINVAL also orders MPT stores against subsequent implicit reads, so I'm not sure that's right.

[Ved Shanbhogue (ved-rivos)]

The correct sequence in this case is to issue the IOFENCE as step 2 and MPTINVAL at step 4. The IOFENCE at step 2 makes the root B visible to the MPT checker. Between step 2 and 4, the implementation may use information cached from either the new MPT or the old MPT. The MPTINVAL at step 4 then invalidates all cached translations and all subsequent translations will be always from root B. In this example, if the page P was denied with the intent to reclaim that page for some other use then software may need to issue a second IOFENCE after the MPTINVAL. That second IOFENCE guarantees that there are no read or writes in flight to the page P. If that second IOFENCE is omitted, then the following could occur a) software puts some data unrelated to SDID S in page P b) that data gets overwritten by a write that was already in flight - post MPT check - somewhere in the fabric or that data gets read by a read that was already in flight and exposes that data.

[Abhinav Agarwal (abhinavagarwal07)]

Thanks, that helps.

I'm trying to square it with the text, which says that on an MPT_MODE/PPN change the MPTINVAL has to be issued before the IOFENCE. In the sequence you describe it comes after, so I think I'm misreading something.

Is the pre-fence one doing something different, like ordering software's stores to the new MPT against the checker's reads of it?