We want to make sure we never re-use a manifest key/nonce pair, so these need to be tracked in the manifest as well.