CAS: drop KeyStrictness, readers skip unknown ordinary keys

docs/en/antalya/cas/architecture/storage-layout.md

@@ -15,8 +15,9 @@ control-plane bodies are JSON Lines — one JSON object per line, sorted where t
 or a set of entries (`Formats/README.md`; see [Envelope format](#envelope-format) below for which
 parts are a single JSON object versus JSON Lines versus raw payload bytes). The format is
 deliberately this plain: any object can be fetched and read with ordinary line-oriented tools
-while debugging, and a new field is additive — a tolerant reader skips it — so the format evolves
-without a migration.
+while debugging, and the format evolves without a migration. A new optional field is a plain key,
+and an old reader skips it. A field that changes a reader's decisions is written with the `!`
+prefix, and an old reader fails with `UNKNOWN_FORMAT_VERSION` instead of half-reading the object.
 
 ## Key table {#key-table}
 
@@ -88,30 +89,27 @@ bytes.
 Condensed from the authoritative traits table in `CasFormat.cpp` (`TRAITS`, asserted complete by
 `gtest_cas_text_format.cpp`).
 
-| Type string | Family | Key strictness | Compression |
+| Type string | Family | Compression |
 |---|---|---|---|
-| `cas_blob` | `PayloadHybrid` | tolerant | never (raw, fixed offset) |
-| `cas_blob_meta` | `Control` | tolerant | never |
-| `cas_pool_meta` | `Control` | tolerant | never |
-| `cas_ref_log` | `Control` | tolerant | always (`.zst`) |
-| `cas_ref_snap` | `Control` | tolerant | always (`.zst`) |
-| `cas_ref_ckpt` | `Control` | strict | never |
-| `cas_ref_catalog` | `Control` | strict | never |
-| `cas_part_manifest` | `PayloadHybrid` | tolerant | always (`.zst`) |
-| `cas_run` | `RecordStream` | strict | pinned raw |
-| `cas_fold_seal` | `Control` | strict | pinned raw |
-| `cas_gc_state` | `Control` | tolerant | never |
-| `cas_gc_hb` | `Control` | tolerant | never |
-| `cas_gc_outcomes` | `Control` | tolerant | always (`.zst`) |
-| `cas_gc_maintenance_state` | `Control` | strict | never |
-| `cas_owner` | `Control` | tolerant | never |
-| `cas_epoch` | `Control` | tolerant | never |
-| `cas_mount_lease` | `Control` | tolerant | never |
-
-"Strict" means unknown keys are rejected rather than skipped, used for objects where every field
-decides a durability or cleanup decision (`cas_ref_ckpt`, `cas_ref_catalog`, `cas_fold_seal`,
-`cas_run`, `cas_gc_maintenance_state`); a `!`-prefixed key is always critical regardless of the
-kind's strictness. "Pinned raw" objects (`cas_run`, `cas_fold_seal`) need stable bytes across
+| `cas_blob` | `PayloadHybrid` | never (raw, fixed offset) |
+| `cas_blob_meta` | `Control` | never |
+| `cas_pool_meta` | `Control` | never |
+| `cas_ref_log` | `Control` | always (`.zst`) |
+| `cas_ref_snap` | `Control` | always (`.zst`) |
+| `cas_ref_ckpt` | `Control` | never |
+| `cas_ref_catalog` | `Control` | never |
+| `cas_part_manifest` | `PayloadHybrid` | always (`.zst`) |
+| `cas_run` | `RecordStream` | pinned raw |
+| `cas_fold_seal` | `Control` | pinned raw |
+| `cas_gc_state` | `Control` | never |
+| `cas_gc_hb` | `Control` | never |
+| `cas_gc_outcomes` | `Control` | always (`.zst`) |
+| `cas_gc_maintenance_state` | `Control` | never |
+| `cas_owner` | `Control` | never |
+| `cas_epoch` | `Control` | never |
+| `cas_mount_lease` | `Control` | never |
+
+"Pinned raw" objects (`cas_run`, `cas_fold_seal`) need stable bytes across
 re-encodes for deterministic-artifact adoption, so their bytes are never recompressed once
 written. `cas_blob` and `cas_part_manifest` are the `PayloadHybrid` family: a text descriptor
 followed by a raw payload zone, rather than a single JSON body.

src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobEnvelopeFormat.cpp

@@ -235,7 +235,7 @@ String encodeEnvelopeHeader(EnvelopeHeader & header, uint32_t blob_header_len,
 EnvelopeHeader decodeEnvelopeHeader(std::string_view head_bytes, uint64_t /*object_size*/, ObjectKind expected_kind)
 {
     ReadBufferFromMemory in(head_bytes.data(), head_bytes.size());
-    JsonObjectReader r(in, KeyStrictness::Tolerant, "blob envelope");
+    JsonObjectReader r(in, "blob envelope");
 
     EnvelopeHeader h;
     h.kind = ObjectKind::Blob;
@@ -287,7 +287,7 @@ EnvelopeHeader decodeEnvelopeHeader(std::string_view head_bytes, uint64_t /*obje
         else if (key == EnvelopeWire::ref)
             h.intended_ref = r.readString();
         else
-            r.skipUnknown(key);   /// `!`-key -> UNKNOWN_FORMAT_VERSION; unknown plain key -> skipped (tolerant)
+            r.skipUnknown(key);   /// `!`-key -> UNKNOWN_FORMAT_VERSION; unknown plain key -> skipped
     }
     if (!saw_type)
         throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS blob envelope: missing type");

src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasBlobMetaFormat.cpp

@@ -65,7 +65,7 @@ BlobMeta decodeBlobMeta(std::string_view bytes)
     expectHeaderLine(in, FormatId::BlobMeta);
     const String body = readLine(in, traitsFor(FormatId::BlobMeta).line_cap, "blob meta");
     ReadBufferFromMemory body_in(body.data(), body.size());
-    JsonObjectReader r(body_in, KeyStrictness::Tolerant, "blob meta");
+    JsonObjectReader r(body_in, "blob meta");
 
     // Start with the documented defaults. In particular, `version` stays at 1 because the header's
     // version is authoritative and is not copied into the body struct.

src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFoldSealFormat.cpp

@@ -380,13 +380,13 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional<uint64_t> expect
     {
         const String meta = readLine(in, line_cap, "fold seal");
         ReadBufferFromMemory m(meta.data(), meta.size());
-        JsonObjectReader r(m, KeyStrictness::Strict, "fold seal");
+        JsonObjectReader r(m, "fold seal");
         String key;
         while (r.nextKey(key))
         {
             if (key == FoldSealWire::generation) seal.generation = r.readU64String();
             else if (key == FoldSealWire::parent_generation) seal.parent_generation = r.readU64String();
-            else r.skipUnknown(key);   /// Strict => any unknown key is CORRUPTED_DATA
+            else r.skipUnknown(key);
         }
     }
 
@@ -400,7 +400,7 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional<uint64_t> expect
     {
         readLineInto(in, row_line, line_cap, "fold seal");
         ReadBufferFromMemory l(row_line.data(), row_line.size());
-        row_reader.reset(l, KeyStrictness::Strict, "fold seal");
+        row_reader.reset(l, "fold seal");
         JsonObjectReader & r = row_reader;
         String key;
         if (!r.nextKey(key))
@@ -409,8 +409,8 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional<uint64_t> expect
         if (key == "n")
         {
             const uint64_t n = r.readU64Number();
-            if (r.nextKey(key))
-                throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: trailer has extra keys");
+            while (r.nextKey(key))
+                r.skipUnknown(key);
             if (!l.eof() || !in.eof())
                 throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: bytes after trailer");
             if (n != seen)
@@ -458,7 +458,7 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional<uint64_t> expect
                 else if (key == FoldSealWire::retry_round) hold_next_retry_round = r.readU64String();
                 else if (key == FoldSealWire::remove_epoch) remove_txn_epoch = r.readU64String();
                 else if (key == FoldSealWire::remove_seq) remove_txn_sequence = r.readU64String();
-                else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown ref_life key '{}'", key);
+                else r.skipUnknown(key);
             }
 
             if (!life_id || *life_id == 0)
@@ -539,7 +539,7 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional<uint64_t> expect
                 else if (key == FoldSealWire::checksum) checksum = r.readHex128();
                 else if (key == FoldSealWire::shard) shard = r.readU64Number();
                 else if (key == FoldSealWire::key_generation) generation = r.readU64String();
-                else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown run key '{}'", key);
+                else r.skipUnknown(key);
             }
             if (!run_key || !checksum || !shard || !generation)
                 throw Exception(ErrorCodes::CORRUPTED_DATA,
@@ -559,7 +559,7 @@ CasFoldSeal decodeFoldSeal(std::string_view data, std::optional<uint64_t> expect
                 else if (key == FoldSealWire::condemned_total) condemned_total = r.readU64Number();
                 else if (key == FoldSealWire::pending_total) pending_total = r.readU64Number();
                 else if (key == FoldSealWire::oldest_round) oldest_nonpending_condemn_round = r.readU64String();
-                else throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS fold seal: unknown condemned key '{}'", key);
+                else r.skipUnknown(key);
             }
             if (!shard || !condemned_total || !pending_total || !oldest_nonpending_condemn_round)
                 throw Exception(ErrorCodes::CORRUPTED_DATA,

src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.cpp

@@ -100,39 +100,36 @@ constexpr uint64_t kMiB = 1024 * 1024;
 /// small raw singletons.
 constexpr FormatTraits TRAITS[] =
 {
-    {FormatId::Blob,         "cas_blob",          TextFamily::PayloadHybrid, KeyStrictness::Tolerant, CompressionPolicy::Never,     256,        256},
-    {FormatId::BlobMeta,     "cas_blob_meta",     TextFamily::Control,       KeyStrictness::Tolerant, CompressionPolicy::Never,     1 * kMiB,   64 * kKiB},
-    {FormatId::PoolMeta,     "cas_pool_meta",     TextFamily::Control,       KeyStrictness::Tolerant, CompressionPolicy::Never,     1 * kMiB,   64 * kKiB},
-    {FormatId::RefLog,       "cas_ref_log",       TextFamily::Control,       KeyStrictness::Tolerant, CompressionPolicy::Always,    64 * kMiB,  64 * kMiB},
-    {FormatId::RefSnapshot,  "cas_ref_snap",      TextFamily::Control,       KeyStrictness::Tolerant, CompressionPolicy::Always,    64 * kMiB,  64 * kMiB},
+    {FormatId::Blob,         "cas_blob",          TextFamily::PayloadHybrid, CompressionPolicy::Never,     256,        256},
+    {FormatId::BlobMeta,     "cas_blob_meta",     TextFamily::Control,       CompressionPolicy::Never,     1 * kMiB,   64 * kKiB},
+    {FormatId::PoolMeta,     "cas_pool_meta",     TextFamily::Control,       CompressionPolicy::Never,     1 * kMiB,   64 * kKiB},
+    {FormatId::RefLog,       "cas_ref_log",       TextFamily::Control,       CompressionPolicy::Always,    64 * kMiB,  64 * kMiB},
+    {FormatId::RefSnapshot,  "cas_ref_snap",      TextFamily::Control,       CompressionPolicy::Always,    64 * kMiB,  64 * kMiB},
     /// `cas_ref_ckpt` is a three-field mutable singleton read by a point GET on every recovery and on
     /// every cleanup decision, so its caps are deliberately TIGHT (64 KiB / 4 KiB rather than the
     /// megabyte scale its Control-family siblings use): nothing legitimate approaches them, and the cap
-    /// is the first thing that fires if a foreign object ever lands at the key. STRICT for the same
-    /// reason its decoder is -- every field changes what cleanup may delete, so nothing in it may be
-    /// skipped. Raw (`Never`): a small singleton, and `publishCkpt` re-encodes it on every attempt.
-    {FormatId::RefCkpt,      "cas_ref_ckpt",      TextFamily::Control,       KeyStrictness::Strict,   CompressionPolicy::Never,     64 * kKiB,  4 * kKiB},
+    /// is the first thing that fires if a foreign object ever lands at the key. Raw (`Never`): a small
+    /// singleton, and `publishCkpt` re-encodes it on every attempt.
+    {FormatId::RefCkpt,      "cas_ref_ckpt",      TextFamily::Control,       CompressionPolicy::Never,     64 * kKiB,  4 * kKiB},
     /// `cas_ref_catalog` (INV-3): one object for the whole pool, token-CAS like `gc/state`, read on
-    /// every fold round and every recovery. STRICT for the same reason `cas_ref_ckpt` is -- every
-    /// field decides a namespace's lifecycle, so nothing in it may be skipped. Raw (`Never`): the
-    /// admission gate measures `encodeRefCatalog`'s own output directly, so a compressed size would
-    /// answer the wrong question. The object cap is the fold-seal's own 256 MiB (predicate (2) of the
+    /// every fold round and every recovery. Raw (`Never`): the admission gate measures
+    /// `encodeRefCatalog`'s own output directly, so a compressed size would answer the wrong question. The object cap is the fold-seal's own 256 MiB (predicate (2) of the
     /// additive admission check bounds it further via the entry count); the line cap is tight (4 KiB)
     /// because one entry's record is ordinarily small -- but not always small enough: a namespace or
     /// `server_root_id` near their own byte bounds, worst-case escaped, can push a single line past
     /// 4 KiB, and `encodeRefCatalog` REFUSES that entry (`LIMIT_EXCEEDED`, `CasRefCatalogFormat.cpp`'s
     /// `checkLineBytes`) rather than writing an object no reader could later decode.
-    {FormatId::RefCatalog,   "cas_ref_catalog",   TextFamily::Control,       KeyStrictness::Strict,   CompressionPolicy::Never,     256 * kMiB, 4 * kKiB},
-    {FormatId::GcMaintenanceState, "cas_gc_maintenance_state", TextFamily::Control, KeyStrictness::Strict, CompressionPolicy::Never, 512 * kKiB, 512 * kKiB},
-    {FormatId::PartManifest, "cas_part_manifest", TextFamily::PayloadHybrid, KeyStrictness::Tolerant, CompressionPolicy::Always,    256 * kMiB, 64 * kKiB},
-    {FormatId::RunFile,      "cas_run",           TextFamily::RecordStream,  KeyStrictness::Strict,   CompressionPolicy::PinnedRaw, 0,          4 * kKiB},
-    {FormatId::FoldSeal,     "cas_fold_seal",     TextFamily::Control,       KeyStrictness::Strict,   CompressionPolicy::PinnedRaw, 256 * kMiB, 64 * kKiB},
-    {FormatId::GcState,      "cas_gc_state",      TextFamily::Control,       KeyStrictness::Tolerant, CompressionPolicy::Never,     1 * kMiB,   64 * kKiB},
-    {FormatId::GcHeartbeat,  "cas_gc_hb",         TextFamily::Control,       KeyStrictness::Tolerant, CompressionPolicy::Never,     1 * kMiB,   64 * kKiB},
-    {FormatId::GcOutcomes,   "cas_gc_outcomes",   TextFamily::Control,       KeyStrictness::Tolerant, CompressionPolicy::Always,    256 * kMiB, 64 * kKiB},
-    {FormatId::Owner,        "cas_owner",         TextFamily::Control,       KeyStrictness::Tolerant, CompressionPolicy::Never,     1 * kMiB,   64 * kKiB},
-    {FormatId::ServerEpoch,  "cas_epoch",         TextFamily::Control,       KeyStrictness::Tolerant, CompressionPolicy::Never,     1 * kMiB,   64 * kKiB},
-    {FormatId::MountLease,   "cas_mount_lease",   TextFamily::Control,       KeyStrictness::Tolerant, CompressionPolicy::Never,     1 * kMiB,   64 * kKiB},
+    {FormatId::RefCatalog,   "cas_ref_catalog",   TextFamily::Control,       CompressionPolicy::Never,     256 * kMiB, 4 * kKiB},
+    {FormatId::GcMaintenanceState, "cas_gc_maintenance_state", TextFamily::Control, CompressionPolicy::Never, 512 * kKiB, 512 * kKiB},
+    {FormatId::PartManifest, "cas_part_manifest", TextFamily::PayloadHybrid, CompressionPolicy::Always,    256 * kMiB, 64 * kKiB},
+    {FormatId::RunFile,      "cas_run",           TextFamily::RecordStream,  CompressionPolicy::PinnedRaw, 0,          4 * kKiB},
+    {FormatId::FoldSeal,     "cas_fold_seal",     TextFamily::Control,       CompressionPolicy::PinnedRaw, 256 * kMiB, 64 * kKiB},
+    {FormatId::GcState,      "cas_gc_state",      TextFamily::Control,       CompressionPolicy::Never,     1 * kMiB,   64 * kKiB},
+    {FormatId::GcHeartbeat,  "cas_gc_hb",         TextFamily::Control,       CompressionPolicy::Never,     1 * kMiB,   64 * kKiB},
+    {FormatId::GcOutcomes,   "cas_gc_outcomes",   TextFamily::Control,       CompressionPolicy::Always,    256 * kMiB, 64 * kKiB},
+    {FormatId::Owner,        "cas_owner",         TextFamily::Control,       CompressionPolicy::Never,     1 * kMiB,   64 * kKiB},
+    {FormatId::ServerEpoch,  "cas_epoch",         TextFamily::Control,       CompressionPolicy::Never,     1 * kMiB,   64 * kKiB},
+    {FormatId::MountLease,   "cas_mount_lease",   TextFamily::Control,       CompressionPolicy::Never,     1 * kMiB,   64 * kKiB},
 };
 }
 

src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasFormat.h

@@ -98,10 +98,6 @@ std::span<const FormatChangePoint> changePoints(FormatId id);
 /// or a descriptor followed by raw payload bytes.
 enum class TextFamily : uint8_t { Control = 1, RecordStream = 2, PayloadHybrid = 3 };
 
-/// Whether a decoder skips unknown ordinary keys or rejects them. Critical keys prefixed with `!`
-/// are rejected by all families because they signal a required extension.
-enum class KeyStrictness : uint8_t { Tolerant = 1, Strict = 2 };
-
 /// Deterministic storage policy. `Always` uses whole-object zstd and a `.zst` key suffix; `Never`
 /// remains raw; `PinnedRaw` is raw because byte adoption compares the serialized bytes.
 enum class CompressionPolicy : uint8_t { Never = 1, Always = 2, PinnedRaw = 3 };
@@ -113,7 +109,6 @@ struct FormatTraits
     FormatId id;
     std::string_view type;      /// header-line "type" value
     TextFamily family;
-    KeyStrictness strictness;
     CompressionPolicy compression;
     uint64_t object_cap;        /// max DECOMPRESSED object bytes; 0 = uncapped (streamed)
     uint64_t line_cap;          /// max bytes of one text line

src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcMaintenanceStateFormat.cpp

@@ -43,7 +43,7 @@ GcMaintenanceState decodeGcMaintenanceState(std::string_view data)
     expectHeaderLine(in, FormatId::GcMaintenanceState);
     const String body = readLine(in, traitsFor(FormatId::GcMaintenanceState).line_cap, "cas_gc_maintenance_state");
     ReadBufferFromMemory body_in(body.data(), body.size());
-    JsonObjectReader reader(body_in, KeyStrictness::Strict, "cas_gc_maintenance_state");
+    JsonObjectReader reader(body_in, "cas_gc_maintenance_state");
 
     GcMaintenanceState result;
     bool has_cursor = false;

src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcOutcomesFormat.cpp

@@ -79,7 +79,7 @@ OutcomeLog decodeOutcomeLog(std::string_view data)
     {
         readLineInto(in, row_line, line_cap, "outcome log");
         ReadBufferFromMemory line_in(row_line.data(), row_line.size());
-        row_reader.reset(line_in, KeyStrictness::Tolerant, "outcome log");
+        row_reader.reset(line_in, "outcome log");
         JsonObjectReader & r = row_reader;
 
         String key;

src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasGcStateFormat.cpp

@@ -60,7 +60,7 @@ GcState decodeGcState(std::string_view data)
     expectHeaderLine(in, FormatId::GcState);
     const String body = readLine(in, traitsFor(FormatId::GcState).line_cap, "gc/state");
     ReadBufferFromMemory body_in(body.data(), body.size());
-    JsonObjectReader r(body_in, KeyStrictness::Tolerant, "gc/state");
+    JsonObjectReader r(body_in, "gc/state");
 
     GcState state;
     bool saw_gcs = false;
@@ -118,7 +118,7 @@ GcHeartbeat decodeGcHeartbeat(std::string_view data)
     expectHeaderLine(in, FormatId::GcHeartbeat);
     const String body = readLine(in, traitsFor(FormatId::GcHeartbeat).line_cap, "gc heartbeat");
     ReadBufferFromMemory body_in(body.data(), body.size());
-    JsonObjectReader r(body_in, KeyStrictness::Tolerant, "gc heartbeat");
+    JsonObjectReader r(body_in, "gc heartbeat");
 
     GcHeartbeat hb;
     bool saw_by = false;

src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPartManifestFormat.cpp

@@ -148,7 +148,7 @@ PartManifest decodePartManifest(std::string_view data)
     {
         const String meta = readLine(in, line_cap, "cas_part_manifest");
         ReadBufferFromMemory mm(meta.data(), meta.size());
-        JsonObjectReader r(mm, KeyStrictness::Tolerant, "cas_part_manifest");
+        JsonObjectReader r(mm, "cas_part_manifest");
         ManifestRefFields fields;
         std::optional<String> ns;
         std::optional<UInt128> pd;
@@ -185,7 +185,7 @@ PartManifest decodePartManifest(std::string_view data)
     {
         readLineInto(in, row_line, line_cap, "cas_part_manifest");
         ReadBufferFromMemory l(row_line.data(), row_line.size());
-        row_reader.reset(l, KeyStrictness::Tolerant, "cas_part_manifest");
+        row_reader.reset(l, "cas_part_manifest");
         JsonObjectReader & r = row_reader;
         String key;
         if (!r.nextKey(key))

src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Formats/CasPoolMetaFormat.cpp

@@ -110,7 +110,7 @@ PoolMeta decodePoolMeta(std::string_view data)
 
     const String body = readLine(in, traitsFor(FormatId::PoolMeta).line_cap, "pool meta");
     ReadBufferFromMemory body_in(body.data(), body.size());
-    JsonObjectReader r(body_in, KeyStrictness::Tolerant, "pool meta");
+    JsonObjectReader r(body_in, "pool meta");
 
     PoolMeta pm;
     bool saw_pid = false;