Serialization format

Serialization format

Implementation as proposed in Serialization Format Idea.

Comparison with current method

Call data cost:

Test case	Current	Proposed
single 2-size ring, everything unique	42,276	43,856
single 2-size ring, wallet shared	42,340	42,496
single 2-size ring, fee token the same (not LRC)	45,060	45,216
single 2-size ring, prices match	42,212	42,704
single 2-size ring, dual author shared	42,276	37,680
single 3-size ring, everything unique	59,172	61,712
single 3-size ring, wallet shared	59,492	58,992
single 3-size ring, fee token the same (not LRC)	63,252	63,520
single 3-size ring, dual author shared	59,236	49,424
order filled by 3 other orders	76,796	73,848

There's a an extra cost store the offsets so if everything is unique in the orders the call data costs more. But as soon as even a single address is shareable the proposed method is just as good. If large parts of data can be reused (like the dual author signature) than the proposed method is a lot better.

The actual deserialization gas cost will approximatly be the same in both methods when deserializing all orders (only the proposed method is currently well optimized, but most optimizations can be used in the current method).

The proposed method supports random access. If only a couple of properties need to be extracted from the order it will cost significantly less gas. It costs ~4000 gas to deserialize a single order.

Serialization Format Overview:

This is how the data is stored in memory:

Header
Mining Data
Order Data
Ring Data
Data Blob

Header

uint16 serializationFormatVersion;
uint16 numOrders;
uint16 numRings;
uint16 numSpendables;

The header contains serializationFormatVersion. This version tells us how we can expect the data to be stored in memory. This allows us to support multiple formats inside the smart contract as long as we want to support them. Miners can still keep using an old version until they update to the latest version.

Mining data:

uint16 feeRecipientOffset;
uint16 minerOffset;
uint16 minerSignatureOffset;

Order Data

uint16 orderVersion;
uint16 ownerOffset;
uint16 tokenSOffset;
uint16 tokenBOffset;
uint16 amountSOffset;
uint16 amountBOffset;
uint16 validSinceOffset;
uint16 tokenSpendableSId;
uint16 tokenSpendableFeeId;
uint16 dualAuthorAddressOffset;
uint16 brokerOffset;
uint16 orderInterceptorOffset;
uint16 walletOffset;
uint16 validUntilOffset;
uint16 signatureOffset;
uint16 dualAuthorSignatureOffset;
uint16 allOrNone;
uint16 feeAmountOffset;
uint16 feePercentage;
int16 waiveFeePercentage;
uint16 tokenSFeePercentage;
uint16 tokenBFeePercentage;
uint16 tokenRecipientOffset;
uint16 walletSplitPercentage;
// Now properties added at the bottom

orderVersion is used to let the protocol know how the order was signed. This lets us know how the hash will need to generated and which order properties we are allowed to use. Depending on the serializationFormatVersion more order properties can be sent by the miner than the order had signed, so the protocol needs to ignore those new order properties. This will allow us to support multiple order versions, as long as the protocol still allows for that order version to be used.

New properties are added to the bottom. We can easily support multiple versions like this:

Load properties version 0
if (version >= 1) {
    Load properties version 1
}
if (version >= 2) {
    Load properties version 2
}
...

Properties from older versions that aren't used anymore in new versions can set their offset to 0 to cheaply not store any data for the property anymore.

Ring Data

For each ring, we always store 9 bytes like this:

uint8 ringSize;
uint8 orderIndex0;
uint8 orderIndex1;
uint8 orderIndex2;
uint8 orderIndex3;
uint8 orderIndex4;
uint8 orderIndex5;
uint8 orderIndex6;
uint8 orderIndex7;

Always storing the full 9 bytes for every ring allows us to randomly access any ring we want without having to deserialize the previous one.

Data Blob

Blob of data storing the data in any order.

Implementation details

• The data blob starts with 64 bytes of zeros. This allows us to store the default value at offset 0
• bytes is stored in the call data just like it is stored in memory. This allows us to copy the memory pointer from the input data to the order instead of having to copy any data
• Offsets are stored in a multiple of 4 to make the offsets smallers so there is only 1 non-zero byte for the offset most of the time