`force_align` on vectors missing (or: how to align vectors in flatcc?)

[benvanik]

[ Summary: flatcc should eventually support force_align on vectors - meanwhile use vectors of structs and force_align structs, or use low level builder calls to create aligned vectors access as discussed below - the problems investigated were eventually decided to be incorrect use of API - low level alignment doesn't work if the buffer isn't started first. ]

Hello! I have some variable-length data that must be aligned on a 16-byte boundary that I'm storing in a [uint8]:

table Buffer {
  data:[uint8];
}

At some point it looks like the google flatbuffers library started supporting force_align on these vector fields, so this works there:

table Buffer {
  data:[ubyte] (force_align: 16);
}

The current flatcc compiler doesn't support this new usage of force_align yet though:

[build] bytecode_module_def.fbs:103:17: error: 'force_align': known attribute not expected in this context

My use requires me being able to ensure alignment for compatibility with DMA operations and SIMD code, but there are also some other big flatbuffers users that are now requiring it: https://github.com/tensorflow/tensorflow/blob/1e66b5790c93a1752fd8ea92a146aff811f2ee95/tensorflow/lite/schema/schema_v3a.fbs

If force_align on vectors could be supported in the flatcc parser/builder that'd be fantastic, but I was also wondering if there was a workaround that could be used (I suspect this isn't the first time alignment has come up as a question :). I tried to directly do this by passing an alignment during construction of the vector but that doesn't seem to do what I expected (16 byte alignment of the data in the flatbuffer):

flatcc_builder_start_vector(fbb, 1, /*align=*/16, FLATBUFFERS_COUNT_MAX(1));
uint8_t* ptr = flatbuffers_uint8_vec_extend(fbb, length);
memcpy(ptr, source_data, length);
flatbuffers_uint8_vec_end(fbb);

I'd expect the flatbuffers_uint8_vec_t when loaded from the buffer to be at a 16 byte offset (or, ignoring file header weirdness, would at least be consistently stride 16) however the pointers I get back seem rather arbitrary (4-, 8-, and 12-byte alignment).
I've started trying to use vectors of structs (that do support force_align in flatcc) but that also doesn't seem to do what I expect, but would love to hear if any of you have seen solutions in the past that are known to work. Also I could be doing something very wrong - I just started digging into the internals of flatcc and don't know much yet :)

[benvanik]

it looks like the google flatbuffers compiler turns the force_align into a ForceVectorAlignment before starting the vector:
https://github.com/google/flatbuffers/blob/78f0c0d1d96a220163a03174d3240864a6a139da/include/flatbuffers/flatbuffers.h#L1667-L1675

Is there an equivalent on the flatcc builder API?

[mikkelfj]

Hi Ben, to your last question first. Flatcc does not have ForceVectorAlignment equivalent. Alignment is taken from the schema during construction where supported. (EDIT: not entirely true, see my follow up comment). (Though there is force assignment to table field members similar to what flatcdoes to bypass skipping default values, but that is something different.

I don't really recall exactly what is supposed to be supported. You report either suggests a regression or something that hasn't yet been implemented. If it is supported by Googles flatc compiler it certainly should be in flatcc as well.

In either case I fully support adding force_align to vectors, I'm just not sure how much time I can commit to that for the time being.

[mikkelfj]

If this is really important to you, you can call the low level builder interface to create the vector without generated code. This can be freely mixed with other code, you just have to be somewhat careful.

flatcc/include/flatcc/flatcc_builder.h

Lines 1407 to 1464 in 988b149

	/**
	* Creates a vector in a single operation using an externally supplied
	* buffer. This completely bypasses the stack, but the size must be
	* known and the content must be little endian. Do not use for strings
	* and offset vectors. Other flatbuffer vectors could be used as a
	* source, but the length prefix is not required.
	*
	* Set `max_count` to `FLATBUFFERS_COUNT_MAX(elem_size)` before a call
	* to any string or vector operation to the get maximum safe vector
	* size, or use (size_t)-1 if overflow is not a concern.
	*
	* The max count property is a global property that remains until
	* explicitly changed.
	*
	* `max_count` is to prevent malicous or accidental overflow which is
	* difficult to detect by multiplication alone, depending on the type
	* sizes being used and having `max_count` thus avoids a division for
	* every vector created. `max_count` does not guarantee a vector will
	* fit in an empty buffer, it just ensures the internal size checks do
	* not overflow. A safe, sane limit woud be max_count / 4 because that
	* is half the maximum buffer size that can realistically be
	* constructed, corresponding to a vector size of `UOFFSET_MAX / 4`
	* which can always hold the vector in 1GB excluding the size field when
	* sizeof(uoffset_t) = 4.
	*/
	flatcc_builder_ref_t flatcc_builder_create_vector(flatcc_builder_t *B,
	const void *data, size_t count, size_t elem_size, uint16_t align, size_t max_count);
	/**
	* Starts a vector on the stack.
	*
	* Do not use these calls for string or offset vectors, but do store
	* scalars, enums and structs, always in little endian encoding.
	*
	* Use `extend_vector` subsequently to add zero, one or more elements
	* at time.
	*
	* See `create_vector` for `max_count` argument (strings and offset
	* vectors have a fixed element size and does not need this argument).
	*
	* Returns 0 on success.
	*/
	int flatcc_builder_start_vector(flatcc_builder_t *B, size_t elem_size,
	uint16_t align, size_t max_count);
	/**
	* Emits the vector constructed on the stack by start_vector.
	*
	* The vector may be accessed in the emitted stream using the returned
	* reference, even if the containing buffer is still under construction.
	* This may be useful for sorting. This api does not support sorting
	* because offset vectors cannot read their references after emission,
	* and while plain vectors could be sorted, it has been chosen that this
	* task is better left as a separate processing step. Generated code can
	* provide sorting functions that work on final in-memory buffers.
	*/
	flatcc_builder_ref_t flatcc_builder_end_vector(flatcc_builder_t *B);

Also, if you can help investigate if this is a regression or a missing feature, it would be helpful.

[mikkelfj]

Another way to work around this is to create a struct with the necessary alignment and store that struct in a vector. It slowly comes back now - I think flatc originally skipped vector force_align because you could do this instead.

Using a struct in this way will not physically change the buffer and you can cast from a simple array if you create the vector in one operation from a C array, and likewise when reading.

EDIT: I should read more close:

I've started trying to use vectors of structs (that do support force_align in flatcc) but that also doesn't seem to do what I expect, but would love to hear if any of you have seen solutions in the past that are known to work. Also I could be doing something very wrong - I just started digging into the internals of flatcc and don't know much yet :)

Can you please elaborate on why this doesn't work?

You can also use the verifier to test buffer alignment. (For the same reason the verifier must be updated upon changes to force_align support).

[benvanik]

Thanks for the suggestions! I agree that the support for force_align on vectors is not a regression in flatcc but was just a late addition to flatbuffers at some point - I suspect someone came along and ran into the same thing I did and threw it in there :)

I think I noticed what was throwing me off about my direct flatcc_builder_start_vector use: it is aligning correctly, however everything is offset 8 bytes in the final buffer due to the file identifier. If I strip the first 8 bytes with the identifier then everything is correctly aligned to what I specify. I suppose the question then becomes: is there a way to tell flatcc to pad out the buffers to ensure that alignment is still respected when the file identifier is prefixed? I think even if force_align was supported this may still be an issue that prevents the intended alignment from being met.

Today I'm doing:

file_identifier "FILE";
table MyRoot {
  data:[uint8];
}
root_type MyRoot;

flatcc_builder_init(&fbb);
flatcc_builder_start_vector(&fbb, 1, 16, FLATBUFFERS_COUNT_MAX(1));
uint8_t* p = flatbuffers_uint8_vec_extend(&fbb, 1234);
memset(p, 'x', 1234);
flatbuffers_uint8_vec_ref_t data = flatcc_builder_end_vector(&fbb);
MyTable_start_as_root(&fbb);
MyTable_data_add(&fbb, data);
MyTable_end_as_root(&fbb);
// etc

The output buffer then has 08000000 46494C45 for the identifier and later on the xxxxxxx at a +8 alignment.

Thoughts? And thanks for the help! If I can get something working I'm happy to write it up in the docs so that others coming along can use it too :)

[mikkelfj]

I'd be happy to help you progress on this. The following is an explanation of how things work or are supposed to work, but it probably won't help much. But it is a start:

I think I noticed what was throwing me off about my direct flatcc_builder_start_vector use: it is aligning correctly, however everything is offset 8 bytes in the final buffer due to the file identifier. If I strip the first 8 bytes with the identifier then everything is correctly aligned to what I specify.

This is not correct. The header is included in the alignment. Are you sure the buffer is placed in sufficiently aligned memory? There are subtle differences in the way a buffer is created if you create it with an identifier or with a null identifier (I don't recall all the details up front). The buffer can also be created with an optional size prefix.

The buffer can have a header in a few different ways: there can optionally be a file identifer, and there can optionally be a size prefix. You need to all the verifier in the proper way, and then it will very from the buffer start.

There are some variants start_as_root_with_identifier and with _size.

The header format is : [<size:4 bytes>]<offset to root object:4 bytes>[<file identifier: 4 bytes>]

(personally I think this is a mess, but it is what we have)

It is actually rather complex, but when creating a buffer, flatcc adjusts alignment according these different kinds of headers. Technically it builds the buffer backwards, properly aligned relative to a virtual zero point, and then inserts enough prepadding between the header and data to ensure that everything keeps staying aligned from the buffer start based on the largest alignment requirement seen during construction.

Related, but not very important here:
You may also notice that vectors are aligned to their elements and that flatcc reader returns a pointer to the first element, which should be aligned. But the actual vector starts 4 bytes earlier, and this vector header is not ailgned if elements are larger than 4 bytes because the vector size field is 4 bytes.

[mikkelfj]

Here is the padding before the header:

flatcc/src/runtime/builder.c

Line 769 in e327f24

header_pad = front_pad(B, field_size + id_size + (uoffset_t)(with_size ? field_size : 0), align);

[mikkelfj]

Thoughts? And thanks for the help! If I can get something working I'm happy to write it up in the docs so that others coming along can use it too :)

Documentation is always welcome. Buf if you dig sufficiently into this, you might consider implementing support for vector force align. I think it might be relatively easy, albeit hard to grasp.

flatcc/src/compiler/symbols.h

Lines 223 to 228 in e327f24

	/*
	* `align`, `offset` are for structs only. 64-bit allows for
	* dynamically configured 64-bit file offsets. Align is restricted to
	* at most 256 and must be a power of 2.
	*/
	uint16_t align;

Here struct sets the align field

flatcc/src/compiler/semantics.c

Lines 668 to 675 in e327f24

	if ((m = knowns[fb_attr_force_align])) {
	if (!is_valid_align(m->value.u)) {
	error_sym(P, &ct->symbol, "'force_align' exceeds maximum permitted alignment or is not a power of 2");
	} else {
	/* This may still fail on natural alignment size. */
	ct->align = (uint16_t)m->value.u;
	}
	}

In code generation, the align field might be ignored where not supported, i.e. non structs, but it could also already be supported, since e.g. integers have different alignment requirements already, so merely setting the align field could possibly work.

Complications: The generated verifier must also support this, and the JSON parser must also, but again, if the align field is already inspected, it might happen automatically.

On the other hand, if structs are not aligned correctly in vectors right now, there are bigger problems.

Also important to never reduce alignment below natural alignment.
I think this has something to do with it:

flatcc/src/compiler/semantics.c

Line 112 in e327f24

static inline uint64_t fb_align(uint64_t size, uint64_t align)

[benvanik]

Thank you for explaining (I really appreciate it!) - but I'm realizing that I'm still missing something :)

I'm using flatcc_builder_copy_buffer to get the final builder bytes and fwrite-ing them to a file which then verifies/loads correctly with flatcc so I'm (fairly) sure it's ok. I'm looking at the bytes in the file as written to disk in a hex editor and definitely seeing that they are not 16 byte aligned as I had specified (the 'xxxx's being the data as in the example above):

There's dozens of these vectors in the file and all are exactly +8, which is great as it means they are all consistently aligned whereas before using the vector_start with align=16 they were all over the place, but also means I don't understand what you mean about the file header being included in the alignment. Is it possible there's a bug in flatcc's handling of this alignment, or something I need to set to ensure it is included?

I have been hacking around in my own code but if this seems like it should be working then I'll build out a reproducer based on samples/bugreport/ to help isolate whether it's my code producing the buffers or not.

[mikkelfj]

I wrote the following just before receiving your last comment:

I was wondering: there could be a bug in the builder api where it does align vectors correctly but it fails to record the alignment size in the buffer state such that the prepadding works on wrong assumptions. This would also affect 64-bit integers because their alignment are larger than what is minimally required, so it ought to have been detected by now if that were the case.

[mikkelfj]

I don't understand what you mean about the file header being included in the alignment

This means that if a something is aligned to, say, 16 bytes, then that something is placed at multiple of 16 bytes relative to the start of buffer where the header is placed. At least it is supposed to.

[benvanik]

Ok cool - sounds like a reproducer is worth it! I'll put one together now.

[mikkelfj]

flatcc/include/flatcc/flatcc_builder.h

Lines 407 to 408 in e327f24

	uint16_t min_align;
	/* The current active objects alignment isolated from nested activity. */

This min_align should be updated whenever adding something to the builder.

You could try adding a function flatcc_builder_set_min_align(B, align); and see if that helps. It just needs to set min_align to max of existing min_align and the new argument. Call it after starting the buffer and before ending it. See if that helps.

[mikkelfj]

It is called for create_vector:

flatcc/src/runtime/builder.c

Lines 1380 to 1381 in e327f24

	get_min_align(&align, field_size);
	set_min_align(B, align);

But it appears to be missing from start_vector:

flatcc/src/runtime/builder.c

Line 1026 in e327f24

get_min_align(&align, field_size);

Does it help to add it there?