Question about space like chunking

[frank6200db]

Again, thanks to the authors for open-sourcing some of the training codes that benefit reproduction.

How to implement rule based chunking: Space like chunking mentioned in the paper? Some examples would be great!

As I obtained from the paper is that the chunking mechanism is operated by trained linear Q and linear K, but how to use space like chunking in training?

Thanks.

[brwa-cartesia]

for our space-like chunking, we copied spacebyte, so strictly rule-based depending on whether input bytes are in some fixed set of "space-like bytes". in this case, there is no dynamically learned chunking, so no learned router (and correspondingly no EMA dechunking procedure)

[frank6200db]

for our space-like chunking, we copied spacebyte, so strictly rule-based depending on whether input bytes are in some fixed set of "space-like bytes". in this case, there is no dynamically learned chunking, so no learned router (and correspondingly no EMA dechunking procedure)

Thank you. And for clearance, for Hnet (space) mentioned in table.3 , the difference is that we still use Mamba as encoder and decoder but no routing or dechunking procedure?

[brwa-cartesia]

for Hnet (space) mentioned in table.3 , the difference is that we still use Mamba as encoder and decoder but no routing or dechunking procedure?

yes, that's correct. there are a few other changes from vanilla spacebyte (e.g. norm placement, projection on the residual), so that it matches h-net exactly except in the router and the dechunking. we found that all of these changes were helpful.

[frank6200db]

Thank you. But i have another question regarding the code evaluation and math benchmark evaluation.

Since I found this model is not currently supported by vllm or other accelerating frameworks. Can you kindly provide a reference script demo? Thank you.

[brwa-cartesia]

Since I found this model is not currently supported by vllm or other accelerating frameworks. Can you kindly provide a reference script demo? Thank you.

you can use generate.py in the base of the directory to generate (with batch size B=1), though i expect it will not perform well on code benchmarks as it's not instruction tuned.