Training code?

[Ryu1845]

I see there's code to generate from the pretrained models and modeling code but I don't see how to train a model. Is releasing the training code planned?

[Himanshunitrr]

👀

[sukjunhwang]

We are considering releasing something for training, but it might take a few (or several) weeks.

[ychuest]

Very anxious to see the pre-training scripts sooner rather than later.

[PatrickEleeve]

We are considering releasing something for training, but it might take a few (or several) weeks.

That would be crazy... Looking forward

[main-horse]

#13

[leedrake5]

Hi all,

I'm working with a team of folks trying to build an LLM to translate cuneiform documents. These are typically transliterated in the following way (using Hittite as an example):

is ka4 ru uh k u babbar kas da a i ar ha -> sacrificial vessel take silver beer from * away

The nature of transliterated cuneiform - the gold standard in academic research, follows the way your model processes data. This, plus getting away from sentencepiece, makes us optimistic that HNet could outperform other model types for these data. Since most tablets remain untranslated, this would be a model that could be put to good work making history more accessible to researchers and the public alike. I've got T5 family models like this one finished, but I'm not convinced it's the best we can do.

I've spent today trying to load HNet to train it, but I just can't get the model to load and run. Pretty much every attempt results in a TypeError that expects a from_pretrained function for HNetForCasualLM. I'd appreciate any guidance on how to get this working.

[malcolmsharpe]

@leedrake5 @main-horse's most recent repo https://github.com/main-horse/hnet-impl has worked for me for training. (It doesn't have a full training loop with data loading, but it's not too hard to add that.) I can't speak to the quality it gets for a serious run, but tiny runs seem to train successfully.

[CharlesDDDD]

For those who want to train byte-level LMs, aunet (https://github.com/facebookresearch/lingua/tree/main/apps/aunet) (essentially an advanced version of SpaceByte) is a good option to use. You can easily modify the chunker to support dynamic chunking or design your own approach as we did in our BFlowNet.