Looking forward to `multi` README

[AmitMY]

As discussed in person, I am interested in using this to translate from SignWriting to spoke language text.

SignWriting, can be linearly represented like so:

However the way it is intended to be read by humans is like so:

While the specification allows for up to 500x500 representation for each sign, in practice, signs are a lot smaller.
For example, the white space in this sign is huge:

I will need to find what size covers 99% of the data.

Looking forward to multi README so I know how to prep the image data, increase the patch size/use multiple vertical and horizontal patches, and train the model

[esalesky]

Hi Amit,

The README is up. Some of your questions may be addressed by the README, but a few would require extensions beyond what is implemented here. Hope the below helps:

• Patch size is set by the arg --pixel-per-patch
• Scripts to train with the multilingual pixel_translation_multi_simple_epoch task can be found in the grid_scripts directory; for a single language pair without temperature sampling across languages, the visual-text task may be sufficient, scripts for which are shown in grid_scripts on the main branch
• It is not implemented to read in image data from disk, only either raw text or binarized rendered images in the fairseq format, or to tokenize images with multiple rows and columns of patches. Here are where you'd minimally need to make modifications to do so:
  • If you'd like to read in image files, you'd want to modify the VisualTextDataset to read in your images in place of text and tokenize them into format expected by the downstream tasks with image_generator. One option would be to have the source text files contain the image file paths per line aligned to the target text
    • The tensor functions in image_generator can be modified to appropriately tokenize your images both horizontally and vertically. When training with raw data, when constructing a batch, the code calls get_tensors() to generate a sequence of tensors representing the tokenized image for a given text sequence. This and the 3 fns it calls (get_tensor, get_image, slice) are what would be necessary to modify.
  • If the above have been modified to yield the same output format as ours, the data could be used directly for training or binarized the same way as our synthetic text images potentially without further modification
  • You'd likely also want to modify the positional embeddings to be 2D