Obtaining transformation matrix (for other languages)

[lwarode]

Hi,

First of all, I would like to thank you for this great package and inspiring paper.

I'm just following your Quick Start Guide (https://github.com/prodriguezsosa/conText/blob/master/vignettes/quickstart.md) to get familiar with the procedure of applying embedding regression.

I have mainly 2 questions right now.

1. Do you provide a formula to learn the transformation matrix? In the tutorial you use cr_transform, while you mention that this is based on an estimation by Khodak et al. (2018) (khodakA.rds). However, is the formula below (from your 2023 paper) somehow implemented in conText?

2. This gets even more important if one tries to move to another language. If I would analyze the semantics of English texts, this would probably work with the provided data from the dropbox, however, I would like to analyze German text with respect to political ideology.

While German word embeddings are available: https://www.deepset.ai/german-word-embeddings, I haven't found corresponding transformation matrices for ALC so far. So I thought I could maybe rely on a function within your package or have to train the learn the transformation matrix on my own with the above formula? Are there any further obstacles when moving to another language beyond the pre-trained embeddings and the (yet to be estimated) transformation matrix?

Thanks in advance!

Best,
Lukas

[mweylandt]

Is it bad etiquette to jump in here as an outsider? Hope not!

conText::compute_transform() might be what you're looking for, unless I misunderstand.

But it does need more than just the final embeddings; because we need corpus frequencies to do some weighing.

I recently helped a friend make a transform for spanish, and I ended up training embeddings from scratch and then doing the transformation using the tutorial at the end of the readme. Retraining from scratch gave me access to the fcm compute_transform needs. I just wanted to chime in here to let you know that those embeddings (just following their code) and transformation matrix seemed to do a good job on a very small sample of texts.

[lwarode]

Oh sorry, you're right. Thanks!

[ArthurSpirling]

Thanks, @lwarode and @mweylandt. Just FYI we have specific resources for ALC transformation matrices and embeddings in many (non-English) languages here: https://alcembeddings.org/ Specifically for Spanish and German you can grab the relevant items here: https://alcembeddings.org/alcdata

HTH.

[mweylandt]

what an amazing resource, thanks so much! Didn't realize you had already put them up, many thanks

[lwarode]

Thanks @ArthurSpirling, this is really, really great. Just out of curiosity, do you think there are cases where the "lightweight" ALC transformation matrices are not sufficient and one should move to estimate larger transformation matrices, e.g. in a highly specialized context?

[ArthurSpirling]

Thanks @ArthurSpirling, this is really, really great. Just out of curiosity, do you think there are cases where the "lightweight" ALC transformation matrices are not sufficient and one should move to estimate larger transformation matrices, e.g. in a highly specialized context?

@lwarode thanks -- you mean, when should you estimate your own?

I would say that this depends on how similar you think your use-case is to our training data (wikipedia). If it's close, use our matrix; if further way, train your own.

Obviously, that's a vague answer; you can make it more concrete by perhaps using our resources as a first cut, and then looking at substantive information like nearest neighbors to see if it "makes sense" for your problem.

[yucluo]

so sad that there isn't simplied Chinese here https://alcembeddings.org/
Do folks know any resources for simplified Chinese embeddings?
BTW thanks a lot for the awesome resources

[ElisaWirsching]

so sad that there isn't simplied Chinese here https://alcembeddings.org/ Do folks know any resources for simplified Chinese embeddings? BTW thanks a lot for the awesome resources

Unfortunately, the Wiki dump lumps all versions of Chinese together in one (which is what we used for the Chinese embeddings and resources we provide). It looks like the conversion of the Wiki dump to simplified Chinese is non-trivial (see here). However, since the grammar is identical, have you tried subsetting or converting our Chinese embeddings to simplified Chinese (e.g., using this)? If this doesn't work, we can retrain the Chinese embeddings by doing the conversion of the raw text first. @yucluo please let us know.

[yucluo]

so sad that there isn't simplied Chinese here https://alcembeddings.org/ Do folks know any resources for simplified Chinese embeddings? BTW thanks a lot for the awesome resources

Unfortunately, the Wiki dump lumps all versions of Chinese together in one (which is what we used for the Chinese embeddings and resources we provide). It looks like the conversion of the Wiki dump to simplified Chinese is non-trivial (see here). However, since the grammar is identical, have you tried subsetting or converting our Chinese embeddings to simplified Chinese (e.g., using this)? If this doesn't work, we can retrain the Chinese embeddings by doing the conversion of the raw text first. @yucluo please let us know.

Thank you for your reply! I tried instead using FastText's Chinese pre-trained embeddings, and then computed my own transformation matrix using compute_transform(). It worked out fine! In my inspection, FastText's model contains both simplified and traditional Chinese so it should be fine?

[ElisaWirsching]

Glad to hear that the existing resources worked for you! Note that fastText uses the same Wikipedia dump as we do, i.e. a combination of traditional and simplified. So, theoretically, if the older fastText resources worked for you, ours should too (and possibly have a better fit based on our experience with other languages). I would recommend also checking the fit with our pre-trained embeddings and transformation matrix.

[yucluo]

Glad to hear that the existing resources worked for you! Note that fastText uses the same Wikipedia dump as we do, i.e. a combination of traditional and simplified. So, theoretically, if the older fastText resources worked for you, ours should too (and possibly have a better fit based on our experience with other languages). I would recommend also checking the fit with our pre-trained embeddings and transformation matrix.

I just tried out the Chinese embeddings on https://alcembeddings.org/ and I can confirm this works with simplified Chinese too. Hope this provides a useful data point for others.