Score distribution plots

[cleong110]

Our README originally had this as a potential qualitative way to analyze metrics. We actually can generate this plot, we have all the scores and everything, we just didn't get around to it for the study.

We would expect a plot like this to look mostly high for a good metric, but somewhat lower for in-class samples.

(note: image is from https://www.researchgate.net/publication/385010313_signwriting-evaluation_Effective_Sign_Language_Evaluation_via_SignWriting)

[cleong110]

Score distributions for ASL Citizen, the full matrix

[cleong110]

[cleong110]

score_distributions_1k.zip

Here's score distributions for 109 metrics but only the first 1.1k test files

[cleong110]

Gloss-Pair Mean Score Distribution Plots

Here's the result if you:

• Calculate distances from every member of the ASL Citizen train set to every member of the ASL Citizen test set.
• categorize ASL Citizen gloss pairs into Semantically Related, Visually Related, Both, and Neither, and Self-Scores
• Find the mean dot product of every asl_finetune_checkpoint_best embedding for each pair, or the mean DTWp or nDTWp. Note that order matters: APPLE/BANANA and BANANA/APPLE are not symmetric for DTWp or nDTWp. This part is hard and requires Dask and a big computer, or a long time and very difficult coding. The table does not fit in memory.
• Make plots

semantic_vs_visual_similarity_plots.zip

All told that's about 2731^2=74 million gloss pairs.

Univariate (one-metric)

DTWp

nDTWp

SignCLIP asl finetune checkpoint best (dot-products)

Bivariate Plots

with max of 100k samples per group.

Replication details

How'd I get this?

Scores

The parquet file containing all 74 million pairs, with categorization groups, can be found here: https://drive.google.com/file/d/1jFK9QHIWwYoxMhoFirvDE0YNIEWEWzEo/view?usp=sharing

Semantically Similar Signs

Notebook demonstrating how to get this: https://colab.research.google.com/drive/1KUOnOvQpjsHmm6fwdqW3_lKX5oIpw4NI

Visually Similar Signs

Notebook about how to compile the pairs: https://colab.research.google.com/drive/1er0ivuwyGo0FTUVEZ57od5sCBfMtJKCF?usp=sharing