Pose evaluation: how to handle "glosses" from PopSign ASL

[cleong110]

Trying to think about how/if to integrate PopSign ASL into experiments.

The PopSign ASL dataset doesn't actually use ASL Glosses for its categories, but English signs. So the vocabulary is not necessarily compatible with datasets such as ASL Citizen and Sem-Lex.

One possibility is to use the "ASL Knowledge Graph", which has relations like:

                         subject         relation             object object_dtype  source      type
id                                                                                                 
NaN                asllrp:rocket  has_translation          en:rocket       string  asllrp  semantic
NaN  asllrp:think+(2h)pass-out_2  has_translation  en:think_pass_out       string  asllrp  semantic
NaN         asllrp:female-figure  has_translation   en:female_figure       string  asllrp  semantic
NaN                    asl:stuck  has_translation           en:stuck       string  asllrp  semantic
NaN                    asl:stuck  has_translation           en:stuck       string  asllex  semantic

So we could

But there are issues with this:

1. It's not one-to-one. There can be multiple gloss translations for each english word. So for example here are possibilities for "en:lamp"

            subject         relation   object object_dtype     source      type
id                                                                             
NaN  iconicity:lamp  has_translation  EN:LAMP       string  iconicity  semantic
NaN     asllex:lamp  has_translation  EN:LAMP       string     asllex  semantic
NaN  asllrp:fs-lamp  has_translation  EN:LAMP       string     asllrp  semantic
NaN   asllex:lamp_2  has_translation  EN:LAMP       string     asllex  semantic

2. The ASL Knowledge Graph has some interesting cases that might take extra checking

• separate entries for "asllex:release, rescue" (with a space) and "asllex:release,_rescue" (with an underscore)
• special cases that made need extra parsing like "engine_3/cylinder", 'asllrp:#all', 'asllrp:(1h)good/thank-you+honeymoon','asllrp:#will', 'asllrp:#sept', 'asllrp:#dec', 'asllrp:(s)old+#ff',
• etc

However if...

1. there is only one gloss translation or all the gloss translations are the same
2. the gloss translation is exactly the same spelling as the english

Then we could get some pretty confident translations. For example there are 2 Translations of en:orange: ['asllrp:orange', 'asllex:orange'], so we can guess that "ORANGE" would be a valid gloss for popsign ASL videos with label "orange".

If we do this, we get 121 mappings, listed below:

[('EN:STUCK', 'STUCK'), ('EN:FACE', 'FACE'), ('EN:NIGHT', 'NIGHT'), ('EN:GIFT', 'GIFT'), ('EN:BOOK', 'BOOK'), ('EN:FLOWER', 'FLOWER'), ('EN:APPLE', 'APPLE'), ('EN:BOAT', 'BOAT'), ('EN:GREEN', 'GREEN'), ('EN:BLUE', 'BLUE'), ('EN:NOW', 'NOW'), ('EN:HOME', 'HOME'), ('EN:GIRAFFE', 'GIRAFFE'), ('EN:BROWN', 'BROWN'), ('EN:NOSE', 'NOSE'), ('EN:FIRST', 'FIRST'), ('EN:TONGUE', 'TONGUE'), ('EN:MOON', 'MOON'), ('EN:ANY', 'ANY'), ('EN:RED', 'RED'), ('EN:HEAD', 'HEAD'), ('EN:OWL', 'OWL'), ('EN:UP', 'UP'), ('EN:DONKEY', 'DONKEY'), ('EN:MANY', 'MANY'), ('EN:PERSON', 'PERSON'), ('EN:AUNT', 'AUNT'), ('EN:RADIO', 'RADIO'), ('EN:YESTERDAY', 'YESTERDAY'), ('EN:DRY', 'DRY'), ('EN:WET', 'WET'), ('EN:WOLF', 'WOLF'), ('EN:ON', 'ON'), ('EN:ELEPHANT', 'ELEPHANT'), ('EN:BOY', 'BOY'), ('EN:TREE', 'TREE'), ('EN:FISH', 'FISH'), ('EN:LION', 'LION'), ('EN:WHY', 'WHY'), ('EN:BEFORE', 'BEFORE'), ('EN:HIDE', 'HIDE'), ('EN:ALLIGATOR', 'ALLIGATOR'), ('EN:READ', 'READ'), ('EN:MAKE', 'MAKE'), ('EN:FARM', 'FARM'), ('EN:THIRSTY', 'THIRSTY'), ('EN:HAPPY', 'HAPPY'), ('EN:HIGH', 'HIGH'), ('EN:COW', 'COW'), ('EN:BLACK', 'BLACK'), ('EN:WAIT', 'WAIT'), ('EN:AIRPLANE', 'AIRPLANE'), ('EN:NOT', 'NOT'), ('EN:TOMORROW', 'TOMORROW'), ('EN:DROP', 'DROP'), ('EN:UNCLE', 'UNCLE'), ('EN:BECAUSE', 'BECAUSE'), ('EN:FROG', 'FROG'), ('EN:DANCE', 'DANCE'), ('EN:BUG', 'BUG'), ('EN:PIG', 'PIG'), ('EN:DIRTY', 'DIRTY'), ('EN:HAVE', 'HAVE'), ('EN:HOT', 'HOT'), ('EN:DOWN', 'DOWN'), ('EN:PENCIL', 'PENCIL'), ('EN:BIRD', 'BIRD'), ('EN:MORNING', 'MORNING'), ('EN:WHERE', 'WHERE'), ('EN:RAIN', 'RAIN'), ('EN:FINGER', 'FINGER'), ('EN:BROTHER', 'BROTHER'), ('EN:TOUCH', 'TOUCH'), ('EN:HORSE', 'HORSE'), ('EN:TIGER', 'TIGER'), ('EN:ZIPPER', 'ZIPPER'), ('EN:GUM', 'GUM'), ('EN:WHITE', 'WHITE'), ('EN:ANIMAL', 'ANIMAL'), ('EN:TOOTHBRUSH', 'TOOTHBRUSH'), ('EN:GRASS', 'GRASS'), ('EN:STICKY', 'STICKY'), ('EN:IF', 'IF'), ('EN:TIME', 'TIME'), ('EN:ORANGE', 'ORANGE'), ('EN:CEREAL', 'CEREAL'), ('EN:SHOWER', 'SHOWER'), ('EN:GIRL', 'GIRL'), ('EN:YELLOW', 'YELLOW'), ('EN:COWBOY', 'COWBOY'), ('EN:DOLL', 'DOLL'), ('EN:ROOM', 'ROOM'), ('EN:HEN', 'HEN'), ('EN:EAR', 'EAR'), ('EN:MOUSE', 'MOUSE'), ('EN:BYE', 'BYE'), ('EN:ZEBRA', 'ZEBRA'), ('EN:HUNGRY', 'HUNGRY'), ('EN:YOURSELF', 'YOURSELF'), ('EN:FIND', 'FIND'), ('EN:MOUTH', 'MOUTH'), ('EN:PENNY', 'PENNY'), ('EN:DRYER', 'DRYER'), ('EN:PEN', 'PEN'), ('EN:BACKYARD', 'BACKYARD'), ('EN:POOL', 'POOL'), ('EN:LIPS', 'LIPS'), ('EN:CHAIR', 'CHAIR'), ('EN:STAIRS', 'STAIRS'), ('EN:AWAKE', 'AWAKE'), ('EN:YUCKY', 'YUCKY'), ('EN:PLEASE', 'PLEASE'), ('EN:LOUD', 'LOUD'), ('EN:CHEEK', 'CHEEK'), ('EN:PRETTY', 'PRETTY'), ('EN:DRAWER', 'DRAWER'), ('EN:PRETEND', 'PRETEND'), ('EN:REFRIGERATOR', 'REFRIGERATOR'), ('EN:BEDROOM', 'BEDROOM'), ('EN:FRENCHFRIES', 'FRENCHFRIES'), ('EN:ICECREAM', 'ICECREAM')]

[cleong110]

Related #33