Issue with ncs function

[ElisaWirsching]

Hi @prodriguezsosa,

I just realized a small issue with the ncs function. It happens that the same context is assigned to both groups. So, for example, the context appearing in a Republican speech might be assigned as a top Democrat context. This is because the ncs function calculates cosine similarities between the group-specific embedding for a term and all contexts. I guess you would want to subset the eligible contexts for each group to only those actually appearing in a group-specific text? Here is a suggestion on how to change the code:

cos_sim <- text2vec::sim2(x = as.matrix(contexts_dem), y = as.matrix(x), method = "cosine", norm = "l2") %>% data.frame()
contexts_df <- data.frame(docid = quanteda::docid(contexts), context = sapply(contexts, function(i) paste(i, collapse = " ")),
                          docgroup = docvars(contexts)[,2])
cos_sim <- cos_sim %>% dplyr::mutate(docid = rownames(cos_sim)) %>% dplyr::left_join(contexts_df, by = 'docid')

result <- tidyr::pivot_longer(cos_sim, -c(docid, context, docgroup), names_to = "target") %>%
  dplyr::filter(docgroup==target) %>% 
  dplyr::group_by(target) %>%
  dplyr::slice_max(order_by = value, n = N) %>%
  dplyr::mutate(rank = 1:dplyr::n()) %>%
  dplyr::arrange(dplyr::desc(value)) %>%
  dplyr::ungroup() %>%
  dplyr::select('target', 'context', 'rank', 'value')

[prodriguezsosa]

Thanks @ElisaWirsching, good point. I had considered something along those lines but I opted to leave it as is for a couple of reasons:

i. x won't always have multiple (group) embeddings e.g. it could be a single full corpus embedding. In the current setup, target is set to NULL when it's a single embedding. But we can adjust your proposed fix to account for this case.

ii. flexibility, i.e. you might plug in other embeddings into x that don't belong to any of the groups in docvars(context). This is perhaps not the typical use-case but the current setup allows for it. Again we could adjust your fix to account for this.

iii. it could be we don't want to limit candidate nearest contexts to each group's contexts i.e. it's not uninformative that some of the nearest contexts are shared (similar to nearest neighbors where we usually don't limit the candidate terms to each group's respective vocabularies).

That said, in our paper I did limit each party's candidate contexts to those appearing in their respective corpora. The way I did this (and expected users to do if they thought it necessary) is:

ncs(x = immig_wv_party["R",], contexts_dem = immig_dem[immig_dem@docvars$party == "R",], contexts = immig_toks[docvars(immig_toks, 'party') == "R",], N = 5)

But, open to pushback/other suggestions on the above.

cc. @ArthurSpirling @bstewart

[ArthurSpirling]

Thanks, @prodriguezsosa and @ElisaWirsching. This basically boils down to "expected" behavior, and fwiw, I had the same instincts as Elisa.

In my case, I was looking at the ncs for two different political groups, and was completely confused to find the same nearest contexts in both. This is especially confusing for an end-user IMO because I think the primary use of ncs is try to interpret how this group is using this term. So the consequence of seeing the same context in the printed object is believing that they use the term the same way---which is true in a somewhat roundabout way (the meanings are "close" in cosine space), but not true in a much more direct way: this is not a context this group uses.

So for me: I think default behavior, if groups are specified, is to only return within group instantiations.

[ElisaWirsching]

Thank you @prodriguezsosa for clarifying! I do see your arguments, especially the comparison to nearest neighbors, and why the current setup offers more flexibility. I agree with @ArthurSpirling that it remains a bit confusing. If you keep the function as is, would it make sense to make this aspect very clear in the quickstart md file and/or the description of the function, and to show the manual adjustment you mentioned explicitly?

[prodriguezsosa]

Thanks @ElisaWirsching @ArthurSpirling. I agree on your intuitions as to expected behavior. I'll think of an appropriate solution. At a minimum, agreed that if it stays as is, then:

would it make sense to make this aspect very clear in the quickstart md file and/or the description of the function, and to show the manual adjustment you mentioned explicitly

[ArthurSpirling]

Is there a straightforward way for the ncs function to check if the argument for x= inherits some groups, or more directly if the argument supplied to x= was created via dem_groups and then restrict to that grouping by default?

On the ground this would just be a

ncs(..., ..., bygroup=TRUE )

default, but it could also issue a warning? or some sort of reminder to screen?

[prodriguezsosa]

Is there a straightforward way for the ncs function to check if the argument for x= inherits some groups, or more directly if the argument supplied to x= was created via dem_groups and then restrict to that grouping by default?

Yes, we can check the rownames of x correspond to the groups in docvars(contexts). This sounds reasonable to me.

[prodriguezsosa]

Yes, we can check the rownames of x correspond to the groups in docvars(contexts).

This is somewhat less straightforward than I implied above, it would require checking if the rownames of x align with any of the variables in contexts_dem@docvars. We could ask the user to specify which is the grouping variable and check whether it includes rownames(x).

I'm also considering including contexts (i.e. the tokenized contexts) as an attribute in context_dem, so we'd remove the need for the context arg. The tradeoff is the size of the resulting context_dem object.