Clean up series information

[aazaff]

Right now searches on the series field are completely inflexible if it is not an exact match.

So if it is DGM-35 in the metadata, a search for DGM 35 or DGM35 will not work. While this probably could and should be handled by better handling of fuzzy match in the API parameter (separate issue), I think an easy way to do this would be to clean up - in SQL - all series to use the same format.

Secondarily, I think that series should ALSO just always be added to keywords so it shows up in the general keywords search.

[NoisyFlowers]

Fuzzy search in the api appears to be working as expected.

• Entering "DGM-35" yields two collections. Database inspection confirms that is correct.
• Entering just "DGM" gives me all DGMs.
• Entering "-35" gives me the two DGM-35 collections plus others that have "-35" in them.
• Entering "35" does not yield these two collections. Looking at the ts_vectors for these, it appears that the tokenizing algorithm creates a "-35" but no a "35". I don't think we have a way to influence how the tokens are created.
• For collections that have "DGM-" plus other things in the series (e.g. "DGM-87, v1.0"), entering "DGM", "-87", or "v1.0" yields results that include that series.

There are no "DGM35" or "DGM 35" series in the database at this time. Running the following query to find anything similar:

select 
	json_data->>'series',
	json_data->>'title'
from
	metadata.azgs
where
	json_data->>'series' not like 'DGM-%' and
	json_data->>'series' like 'DGM%'

yields fifteen records, so it would not be hard to correct their series format.

I should add that fuzzy search still works as expected for those 15 records. For instance, there is a "DGM 140". Searching for either "DGM" or "140" returns that collection, among others that contain those tokens. There is also a "DGM126". Searching for either "DGM" or "126" does not return that collection, because there is no way for the tokenizer to determine how to break DGM126 into tokens.

To summarize: I think the fuzzy text search is working about as well as it can. I think we can modify the 15 errant series if we want to.

I'm not sure what is meant by "adding series to keywords so it shows up in the general keywords search".

[aazaff]

The keywords bit is a legacy statement from like x billion iterations of the system ago. It absolutely makes no sense in the current framework - you can ignore it.

By clean up those 15 records, do you just mean put a space between - and the number? That sounds reasonable.

[NoisyFlowers]

Here are the 15 weird DGMs:
WeirdSeries.csv

I don't know. It's up to you. I'd probably get rid of spaces and just make them all meet "DGM-xx" as a template, since that's the way all the others look.

But that's just DGMs. There are a bunch of other series. How do we define what they should properly look like? No idea.

[aazaff]

I think your solution is the one I wanted originally across the board. No spaces surrounding dashes.

[NoisyFlowers]

Those 15 DGMs patched in 85ba149

Live on prod

[aazaff]

We will come back to this when I have a student available to manually look through series we identify and clean them up.