Łukasiewicz / Lukasiewicz

[colinallen]

https://www.inphoproject.org/thinker/3495 - Lukasiewicz
https://www.inphoproject.org/thinker/5554 - Łukasiewicz

Only the latter has associated data mining from SEP (it has sepdir)

Only the former is visible by searching

Before merging these records, need to fix searching to find Łukasiewicz by typing Luka...

[JaimieMurdock]

Should our general policy be to display Unicode and Unidecode all input and regexes?

[colinallen]

I think that’s the best way to do it — most non-Poles won’t know how to type ‘Ł’ so will start with ‘L'

…

On Dec 26, 2017, at 11:11, Jaimie Murdock ***@***.***> wrote: Should our general policy be to display Unicode and Unidecode all input and regexes? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#167 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AArCqw6QGX9-RTk_Ylig-ibKvwH4yPa2ks5tERq1gaJpZM4RM2Ku>.

[JaimieMurdock]

As a global policy across all inputs and all searches?

[colinallen]

I can’t think of any downsides to this, can you?

…

On Dec 26, 2017, at 11:24, Jaimie Murdock ***@***.***> wrote: As a global policy across all inputs and all searches? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#167 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AArCq4XgxJNWwPEvM-l-M8qyiL2mf51Tks5tER2ugaJpZM4RM2Ku>.

[JaimieMurdock]

None. It will take some significant refactoring to clean up the code to make this universal, but doable.

…

On Dec 26, 2017 10:42 AM, "colinallen" ***@***.***> wrote: I can’t think of any downsides to this, can you? > On Dec 26, 2017, at 11:24, Jaimie Murdock ***@***.***> wrote: > > As a global policy across all inputs and all searches? > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub < #167 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe- auth/AArCq4XgxJNWwPEvM-l-M8qyiL2mf51Tks5tER2ugaJpZM4RM2Ku>. > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#167 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAoqgIxkBrQmODN2ugQGM-KH5DstUliEks5tESIPgaJpZM4RM2Ku> .

[colinallen]

I deleted Lukasiewicz from the database, which means that Łukasiewicz is now not searchable from the header search box, when beginning with 'L'.

Assigning this to @kirtansakariya to have a go. Kirtan, the task here is to make all unicoded names for entities searchable via the closest ascii.

This is tricky to accomplish, however, because there's no standard conversion dictionary.

With import unicodedata you can do something like this:
unicodedata.normalize('NFKD','égal').encode('ascii','replace') which returns b'e?gal'
which if you strip out the unprintable '?' gives something reasonable.

Unfortunately this does not work for most characters. E.g.,
unicodedata.normalize('NFKD','Łukasiewicz').encode('ascii','replace')
returns
b'?ukasiewicz'
where the first character is now unprintable and there's no 'L'.

Or you can use 'ignore' instead of 'replace' and that returns
b'ukasiewicz'

See https://docs.python.org/2/library/unicodedata.html and https://stackoverflow.com/questions/517923/what-is-the-best-way-to-remove-accents-in-a-python-unicode-string for other suggestions, none of which are ideal.

Brainstorm other approaches here!