When there is a line ending in the text excerpts field (carriage return, new line), the stitched file may have different (hidden) character representing those line ending: Windows line ending "\r" and some on mac/linux have "\n"
It should not affect the data analysis in anyway, unless the text_excerpt field is counted for characters or combined or processed somehow.
However, this can cause an issue when doing a strict comparison of two versions of the stitched file (for error checking) making the strings in these fields to appear to be different by any table 'diff' process except for the invisible line ending character. It only happens on those rows with text excerpts that have a line ending. For example:
as.character(text_excerpt_v1)
[1] "HOST LIST OF AVIAN BROOD PARASITES - 2 - CUCULIFORMES - Old World cuckoos. Asian Emerald Cuckoo, Chrysococcyx maculatus. Species: Gould's Sunbird, Aethopyga gouldiae \n."
as.character(text_excerpt_v2)
[1] "HOST LIST OF AVIAN BROOD PARASITES - 2 - CUCULIFORMES - Old World cuckoos. Asian Emerald Cuckoo, Chrysococcyx maculatus. Species: Gould's Sunbird, Aethopyga gouldiae \r."
Solution:
ensure to strip unecessary line endings from text_excerpt or perhaps all fields when doing L0 cleaning: we need a new field cleaning function that gets called inside the read_and_amend() function the makes other corrections for typos when stitching.
Background:
While working on the repository, I ran the workflow to ensure the same data was generated, comparing the current stitched file that is in the repo ( Avian-Interaction-Database-Working/L0/ain_all_raw.csv ) with any new version to ensure they are identical using a table difference method using numeric fingerprint of rows, using the SHA256 hash like digest::digest(my_df[x,] algo="sha256")
When there is a line ending in the text excerpts field (carriage return, new line), the stitched file may have different (hidden) character representing those line ending: Windows line ending "\r" and some on mac/linux have "\n"
It should not affect the data analysis in anyway, unless the
text_excerptfield is counted for characters or combined or processed somehow.However, this can cause an issue when doing a strict comparison of two versions of the stitched file (for error checking) making the strings in these fields to appear to be different by any table 'diff' process except for the invisible line ending character. It only happens on those rows with text excerpts that have a line ending. For example:
Solution:
ensure to strip unecessary line endings from text_excerpt or perhaps all fields when doing L0 cleaning: we need a new field cleaning function that gets called inside the
read_and_amend()function the makes other corrections for typos when stitching.Background:
While working on the repository, I ran the workflow to ensure the same data was generated, comparing the current stitched file that is in the repo ( Avian-Interaction-Database-Working/L0/ain_all_raw.csv ) with any new version to ensure they are identical using a table difference method using numeric fingerprint of rows, using the SHA256 hash like
digest::digest(my_df[x,] algo="sha256")