agrep does not handle encoding errors gracefully

[pat-mccavery]

Hi and thanks for providing us with agrep.

I am trying to run it on a file with > 9000 lines. The results appear truncated. If i cat and pipe the file into agrep I end up with about 100 results but if I use tac instead of cat, I end up with different results from the end of the file instead. I ran ./configure --enable-largefile but the problem remains.

Please help-Patrick

[dag-erling]

Hi Patrick,

Any chance you can share the file with us?

[pat-mccavery]

Hi Dag
I have uploaded it now. It is Menonnite Low German and English. I was making a small translator for myself to study this language. Thanks

TIDY1TO2.CSV

[dag-erling]

Thanks, I can say without hesitation that the bug is confirmed:

% wc -l tidy1to2.csv
9287 tidy1to2.csv
% ./src/agrep . tidy1to2.csv | wc -l
9287
% ./src/agrep -c . tidy1to2.csv
2109
% ./src/agrep . tidy1to2.csv | tail -1
zoology             UNKNOWN   tiakonnst.          UNKNOWN
% grep -c . tidy1to2.csv
9288
% grep . tidy1to2.csv | tail -1
zoology             UNKNOWN   tiakonnst.          UNKNOWN
% ./src/agrep NEUT-MASC tidy1to2.csv | wc -l
711
% ./src/agrep -c NEUT-MASC tidy1to2.csv
711
% ./src/agrep NEUT-MASC tidy1to2.csv | tail -1
doubter             NEUT-MASC twiewla             UNKNOWN
% grep -c NEUT-MASC tidy1to2.csv
2937
% grep NEUT-MASC tidy1to2.csv | tail -1
zone                NEUT-MASC eadstrijch          NEUT-MASC kjreiss

[dag-erling]

The actual issue is that your file is corrupt. See for instance the entry for dowager:

% grep -aw dowager tidy1to2.csv | hexdump -C
00000000  64 6f 77 61 67 65 72 20  20 20 20 20 20 20 20 20  |dowager         |
00000010  20 20 20 20 46 45 4d 20  20 20 20 20 20 20 72 69  |    FEM       ri|
00000020  6b 6a 65 20 20 20 20 20  20 20 20 20 20 20 20 20  |kje             |
00000030  20 20 55 4e 4b 4e 4f 57  4e 20 20 20 20 65 6c 6c  |  UNKNOWN    ell|
00000040  65 72 61 63 68 74 6a 65  20 77 c3 a4 74 66 72 c3  |erachtje w..tfr.|
00000050  0a                                                |.|
00000051

I assume the final word supposed to read “wätfrü” (widow) which would be encoded 77 c3 a4 74 66 72 c3 bc; the bc is missing. Here is a complete list of truncated entries:

% LC_CTYPE=C grep $'\xc3$' tidy1to2.csv
dowager             FEM       rikje               UNKNOWN    ellerachtje wätfr�
forage              UNKNOWN   fooda               UNKNOWN   ' no fooda e''romms�
grave               NEUT-MASC grauf               UNKNOWN   ' een eanstet be''n�
hoarfrost           UNKNOWN   riep                UNKNOWN   ' wan jewauss met n�
plaster             NEUT-MASC plosta              UNKNOWN   ' met miasse''ment �
wile                UNKNOWN   list                NEUT-MASC 'hinjalestjet be''n�

However, agrep should recover gracefully from this, like GNU grep does:

% ./src/agrep grave tidy1to2.csv
% grep grave tidy1to2.csv
gravedigger         NEUT-MASC graufgrowa          UNKNOWN
gravel              UNKNOWN   graunt              UNKNOWN    kjiess
grep: tidy1to2.csv: binary file matches
% grep -a grave tidy1to2.csv
grave               NEUT-MASC grauf               UNKNOWN   ' een eanstet be''n�
gravedigger         NEUT-MASC graufgrowa          UNKNOWN
gravel              UNKNOWN   graunt              UNKNOWN    kjiess

[pat-mccavery]

Hi Dag. I see, sorry for the the noise. Thanks for your time !