The idea is that you give it a mix of paths/regexes etc and it creates a (per bucket) database and keeps track of all the file times (on a per directory level maybe?) and will copy new files into the batch enrichment directory (or have mode where copies somewhere else, eg longer term might want to make part of a multi harvester together with flume/logstash)
Note: not intended to handle streaming files, only complete ones
Could also check when a file is still being accessed (/was accessed in the last N minutes) to make life easier as far as atomicity is concerned
The idea is that you give it a mix of paths/regexes etc and it creates a (per bucket) database and keeps track of all the file times (on a per directory level maybe?) and will copy new files into the batch enrichment directory (or have mode where copies somewhere else, eg longer term might want to make part of a multi harvester together with flume/logstash)
Note: not intended to handle streaming files, only complete ones
Could also check when a file is still being accessed (/was accessed in the last N minutes) to make life easier as far as atomicity is concerned