Handle storage_service.json vs storage_service.raw more consistently

[Alex-Ikanow]

The idea is that storage_service.raw is the pure raw file with no processing at all whereas storage_service.json is after input parsing but before any enrichment has occurred.

This is handled in a few different places:

• After processing, the hadoop batch enrichment will move the files in its input queue into raw if that is enabled - this makes sense if they are binary/XML/csv etc (*), but is a bit counter-intuitive if they are JSON
  • Though later on if we add JSON parsing options then it might make more sense
  • eg if you are running Flume then I think you'll get two v similar copies of the JSON in raw and json
• I believe that Storm Stream enrichment taps the kafka input queue and writes into json if enabled (though it's possible I was doing that using Flume, see below, you'd have to check the code)
• I am not sure what the batch/streaming analytic versions will do with raw/json, I think nothing
• Currently spark (which is nearly a combined batch/streaming enrichment engine, though the interface is not wired up) does not do anything
• (I believe the Flume harvester can be configured to (/will always?) create the json files - but it will never create the raw files - see below, batch enrichment will currently may then take its HDFS input queue and move that into raw)

(note that processed works fine, that is hooked up to the MultiDataServiceOutput class)

(*) raw csv input hasn't yet been ported from v1 (or flume, which contains the v1 logic), but it's trivial eg XML took a couple of hours to add.