Refactor common and goodbye readers....

[ipmach]

• Minor refactor main structure
• Removal of readers
• Removal of outputs
• Added From, To and FromTo library to support file reads from multiple formats for the different components.

[ipmach]

Use Case

Two examples of how to load and save data.

Methods use

 detector = DummyDetector()
 parser = DummyParser()

Standard use

Load data from a log file.

for parsed_log in From.log(parser, in_path=log_path):
    alerts.append(detector.process(parsed_log))

or run them independently:

for parsed_log in From.log(parser, in_path=log_path):
    parsed_logs.append(parsed_log)

for parsed_log in parsed_logs:
    To.json(detector.process(parsed_log), out_path=temp1)

Debug use

We want to save the data in every step

for parsed_log in FromTo.log2json(parser, in_path=log_path, out_path=temp1):
    pass

for alerts in FromTo.json2json(detector, in_path=temp1, out_path=temp2):
    pass

[ipmach]

@viktorbeck98 can you review this PR?

[viktorbeck98]

I left a comment for src/detectmatelibrary/utils/from_to.py. Looks good!

[viktorbeck98]

I like the functionality and the names but I am unsure whether the file location fits. This is more supposed to be a helper class for the developer / library user use case., no? @ipmach what do you think?

[ipmach]

Mm... yes. Should we create a helper folder? @viktorbeck98

[ipmach]

but following that logic, shouldn't be the persistency the same?

[viktorbeck98]

Just posting here again: Persistency will be used a lot in the detectors, whereas from_to.py will not, so that's where I would differ