Migrate splunk-forwarder service to k8s

[efinlay24]

We currently have the splunk-forwarder service as a systemd unit, just like in coco.

We should take a closer look to see if we can migrate it to k8s.
The difficulty comes from 2 parts:

1. The command of starting the service has 2 bash pipes:

 journalctl -a -f --since=now --output=json \
        | docker run -i --log-driver=none -e=ENV=$ENV --rm --name %p-filter_$(uuidgen) --memory="256m" coco/coco-logfilter:$DOCKER_LOGFILTER_VERSION \
        | docker run -i --log-driver=none --rm --name %p-http_$(uuidgen) --memory="256m" -e="FORWARD_URL=$FORWARD_URL" -e="ENV=$ENV" -e="HOSTNAME=$(hostname -s)" -e="WORKERS=8" -e="BUFFER=256" -e="TOKEN=$TOKEN" -e="BATCHSIZE=$BATCHSIZE" -e="BATCHTIMER=5" coco/coco-splunk-http-forwarder:$DOCKER_FORWARDER_VERSION'

2. accessing the journal is done by using the journalctl command. Not sure how we'd do this from a container.

Look at fluentd as a possible solution for this.See https://financialtimes.slack.com/archives/C96U2ERJL/p1520953327000216

---

As suggested by @GeorgeAntoniadis for point 1 we can use Docker Inside Docker by mounting the docker socket to communicate with the host daemon, and so we would start piped docker containers inside the k8s container. For point 2 we can mount the journal socket and have access from the container.

---

See also:
https://github.com/Financial-Times/resilient-splunk-forwarder/issues/3
https://trello.com/c/Q2rpYaFH/113-migrate-splunk-forwarder-service-to-k8s