Stateful Operations

[lzwaan]

Is there an option to restore states during restarts/recovery?
And can i use the confluent python kafka schema-registry inside CSP?

[robambalu]

Not quite sure what youre asking for here. If you mean to say can CSP recover state specifically from kafka, then the answer is yes, you can use the start_offset option and it it to START_TIME:
https://github.com/Point72/csp/wiki/Input-Output-Adapters-API#api
https://github.com/Point72/csp/blob/main/csp/adapters/kafka.py#L35

[lzwaan]

So i quess it keep state in memory?,Some other stream processors like arroyo or Bytewax are using AWS S3 resp. sqlite to keep track of state. In the case of CSP you start from a kafka offset to build up the state again and then it automatically continue when the current time or is it a 2 step process first historical and then realtime?

I never heard about CSP till i saw the youtube presentation, so i start from scratch with it, so forgive me if it sounds stupid...;-)
We currently using redpanda connect, but using window operations is basic and using python for ML/anomaly detection is much more flexible.

[robambalu]

CSP doesnt (currently) have any external state keeping. At the moment the only choice is to rebuild state yourself from external sources.
Kafka is a very good fit for this as you can just replay the kafka stream to rebuild your in-memory state in CSP completely deterministically.
If you use start_offset : START_TIME it will replay the historical kafka message for you in historical time and then seamlessly switch to realtime once it has caught up.
oh, and welcome aboard! :)

[lzwaan]

Thanks Rob, that sounds great, kafka is our main source/sink, we makes it easy for SCP to start from any streaming server and not being dependent on external db/object storage ;-)