StreamVizzard ist an interactive and explorative stream processing editor written in Python, designed to simplify the pipeline development process. The system is divided into a Frontend, that is responsible for user interaction and visualization, and a Backend, that brings its own custom stream processing engine for execution and modification of the pipeline.
Please note that StreamVizzard is currently in active development. Some features might not yet be completed or may be subject to future changes.
StreamVizzard comes with its custom single-node Python stream processing engine that was designed to allow instantaneous runtime modifications without the need to restart the pipeline.
The adaptive visualizer allows to explore the processed data by displaying the data each operator processed as well as various different statistics.
The integrated debugger allows to explore the execution history of the pipeline and travel back to any previous state in the execution to review the pipeline behavior.
StreamVizzard comes with an integrated pipeline generator that allows to transform the pipelines created within the editor into various different representation for export, such as PyFlink.
The simplest way to run the StreamVizzard system is to build a docker container based on the provided docker-compose file. Please adjust the volume mounts in the compose file to access the output files generated by the system or to provide input data for the pipeline execution.
The StreamVizzard system can also be build manually by following the described steps in the Frontend and Backend subdirectories.
After startup, the Frontend can be reached at http://localhost:8080 (default) to construct and execute pipelines. For accessing the StreamVizzard system without visual interface, please refer to the CLI section in the Backend readme.
To construct a pipeline within the visual interface, the right-click context menu can be employed to choose from a variety of supported operators. By clicking on the operator input and output sockets, connections between operators can be established.
Each pipeline must start with a data source that ingests data into the pipeline.
For quick playarounds, the Base -> Sources -> Random Data source can be added to produce random generated data.
The most common operator type are user-defined functions, which allow to enter arbitrary Python code to process the incoming data tuples.
After setting up a first pipeline, the execution can be started from the interface, which will execute the constructed pipeline using the internal stream processing engine of the backend.
An Examples folder contains various pipeline configurations and supplementary files for demonstration and evaluation purposes.
An illustration of the full pipeline development and generation workflow is available as a short (4min) YouTube video.
-
Timo Räth, Kai-Uwe Sattler: StreamVizzard: An Interactive and Explorative Stream Processing Editor, DEBS 2022
-
Timo Räth, Kai-Uwe Sattler: Interactive and Explorative Stream Processing, DEBS 2022
-
Timo Räth, Kai-Uwe Sattler: Traveling Back in Time: A Visual Debugger for Stream Processing Applications, ICDE 2023
-
Timo Räth, Ngozichukwuka Onah, Kai-Uwe Sattler: Interactive Data Cleaning for Real-Time Streaming Applications, HILDA 2023
-
Timo Räth, Francesco Bedini, Kai-Uwe Sattler, Armin Zimmermann: Demo: Interactive Performance Exploration of Stream Processing Applications Using Colored Petri Nets, DEBS 2023
-
Timo Räth, Marius Schlegel, Kai-Uwe Sattler: Everything Everyway All at Once - Time Traveling Debugging for Stream Processing Applications, ICDE 2024