Gehören Sie zu den Ersten, denen das gefällt!
This presentation was held at Code.Talks 2019 in Hamburg.
A video is available at: https://www.youtube.com/watch?v=K1y5dJvP1jM
Window aggregation is a core operation in data stream processing.
Stream Processing Systems, like Flink or Storm, implement general aggregation techniques which perform poorly under specific workloads (e.g. Sliding Windows).
To this end, we present Scotty, a new highly-efficient window operator.
Scotty exploits specific workload properties such as the type of aggregation functions (e.g., invertible, associative), window types (e.g., sliding, sessions), windowing measures (e.g., time- or countbased), and stream (dis)order. This allows Scotty to outperform systems like Flink by up to one order of magnitude.
The structure of this talk is threefold:
First, we give an introduction to the semantics and implementations of window aggregations in modern Stream Processing Systems.
Second, we discuss the design of Scotty and show why Scotty is able to outperform the default window operators of many stream processing systems.
Third, we give a hands-on introduction to Scotty and demonstrate how it can be integrated into standard Flink, Storm, or Beam stream processing pipelines.
Scotty and its connectors are available as open-source (https://github.com/TU-Berlin-DIMA/scotty-window-processor) and contributions are highly welcome.