Netflix’s playback data records every user interaction with video on the service, from trailers on the home page to full-length movies. This is a critical dataset with high volume that is used broadly across Netflix, powering product experiences, AB test metrics, and offline insights. In processing playback data, we depend heavily on event-time partitioning to handle a long tail of late arriving events. In this talk, I’ll provide an overview of our recent implementation of generic event-time partitioning on high volume streams using Apache Flink and Apache Iceberg (Incubating). Built as configurable Flink components that leverage Iceberg as a new output table format, we are now able to write playback data and other large scale datasets directly from a stream into a table partitioned on event time, replacing the common pattern of relying on a post-processing batch job that “puts the data in the right place”. We’ll talk through what it took to apply this to our playback data in practice, as well as challenges we hit along the way and tradeoffs with a streaming approach to event-time partitioning.

1. Streaming Event-Time
Partitioning With Apache
Flink And Apache Iceberg
Julia Bennett
Flink Forward
October 2019

2. What is this talk about?
● Story of building event-time partitioning in
Flink and Iceberg
○ Netflix playback data
○ Why event-time partitioning matters
○ Introduce Iceberg
○ Implementation details
○ Tradeoffs

4. Playback
Data

5. Processing Time:
2019-08-15, 11:53
Event Time:
2019-08-15, 9:17
Event
Data
2019-08-15, 11:00
2019-08-15, 10:00
2019-08-15, 9:00
2019-08-15, 8:00
Table Partitioning
Processing Time?
Event Time?✔

6. Before: Playback Data
Event
Time
Streaming (Near Real-Time) Batch (Hourly)
Process
Time
Raw Events
Session
Summaries
Business
Logic
**Lots of late events

7. Common Pattern
Streaming (Near Real-Time) Batch (Hourly)
Event
Time
Process
Time
Business
Logic
?

8. Better Pattern?
Streaming (Near Real-Time)
Event
Time
RouterBusiness
Logic
?

9. Why is this hard?
● Exactly once processing
○ Dedupe incoming stream
○ Append to table without duplicates (or loss)
● Out of order and late arriving events
○ Partition-level modifications
○ Efficiently modify long tail of partitions

10. Iceberg Sink
● Apache Iceberg (Incubating) is new table
format developed at Netflix
○ Atomic commits and file-level changes
● Flink + Iceberg: Exactly once partition appends
○ Buffers records into partitioned files
○ Commits with each checkpoint

11. This should be easy...
Event
Time
Router
.sink(Iceberg)

12. Problem: Too many files
Oh, and dedupe too
(… e.g. 250 MILLION per day)

13. Source
Dynamic
Partitioner
Traffic Metrics
State
Iceberg
Sink
Late Events
State
Late Batching
Punctual Stream
Dedupe Key
State
Dedupe
Reduces # Files & Skew
Offline
Compaction

14. Dedupe Key
State
Dedupe
Keyed Process
Function
+ Timer Service
Dynamic
Partitioner
Traffic Metrics
State
Side Output
+ Window
+ Broadcast
Late Events
State
Late Batching
Punctual Stream
Keyed Process
Function
+ Timer Service

15. After: Playback Data
Streaming (Near Real-Time)
Event
Time
RouterBusiness
Logic
Raw Events
Session
Summaries
~100K Files
Per Day

16. Source
Dynamic
Partitioner
Traffic Metrics
State
Iceberg
Sink
Offline
Compaction
Late Events
State
Late Batching
Punctual Stream
Dedupe Key
State
Dedupe
-Generic event-time partitioning out of the box -

17. New Pattern
Streaming (Near Real-Time)
Event
Time
RouterBusiness
Logic
?

18. Tradeoffs
● Low-latency streaming output
● Remove operation of offline batch job
● Failure recovery is hard
○ Multiple long-lived states
○ Writing directly to production table
● Complexity, even if abstracted away

19. Team Effort
Lokesh
Balakrishnan

20. And next time...
Event
Time
Router
.sink(Iceberg)