Scalable Streaming Data Pipelines with Redis
•Als PPTX, PDF herunterladen•
3 gefällt mir•1,791 views
Talk at Redis Conference 2016. Video here: https://www.youtube.com/watch?v=Cj0ChmpkoZU
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Scalable Streaming Data Pipelines with Redis
Ähnlich wie Scalable Streaming Data Pipelines with Redis (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Scalable Streaming Data Pipelines with Redis
- 1. Scalable Streaming Data Pipelines with Redis Avram Lyon @ajlyon / github.com/avram redisconf / May 10, 2016
- 2. MOBILE GAMES - PUBLISHER AND DEVELOPER
- 3. What kind of data? • App opened • Killed a walker • Bought something • Heartbeat • Memory usage report • App error • Declined a review prompt • Finished the tutorial • Clicked on that button • Lost a battle • Found a treasure chest • Received a push message • Finished a turn • Sent an invite • Scored a Yahtzee • Spent 100 silver coins • Anything else any game designer or developer wants to learn about
- 4. How much? Recently: Peak: 2.8 million events / minute 2.4 billion events / day
- 5. Primary Data Stream Collection Kinesis Warehousing Enrichment Realtime MonitoringKinesisPublic API
- 6. Collection HTTP Collection SQS SQS SQS Studio A Studio B Studio C Kinesis SQS Failover Redis Caching App Configurations System Configurations
- 8. What’s in the box? =
- 9. Where does this flow? Ariel / Real-Time Operational monitoring Business alerts Dashboarding Data Warehouse Funnel analysis Ad-hoc batch analysis Reporting Behavior analysis Elasticsearch Ad-hoc realtime analysis Fraud detection Top-K summaries Exploration Ad-Hoc Forwarding Data integration with partners Game-specific systems
- 11. Kinesis • Distributed, sharded streams. Akin to Kafka. • Get an iterator over the stream— and checkpoint with current stream pointer occasionally. • Workers coordinate shard leases and checkpoints in DynamoDB (via KCL) Shard 0 Shard 1 Shard 2
- 12. Shard 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 Checkpointing Checkpoint for Shard 0: 10 Given: Worker checkpoints every 5 Worker A 🔥 Worker B
- 13. Auxiliary Idempotence • Idempotence keys at each stage • Redis sets of idempotence keys by time window • Gives resilience against various types of failures
- 15. Auxiliary Idempotence • Gotcha: Set expiry is O(N) • Broke up into small sets, partitioned by first 2 bytes of md5 of idempotence key
- 17. 1. Deserialize event batch 2. Apply changes to application properties 3. Get current device and application properties 4. Get known facts about sending device 5. Emit to each enriched event to Kinesis Collection Kinesis Enrichment
- 18. Kinesis SQS Failover Kinesis S3 Elasticsearch ? S3 Backups to HDFS Enricher Data Warehouse Forwarder Idempotence Ariel Realtime Idempotence Aggregation Idempotence
- 19. Now we have a stream of well- described, denormalized event facts.
- 20. Pipeline to HDFS • Partitioned by event name and game, buffered in-memory and written to S3 • Picked up every hour by Spark job • Converts to Parquet, loaded to HDFS
- 21. A closer look at Ariel
- 23. Ariel Goals • Low time-to-visibility • Easy configuration • Low cost per configured metric
- 24. Configuration
- 25. Live Metrics (Ariel) Enriched Event Data name: game_end time: 2015-07-15 10:00:00.000 UTC _devices_per_turn: 1.0 event_id: 12345 device_token: AAAA user_id: 100 name: game_end time: 2015-07-15 10:01:00.000 UTC _devices_per_turn: 14.1 event_id: 12346 device_token: BBBB user_id: 100 name: Cheating Games predicate: _devices_per_turn > 1.5 target: event_id type: DISTINCT id: 1 name: Cheating Players predicate: _devices_per_turn > 1.5 target: user_id type: DISTINCT id: 2 name: game_end time: 2015-07-15 10:01:00.000 UTC _devices_per_turn: 14.1 event_id: 12347 device_token: BBBB user_id: 100 PFADD /m/1/2015-07-15-10-00 12346 PFADD /m/1/2015-07-15-10-00 123467 PFADD /m/2/2015-07-15-10-00 BBBB PFADD /m/2/2015-07-15-10-00 BBBB PFCOUNT /m/1/2015-07-15-10-00 2 PFCOUNT /m/2/2015-07-15-10-00 1 Configured Metrics Collector
- 27. HyperLogLog • High-level algorithm (four bullet-point version stolen from my colleague, Cristian) • b bits of the hashed function is used as an index pointer (redis uses b = 14, i.e. m = 16384 registers) • The rest of the hash is inspected for the longest run of zeroes we can encounter (N) • The register pointed by the index is replaced with max(currentValue, N + 1) • An estimator function is used to calculate the approximated cardinality http://content.research.neustar.biz/blog/hll.html
- 28. Live Metrics (Ariel) Enriched Event Data name: game_end time: 2015-07-15 10:00:00.000 UTC _devices_per_turn: 1.0 event_id: 12345 device_token: AAAA user_id: 100 name: game_end time: 2015-07-15 10:01:00.000 UTC _devices_per_turn: 14.1 event_id: 12346 device_token: BBBB user_id: 100 name: Cheating Games predicate: _devices_per_turn > 1.5 target: event_id type: DISTINCT id: 1 name: Cheating Players predicate: _devices_per_turn > 1.5 target: user_id type: DISTINCT id: 2 name: game_end time: 2015-07-15 10:01:00.000 UTC _devices_per_turn: 14.1 event_id: 12347 device_token: BBBB user_id: 100 PFADD /m/1/2015-07-15-10-00 12346 PFADD /m/1/2015-07-15-10-00 123467 PFADD /m/2/2015-07-15-10-00 BBBB PFADD /m/2/2015-07-15-10-00 BBBB PFCOUNT /m/1/2015-07-15-10-00 2 PFCOUNT /m/2/2015-07-15-10-00 1 Configured Metrics We can count different things Collector
- 29. Kinesis Aggregation Ariel PFCOUNT Are installs anomalous? Collector Idempotence PFADD Web Workers
- 30. Pipeline Delay • Pipelines back up • Dashboards get outdated • Alarms fire!
- 31. Alarm Clocks • Push timestamp of current events to per-game pub/sub channel • Worker takes 99th percentile age of last N events per title as delay • Use that time for alarm calculations • Overlay delays on dashboards
- 32. Ariel, now with clocks Event ClockKinesis Aggregation PFCOUNT Are installs anomalous? Collector Idempotence PFADD Web Workers
- 33. Ariel 1.0 • ~30K metrics configured • Aggregation into 30-minute buckets • 12 kilobytes per HLL set (plus overhead)
- 34. Challenges • Dataset size. RedisLabs non-cluster max = 100GB • Packet/s limits: 250K in EC2-Classic • Alarm granularity
- 35. Hybrid Datastore: Requirements • Need to keep HLL sets to count distinct • Redis is relatively finite • HLL outside of Redis is messy
- 36. Hybrid Datastore: Plan • Move older HLL sets to DynamoDB • They’re just strings! • Cache reports aggressively • Fetch backing HLL data from DynamoDB as needed on web layer, merge using on-instance Redis
- 37. Ariel, now with hybrid datastore DynamoDB Report Caches Old Data Migration Event Clock Kinesis Aggregation PFCOUNT Are installs anomalous? Collector Idempotence PFADD Web Workers Merge Scratchpad
- 39. Redis Roles • Idempotence • Configuration Caching • Aggregation • Clock • Scratchpad for merges • Cache of reports • Staging of DWH extracts
- 40. Other Considerations • Multitenancy. We run parallel stacks and give games an assigned affinity, to insulate from pipeline delays • Backfill. System is forward-looking only; can replay Kinesis backups to backfill, or backfill from warehouse
- 41. Why Not _____? • Druid • Flink • InfluxDB • RethinkDB
Hinweis der Redaktion
- We also expect this to grow with the growth of our userbase, the launch of new titles, and of course with every addition of new, useful functionality.
- We’re just looking at one simple transformation of a stream, and the consumption of that stream by a variety of consumers. Since we’re using Kinesis, we can read the same stream in parallel from multiple applications safely. We’ll consider major challenges moving from left to right across this architecture.
- Primary collection is intended to be at-least-once; currently support SQS and HTTP; all batches have idempotence information to allow deduplication. At this stage, we have minimal logic— we are focused on letting game servers and clients successfully unload their batches of user events, so they can be durably stored in our systems. System configuration lives in DynamoDB; we use Netflix Archaius App configuration lives in DynamoDB; we cache in-memory on instances and in Redis Goals of SQS: Goal: Register and receive events asynchronously Goal: Provide elasticity when senders spike Goal: Reduce CPU burn for senders
- Autoscaling group containing a simple Java service, deployed as a golden AMI provisioned with Packer and Ansible, using Cloudformation. We make lots of these — we call them our satellites. Usually we name them after moons. The little orange symbol means we’re using Amazon’s KCL, so the fleet negotiates workers’ shard control using a lease table in DynamoDB. Monitoring is New Relic and lots of StatsD sent to Datadog. So every time we see a gray square, assume we’re talking about 1-50 EC2 instances across several availability zones in one AWS region.
- But first an aside on Kinesis.
- Checkpointing and auxiliary idempotence The data in our stream has monotonically increasing pointers (huge, huge numbers!). In our case, 1-22 and beyond. A worker on this shard appears and checkpoints every 5 successfully processed records. But it dies after processing record 12. When Worker B appears, it sees the checkpoint at 10 and picks up processing the shard at 11. But this means we’ll reprocess 11 and 12! Similar issues can occur with out-of-order processing of data.
- Expensive. Bloom filters may be a viable option some day
- Expensive. Bloom filters may be a viable option some day
- this stage is the latency-sensitive.
- This lets all downstream systems act on data without needing to hit any more systems.
- We have considered a streaming ingest, but this has proven easier to reason about and has sufficient liveness at the moment.
- Introduced in Redis 2.8.9 (http://antirez.com/news/75) But I don’t want to really get into this too much…
- The first complete implementation of this had three major components: collector, web and workers.
- Caveat— not all metrics were HLL; we also support sums, which take only several bytes. But only the sparsest of distinct metrics would require less than 12KB for a time window