Converging data-in-motion & data-at-rest with MapR Streams
•
16 gefällt mir•3,299 views
The document discusses MapR Streams, a global publish/subscribe event streaming system. It provides converged, continuous, and global capabilities. MapR Streams allows producers to publish billions of messages per second to topics, and guarantees immediate and reliable delivery to consumers. It also enables tying together geo-dispersed clusters globally. The document demonstrates MapR Streams capabilities with a live demo and discusses use cases for event streaming across various industries.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Converging data-in-motion & data-at-rest with MapR Streams
Ähnlich wie Converging data-in-motion & data-at-rest with MapR Streams (20)
Mehr von MapR Technologies
Mehr von MapR Technologies (17)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Converging data-in-motion & data-at-rest with MapR Streams
- 1. © 2015 MapR Technologies 1© 2015 MapR Technologies
- 2. © 2015 MapR Technologies 2 Topics • Business imperative: Converging data-in-motion & data-at-rest • Introducing MapR Streams • Live Demo: MapR Streams in action • Use cases: putting event streaming to work for your business
- 3. © 2015 MapR Technologies 3 Speakers Steve Wooledge VP, Product Marketing Anil Gadre SVP, Product Management Will Ochandarena Director, Product Management Nick Amato Director, Technical Marketing
- 4. © 2015 MapR Technologies 4 Empowering the “as-it-happens” business by speeding up the data-to-action cycle
- 5. © 2015 MapR Technologies 5 Top Brands Across All Major Industries FINANCIAL SERVICES RETAIL & CPG SECURITY ONLINE SERVICES & SOFTWARE MEDIA & ENTERTAINMENT MANUFACTURING, UTILITIES, OIL & GAS ADVERTISING HEALTH COMMUNICATIONS GOVERNMENT Fortune 10 Retailer
- 6. © 2015 MapR Technologies 6© 2015 MapR Technologies Converging data-in-motion & data-at-rest
- 7. © 2015 MapR Technologies 7 The Rise of IoT and Data in Motion 1 Billion 6 Billion 50 Billion 2000s: Mobile Internet 2020: Internet of People and Things 1990s: Fixed Internet Connected Devices Worldwide By 2020, 21% of all “high value” data will come from IoT - IDC
- 8. © 2015 MapR Technologies 8 Introducing MapR Streams Global Publish/Subscribe Event Streaming Producers Publish billions of messages/sec to a topic. Consumers Reliable delivery to all consumers. Immediately. Global Tie together geo-dispersed clusters. Worldwide.
- 9. © 2015 MapR Technologies 9 The Struggle to Deliver New Apps as Big Data Grows More kinds of data More data sources Batch to more real-time analytics More consuming apps • Multiple technologies • Harder to develop the apps • Latencies everywhere • Silos coming back • Data copies proliferating • Even harder to maintain apps • More management issues • Multiple cluster expense How to create breakthrough value for the business rapidly
- 10. © 2015 MapR Technologies 10 A Once-in-30-Year Shift is UnderwayLegacy Data-to-action Applications Middleware Expensive Specialized Compute / Storage Enterprise Applications Commodity Hardware RDBMs Batch Analytics Message Bus Global Event Streaming BigDataAge Single Namespace Operational Analytics Structured data Semi- structured Data Unstructured Data
- 11. © 2015 MapR Technologies 11 ProcessingData Batch Streaming SQL MapR Converged Data Platform Problem: A Patchwork of Data Silos is EmergingApps Customer ExperienceData Architecture Optimization Security Investigation & Event Management Operational Intelligence Managed Services & Custom Apps
- 12. © 2015 MapR Technologies 12 ProcessingData MapR Converged Data Platform Files Tables Documents Streams Design Goal: Common Data Services for All ApplicationsApps Customer ExperienceData Architecture Optimization Security Investigation & Event Management Operational Intelligence Managed Services & Custom Apps Batch Streaming SQL MapR Converged Data Platform MapR Converged Data Platform Converged platform with the power of Hadoop, Spark, NoSQL database, SQL, event streaming, and Web-scale storage
- 13. © 2015 MapR Technologies 13 MapR Converged Data Platform Open Source Engines & Tools Commercial Engines & Applications Utility-Grade Platform Services DataProcessing Enterprise Storage MapR-FS MapR-DB MapR Streams Database Event Streaming Global Namespace High Availability Data Protection Self-healing Unified Security Real-time Multi-tenancy Search & Others Cloud & Managed Services Custom Apps UnifiedManagementandMonitoring
- 14. © 2015 MapR Technologies 14 Hadoop + Enterprise Storage Hadoop + Enterprise Storage + NoSQL Hadoop + Enterprise Storage + NoSQL + Interactive SQL Hadoop + Enterprise Storage + NoSQL + Interactive SQL + Event Streaming 2011 2013 2014 2015 Top Ranked Hadoop Top Ranked NoSQL Top Ranked SQL Consistently Executing On Our Converged Vision
- 15. © 2015 MapR Technologies 15 MapR Platform Services: Open API Architecture Assures Interoperability, Avoids Lock-in MapR-FS Enterprise Storage MapR-DB NoSQL Database MapR Streams Global Event Streaming HDFS API POSIX NFS SQL, Hbase API JSON API Kafka API
- 16. © 2015 MapR Technologies 16 MapR Streams Creates the Industry’s First and Only Converged Data Platform High-throughput streaming converges data-in-motion with data-at-rest 1 2 3 Converged : Single cluster for event streams, database, and file-based apps with unified security Continuous: Continuous analytics with always-on reliability enables “streams of record” Global: Optimized for global real-time applications including “intermittent” IoT connections
- 17. © 2015 MapR Technologies 17 Big Data is Generated One Event at a Time “time” : “6:01.103”, “event” : “RETWEET”, “location” : “lat” : 40.712784, “lon” : -74.005941 “time: “5:04.120”, “severity” : “CRITICAL”, “msg” : “Service down” “card_num” : 1234, “merchant” : ”Apple”, “amount” : 50
- 18. © 2015 MapR Technologies 18 Batch Processing Has Many Use Cases ● Clickstream analysis ● Predictive maintenance ● Fraud detection ● Coupon offers ● Risk models ● Customer 360 ● Sentiment analysis
- 19. © 2015 MapR Technologies 19 Real-time Processing is Complementary ● Ops dashboards ● Failure alerts ● Breach detection ● Real-time fraud detection ● Real-time offers ● Push notifications ● Trending now ● News feed
- 20. © 2015 MapR Technologies 20 The Challenge with Data Pipelines Filtering & Aggregation Alerting Processing
- 21. © 2015 MapR Technologies 21 Streams Simplify Data Movement Filtering & Aggregation Alerting Processing Streams Reliable publish/subscribe transport between sources and destinations.
- 22. © 2015 MapR Technologies 22 Legacy Systems: Message Queues IBM MQ, TIBCO, RabbitMQ OrdersFront End Order Processing Order Processing Usage/Requirements ● Tight, transactional conversations between systems ● 1:1 or Few:Few ● Low data rates ● Mission-critical delivery Approach ● Queue-oriented design ● Each message replicated to N output queues ● Messages popped when read ● Scale-up, master/slave Doesn’t Do ● High message rates (>100K/s) ● Slow consumers ● Queue replay/rewind
- 23. © 2015 MapR Technologies 23 Evolving “Big Data” Event Streams: Distributed Logs Kafka, Hydra, DistributedLog Usage/Requirements ● High throughput data transferred from decoupled systems ● Many->1 ● 1->Many ● Different speeds Approach ● Log-oriented design ● Write messages to log files ● Consumers pull messages at their own pace ● Scale-out Doesn’t Do ● Global applications ● Message persistence ● Integrated analytics (data movement required) DB_Changes Stream Processing Search/ EDW DB
- 24. © 2015 MapR Technologies 24 MapR: Rethinking a Platform for Event Streams ● “Big data” scale and performance ● Global applications and data collection ● Multi-tenant and multi-application ● Secure ● Analytics-ready (no movement) ● Converged: no cluster sprawl Stream Processing Analytics Ad Impressions App Logs Sensor Data
- 25. © 2015 MapR Technologies 25© 2015 MapR Technologies MapR Streams Converged, Continuous, Global
- 26. © 2015 MapR Technologies 26 MapR Streams: Global Pub-sub Event Streaming System for Big Data Producers publish billions of messages/sec to a topic in a stream. Guaranteed, immediate delivery to all consumers. Tie together geo-dispersed clusters. Worldwide. Standard real-time API (Kafka). Integrates with Spark Streaming, Storm, Apex, and Flink Direct data access (OJAI API) from analytics frameworks. To pi c Stream TopicProducers Consumers Remote sites and consumers Streaming Batch analytics
- 27. © 2015 MapR Technologies 27 Global Provides ● Arbitrary topology of thousands of clusters ● Automatic loop prevention ● DNS-based discovery ● Globally synchronized message offsets and consumer cursors Enables ● Global applications & data collection ● Producer & consumer failover ● Analysis/filtering/aggregation at the edge ● “Occasional” connections Producers Consumers
- 28. © 2015 MapR Technologies 28 Top Differentiators MapR Streams Converged Global Secure & Multi-Tenant Single cluster for files, tables, and streams. Global, IoT-scale “fabrics” with failover. Tenant-owned streams, logical grouping of topics and messages. Authentication, authorization, encryption. Unified policy with all other platform services. Data persistence and direct data access to batch processing frameworks.
- 29. © 2015 MapR Technologies 29© 2015 MapR Technologies See MapR Streams in Action – Demo Nick Amato
- 30. © 2015 MapR Technologies 30© 2015 MapR Technologies Event Streaming & Processing Use Cases Will Ochandarena
- 31. © 2015 MapR Technologies 31 All Industries Web 2.0 Healthcare Telecom • ETL / DW optimization • Mainframe optimization • Application & network monitoring • Security information & event management • Recommendation engines & targeting • Customer 360 • Click-stream analysis • Social media analysis • Ad optimization • Smart hospitals • Biometrics • Patient vital monitoring • Fraud detection • Antenna optimization • Charging & billing • Equipment monitoring & preventative maintenance • Smart meter analysis Sample Verticals & Use Cases Oil & Gas Financial Services Retail Ad Tech • Pump monitoring & alerting • Seismic trace identification • Equipment maintenance • Safety & environment • Security • Real-time fraud/risk monitoring • Mobile notifications of transactions • Real-time supply chain optimization • Inventory management • Real-time coupons • Ad targeting & optimization • Global campaign dashboards
- 32. © 2015 MapR Technologies 32 USE CASE Stream Application/Infrastructure Monitoring Logs Metrics Business Results ● Real-time detection and alerting on failures, security breaches ● Global dashboards on utilization, availability, performance Why Streaming ● Real-time delivery from apps/infra to ETL & processing systems. ● Reliable buffering of data in case of slow/failing systems Why MapR ● Converged platform brings all components together. ● Global event replication enables centralized monitoring. ● Secure multi-tenancy allows cluster sharing.
- 33. © 2015 MapR Technologies 33 Database Change Capture ForSmart Credit Card Processing Business Results ● Improved user satisfaction with real-time mobile notifications of purchases. ● More fraud detected in real-time. ● More productive staff with data exploration. Why Streams ● Seamless, real-time connection between mainframe RDBMS and ETL/processing. Why MapR ● Utility-grade reliability means no transactions are lost. ● Converged platform security gives unified authentication, authorization, encryption. USE CASE Transactions Fraud Detection Streaming 1
- 34. © 2015 MapR Technologies 34 Stream System of Record for Healthcare, Finance Business Results ● Data agility - up-to-date views in JSON, Graph, Search formats. ● HIPAA, PCI Compliance. ● Compliance with in-country data regulations. Records JSON DB (MapR-DB) Graph DB (Titan on MapR-DB) Search Engine (Elastic-Search) Write APIs Read APIs Why Streams ● Streams are immutable, re-windable, and auditable data structures. ● Pub-sub allows real-time replication to JSON-DB, Graph DB, ElasticSearch. Why MapR ● Converged platform gives single cluster, single security model for data in motion and at rest. ● Selective, reliable global replication for DR. EU USE CASE
- 35. © 2015 MapR Technologies 35 Global, Consolidated Analytics in Ad Tech … + APJ, EMEA Business Results ● New, real-time customer dashboards for ad spend and performance. ● Reduced global time-to-insight - hours to minutes. ● Addition of disaster recovery capability. Why Streams ● Simpler, more reliable data/ETL pipelines than legacy log-shipping method. Why MapR ● Converged platform gives single cluster, single security model for data in motion and at rest. ● Reliable global replication for distributed collection, analysis, and DR. US1 Ad Application HQ US2 USE CASE
- 36. © 2015 MapR Technologies 36 IoT Data Transport & Processing USE CASE Business Results ● New revenue streams from collecting and processing data from “things”. ● Low response times by placing collection and processing near users. Why Streams ● IoT is event-based, and needs an event streaming architecture. Why MapR ● Converged platform gives single cluster, single security model for data in motion and at rest. ● Reliable global replication for distributed collection, analysis, and DR. Global Dashboards, Alerts, Processing Local Collection, Filtering, Aggregation
- 37. © 2015 MapR Technologies 37 Q&A MapR Streams Creates Industry’s First and Only Converged Data Platform 1 2 3 Converged : Single cluster for event streams, database, and file-based apps with unified security Continuous: Continuous analytics with always-on reliability enables “streams of record” Global: Optimized for global real-time applications including “intermittent” IoT connections Learn more at www.mapr.com/streams
Hinweis der Redaktion
- FS – Eastern Bank; Experian; TransUnion; Zions Bank Security – Solutionary Online services & software – Xactly; Liason; ancestry.com; Live Nation; Razorsight; Datasong Media & entertainment – Beats Music
- IoT is the third big wave of the Internet. To put this in perspective, the fixed Internet, which is really what we mostly thought about back in the 1990s, connected about a billion users to the Internet, primarily via their desktops. In the 2000s, we had the second wave, which connected about two billion people to the Internet via their mobile devices which has grown to 6B devices. What we’re talking about now with the Internet of Things is connecting about 50 billion or more things to the Internet by 2020
- MapR provides a converged data application platform which consolidates compute engines on unified web-scale storage (maybe reuse the opening statement about what MapR provides, but shorten for public speaking so it’s not rote and stale) If you double-click into MapR Enterprise, you see that we have the “MapR Alloy Data Operating System” (final name TBD) providing key data services to the compute engines, managed centrally by MapR “Management Services” (final name TBD). MapR provides the big data application platform that is the premier choice for leading enterprises building out the next phase of their data-driven business strategy. First, let’s look at the requirements of these data-driven applications you see at the top of the diagram. Whether it’s analytical applications such as personalized recommendations on a website, fraud detection, or powering operational applications such as with managed service providers, email services and others, there are powerful compute/processing engines required to support them. These include big data analytics engines from the Hadoop and Spark ecosystems, global messaging which users and applications can publish/subscribe to, search, real-time database operations, interactive SQL, stream processing, and Web-scale network-attached storage (NAS). Underneath these compute engines is the “MapR Data Platform Services” are the data services provided transientially (sp?) to provide performance, security, reliability, storage, resource management, and more. MapR Data Services (MapR DPS? – will need to think about how this gets shortened) was designed and engineered from the hardware up for modern big data workloads and scale with utility-grade reliability, performance, and unified administration. It is the production choice of the Global 2000 for mission-critical Hadoop and Spark workloads and is the reason why Forrester and other industry analyst firms consistently rank us as the top-ranked Hadoop distribution. (from here, suggest having a separate slide which blows up and shows the internals of the MapR Data Services layer, how it works, why it’s better, and can have a loop supporting that, such as the updated 2nd call deck. Similarly, should have a double-click on the individual projects in the distro for people that care about that and “what we support”)
- MapR provides a converged data application platform which consolidates compute engines on unified web-scale storage (maybe reuse the opening statement about what MapR provides, but shorten for public speaking so it’s not rote and stale) If you double-click into MapR Enterprise, you see that we have the “MapR Alloy Data Operating System” (final name TBD) providing key data services to the compute engines, managed centrally by MapR “Management Services” (final name TBD). MapR provides the big data application platform that is the premier choice for leading enterprises building out the next phase of their data-driven business strategy. First, let’s look at the requirements of these data-driven applications you see at the top of the diagram. Whether it’s analytical applications such as personalized recommendations on a website, fraud detection, or powering operational applications such as with managed service providers, email services and others, there are powerful compute/processing engines required to support them. These include big data analytics engines from the Hadoop and Spark ecosystems, global messaging which users and applications can publish/subscribe to, search, real-time database operations, interactive SQL, stream processing, and Web-scale network-attached storage (NAS). Underneath these compute engines is the “MapR Data Platform Services” are the data services provided transientially (sp?) to provide performance, security, reliability, storage, resource management, and more. MapR Data Services (MapR DPS? – will need to think about how this gets shortened) was designed and engineered from the hardware up for modern big data workloads and scale with utility-grade reliability, performance, and unified administration. It is the production choice of the Global 2000 for mission-critical Hadoop and Spark workloads and is the reason why Forrester and other industry analyst firms consistently rank us as the top-ranked Hadoop distribution. (from here, suggest having a separate slide which blows up and shows the internals of the MapR Data Services layer, how it works, why it’s better, and can have a loop supporting that, such as the updated 2nd call deck. Similarly, should have a double-click on the individual projects in the distro for people that care about that and “what we support”)