MongoDB and Spark
•Als PPTX, PDF herunterladen•
1 gefällt mir•1,639 views
The MongoDB Spark Connector integrates MongoDB and Apache Spark, providing users with the ability to process data in MongoDB with the massive parallelism of Spark. The connector gives users access to Spark's streaming capabilities, machine learning libraries, and interactive processing through the Spark shell, Dataframes and Datasets. We'll take a tour of the connector with a focus on practical use of the connector, and run a demo using both Spark and MongoDB for data processing.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie MongoDB and Spark
Ähnlich wie MongoDB and Spark (20)
Mehr von Norberto Leite
Mehr von Norberto Leite (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
MongoDB and Spark
- 1. MongoDB and Spark Getting MongoDB and Spark to play nice together
- 3. Agenda What we will unravel today!
- 4. 4 Agenda What Is Spark Overview Spark Stack Spark + MongoDB How to set up MongoDB and Spark? Integration Use Cases / Demo Datascience, Analytics Others
- 5. 5 Howdy! Who's this guy? Norberto Leite Lead Engineer @nleite MongoDB
- 9. 9 Delivering User Relevancy • Integrate data from many sources • Fast-cycle analytics • Real-time • Reliable
- 10. 10 Wearable Devices Embedded Systems Internet of Things Embedded Medical Devices
- 11. 11 Access complete patient history Avoid of conflicting prescriptions Clinical trials
- 12. wget https://www-eu.apache.org/dist/spark/spark-2.4.0 /spark-2.4.0-bin-hadoop2.7.tgz tar -xzvf spark-2.4.0-bin-hadoop2.7.tgz spark-2.4.0-bin-hadoop2.7/bin/pyspark Python 2.7.10 (default, Aug 17 2018, 17:41:52) [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)] on darwin ... Using Python version 2.7.10 (default, Aug 17 2018 17:41:52) SparkSession available as 'spark'. >>> Getting started with Spark - pyspark
- 13. spark-2.4.0-bin-hadoop2.7/bin/spark-shell ... Welcome to ____ __ / __/__ ___ _____/ /__ _ / _ / _ `/ __/ '_/ /___/ .__/_,_/_/ /_/_ version 2.4.0 /_/ ... scala> Getting started with Spark - spark-shell (scala)
- 14. Spark Stack
- 16. Spark Stack Spark SQL Spark Streaming MLIB GraphX Apache Spark Seamless integration with SQL using DataFrame API. Also supports HIVE SQL Fast Feed data processing API. Designed for Fault Tolerance and bridges streaming with batch processing MLib is Spark machine learning algorithms trick bag. Spark graph library
- 25. How can we use Spark with MongoDB?
- 27. Parallelism Machine Learning Stream Processing Aggregation Native Processing Horizontal Scalling
- 30. 30 What do we need ?
- 32. 32 MongoDB Spark Connector Spark SQL Spark Streaming Apache Spark MongoDB Spark Connector
- 33. 33 Input Cluster Spark SQL Spark Streaming Apache Spark MongoDB Spark Connector
- 34. 34 Output Cluster Spark SQL Spark Streaming Apache Spark MongoDB Spark Connector
- 35. 35 Which can be the same! Spark SQL Spark Streaming Apache Spark MongoDB Spark Connector
- 37. What we can do with Spark and MongoDB
- 38. Spark SQL
- 41. Fraud Detection I'm so in love!
- 42. Fraud Detection I'm so in love! Me, too<3 Now send me your CC number ? Ok, XXXX-123-zzz $$$
- 43. Fraud Detection
- 44. Workloads Chat App Login User Profile Contacts Messages … Spark Fraud Detection Segmentation Recommendations HDFS HDFS HDFS Archiving Data Crunching
- 45. Workloads Chat App Spark Real-time data processing HDFS HDFS HDFS
- 46. Spark Streaming
- 48. 48 Spark Streaming Twitter Feed { "statuses": [ { "coordinates": null, "favorited": false, "truncated": false, "created_at": "Mon Sep 24 03:35:21 +0000 2012", "id_str": "250075927172759552", "entities": { "urls": [ ], "hashtags": [ { "text": "freebandnames", "indices": [ 20, 34 ] } ], "user_mentions": [] } } }
- 49. 49 Spark Streaming Spark { "statuses": [ { "coordinates": null, "favorited": false, "truncated": false, "created_at": "Mon Sep 24 03:35:21 +0000 2012", "id_str": "250075927172759552", "entities": { "urls": [ ], "hashtags": [ { "text": "freebandnames", "indices": [ 20, 34 ] } ], "user_mentions": [] } } } { "time": "Mon Sep 24 03:35", "freebandnames": 1 } { "statuses": [ { "coordinates": null, "favorited": false, "truncated": false, "created_at": "Mon Sep 24 03:35:21 +0000 2012", "id_str": "250075927172759552", "entities": { "urls": [ ], "hashtags": [ { "text": "freebandnames", "indices": [ 20, 34 ] } ], "user_mentions": [] } } } { "statuses": [ { "coordinates": null, "favorited": false, "truncated": false, "created_at": "Mon Sep 24 03:35:21 +0000 2012", "id_str": "250075927172759552", "entities": { "urls": [ ], "hashtags": [ { "text": "freebandnames", "indices": [ 20, 34 ] } ], "user_mentions": [] } } } { "statuses": [ { "coordinates": null, "favorited": false, "truncated": false, "created_at": "Mon Sep 24 03:35:21 +0000 2012", "id_str": "250075927172759552", "entities": { "urls": [ ], "hashtags": [ { "text": "freebandnames", "indices": [ 20, 34 ] } ], "user_mentions": [] } } } { "time": "Mon Sep 24 03:35", "freebandnames": 4 }
- 50. Clear, right?
- 51. Demo Time
- 54. 54 Steps Describing Demo ● SRT text messages in the network ● Spark collects those messages ○ Defines a processing Window ○ Performs word count ● Store DataFrame into MongoDB https://github.com/nleite/mdb.local
- 55. In Short
- 56. 56 ● An extremely powerful combination ● Many possible use cases ● Evolving all the time Spark and MongoDB