Here is Matt Brender's presentation at Big Data TechCon centered on understanding how distributed systems play a role in Big Data. Full description: Whether you’re an experienced user of Hadoop or a recent convert to Spark, you recognize that data is powerful when stored and analyzed. Analysis, as a workload, can be contrasted with the initial creation and storage of that data. These “active” workloads are what generate the data we covet. Understanding this persistence of data as workload requires an appreciation of distributed systems. We will explore what factors affect your choice in database technology and particularly how to prioritize the choice in core architectural underpinnings present in NoSQL designs. We will also explore what these technologies solve and suggestions for how to align them with your business objectives.   You’ll leave this session with an understanding of the basic principles of NoSQL architectural design and a deeper understanding of the considerations when identifying a persistence solution for your active workloads.

1. This presentation includes information that is confidential and
proprietary to Basho Technologies and should not be forwarded or
distributed without Basho's prior written consent. © 2014. Basho
Technologies, Inc. All Rights Reserved.
This presentation includes information that is confidential and proprietary to
Basho Technologies and should not be forwarded or distributed without Basho's
prior written consent. © 2014. Basho Technologies, Inc. All Rights Reserved.
Matt Brender
Developer Advocate
Taming
Big Data with NoSQL

2. Relational databases are
not bad
Data Scientist
Big Data

3. Basho Confidential 3
Data Scientist
Big Data

4. 4
Big Data is

5. Basho Confidential 5

6. 6
And it’s a distributed
systems problem

7. Basho Confidential 7
Beyond the
Scope of
one tool
Beyond the
Scope of
one file
system
Beyond the
Scope of
one
database

8. 8
Ergo, NoSQL

9. 9
NoSQL is

10. 10

11. 11
For Good Reason

12. Basho Confidential 12

13. 13
Consistency Level
Conflict Resolution
Partitioning Strategy

14. 14
Consistency Level

15. Eventually Consistent
C = Consistency
A = Availability
P = Partition Tolerance
Client Client
DBDBDB
Network Partition
Cap theorem states that a distributed system can at most support
2 out of these 3 properties

16. 16
Consistency Level

17. 17
Conflict Resolution
Last Write Wins
vs.
Causal Context

18. 18
Conflict Resolution

19. 19
Partition Strategy
Master
Slave Slave Slave
OR
Node
1
Node
2
Node
3

20. 20
Partition Strategy

21. 21
“data is powerful when
stored and analyzed”

22. Relational databases are not
bad

23. 23
Storing

24. 24
Report on this

25. Basho Confidential 25
Report on this

26. 26
vs

27. 27
A single scalable system

28. 28
A single scalable system

29. 29
Analyzing

30. What kind of questions
do you need to ask?

31. Basho Confidential 31

32. 32
Error Analysis?
As simple as NoSQL + Solr

33. 33
Patterns?
Machine Learning?
Multi-client writes to NoSQL
& HDFS

34. OR

35. 35
NoSQL + ETL process =>
other datastore + Spark
and/or Hadoop M/R

36. OR

37. 37
NoSQL & Apache Storm to
Kafka to HDFS

38. 38

39. LAMDA
39

40. So we agree.
NoSQL is helpful.

41. Side Note: NoSQL is a
terrible term

42. 42
In Review

43. 43
You can’t analyze what you
don’t have.
And you don’t want an analysis
system to be unreliable.

44. THE COST OF DOWNTIME
44

45. Basho Confidential 45

46. 46
everything works
at small scale

47. 47
Nothing matters if..

48. Basho Confidential 48

49. • Hadoop
A framework that allows for
the distributed processing of
large data sets across
clusters
• Spark
A fast, general engine for
large-scale data processing
• Storm
A distributed real-time
computation system
• NoSQL
A collection of highly
scalable, highly available
systems that fall within CAP
theorem
• Solr
Apache project for indexing
text for search
• Kafka
Distributed scalable pub/sub
messaging queue
Summary

50. 50
Thank You!
Matt Brender
@mjbrender