What is data-driven architecture? And if we use one, what data should we use to drive it? A data-driven architecture should provide many real advantages - timeliness, self-adapting to change, and more anchored in the real-world context. Yet we can only reach those advantages when we have the right data - so how do we identify the right data to use? The danger with ‘data-driven’ is that it often points us towards the wrong end of that challenge - the ‘What’ of the data, rather than the ‘Why’ and ‘How’ that underpins the architecture itself. For example, one common trap is saying “We have this data-source: how can we use it in our architecture?” - the classic architecture-error called ‘solutioneering’. Instead, we need to start our architecture at the other end, moving from stakeholders to story to solution. In this webinar we’ll re-purpose the classic DIKW set - data information, knowledge, wisdom - to help us make sense of how a data-driven architecture actually operates, and thence point us towards the data-sources and sensors that we need to make it all work. (Webinar for The Bridge / MongoDB, organised by Andrew Blades, Sydney, Australia, 06 August 2020.)

1. Making sense of Data-Driven Architecture
Tom Graves, Tetradian
The Bridge, August 2020

2. Hi.
I’m Tom.
(yeah, I’ve
been around
for a while…)

3. These days
I’d describe myself as
a maker of tools for change...
also the architecture of change,
linking strategy to execution
and back again, as needed...

4. Part 1:
What is ‘data-driven architecture’?

5. What the heck is
‘data-driven architecture’?
Apply the classic mantra
“I don’t know”
- and go search for answers...

6. Looking for the real…
Is it just Object Oriented Programming
repackaged into a new guise?
- and if so, where does ‘architecture’
come into the story?

7. Looking for the real…
Is it mostly about use of big-data?
• Reporting: ‘what happened’
• Analysis: ‘why did it happen’
• Predictive: ‘what will happen’
• Operational: ‘what is happening now?’
• Influential: ‘how to influence what happens next’
(source: Matt Aslett / Steven Noels, ‘From Data to Data Driven - Applications that will change your business’,
https://www.slideshare.net/MarketingNGDATA/combined-datadriven090414final )

8. Looking for the real…
Is it more about applications?
• “Despite all the focus on data platforms, it is the
applications that deliver the value to the business
and the user”
• Support for personalisation, recommendations,
preferences, customer services, micro-campaigns
(source: Matt Aslett / Steven Noels, ‘From Data to Data Driven - Applications that will change your business’,
https://www.slideshare.net/MarketingNGDATA/combined-datadriven090414final )

9. Looking for the real…
Is it about architectures for
data-platforms and applications?
Or about architectures themselves?

10. Looking for the real…
Is it mostly about IT-strategy
and IT-architectures?
Or more about business-strategy
and business-architectures?

11. Has ‘data-driven architecture’
become merely another buzzword
for meaningless sales-hype?
For too much of what I’ve seen so far,
the answer seems to be ‘Yes’...

12. To make any sense of this mess,
we’ll need to go back
to first-principles…

13. Tackle the problem as
a data-architecture issue:
split the term ‘Data Driven Architecture’
into each of its component parts

14. Part 2:
‘Data Driven Architecture’

15. What does ‘data’ mean
in this context?

16. A maze of competing terms
- data, information, knowledge,
wisdom, intelligence, and more -
each with conflicting definitions...

17. The DIKW set…
• Data
• Information
• Knowledge
• Wisdom
- is it a stack? a hierarchy? or what?

18. Use a dimensional view…
- DIKW as discrete dimensions
with intelligence as the factor
that links the dimensions together

19. A dimensional view…
Wisdom(s)
(prepackaged decisions
or interpretations)
Knowledge
(human engagement,
embedded action)
Data
(raw unstructured evidence)
Information
(metadata, context,
schemas)
Intelligence
(link all dimensions to
support meaningful
decision-making)

20. Data dimension…
• presents content for intelligence
• represents real-world fact
– sensory evidence: is literally ‘that which is seen’
– only fact is real: everything else is an interpretation
• derived from past or present (real-time)
– no ‘future facts’: ‘the future’ is always an assumption
• cannot make sense on its own – it’s just data

21. Data dimension…
Wisdom(s)
(prepackaged decisions
or interpretations)
Knowledge
(human engagement,
embedded action)
Data
(raw unstructured evidence)
Information
(metadata, context,
schemas)

22. NOTE:
Feelings are facts...
Too many people
get this the wrong way round...
Interpretations about feelings
are not facts...

23. Information dimension…
• presents context for intelligence
– represented and anchored by metadata
• provides frames and filters for fact
– distinguish between ‘signal’ and ‘noise’
• provides schemas for sensemaking
• depends on availability of real-world data
• cannot test its own assumptions or logic

24. Information dimension…
Wisdom(s)
(prepackaged decisions
or interpretations)
Knowledge
(human engagement,
embedded action)
Data
(raw unstructured evidence)
Information
(metadata, context,
schemas)

25. WARNING:
Beware the trap of
‘policy-based evidence’...
- pre-filtering all fact to align with
existing assumptions...

26. Knowledge dimension…
• provides connections for intelligence
• gives personal form to data / information
– example: ‘body-knowledge’, physical skill
• provides anchor for meaning
– personal meaning can only be learnt, not taught
• provides anchor for understanding, action

27. Knowledge dimension…
Wisdom(s)
(prepackaged decisions
or interpretations)
Knowledge
(human engagement,
embedded action)
Data
(raw unstructured evidence)
Information
(metadata, context,
schemas)

28. WARNING:
Meaningful knowledge depends
on practical understanding...
“I know” used as a synonym for
“I have that information”
is essentially meaningless...

29. ‘Wisdoms’ dimension…
• presents prepackaged interpretations
– example: proverbs, ‘best practices’ etc
• underpins choices and decisions
• alternatively, provides placeholders for
concerns, drivers, purpose
• relies on contextuality - ‘adapt, then adopt’
– must always adapt to the current applicable
content, context and connections

30. ‘Wisdoms’ dimension…
Wisdom(s)
(prepackaged decisions
or interpretations)
Knowledge
(human engagement,
embedded action)
Data
(raw unstructured evidence)
Information
(metadata, context,
schemas)

31. WARNING:
To be useful, ‘wisdoms’ must be
anchored in real-world data,
information, knowledge...
- on their own, ‘wisdoms’ are
inherently meaningless and useless...

32. Intelligence as integration…
• links all of the dimensions together,
as a unified whole
• represents chosen pathway through
the dimension-set
– pathway is often (usually?) iterative, fractal
• connects / bridges between sensing,
choices, action and learning

33. Intelligence as integration…
Wisdom(s)
(prepackaged decisions
or interpretations)
Knowledge
(human engagement,
embedded action)
Data
(raw unstructured evidence)
Information
(metadata, context,
schemas)
Intelligence
(link all dimensions to
support meaningful
decision-making)

34. Part 3:
‘Data Driven Architecture’

35. What does ‘driven’ mean
in this context?

36. SUGGESTION:
‘Driven’ relates to how we use
continuously-updated intelligence
to guide action and learning
- drivers for purposive learning-loops

37. Sense, make-sense, decide, act…
• Systematic loop for
action-learning
• Decisions based on
sensing, sense-
making
• Each action triggers
new sensory data
• Self-adapt to change

38. Sense, make-sense, decide, act…
• Loops are
iterative, fractal
• Sensors may
differ for each
type of loop
• Fractal loops
may interact with
each other

39. Loops: Certainty versus uncertainty

40. Limitations of IT…
• Most IT is still rule-
bound – hence may
only be able to work
on certainties
• ‘Data-driven’
architectures may
enable some ability
to work on
uncertainties

41. REMINDER:
Beware the trap of
‘policy-based evidence’...
- over-dependence on assumptions
may be dangerous...

42. Effect of ‘policy-based evidence’
• Absence of
sensing, sense-
making
• Decisions based on
belief, assumption
• No ability to adapt
to real-world
change

43. Part 4:
‘Data Driven Architecture’

44. What does ‘architecture’ mean
in this context?

45. • Architecture is an exercise in truth
A proper building is responsible to universal
knowledge and is wholly honest in the
expression of its functions and materials
• Architecture is an exercise in narrative
Architecture is a vehicle for the telling of
stories, a canvas for relaying societal myths,
a stage for the theatre of everyday life
“Two points of view on architecture”
(Chapter 84, in Matthew Frederick, 101 Things I Learned In Architecture School, MIT Press, 2007

46. • “Architecture is an exercise in truth”
- architecture is about structure
– (IT-architecture is often really good at this)
• “Architecture is an exercise in narrative”
- architecture is about story
– (IT-architecture is often really bad at this…)
• We need balance between structure and story
TL;DR version of ‘Two points of view’

47. Domains for data-driven architecture
• IT-infrastructure architecture
– example: self-adapting system-configuration
• IT data-architecture
– example: data-platforms, big-data
• IT applications-architecture
– example: big-data applications, self-adapting systems
• Business architecture
– example: self-adapting business-models, service-designs
• Enterprise architecture
– whole-of-IT and/or whole-of-enterprise

48. WARNING:
On ‘enterprise-architecture’:
‘enterprise-wide IT architecture’
is not the same as
‘the architecture of the enterprise’...
- DON’T MIX THEM UP!

49. Part 5:
Where does this take us?

50. Summarise lessons-learned
about the current state of
‘data-driven architecture’…

51. CAUTION:
• As a general term, ‘Data Driven
Architecture’ is perhaps misleading
• Without some schema,
data are essentially meaningless
(it doesn’t have to be a classic DB schema,
but some schema nonetheless)

52. CAUTION:
• A risk of ‘Data Driven Architecture’
becoming another way for IT
to over-focus on structure,
and lose connection to the story
(Example: a CRM is a one-sided view of a
business-relationship, not the whole story!)

53. CAUTION:
• Huge hidden risks for Data Driven
Architecture becoming reliant on
‘policy-based evidence’
(Example: training for machine-learning that
reinforced social stereotypes on race/gender)

54. CAUTION:
• Only raw-data is ‘real’: anything else
is inherently based on assumptions
that may not be valid
(and choices on sources for raw-data
will themselves be based on assumptions...)

55. Checklist: Data-driven architecture
• Data: What sensors and sources do we need?
• Information: What frames, schemas and metadata do
we need?
• Knowledge: What connections do we need? How will
we connect to each person in the context?
• Wisdom: What ‘best practices’ and other patterns will
we use? How will we adapt these to our needs? What are
our choices? What is our overall guiding-purpose?
• Intelligence: How will we link all these elements
together, continuously, as a self-adapting, unified whole?

56. Thank you!