Rethinking intelligent resilient systems. Re-framing problems changes how we see and solve them. The intersection of scientific thought and principles parallels much of what we solve as engineers of information (e.g. uncertainty, time, distribution) and need. This talk is an interdisciplinary look at complex adaptive systems and how they innately solve things like resource distribution, growth and rebalancing. From the context of intelligence and systems, this talk will look at ideas around entropy and time, ensemble forecasting, self-organization theory, the butterfly effect, virus-human co-evolution and adaption, natural feedback loops, self-balancing, and adaptation.
Can we leverage these principles, behaviors and strategies to design intelligent systems at scale?
Can seeing things in an interdisciplinary way benefit solving common problems and speed innovation?
2. @helenaedelson
Seen In The Wild
Committer/Contributor
FiloDB, Akka, Spark Cassandra
Connector, Kafka Connect Cassandra,
Spring Integration
Helena Edelson
twitter.com/helenaedelson
Program Committee Member
Kafka Summit 2018
Reactive Summit 2016-2017
Speaker
Kafka Summit, Spark Summit (EU, NYC),
Strata (NYC, SJ), QCon SF, Scala Days
(EU, NYC), Reactive Summit (’16, ’17),
Philly ETE, Scale by the Bay!
linkedin.com/in/helenaedelson
3. @helenaedelson
• Interdisciplinary look at how complex adaptive systems apply
to distributed systems and information engineering
• Systems, intelligence and theories
• Entropy, Events and Time
• Rethinking adaptive systems, complexity and resilience
Different Approaches
4. @helenaedelson
Inspired By
• My scientific research before
working in tech
• What I've noticed in the industry
over almost two decades
• Questioning how we approach
distributed systems, balance and
disorder
Finding better ways to handle
system dynamics
• Creating models to predict
system dynamics
• Re-engineer energy flows in
biological systems
• Slow the rate of entropy in
those systems
9. @helenaedelson
sys·tem
• An entity comprised of interdependent
elements and subsystems
• More than the sum of its parts
• Has feedback loops
• Defined by its distinguishing edges
In this talk we refer to open systems
10. @helenaedelson
Systems Theory
• Discovering how elements of a system and its sub-
systems interact to produce given end states
• To understand a system's dynamics
• Changing one part affects others in the system
• Many systems-related theories developed out of this
Interdisciplinary study of systems
11. @helenaedelson
Bertalanffy proposed that Systems Theory needed a much
broader, unified approach
• Transcending technical problems
• Applicable to all scientific study (biology, physics...)
General System Theory
Was a new paradigm for scientific inquiry
12. @helenaedelson
Complex Adaptive Systems Theory
• Used to model an array different systems
• Complex, Non-Linear Systems: how order
emerges, e.g. in neural networks, galaxies,
ecosystems
• Self-organization - suggests living systems
can migrate to a dynamic state, the ”edge of
chaos”
- This discipline suggests living systems migrate to a state of dynamic stability they call the
"edge of chaos" or balance point.
Complexity Theory
13. @helenaedelson
Distributed Systems
• With increasing scale comes increased complexity and
potential for disorder
• The more moving parts in a system, the more things that
can fail
• In biological systems, the greater the diversity and/or
complexity, the greater the overall resilience
The larger the scale, the greater potential to fail
19. @helenaedelson
Second Law of Thermodynamics
• The law from physics stating that entropy increases
• Measures the degree of disorder of a system
• The increase in entropy accounts for the irreversibility of
natural processes, and the asymmetry between future and
past
Entropy
20. @helenaedelson
Entropy And The Arrow Of
Time
"If given complete knowledge of the universe for two instances of
time, how would you solve which instance happened first?
Order Disorder
Time
Calculate the entropy of the two snapshots. The one with lower entropy was first."
- Muller, Richard A, The Physics of Time
21. @helenaedelson
Future Light Cone
"If the sun were to cease to shine at this very
moment, it would not affect things on earth at the
present time because they would be in the
elsewhere of the event when the sun went out."
- Stephen Hawking, A Brief History of Time, 1988
Stephen Hawking, A Brief History of Time
22. @helenaedelson
Stephen Hawking, A Brief History of Time
• Events lie in the future light cone
everywhere that is not its origin
• When we look at the universe we are
seeing the past
28. @helenaedelson
The Immune System
• Exhibits a highly distributed, adaptive and self-organizing behavior
• Is a self-programming system
• Infinite ability to re-program itself to destroy threatening microbes
• Is a self-learning system
• Learns in parallel to fight the many forms of virus
30. @helenaedelson
Domino Effect
• Change of one can trigger
change in others
• Genesis event
• As elements of the system are
effected, they generate more
events
• E.g. cascading failure
32. @helenaedelson
Self-Organization
• We tend to assume that organization and
order need to be imposed by some external
force.
• Self-organization is the idea that this type of
global organization can instead be the result
of local interactions.
35. @helenaedelson
Emergence
Ant colonies are governed by very simple rules, and only local
interactions. Through combined activities, generate colonies that
• Exhibit complex structures and behavior
• Far exceed intelligence or capability of the individual
• Decentralized structure to self-organizing systems
• Organization is distributed over the whole system
• All parts contribute equally
Case Study
37. @helenaedelson
Cyclic, Predictable Patterns &
Resilience
Biological systems have natural feedback loops and strategies that enable
resilience to fluctuation.
The Three Rs
• Replication
• Regeneration
• Rebalance
40. @helenaedelson
Daily Pattern of
Movement
Arctic Wolves
• Top of their food chain
• Operate in packs, 30+
• Pack roams its territory daily
• Travel 40-100 miles per day
• Follows herd food sources
annually in their migration
43. @helenaedelson
Resilient Systems & Diversity
Variety of entities makes the systems more effective at absorbing change.
and variations in its environment.
44. @helenaedelson
Role Niche
• Organisms role in an
ecosystem
• The environment of the entity
• What it consumes
• How it interacts with other
elements or entities
• Entities role in a system
• Data ingestion
• Functions in the system
• How it interacts with other
elements or entities
If the number of entities performing a necessary function in a
system decrease, the system can fall into imbalance.
48. @helenaedelson
Tropic Cascade Case Study
A complex system in constant change
In 1926 the last wolf in Yellowstone
NP in the US was eliminated.
By 1994 the elk population grew to
roughly 19,000.
49. @helenaedelson
Elimination of the wolves caused a
cascade of changes through the entire
ecosystem.
With no natural predator, Elk
consumed most of their food
resources.
Tropic Cascade Case Study
A complex system in constant change
50. @helenaedelson
Destabilization
As elk increased
• Berries for bear food supply decreased
• Bear population fell to Endangered Species levels
• The coyote population increased to partially fill the niche
left by the wolves
• Tree and plant hight and numbers decreased dramatically
Absence of top predator altered the entire system
51. @helenaedelson
Reintroduction
• In 1995 14 grey wolves from Canada were introduced to
Yellowstone, after being absent for over 60 years
• A year later 17 wolves were introduced
• By December, 2001 their population had grown to 132
Of entities performing the primary regulating role
53. @helenaedelson
Regeneration
Elk started to avoid parts of the park where they were more
exposed for the wolves to hunt.
• Forests of aspen and willow began growing back
• As bushes and grasses grew back, there were more berries
• The diversity and number of birds started increasing
54. @helenaedelson
Repopulation
Trees started to grow taller again as the elk population
decreased.
• Beaver, previously extinct in the region, returned
• The dams beavers built provided habitat for otters and
other animals and reptiles
• Wolves hunted the coyote, decreasing their population 50%
• The numbers of rabbits and mice were able to grow back
• Which brought more red foxes, weasels, badgers
• The bald eagle and hawk populations grew
57. @helenaedelson
Rebalance
With the rebalancing of predator / prey, the populations of
many other species were again able to rebalance.
• The vegetation along rivers and lakes returned
• Erosion decreased
• Which changed the shape of the rivers
• River banks stabilized, channels narrowed
• More pools of water formed
• Increasing habitat for water birds and reptiles
59. @helenaedelson
– Stephen Hawking
“It is a matter of common experience that disorder
will tend to increase if things are left to themselves.”
Self-Balancing Systems
61. @helenaedelson
Research
There was a time when companies weren’t afraid to invest in
basic science.
Companies still invest heavily in innovation, but the focus is
practical applications rather than basic science.
Research and development has become “less R, more D” -
Prof. Ashish Arora, economics of technology and technical
change
62. @helenaedelson
Rate Of Innovation
• Why is information technology seemingly behind
technology in scientific fields such as astrophysics, particle
physics, molecular biology and behavioral neuroscience?
• They have made phenomenal gains but the compute
systems that network and manage them, and also capture,
process, store and query those system's data has not seen
the same speed in innovation.