Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems - Professors Daniel Martin Katz + Michael J Bommarito

Complexity
& Prediction
daniel martin katz
michael j bommarito
Toward a Characterization of
Law +
Legal Systems as Complex Systems

Today we are excited to
return to Ann Arbor …

for us -
this is the place
where it all began …

cut & paste
safety ﬁrst
west hall
stephen
wolfram’s
robot
presence

Here Are a Few
Things from Our Time
@ UM CSCS
3D HD Visualization of Supreme
Court Citation Network
Campaign Contributions and
Legislative Ecosystems
Six Degrees
of
Marbury
v.
Madison
Electronic
World
Treaty
Index
Radial
SCOTUS
Citation
Network

American
Federal
Judiciary
American
Law Professoriate
Building New Algorithms
Large Scale
Judicial Studies
Here Are a Few
@ UM CSCS

Mathematical Formalization of
the United States Code
Congressional Bills
Law as a Seamless Web?
SCOTUS + TAX
Here Are a Few
@ UM CSCS

ComputationalLegalStudies.com
BLOG
Here Are a Few
@ UM CSCS

For Me —
Culminating in this …

And for both of us
ultimately laying a
foundation for this …

So thanks to the
organizers for inviting
us to participate…
http://lsa.umich.edu/cscs/news-events/all-events/complexity-and-the-law.html

Today we would like to
sweep across a bunch
of our projects …

in the hopes that it might
spur a conversation

spur a conversation
and

spur a conversation
and
perhaps a collaboration .?

Today —
Three Areas of Our
Research that would
like to highlight

Network Science + Law
Exploring the ‘Evolution’ of the Law and its Institutions

Experts, Crowds + Algorithms
Three Forms of (Legal) Prediction

Mean-Field Measurement of Risk + Legal Complexity
Measuring Complex Systems

Exploring the ‘Evolution’ of the Law

Network Science has been
one of the fastest growing
areas of science over the
past 15+ years …

Networks are one
manner in which we
might formalize the
interdependencies
between units in
complex systems

Physical and
Biological Sciences

Computer
Science
Mapping
of the
Code
The Internet
The Iranian Blogosphere
code
dependencies

Hiring and Placement of
Political Science PhD’s
Co-Sponsorship in Congress
(Fowler et al)
Spread of Obesity
(christakis and fowler)
High School Friendship
(Moody)
Roll Call Votes in Congress
(Mucha, et al)
Social Science
The Political Blogosphere
(Adamic & Glance)

Among others, we
undertook some of the early
work (exploring merely a
small subset) of possible
applications for law

Network Science
in Legal Studies:
An Overview

The American Federal Judiciary

D. Katz, D. Stafford, Hustle and Flow: A Social
Network Analysis of the American Federal
Judiciary. Ohio St. L. J. 71, 457 (2010)
available at https://papers.ssrn.com/sol3/papers2.cfm?abstract_id=1103573

The American Law Professoriate

Computational Model
of Intellectual Diffusion

Katz, D. Gubler J. Zelner J. Bommarito M.
Provins E. & Ingall E. (2011). Reproduction
of Hierarchy? A Social Network Analysis of
the American Law Professoriate. Journal of
Legal Education, 61(1), 76-103.
https://papers.ssrn.com/sol3/papers2.cfm?abstract_id=1352656available at

Tracking the Development
of Common Law Jurisprudence

Views of the SCOTUS Citation Network
Radial
Layout
SCOTUS
Citation
Network
Horizontal
Layout
SCOTUS
Citation
Network

Six Degrees of Marbury v. Madison

Judicial Citation Networks
are Acyclic Digraphs

There has
Cases Decided by
the Supreme Court
Citations in the
Current Year
Citations from
prior years
watch full video at https://vimeo.com/9427420

Early Growth of the  
SCOTUS Giant Component

This is still a signiﬁcant
area for potential
scientiﬁc exploration …
#LegalScience
#STEMplusLawEqualsFuture

Not unlike many other
Human Socio-Technical systems

we observe is the
highly skewed distribution
of authority in legal systems
one of the common features —

This is some evidence
supporting the thesis that
Complex SystemsLAW =

Katz, et al (2011)
American Legal Academy
Katz & Stafford (2010)
American Federal Judges
Geist (2009)
Austrian Supreme Court
Smith (2007)
U.S. Supreme Court
Smith (2007)
U.S. Law Reviews
Post & Eisen (2000)
NY Ct of Appeals

https://computationallegalstudies.com/network-analysis-and-law-tutorial/
For those who
might want to
learn more …
Given time
is limited
for today …

In 2017, 2018 and beyond -
we hope to revisit and expand
upon our earlier work in a
variety of important ways …
(we are open to collaborating
in appropriate instances with
those who are interested)

Beyond the policy making
sphere where there is
a real case to be made that
prediction > causal inference …

There has been
growing interest in

rigorous
There has been
growing interest in

rigorous
There has been
growing interest in
out of sample

rigorous
There has been
growing interest in
prediction in law
out of sample

rigorous
#AI #LegalTech
#Machine Learning
#LegalAnalytics
There has been
growing interest in
prediction in law
out of sample

Here are just a few
predictions
that lawyers are trying to
accomplish on a daily basis

#Predict Relevant Documents
Data Driven EDiscovery/Due Diligence
(Predictive Coding)

(Predictive Coding)
#Predict Contract Terms/Outcomes
Data Driven Transactional Work

(Predictive Coding)
#Predict Rouge Behavior
Data Driven Compliance

#Predict Case Outcomes
Data Driven Legal Underwriting
(Predictive Coding)

#Predict Case Outcomes
Data Driven Legal Underwriting
(Predictive Coding)
#Predict Regulatory Outcomes
Data Driven Lobbying, etc.

In so much as prediction is
the task in question …
#LegalTech #FinTech
#Fin(Legal)Tech

“The real roll-up of all this isn’t robot lawyers,
it’s financialization, with law becoming an
applied branch of finance and insurance.”
Daniel Martin Katz, professor, Illinois Tech’s Chicago Kent College of Law
http://www.ozy.com/fast-forward/why-artificial-intelligence-might-replace-your-lawyer/75435

#Fin(Legal)Tech
https://computationallegalstudies.com/2016/02/27/ﬁn-legal-tech-
laws-future-from-ﬁnances-past-an-expanded-version-of-the-deck/
GO HERE FOR A DETAILED TREATMENT OF THE QUESTION

Borrowing in part from ﬁn(tech)

There are 3 Known Ways
to Predict Something
Borrowing in part from ﬁn(tech)

predicting the decisions of the
Supreme Court of the United States
#SCOTUS

Columbia Law Review
October, 2004
Theodore W. Ruger, Pauline T. Kim,
Andrew D. Martin, Kevin M. Quinn
Legal and Political
Science Approaches to
Predicting Supreme
Court Decision Making
The Supreme Court
Forecasting Project:

Case Level Prediction
Justice Level Prediction
67.4% experts
58% experts
From the 68
Included
Cases
for the
2002-2003
Supreme
Court Term

these experts probably
overﬁt

they ﬁt to the noise
and
not the signal

if this were
ﬁnance this
would be
trading
worse than
S&P500

like many other forms
human endeavor
law is full of  
noise predictors …

we need to
evaluate
legal experts
and
somehow
benchmark
their
expertise

from a pure
forecasting
standpoint

the best
known
SCOTUS
predictor is

the law
version of
superforecasting

https://fantasyscotus.lexpredict.com/case/list/
We can
generate
Crowd
Sourced
Predictions

not all
members of
crowd are
made equal

we maintain
a ‘supercrowd’
which is the top n
of predictors
up to time t-1

the
‘supercrowd’
outperforms
the overall
crowd
(and even the
best single player)

not
enough
crowd
based
decision
making in
institutions
(law included)

“Software developers were asked on
two separate days to estimate the
completion time for a given task, the
hours they projected differed by 71%,
on average.
W h e n p a t h o l o g i s t s m a d e t wo
assessments of the severity of biopsy
results, the correlation between their
ratings was only .61 (out of a perfect
1.0), indicating that they made
inconsistent diagnoses quite frequently.
Judgments made by different people
are even more likely to diverge.”

Brief Aside
About
Crowd
Sourced
Prediction
#LegalCrowdSourcing

(most pundits did not
identify as a serious
candidate him until
mid-January 2017)
Neil Gorsuch was #1
o n o u r F a n t a s y
Platform 12 Days after
Donald Trump was
elected President
(i.e Nov 20)

Columbia Law Review
October, 2004
Theodore W. Ruger, Pauline T. Kim,
Andrew D. Martin, Kevin M. Quinn
Legal and Political Science
Approaches to Predicting
Supreme Court Decision
Making
The Supreme Court
Forecasting Project:

Ruger, et al (2004)
relied upon
Brieman(1984)
(as partially shown below)

Leo Brieman moved away from
CART in Brieman (2001)

Breiman, L.(2001). Random forests.
Machine learning, 45(1), 5-32.
Published in Machine Learning
(A Springer Science Journal)

One well-known problem with
standard classiﬁcation trees is
their tendency toward overﬁtting

http://machinelearning202.pbworks.com/w/ﬁle/fetch/37597425/
performanceCompSupervisedLearning-caruana.pdf
Random
Forest
(particularly
with special
conﬁg/
optimization)
have proven to
be unreasonably
effective

Random forest is an approach to
aggregate weak learners into
collective strong learners
(using a combo of bagging and random substrates)
(think of it as crowd sourcing of models)

Our algorithm is a special version
of random forest (time evolving)
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2463244
available at
Revise
+
Resubmit
Version 2.02 -
January 16, 2017

We call this a ‘general’ model
of #SCOTUS Prediction
available at
https://arxiv.org/pdf/1612.03473

Not just interested in accuracy
over a short time window
available at

A locally tuned model will
typically lead to overﬁtting
as the dynamics shift
available at

We want a model that is
robust to a large number
of known dynamics …
available at

Version 2.02
January 16, 2017
243,882
28,009
Case Outcomes
JusticeVotes
Current Version of #PredictSCOTUS
1816-2015

Version 2.02
January 16, 2017
Current Version of #PredictSCOTUS
1816-2015
case accuracy
70.2%
71.9%
justice accuracy

But are these results ‘good’ ?

What constitutes ‘good’
performance in this context?

We Craft
Three
Alternative
‘Null’
Models

Our Model Against the Null Models
Some commentators had suggested using a heuristic rule of 
‘always guess reverse’ as a baseline
(Null Model 1 ) the always guess Reverse model
Turns out it is a lousy
model prior to ~1950
Because the reversal rate
is not stable over time

(Null Model 2 ) memory window = inf
This is our model against Null Model 2
What about memory window that selects the most frequent
historical outcome?
(Green = our model out performs)

(Null Model 3 ) ﬁnite memory window = 10
We in-sample optimize using future information to select a
null model that is among the best performing of all null models
as it is using in-sample info this is a deeply unfair null

Over past century, we outperform
M=10 by nearly 5% and have
signiﬁcant temporal stability at both
the justice, case, term level

For most problems ...
ensembles of these streams
outperform any single stream

the non-trivial question
is how to optimally assemble
such streams for particular problems

Humans
+
Machines
Humans
or
Machines
>

Here is what we are
working on right now …

expert
forecast
crowd
forecast
learning problem is to discover how to blend streams of intelligence
algorithm
forecast
ensemble method
ensemble model

expert
forecast
crowd
forecast
learning problem is to discover how to blend streams of intelligence
algorithm
forecast
ensemble method
ensemble model
via back testing we can learn the weights
to apply for particular problems

By the way, you
might ask why does
one care about
marginal improvements
in prediction ?
#Fin(Legal)Tech

Given our ability to offer
forecasts of judicial
outcomes, we wondered
if this information could
inform an event driven
trading strategy ?

Revise + Resubmit @
http://arxiv.org/abs/1508.05751
available at
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2649726

Mean-Field Measurement of Risk + Legal Complexity
Measuring Complex Systems

For years, we have been arguing
social, economic & political
complexity
is generating
legal complexity …

This complexity is challenging
our (legal and other)
long standing institutions

Measuring Complexity
in Complex Systems
is a major challenge …

This is the United States Code

+
Hierarchical
Structure
Citation
Network

M. Bommarito. & D. Katz. A Mathematical Approach to the
Study of the United States Code. Physica A: Statistical Mechanics
and its Applications, 389(19), 4195-4200 (2010).

D. Katz & M. Bommarito. Measuring the complexity of the law: the United
States Code. Artiﬁcial intelligence and law, 22(4), 337-374. (2014)

Mean-Field Measurement Projects -
Mining 10k’s / 8k’s
Exploring Regultory Risk
One of our current

Paper 1
(Currently Under Review)
M. Bommarito., D. Katz,
Measuring the Temperature
and Diversity of the U.S.
Regulatory Ecosystem (2016)
V available at
https://arxiv.org/pdf/1612.09244.pdf
https://papers.ssrn.com/sol3/papers2.cfm?abstract_id=2891494
2

Summary
Available
Here
https://www.law.ox.ac.uk/business-law-
blog/blog/2017/02/measuring-temperature-
and-diversity-us-regulatory-ecosystem

“A Form 10-K is an annual report
required by the U.S. Securities and
Exchange Commission (SEC), that gives
a comprehensive summary of a
company's ﬁnancial performance.”

Data Collection and Pre-Processing
Form 10-K contains
a range of relevant
information about
regulatory exposure 
(and other risks)

“…the Financial Executives
Research Foundation reveals
a mean and median 2015
expense of $1.8M and
$522,205, respectively.”
Real Resources are used to
produced these reports:

Countervailing Incentives:
Report to get a form of
‘securities fraud insurance’

Countervailing Incentives:
Report to get a form of
‘securities fraud insurance’
Do not enumerate all risks
under the sun because it
may scare investors

data is *not* perfect
but it is large scale
characterization of
the manner in which
regulations impact
companies

Text of the
10-k for
Company i
in year y
Synonym +
Fuzzy String
Matching to Act,
Agency, U.S. Code
and CFR Masterlist
Gramm Leach
Bliley
Financial
Services
Modernization
Act
GLBA
Graham Leach Bliley
Financial Services
Modernization” Act
Gramm Leach
Bliley Financial
Services
Modernization
Act of 1999

Data
Publically Traded Companies34,000+

Data
Publically Traded Companies
Total Number of 10-K’s
34,000+
160,000+

Data
Publically Traded Companies
Total Number of 10-K’s
Total Number of
Act / Agency References
34,000+
160,000+
4.5 million

A Mean - Field Characterization
Temperature

What is
the global
average
temperature?
Has this changed
over time?
Paper 1

Paper 1
Temperature =
# of
Acts / Agency
References
as a Function
of Time

Paper 1
Note:
Norms,
Incentives,
Requirements
have changed
over 20+ years…

Diversity

Use BitString Encoding
as an Information Theoretic
Representation for the Self-
Identiﬁed Regulatory Exposure

0
1
1
0
0
0
0
0
1
…
Encoding
Regulatory
Bitstring for
Company i
in year y
For each
{company, year} pair
Encode as 1 for the
presence of an act / agency
Pseudocode for BitString Encoding
Encode as 0 otherwise

0
1
1
0
0
0
0
0
1
…
Encoding
Regulatory
Bitstring for
Company i
in year y
Text of the
10-k for
Company i
in year y
Synonym +
Fuzzy String
Matching to Act,
Agency, U.S. Code
and CFR Masterlist
Gramm Leach
Bliley
Financial
Services
Modernization
Act
GLBA
Graham Leach Bliley
Financial Services
Modernization” Act
Gramm Leach
Bliley Financial
Services
Modernization
Act of 1999

We have 160,000+ of these
{year, company} bitstrings

Is the Diversity of the System
Growing or Contracting?

We use Hamming Distance
as a Mean-Field Characterization

Diversity of the Overall
Regulatory Ecosystem

Regulatory Dynamics Revealed
by the Securities Filings of
Publicly Traded Companies
Paper 2
(Currently In Progress)

In Bommarito & Katz 2016,
we explored the mean-ﬁeld
temperature and diversity of
the regulatory ecosystem

and mentioned that there
were likely to be differential
regulatory dynamics and
microclimates therein

Classiﬁcation Using
Time Series Signatures
Characterizing Dynamics:

In the spirit of
complex systems /
physics, we would
like to try offer a
generalization …

We are
undertaking
a version of
this approach
Riley Crane and Didier Sornette. "Robust dynamic
classes revealed by measuring the response function of
a social system." Proceedings of the National Academy
of Sciences 105, no. 41 (2008): 15649-15653.

Some
Motivating
Examples
Clean Water Act
Sarbanes Oxley
Y2K

Again,
leveraging
the time series
signature …

μ
σ
τ auto-correlation
(can be thought of as memory)
set each parameter to base zero at
time = and indext 0
variance2
mean (H, M, L)
(+, -)
(H, M, L)

taxonomy
of
behavioral
signatures

(‘M’ ‘H’, ‘+’)
Anti Kickback
Fairness In Asbestos Injury Resolution
American Clean Energy And Security
Pension Funding Equity
Medicare, Medicaid, And Schip Beneﬁts Improvement
Clusters that have a Similar
Behaviorial Signature
(Not necessariy topically similar)
(‘H’ ‘H’, ‘+’)
Patient Protection And Affordable Care
Secure And Fair Enforcement For Mortgage Licensing
Dodd Frank Wall Street Reform And Consumer Protection
Energy Independence And Security
Tax Relief, Unemployment Insurance Reauthorization

In sum, we are able to use this
framework to classify any
regulation by its behavior
upon the broader
regulatory ecosystem

Regulatory ‘Proﬁles’
and Mircoclimates

Once we have a distance metric
or some manner to encode edges…

we can generate a network
projection of the overall
company landscape …

Network Generation Procedure
a) Calculate the Hamming distance matrix as described
in Paper #1 over Acts
b) Threshold the matrix by removing all elements whose
distances are greater than D (D=3 in this ﬁgure)
c) Generate graph from resulting edge-weighted
adjacency matrix, where edge weight = 1/(1 + d)
d) Layout is Fruchterman-Reingold weighted showing only
giant component

Sequencing
of
Regulatory
Bitstrings
Universal Profiles
versus
Specific Profiles
(Growing Speciation)

Project is Still in Progress
So More to Come Soon …

Today we have
provided an
overview of three
areas of our
research

We hope to work with
some of you to advance
the state of the science
in this ﬁeld

Associate Professor of Law
The Law Lab @ Illinois-Tech
Afﬁliated Faculty
Stanford CodeX
Center for Legal Informatics
Founder + Director

Fellow
Stanford CodeX
Center for Legal Informatics
Director of Research
The Law Lab
IllinoisTech - Chicago Kent Law

Chief Strategy Ofﬁcer
LexPredict
Chief Executive Ofﬁcer
LexPredict

ComputationalLegalStudies.com
BLOG

Michael J. Bommarito II
@ mjbommar
computationallegalstudies.com
lexpredict.com
bommaritollc.com
illinois tech - chicago kent college of law@
thelawlab.com

Daniel Martin Katz
@ computational
computationallegalstudies.com
lexpredict.com
danielmartinkatz.com
illinois tech - chicago kent college of law@
thelawlab.com

Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems - Professors Daniel Martin Katz + Michael J Bommarito

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems - Professors Daniel Martin Katz + Michael J Bommarito

Similar to Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems - Professors Daniel Martin Katz + Michael J Bommarito (20)

More from Daniel Katz

More from Daniel Katz (15)

Recently uploaded

Recently uploaded (13)

Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems - Professors Daniel Martin Katz + Michael J Bommarito