Pete Newell & Steve Blank on Diagnostics and Metrics for Innovation

The Innovation Pipeline
Diagnostics, Metrics and Doctrine
Steve Blank, Pete Newell
www.steveblank.com
www.bmnt.com 1
2019 BMNT & Steve Blank

• US Army Colonel (Retired)
• Enlisted National Guardsman
• Commanded platoon, company, battalion and brigade Infantry units
• Combat tours in Panama (1989) and Iraq (2004 & 2009)
• Watch Officer on duty in the National Military Command Center (Pentagon) on
9/11
• Led the U.S. Army Rapid Equipping Force
• 360 projects, 115 material deployments, 20 Program of Record transitions
• $1.5B invested
• Defense News 100 most influential people in the Defense Industry (2012)
• Co-Founder and CEO, BMNT (2013)
• Forbes top 25 veteran founded start-ups in America (2015)
• Inc 1000 list of the fastest growing private companies in America (2017)
• Co-Author, Hacking for Defense
• Co-Founder and Board Director H4Di.org (2015)
• Seed Investor, Board Advisor and Board Director
• Visiting Fellow, National Defense University (2014-2016)
Pete Newell
2

Steve Blank
• Innovation consultant to the Intelligence community
• Co-author: Hacking for Defense (H4D)
• Co-author: I-Corps @ NSA
• Author: NSF I-Corps
• Co-creator of the Lean Startup methodology
• Participant/founder 8 startups in Silicon Valley
• Author The Four Steps to the Epiphany and
The Startup Owner’s Manual
• Stanford - Adjunct Professor / Columbia – Senior Fellow
• USAF 1971-1975 EW/ELINT in TAC and SAC
3

What got this conversation started
4

2016
Lots of Activities, Lots of Frustration
5

Individual
Innovation
Innovation
Activities
Single-Team
Innovation
Operational
Innovation
Leadership
Process Examples Deliverables
Heroic CIA Innovators Alliance One-off successes
Frustration w/system
6

Individual
Innovation
Innovation
Activities
Single-Team
Innovation
Operational
Innovation
Leadership
Heroic Innovators Alliance One-off successes
Innovators – invent things
Entrepreneurs – get innovation adopted
7

Individual
Innovation
Innovation
Activities
Single-Team
Innovation
Operational
Innovation
Leadership
Heroic CIA Innovators Alliance One-off successes
8

Individual
Innovation
Innovation
Tools &
Activities
Single-Team
Innovation
Operational
Innovation
Activities/
Training
Lean, Design Thinking,
Hackathons, Incubators,
Outposts, Maker Spaces
Innovation Demos,
Prototypes
People
9

Individual
Innovation
Innovation
Activities
Single-Team
Innovation
Operational
Innovation
Activities/
Training
Maker Spaces
Innovation Demos,
Prototypes
People
Lean/I-Corps I-Corps@NSA, H4D
Problem Curation
Evidence-based MVPs
Incubator-ready Team
We lacked an end-to-end process - from sourcing to deployment
10

Individual
Innovation
Innovation
Activities
Single-Team
Innovation
Mission
Driven
Innovation
Activities/
Training
Maker Spaces
Innovation Demos,
Prototypes
People
Problem Curation
Evidence-based MVPs
Pipeline Innovation Pipeline Deliver more, faster
Decision Points
So we built one
11

2017
We deployed the Innovation Pipeline
& a few new tools
12

An Innovation Pipeline
That solves the Red Queen Problem
Requires a disciplined, evidence-based, data-driven, process for
connecting innovation activities into an accountable system that rapidly
delivers solutions to hard problems
Here is our Example
13

New Tool - Mission Model Canvas
14

New Tool - Portfolio Management
15

2018
We got out of the building and gathered data
…and we learned a ton
16

Individual
Innovation
Innovation
Activities
Single-Team
Innovation
Operational
Innovation
Activities/
Training
Maker Spaces
Innovation Demos,
Prototypes
People
Problem Curation
Evidence-based MVPs
Pipeline Innovation Pipeline – CIA
H4X
Deliver more, faster
Decision Points
Contracts HRFinance LegalProcurement Security
• One of the things we learned was that bottlenecks to
innovation existed everywhere
• We asked, “Why?”
• And how can we fix it?
17

Individual
Innovation
Innovation
Activities
Single-Team
Innovation
Operational
Innovation
Leadership
Activities/
Training
Maker Spaces
Innovation Demos,
Prototypes
People
Problem Curation
Evidence-based MVPs
H4X
Decision Points
Guidance/
Investment/Org
Decisions
• In a perfect world the DNI would extend current guidance
to include Innovation Guidance:
1. Replace capabilities that atrophy (tech, tradecraft, etc.)
2. Create new capabilities that get us ahead of adversaries
3. Create new processes to do more faster
4. Create a way to rapidly reconfigure and standup
new organizations as new threats and tech emerges
5. Create innovation processes, procedures and orgs to do so
Innovation as a
parallel system
Innovation Doctrine
Barrier Reduction
18

Individual
Innovation
Innovation
Activities
Single-Team
Innovation
Operational
Innovation
Leadership
Activities/
Training
Outposts, Maker Spaces
Innovation Demos,
Prototypes
People
Problem Curation
Evidence-based MVPs
Incubator-ready Team
H4X, ONR
Decision Points
Innovation
Network
Guidance/
Investment/Org
Decisions
Innovation as a
parallel system
Innovation Doctrine
Barrier Reduction
19

Individual
Innovation
Innovation
Activities
Single-Team
Innovation
Operational
Innovation
Leadership
Activities/
Training
Maker Spaces
Innovation Demos,
Prototypes
People
Problem Curation
Evidence-based MVPs
H4X
Decision Points
Guidance/
Investment/Org
Decisions
• Innovation Guidance would remove barriers
• By creating fast-track processes for innovation
Innovation as a
parallel system
Innovation Doctrine
Barrier Reduction
Contracts HRFinance LegalProcurement Security
X X XX X X
20©2019 BMNT & Steve Blank

Individual
Innovation
Single-Team
Innovation
Operational
Innovation
Leadership
One-off successes
Problem Curation
Evidence-based MVPs
H4X
Decision Points
Guidance/
Investment/Org
Decisions
Innovation
Network
Problem:
• It’s not a perfect world
• Leadership is focused on execution
• Innovators feels like it is a continual uphill battle
Innovation as a Pillar Coffee Cups,
Lanyards, posters
There is a fix
• Diagnostics
• Metrics
• Doctrine
21

2019
22

When operationalized…
the innovation pipeline creates a funnel for innovation
Ideas
Technology
Problems
People
Deployed
Solutions

24

What We Learned: The Innovation Pipeline
• Leadership needs metrics to defend their decisions and resource requirements
• Goals need to be set, investments made and resources applied at every step of
the pipeline
• Problem curation is more complex and critical than we thought. Prioritization is
the output, not a next step
• Skilled practitioners and experienced entrepreneurs are needed both inside and
outside the agencies
• We need to incubate better: We’ve created a “Bermuda triangle” where nascent
teams are lost because they are not ready for an accelerator or to transition to a
program

There are 5 critical components to each step in the pipeline:
1. Strategy
2. Pipeline Process
3. Operational Support
4. Activities, Methodologies and Tools
5. Metrics and Data
26

There are GO/NO-GO stops between each step
27

… and in the next two days you’re going to hear from others who also
got out of the building and got real data
1:15-2:30 Setting the tone: Doctrine, Diagnostics and Prime
Investments - Part I
2:30-2:45 Break
2:45-4:00 Setting the tone: Doctrine, Diagnostics and Prime
Investments - Part II
4:00-4:15 Break
4:15-5:15 Discussion #1: Internal Innovation (Lead by Kevin
Keaton, NSA; facilitated by Steve Weinstein)
5:15-6:00 Ranch tour, reset, dinner prep
6:00-8:00 Dinner: Discussion #2: Countering Chinese
investment in the US (Lead by Mike Brown, DIU; facilitated by
Steve Blank)
9:00-10:00 Discussion #3: Scaling Innovation (Lead by Rich Carlin,
ONR; facilitated by William Treseder)
10:00-10:30 Break: Birds of a feather #1
10:30-11:30 Discussion #4: Investing in Innovation (Lead by Chris
Moran, LMCO & Steve Bowsher, IQT; facilitated by Heather Richman)
11:30-12:30 Lunch: Discussion #5 (Innovation Education Panel;
facilitated by Pete Newell)
12:30-1:30 Discussion #6: Building an Innovation Strategy (Lead by
Christy Monaco; facilitated by Doug Weinstein)
1:30-2:30 Discussion #7: Diagnosing the Innovation System (Lead by
Doug Cate, DIA; facilitated by Brian Miller)
2:30-2:45 Innovation Pipeline Cohort Results
2:45-3:15 Break: Birds of a feather #2
3:15-4:00 Wrap up - What’s important (Pete / Steve)
Today Tomorrow
28

Diagnostics and Metrics
Overview
29

Diagnostics & Metrics: The Innovation Pipeline
Activities, Methodologies, & Tools
Operational Support
Pipeline & Infrastructure Process
Strategy
Metrics & Data 30

Strategy: The Innovation Pipeline
Strategy
The team responsible for the step needs
established goals for the step
Leadership needs established goals for each
step
31

Pipeline & Infrastructure: The Innovation Pipeline
Process
• What’s the input and where does it
come from?
• What’s the output (deliverable
product of this step)?
Team / Resources
• What team and what expertise is needed
for the team managing the pipeline?
• What about for teams going through the
pipeline?
• What resources are required (funding,
contracts vehicles, facilities)
32

Operational Support: The Innovation Pipeline
Organizational support
• Contracts & contracting officers,
Human Resources, Security, access to
testing facilities, standing exceptions to
policy
Operational Support
Leadership support
• Provide resource & compliance approvals,
conduct firefighting, act at key decision
points between steps.
33

Activities, Methodologies, Skills & Tools: The Innovation Pipeline
Activities and Methodologies
• What methodologies are used to
perform this step
• Skills required and experiences needed
Activities, Methodologies, Skills & Tools
Skills and Tools
• Skills need to enable execution
• Tools applied to collect insight
34

Metrics & Data: The Innovation Pipeline
Metrics are used to measure throughput
and quality of output
Metrics & Data
Vital signs (data) are used to check
performance
35

The Innovation Pipeline: Decision Support Tool
36

TRL+ IRL+ ARL
Measure Readiness
How to Measure Team Progress Through the Innovation Pipeline
37

Measure Team Progress
• Measure Technology Readiness, Investment Readiness and
Acquisition/Deployment Readiness
• All are data-driven and evidence-based
• Each team can track their own progress
• Leadership can measure teams progress
• Can establish funding and deployment milestones based on TRL/IRL/ARL number
38

Technology + Investment + Adoption
Readiness Level
TRL
Problem validation
Complete a first-pass canvas
Problem/Solution Validation
Low Fidelity MVP Experiments
Validate Solution/Mission fit
Validate right side of canvas
Prototype High-Fidelity MVP
Validate left-side of canvas
Validated funding/deployment
IRL ARL
39

Technology Readiness Level
Investment Readiness Level
Adoption Readiness Level
When to Measure Readiness Levels

TRL – Measure
Technology Readiness
41

NASA/DOD
Technology Readiness Level (TRL)
• Formal Way to assess project maturity
• Quantify Relative Risks
• Data Driven
• Adopted by NASA, DOD, FAA, ESA…
42

NASA/DOD: Technology Readiness
Level
43

NASA/DOD Technology Readiness:
Levels 1 & 2
Basic Technology Research
• Basic principles observed
• Technology concept formulated
Concept
44

NASA/DOD Technology Readiness
Levels 3 & 4
Research to prove Feasibility
• Experimental proof of concept
• Breadboard validation in lab
Research
Concept
45

Levels 5 & 6
Demo Prototype
• Breadboard validation outside the building
• System demo in real-world
Research
Concept
Demo
46

Levels 7, 8, 9
Deployment
• System Development
• System deployed in real-world
Research
Concept
Demo
47

TRL Measure Team Progress
• TRLs are data-driven and evidence-based
• Can establish funding milestones based on TRL number
48

IRL – Measure
Investment Readiness
49

• A Formal Way to Quantify Relative Risks
• Data Driven
• Analog to NASA/DOD Technology Readiness Level
50

We can do the same
for new mission-driven
ventures
51

We can do the same
for mission-driven
Ventures
Emphasis is on data
52

Mission Model Canvas drives the –
Mission Model Canvas = hypotheses of how you
create and deliver value for the
DOD/IC and the warfighter
Source: Alexander Osterwalder- Business Model Generation
KEY
PARTNERS
Value
Proposition
CHANNELS
CUSTOMER
RELATIONSHIPS
CUSTOMER
SEGMENTS
REVENUE STREAMSCOST STRUCTURE
KEY
ACTIVITIES
KEY
RESOURCES
Beneficiaries
Deployment
Buy-
in/Support
Mission Achievement
Value
Proposition
Activities
Resources
Partners
Costs
53

Mission Model Canvas
Source: Alexander Osterwalder- Business Model Generation
KEY
PARTNERS
Value
Proposition
CHANNELS
CUSTOMER
RELATIONSHIPS
CUSTOMER
SEGMENTS
KEY
ACTIVITIES
KEY
RESOURCES
Beneficiaries
Deployment
Buy-
in/Support
Mission Achievement Factors
Value
Proposition
Activities
Resources
Partners
Costs
How does the
team get “Buy-
In” from all the
beneficiaries?
How will we deploy
the product to
widespread use?
What constitutes a
successful
deployment?
Who are our
most important
customers?
Stakeholders?
What are their
pains/gains?
What job do
they want us to
get done for
them?
How are we
solving each
customers
pains/gains?
What
product/service
features match
their needs?
What key activities
do we need to be
expert in?
What key resources
do we need to own
or acquire?
Financial? Human?
Who are our key
partners?
Suppliers?
What are we
getting from
them?
Giving them?
What is the Mission Budget/Cost? How will we measure Mission Achievement?
54

KEY
PARTNERS
Value
Proposition
CHANNELS
CUSTOMER
RELATIONSHIPS
CUSTOMER
SEGMENTS
KEY
ACTIVITIES
KEY
RESOURCES
• The TRL answers questions about the technology
• The IRL answers questions about the Mission Model
Canvas in Curation, and Discovery
55

Problem validation
KEY
PARTNERS
Value
Proposition
CHANNELS
CUSTOMER
RELATIONSHIPS
CUSTOMER
SEGMENTS
KEY
ACTIVITIES
KEY
RESOURCES
The IRL answers questions about
the Mission Model Canvas in
Curation, and Discovery
56

IRL Measure Team Progress
• IRLs are data-driven and evidence-based
• Can establish funding milestones based on IRL number
57

Technology Readiness Level + Investment
Readiness Level
Problem validation
TRL IRL
58

ARL – Measure
Adoption Readiness
59

ARL - Adoption
Readiness Level
• The TRL answers questions about the technology
• The IRL answers questions about the Mission Model Canvas
in Curation, Discovery and Incubation
• The ARL answers questions about Incubation & Transition
60

Adoption
Readiness Level
The ARL answers questions
Incubation about Transition
Sponsor $s to figure out
solution/ mission fit
Sponsor & Problem Owner
Commitment to find a solution
Teams Funded
Funded Pilots
Field Trials
*Final DOTMLPF
Deployment
Leadership commitment
Final Requirement
* DOTMLPF is defined in The Joint Capabilities Integration Development System, or JCIDS Process.
The JCIDS process provides a solution space that considers solutions (stated in terms of diplomatic or military
organization) involving a simulation of doctrine, organization, training, materiel, leadership and education,
personnel, and facilities (DOTMLPF) required to accomplish a mission.[2
61

ARL Measure Progress
to get a solution adopted
• ARLs are data-driven and evidence-based
• Can establish funding milestones based on ARL number
62

TRL+ IRL+ ARL
Measure Readiness
63

Technology + Investment + Adoption
Readiness Level
TRL
Problem validation
IRL ARL
64

Self Assessment
65

Strategy
- What’s the goal for the Team
- What’s the goal for Leadership
Process
- What’s the input and where does it come from?
- What’s the output (deliverable product of this step)?
Team / Resources
- What team and what expertise is needed?
- What resources are required (funding, contracts,
exceptions)
Operational Support
- Type of organizational support required (contracts, HR,
Security)
- What leadership is required (resource/compliance decisions,
firefighting etc)
Activities, Methodologies, Tools and Metrics
Activities and Methodologies
- What methodologies are used to perform this step
- Skills required and experiences needed
Tools and Skills
- Tools applied to collect insight
- Skills need to enable execution
Data and Assessments
- What metrics are used to measure throughput
- What vital signs (data) are used to check performance
INNOVATION PIPELINE SELF ASSESSMENT KEY
66

M Indicate missing components
Highlight program weaknesses
Highlight program strengths
Comment
Score
Indicate score for each sub-category
1 - 5
1 – Dysfunctional or missing component
5 – Exceptional
Total
Total all scores from subcategories
INNOVATION PIPELINE SELF ASSESSMENT KEY
Teams’ goals are defined to achieve pipeline stage
Appropriate input is available to garner success
Team & expertise is available to achieve success
Organizational support is available for this stage of
the pipeline
Activities are being executed to generate output
Tools defined to collect insight
Metrics are defined to determine success on this
stage of the pipeline
Leadership’s goal is defined to achieve pipeline
stage
Output is generated to drive next stage of the
pipeline
Resources are available to achieve success
Leadership available & decisions points identified
to have progress on this pipeline stage
Methodologies defined/used to guide discovery
Training is being provided to develop execution
skills
Data is collected to measure vital signs for this
Score Score
Score Score
Score Score
Score Score
Score Score
Score
ScoreScore
Score
67

Sourcing
68

Sourcing – Strategy: The Innovation Pipeline
Strategy
• Goal for the Innovation Team: Build a repeatable
and scalable sourcing process that ensures there is a
fresh pool of ideas, tech, people and problems that
is exponentially larger than the number of solutions
desired to be deployed (using internal and external
resources)
• Goal for Agency Leadership: Gain the peripheral
vision needed to:
• provide early warning that disruptive tech may by
accelerating
• Build systems to enable the org to rapidly seize
opportunities to disrupt our enemies
• Retaining the agility to respond to the unknown (at
scale)
69

Sourcing – Process: The Innovation Pipeline
• Input to Sourcing: Active internal / external
sourcing efforts
• Output to Curation: Large pool of people, tech,
ideas & problems (pool continuously refreshed)
• Innovation team and expertise needed: Active &
aggressive Ecosystem Architects with local
knowledge, internal and external sourcing
programs, great Bus Dev / Connector skills
Resources required:
• Full time & contract ecosystem sourcing team
• Physical space
• Contracting, financing, legal and security
liaisons
• $’s for travel, $ to host programs (ie:
Hackathons, Sprints)
70

Sourcing - Operational Support: The Innovation Pipeline
• Organizational support required: Contracts, HR,
Security, engineering and R&D liaisons with
authority for sourcing exceptions
Operational Support
• Leadership support required: Prioritize current &
future problems for the innovation team. Provide
the support to allow rapid and fast track internal
and external sourcing of tech, problems and
people by providing parallel processes for
security, contracting, HR, finance, policy and
engineering.)
71

Sourcing - Activities, Methodologies, Tools & Metrics:
• Methodologies to perform sourcing: Funnel,
Hackathons, Crowd Sourcing, Kickstarters, external
partnerships, proxy generation, Outreach Services,
Extraction Workshop, H4X
• Skills/experiences to perform sourcing: Aggressive
connector, team/coalition builder, connection to
senior leadership, organizational silo breaker
• Skills for execution: Aggressive ecosystem
development team, H4X, Lean
• Tools applied to collect insight: Osterwalder
Portfolio Management, innovation pipeline model,
input / output ratio / funnel metrics, Problem
Sourcing Canvas
72

Sourcing – Metrics & Data: The Innovation Pipeline
Metrics & Data
Metrics
• # of active internal and external sourcing programs
• Ratio of new sources netted each event to those
actually engaged in other parts of the pipeline
over time (passive vs active connections)
• Ratio of entities sourced vs entities accepted into
next phase (1-1 ration indicates a weak sourcing
mechanism, 4-1 may be more preferable)
Data
• # and diversity of independent sources (tech,
people, problems & ideas) across three horizons
of potential innovation
• # of active engagements from the pool that
remain engaged in follow-on steps (ie; # of
innovators and entrepreneurs moving into
curation)
• # of vetted problems accepted into Curation
73

Self Assessment
74

Strategy
Process
Team &
Resources
Operational
Support
Activities &
Methods
Metrics
& Data
Team’s goals are defined to achieve pipeline stage Leadership’s goal is defined to achieve pipeline
stage
Appropriate input is available to garner success Output is generated to drive next stage of the
pipeline
Team & expertise is available to achieve success Resources are available to achieve success
the pipeline
Activities are being executed to generate output Methodologies defined/used to guide discovery
Tools defined to collect insight Training is being provided to develop execution
skills
Tools &
Skills
Score Score
Score Score
Score Score
Score Score
Score Score
Score Score
Score Score
*Total
Score
SOURCE
75

Curation
76

Curation – Strategy: The Innovation Pipeline
Strategy
• Goal for the Innovation Team: Build a repeatable
and scalable curation and prioritization process
that creates a vetted problem pool that is
exponentially larger than the number of solutions
desired to be deployed
• Goal for Agency Leadership:
• Continuous insight into emerging problems and
the decay of current assets.
• Establish organizational momentum for
innovation based on a prioritized portfolio
(Horizon 1, 2, 3) of current & future problems.
77

Curation – Process: The Innovation Pipeline
• Input from Sourcing: a healthy & diverse pool of
ideas, problems, technology and innovators
from active internal AND external sourcing
programs
• Output to Discovery: prioritized portfolio of
curated problems. Concept sketch for how the
problem presents itself, initial tech & expert
ecosystem map, problem sponsor & champion
for each problem
• Innovation team expertise needed: Problem
Curation team, technology analysis, ecosystem
architects
• Resources required: Full time curation team,
physical space, contracting, financing, legal and
security liaisons. $’s for travel. Organizational OK
for curation team to spend time on
problems/tech in the pipeline, teams into the
pipeline

Curation - Operational Support: The Innovation Pipeline
Operational Support
• Organizational support required: Contracts, HR,
Security, engineering and R&D liaisons with
authority for sourcing exceptions
• Leadership support required: top-level
agreement for fast track sourcing, committed
funding, written exceptions for security,
contracting, policy, legal for innovation sourcing
79

Curation - Activities, Methodologies, Skills & Tools:
• Activities supporting curation: Active outreach
to find problems and sponsors, Problem Curation
Workshop, Technology Exploration
• Methodologies to perform curation: Problem
Curation, Concept (CONOP) development, MVP
development, Mission Model Canvas (MMC),
Portfolio map, Investment Readiness Level (IRL) /
Technology Readiness Level (TRL) / Adoption
Readiness Level (ARL)
• Skills & experiences to perform curation: Problem
Curation, Talking to Humans, Interviewing, Lean,
Technology Assessment, Design Thinking, MVP
development, Iterative Development
• Tools applied to collect insight: Problem curation
workbook, petal diagrams, Value Proposition
Canvas (VPC), Collaboration Map

Curation – Metrics & Data: The Innovation Pipeline
Metrics & Data
Metrics
• Ratio of problems / new concepts curated vs
those accepted into Discovery (1-1 reflects a
weak pool, 4-1 is closer to historical norms from
REF and H4D)
• Ratio of problems / new concepts spread across
Alex O’s portfolio map vs 3 Horizons
• Initial TRL / IRL and ARL established
• Quality of teams in Discovery - how many meet
minimum standard & remain intact
Data
• # of beneficiary, innovators, entrepreneurs and
tech expert interviews conducted to validate
problems & ideas
• Diversity of innovators, entrepreneurs and
experts engaged in validation
81

Self Assessment
82

Strategy
Process
Team &
Resources
Operational
Support
Activities &
Methods
Metrics
& Data
stage
pipeline
the pipeline
skills
Tools &
Skills
Score Score
Score Score
Score Score
Score Score
Score Score
Score Score
Score Score
*Total
Score
CURATE
83

Discovery

Discovery – Strategy: The Innovation Pipeline
• Goal for the Innovation Team:
• Build a repeatable and scalable discovery
process that validates problems and solution
pathways, while also narrowing the potential
solution teams to only the most viable for
further investment.
• Run a Lean/I-Corps model using synchronous
teams, MMC, MVPs
Strategy
Goal for Agency Leadership: Establish a
continuous flow of options to:
• Accelerate to replace decaying/lost
capabilities
• Provide options to disrupt emerging
protagonists
85

Discovery – Process: The Innovation Pipeline
• Input from Curation: prioritized portfolio of curated
problems. For each problem; concept sketch for
how the problem presents itself, initial tech & expert
ecosystem map, problem sponsor & champion
• Output to Incubation: Validated MVPs, test data and
a pathway to deliver a solution; teams with the
potential to deliver; Early adopters/1st customer ID’d
• Innovation Team expertise needed: Lean/I-Corps
Instructors, internal entrepreneurs recruited to
teams to build agency buy-in & coalition.
• Expertise for the team in the pipeline:
Hypothesis generation, MVP development,
MVP testing development, Interviewing
techniques, data capture
• Resources are required – Full time innovation
team, physical space, contracting, financing,
legal and security liaisons. $’s for travel.
Organizational OK for innovators to spend
time on problems/tech in the pipeline, teams
into the pipeline. Other orgs bought into 1)
fast track process 2) adoption of output 86

Discovery - Operational Support: The Innovation Pipeline
• Organizational support required:
• Contracts, HR, Security, engineering and
R&D liaisons with authority for discovery
and MVPs;
• Understanding how to “get to Yes” from
those groups;
• Unfettered access to potential
beneficiaries.
• Coaches and mentors to support teams in
discovery
Leadership support required:
• Top-level agreement for fast track MVPs;
• Funding for teams in the discovery process
(travel, prototyping, stipends, pay); w
• Written exceptions for security, contracting,
policy, legal for innovation sourcing;
• Designated/trained organizational POCs in
contracting, HR, Resource Management
• Firefighting board of organizational leaders
capable of reducing barriers for innovation teams
Operational Support
87

Discovery - Activities, Methodologies, Tools & Metrics:
• Activities supporting discovery: Hacking
for Defense, I-Corps, Exploration
Workshops, Discovery workshop, Product
Jam / Experimentation Workshop
• Methodologies to perform discovery: Lean
StartUp, MVP Development, Mission
Model Canvas (MMC), Business Model
Canvas (BMC), Portfolio Management,
• Skills & experiences to perform discovery: Innovation
teams need innovator + entrepreneurs; Pipeline
leadership team needs to build agency buy-in and
coalition; Internal & external entrepreneurs recruited
to teams; Prototyping ability
• Tools applied to collect insight: Mission Model Canvas
(MMC), Business Model Canvas (BMC), Innovation
Portfolio Map, Investment Decision Support Tool (IRL,
TRL, ARL), Funnel Metrics, Workflow maps, User
journey map, HMW statements 88

Discovery – Metrics & Data: The Innovation Pipeline
Metrics & Data
Metrics
• Ratio of teams entering discovery vs those
moving on to incubation (1-1 ratio is flat,
historical norm from venture due diligence, H4D
and REF is ~4-1)
• TRL, IRL and ARL established as part of decision
to move forward into incubation
• Ratio of teams in discovery spread across Alex
O’s portfolio map vs 3 Horizons
• Quality of team transitioning to Incubation
(focused on left side of MMC)
Data
• # of hypothesis and MVP’s generated
• # of team pivots
• # and diversity of interviews conducted
• TRL, IRL and ARL at completion of Discovery
• # of external partners and internal parts of
organization engaged
89

Self Assessment
90

Strategy
Process
Team &
Resources
Operational
Support
Activities &
Methods
Metrics
& Data
stage
pipeline
the pipeline
skills
Tools &
Skills
Score Score
Score Score
Score Score
Score Score
Score Score
Score Score
Score Score
*Total
Score
DISCOVER
91

Incubation
92

Incubation –The Innovation Pipeline
• Goal for the Innovation Team: Build a
repeatable and scalable incubation process –
establish a supply of investible solutions -
matching the number desired to be deployed
Strategy
• Awareness and visible support of insertion of
new solutions into existing program.
• Funding for the teams in the incubator, active
pipeline participation (mentors, coaching, etc.),
continued support
93

Incubation – Process: The Innovation Pipeline
• Input from Discovery:
• Validated MVPs and test data and a pathway to deliver
a solution; teams with the potential to deliver;
• Early adopters/1st customer ID’d
• Output to Transition – Investible entity with validated
IRL, TRL and ARL
• Innovation Team expertise needed: Domain expert
coaches and mentors, Pipeline leadership to build
agency buy-in and coalition. Understand the politics of
transition
• Resources Required:
• Recruit developers to teams in pipeline,
physical space, contracting, financing, legal and
security liaisons.
• $’s for travel and funding for teams.
Organizational OK for innovators to spend full-
time on problems/tech in the pipeline.
• Other orgs bought into 1) fast track process 2)
adoption of output
94

Incubation - Operational Support: The Innovation Pipeline
Contracts, HR, Security, engineering
and R&D liaisons with authority for
discovery and MVPs.
Understanding how to “get to Yes”
from those groups. Coaches and
mentors
Operational Support
• Leadership support required:
• Written agreement for how incubation moves to
transition from contracting, security, legal, engineering,
etc.;
• Funding for teams; Written exceptions for security,
contracting, policy, legal for innovation sourcing;
• Firefighting board of organizational leaders capable of
reducing barriers for innovation teams;
• Prioritization for testing and evaluation
95

Incubation - Activities, Methodologies, Skills & Tools:
• Methodologies to perform incubation: MVPs,
MMC, active coaching and mentoring,
transition plans, scale and deployment plans
• Skills/experiences to perform incubation:
innovation teams need innovator +
entrepreneurs + developers. Pipeline leadership
team continue to build agency buy-in and
coalition. Recruit developers to teams in
pipeline
• Skills for execution: Investment due diligence,
gov contracts/grants management; prototyping,
Agile Development, Scrum, IP assignment,
partnership building, Business Development
• Tools applied to collect insight: MMC, IRL, TRL,
ARL, Funnel metrics
96

Incubation - Activities, Methodologies, Skills & Tools:
Metrics & Data
Metrics
• Ratio of teams in incubation vs those gaining
follow-on investment (for H1 teams maybe 2-1,
H2 teams 4-1, H3 teams 10-1)
• Change in TRL, IRL and ARL of teams throughout
Incubation and at departure to transition
• Condensation: # of projects in discovery
combined into one effort moving forward to
incubation
Data
• # and ratio of MVP’s moving to prototype
development
• Size / scale of initial and subsequent user pools
gained during incubation
• $ committed to team during incubation
• # and amount of follow-on investment, grants
and gov contracts received
• # of other organizations engaged
97

Self Assessment
98

Strategy
Process
Team &
Resources
Operational
Support
Activities &
Methods
Metrics
& Data
stage
pipeline
the pipeline
skills
Tools &
Skills
Score Score
Score Score
Score Score
Score Score
Score Score
Score Score
Score Score
*Total
Score
INCUBATE
99

Transition
100

Transition – Strategy: The Innovation Pipeline
• Goal for the Innovation Team: Establish a
pathway for off ramping a team from the
innovation team to a transition team or
program; manage organizational technology
refactoring
Strategy
• Support of insertion of new solutions into existing
program – specifically by providing Funding and
headcount for new solution insertion
• Recognize and manage organizational technology
refactoring
• Establish guidance for transition of solutions that
fit in the white space between programs (who is
responsible for accepting?)
101

Transition – Process: The Innovation Pipeline
• Input from Incubation: Investible entity with
validated IRL, TRL and ARL
• Output to the agency: A scalable solution that is
delivered in time to have an impact
• Innovation Team expertise needed: At this step
oversight responsibility should move from the the
innovation team to a transition team capable of
refactoring the solution to deliver at scale
• Resources Required:
• Funds for testing, advanced prototyping and
access to facilities;
• Funds for training, sustainment for 1 year to
support the transition of a prototype into a
program of record (POR);
• Augment the innovation team with members
of the organizational transition team
102

Transition - Operational Support: The Innovation Pipeline
• Contracts, HR, Security, engineering and
R&D liaisons with authority for
prototype development, testing &
evaluations.
• Access to users; Understanding how to
“get to Yes” from those groups. Coaches
and mentors
Operational Support
• Leadership support required:
• Written agreement for how incubation moves to
transition from contracting, security, legal, engineering,
etc.;
• Funding for teams; Written exceptions for security,
contracting, policy, legal for innovation sourcing;
• Firefighting board of organizational leaders capable of
reducing barriers for scaling teams;
• Prioritization for testing, evaluation and forward
deployment 103

Transition - Activities, Methodologies, Skills & Tools:
• Methodologies to perform transition:
Refactoring, Organizational Change
Management, Sustainment packaging, Training
Development, Kaizen for continuous
improvement (Toyota’s engineering way)
• Skills/experiences to perform transition: Agile,
Scrum, Deployment planning, Training
development, integration, supply chain
management
• Skills for execution: Transition management,
budgeting, experimentation design, supply chain
development, POR management, POM planning
• Tools applied to collect insight: User surveys,
adoptions surveys, localization tools (solution
performance at different units, bases etc)
104

Transition - Activities, Methodologies, Skills & Tools:
Metrics & Data
Metrics
• Ratio of teams receiving initial investment
leaving incubation to those receiving follow-on
investment (different ratio for H1, H2 and H3)
• Ratio of teams in transition to those accepted in
POR or theater sustainment (different ratio for
H1, H2, H3)
• Average time spent in transition (shorter for H1,
longer for H3)
Data
• Realized beneficiary value (impact on original
problem)
• IRL, TRL and ARL at transition to POR
• Scaled implementation data (user drop off / pain
points / patches or modification required)
105

Self Assessment
106

Strategy
Process
Team &
Resources
Operational
Support
Activities &
Methods
Metrics
& Data
stage
pipeline
the pipeline
skills
Tools &
Skills
Score Score
Score Score
Score Score
Score Score
Score Score
Score Score
Score Score
*Total
Score
TRANSITION
107

Final Thoughts: Why have an innovation pipeline?
The result of a pipeline are solutions that:
1. Continuously replenish lost capabilities
2. Create new capabilities that match current and future threats
The purpose of innovation pipeline doctrine is to create the
organization / process / budget and people to do so
108

The Innovation Pipeline: Decision Support Tool
109

The Innovation Pipeline: Self Assessment View

181
GOV/COM
entities
engaged
1800
People
Trained
1391
Problems
Sourced
540
Problems
Curated
122
Teams in
Discovery
9
Solutions
Delivered
28
Solutions in
Incubation
1391
Problems
38% 23% 23% 32%
% moving
forward
% of original
total.6%2%9%
US Army REF #’s
2011-2013
840
Problems
360
Projects
initiated
(42%)
115
Solutions
Incubated in theater
(31%)
20
Program of
Record
(17%)
H4Di & BMNT
2018
EOY Snapshot

112

Pete Newell & Steve Blank on Diagnostics and Metrics for Innovation

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Pete Newell & Steve Blank on Diagnostics and Metrics for Innovation

Ähnlich wie Pete Newell & Steve Blank on Diagnostics and Metrics for Innovation (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Pete Newell & Steve Blank on Diagnostics and Metrics for Innovation