Building specialized container-based systems with Moby: a few use cases This talk will explain how you can leverage the Moby project to assemble your own specialized container-based system, whether for IoT, cloud or bare metal scenarios. We will cover Moby itself, the framework, and tooling around the project, as well as many of it’s components: LinuxKit, InfraKit, containerd, SwarmKit, Notary. Then we will present a few use cases and demos of how different companies have leveraged Moby and some of the Moby components to create their own container-based systems.

1. Patrick Chanezon, @chanezon
Build your own container-based system with the
September 2017

2. French
Polyglot
Platforms
Software Plumber
San Francisco
Developer Relations
@chanezon

3. Docker

4. The world needs
tools of mass innovation

5. A programmable Internet would be the ultimate
tool of mass innovation

6. Docker is building a stack to program the Internet
CE
EEA commercial product,
built on
a development platform,
built on
infrastructure,
built on
standards.

7. enterprise edition
Ubuntu
Fedora
Mac
Azure
CentOS
Windows 10
AWS
Debian
community edition
Ubuntu
Windows Server
Azure
CentOS
Suse
Red Hat
AWS
Oracle Linux

8. Image Registry
CI/CD
Security scan
& sign
Traditional
Third Party
Microservices
docker store
DEVELOPERS IT OPERATIONS
Control Plane
Docker EE Container Platform to Modernize Traditional Apps
and beyond
More Info: Docker.com/MTA

9. Orchestration
Container Runtime
OS
Infrastructure Management
Container Platform Layers
Application Services

10. Docker is a platform made of components
Raft Store
Node
Identity
Secrets
Routing
Mesh
Overlay
Networking
Swarm Orchestration
Engine
Application Services

11. containerd
An open and reliable container runtime

12. A Brief History
APRIL 2016 Containerd “0.2” announced, Docker 1.11
DECEMBER 2016Announce expansion of containerd OSS project
Management/Supervisor for the OCI runc executor
Containerd 1.0: A core container runtime project for the industry
MARCH 2017 Containerd project contributed to CNCF

13. runc
containerd
Why Containerd 1.0?
▪ Continue projects spun out
from monolithic Docker engine
▪ Expected use beyond Docker
engine (Kubernetes CRI)
▪ Donation to foundation for
broad industry collaboration
▫ Similar to runc/libcontainer
and the OCI

14. Technical Goals/Intentions
▪ Clean gRPC-based API + client library
▪ Full OCI support (runtime and image spec)
▪ Stability and performance with tight, well-
defined core of container function
▪ Decoupled systems (image, filesystem,
runtime) for pluggability, reuse

15. Requirements
- A la carte: use only what is required
- Runtime agility: fits into different platforms
- Pass-through container configuration (direct OCI)
- Decoupled
- Use known-good technology
- OCI container runtime and images
- gRPC for API
- Prometheus for Metrics

16. Use cases
- CURRENT
- Docker (moby)
- Kubernetes (cri-
containerd)
- SwarmKit (experimental)
- LinuxKit
- BuildKit
- FUTURE/POTENTIAL
- IBM Cloud/Bluemix
- OpenFaaS
- {your project here}

17. LinuxKit
A toolkit for building secure, portable and lean
operating systems for containers

18. What is LinuxKit?
A toolkit for building secure, portable and lean operating systems for
containers.
● uses Moby tooling to build system images
● everything is a container
● runs with Containerd 1.0 branch for over four months
● lightweight, fully customizable

19. Some metrics
● 75 contributors!
● first new maintainer appointed from the community
● 50 commits a week since DockerCon

20. Arm64 support
Thanks to Dennis Chen at ARM
● multi arch base images so system containers can be built
● signed multiarch manifests - thanks to IBM for all their work
● thanks to Packet.net for providing ARM64 machines
● ongoing work on EFI boot that works cross platform
● other architectures now easy to add

21. Linux Containers on Windows
● as announced at DockerCon
● LinuxKit provides build images in blueprints/lcow.yml
● ultra minimal system only 13MB
● blog post soon with HOWTO instructions
● ongoing work with Microsoft on shipping this

22. Platform support
The community added support for so many platforms...
● Azure
● OpenStack
● VMware and vCenter
● Packet.net
● Vultr
● IBM Bluemix
... and improved AWS, GCP, Hyperkit, KVM, Hyper-V...

23. 23

24. Lots of smaller improvements
● TPM support
● containers to run on clean shutdown
● fully immutable images, eg CD-ROM images
● 4.10, 4.11, 4.12 kernels, 4.13 coming soon
● namespace sharing for system containers
● rewrote a lot of shell scripts in Go for better maintainability
● OCI runtime spec 1.0
● static PIE binaries everywhere
● many more tests

25. WireGuard graduated from projects
● fast secure modern VPN tunnel based on Noise framework
● added to the LinuxKit kernels
● now easy to construct network tunnels between system containers
● prototype next stage of container networking

26. Kubernetes about to graduate from projects
● initial port contributed by Weave for DockerCon launch
● maintained since then
● also working on CRI-Containerd support, with shared system containerd
● more work ongoing
● full testing and validation planned

27. LinuxKit Security SIG
• okernel, protecting kernel integrity
https://github.com/linuxkit/linuxkit/tree/master/projects/okernel
• Kernel Self Protection Project
• Alpine Linux
• WireGuard fast, modern, secure vpn tunnel
• Opportunistic Privilege Separation (OPS)
• Landlock LSM (C -> eBPF)
• MirageSDK, Type Safe daemons
https://github.com/linuxkit/linuxkit/tree/master/reports/sig-security

28. LinuxKit Use Cases
● Linux Containers on Windows - announced at DockerCon, in the works
● Docker for Mac: shipping in edge release soon
● Kubernetes with shared system containerd
● Secure appliances
● Network function virtualization

29. https://github.com/linuxkit/linuxkit
Get Started with LinuxKit

30. Moby
An open framework to assemble specialized
container systems without reinventing the wheel.

31. Scaling the Docker production model: share components AND
ASSEMBLIES.

32. It’s time to take our ecosystem to the next level…
By collaborating on components AND COMMON ASSEMBLIES.

34. – Library of 80+ components
– Package your own
components as containers
– Reference assemblies
deployed on millions of nodes
– Create your own assemblies
or start from an existing one
A framework to assemble
specialized container
systems without
reinventing the wheel.

35. Docker uses Moby for its
open-source
– Thousands of contributors,
hundreds of patches/week
– Component development
– Specialized assembly
development
– Integration tests
– Architecture design
– Integration with other projects
– Experimentation and bleeding
edge features

36. Docker uses Moby for its
open-source...
and so can you!
– Community-run
– Open governance inspired by
the Fedora project
– Plays well with existing
projects - no donation
necessary!

37. Moby and Docker
moby-core

38. What it means for you
Moby helps you
innovate without tying
you to Docker
System BuildersDocker Users
Docker will better leverage
the ecosystem to innovate
faster for you

39. Moby transforms multi-month R&D
projects into weekend projects.

40. “RedisOS”
Weekend
project #4:

41. "RedisOS"
for Windows
"RedisOS"
for Mac
"RedisOS"
for bare metal
HyperKit
bare metal

42. SSHD
Kubernetes on
the Mac
Weekend
project #6:
HyperKit

43. http://play-with-moby.com/

44. Getting Started
- Blog https://mobyproject.org/blog
- http://play-with-moby.org
- Twitter @moby
- Github moby/moby

45. Let’s take containers mainstream!

47. Notary & TUF
A Framework for trusted content distribution.

48. What is Notary?
- Framework for trusted content
distribution.
- Golang implementation of The
Update Framework (TUF)
- Created by a group of NYU
researchers.
- Based on the TOR updater Thandy

49. Proposal to contribute to CNCF June 20
- Still waiting for vote
- Proposal and discussion at https://github.com/cncf/toc/pull/38

50. TUF core concepts
- Compromise-resilient software distribution
- Principled, graceful degradation of security
- Focus on key revocation / partial compromise of infrastructure
- Applies security best practices: separation of privilege (roles), threshold signatures,
minimizing risk, selective delegation of trust, etc.
- Flexibility
- Does not prescribe exactly how to perform a task
- Works with existing deployments constraints

51. TUF in the Cloud Native Ecosystem
- Solves trusted data distribution problem.
- Specific opinionated implementations, or uses of existing tools like Notary
can solve vast majority of content trust problems.
- Abstract solution aiming for best security.
- Sets the bar for high expectation of security in ecosystem.

52. TUF Use Cases

53. TUF Community
- Open source since 2010
- 517 GitHub stars, 74 forks
- 26+ Contributors
- 5 maintainers
- 2700+ commits

54. Notary in the Cloud Native Ecosystem
image

55. Notary Architecture

56. Notary in the Cloud Native Ecosystem
- Solves the problem of image provenance
- Can be more generally applied:
- OS/VM images
- Updates/patches
- Shared filesystems
- External resources
- Every piece of deployed code from the OS to the application should be signed

57. Notary Use Cases
- Signing container images for trusted distribution.
- Docker, Quay, Huawei, Motorola, VMWare
- Signing system components/packages for system updates.
- LinuxKit
- Signing filesystem integrity checksums
- moby
- Threshold signing to require quorum for validity
- Docker Data Center, Quay
- Signing service definitions
- Docker Swarm, Kubernetes

58. Notary Community
- Open Sourced at DockerCon SF 2015
- 865 GitHub stars, 156 forks
- 45 Contributors
- 8 maintainers from 3 Companies; Docker, CoreOS, Huawei
- 2600+ commits, 34 releases

59. Notary Community

60. Alignment with CNCF
- Provides state of the art trust and provenance for content distribution.
- Uses existing CNCF projects
- GRPC
- Prometheus
- Enhances existing CNCF projects
- Can provide trusted content acquisition for containerd, kubernetes, rkt

61. InfraKit
A toolkit for building declarative, self-healing
infrastructure.

62. What is it?
62
• Launched at LinuxCon, Berlin in October, 2016.
• Toolkit for building declarative, self-managing distributed
applications
• Active management with active controllers
• scaling groups, rolling updates
• monitoring / health checks
• connecting nodes to L4 / ingress
• Declarative infrastructure
• Proposal to contribute to CNCF 6/20, too soon

63. What is InfraKit
63
• Toolkit for infrastructure automation
• Provisioning and management services for
higher-level systems
• Focus on patterns and automation:
• Convergence to declarative specification
• Scaling groups, rolling updates
• Infrastructure metadata, events
• Immutable infrastructure
Application Definition/ Development
Orchestration & Management
Runtime
Provisioning
Infrastructure (Bare Metal/Cloud)

64. InfraKit in a Cloud Native Ecosystem
64
• Immutable nodes + attached storage
• OS Images - LinuxKit integration
• Devops Deployment Tooling &
Provisioning
• Infrastructure Automation
• Compute - rolling updates, scaling
groups
• Storage
• Network
Provisioning layer + infrastructure automation services

65. InfraKit Use Cases
65
• Day-0 (install), Day-1 (configure) of container orchestrators
• Docker Swarm - Docker for GCP, AWS, Appcelerator/AMP
• Kubernetes
• Day-N automation of infrastructure - provisioning, rolling
updates and capacity scaling.
• A cloud provider for Kubernetes Cluster Autoscaler
• GPU cluster provisioning
• LinuxKit integration for building, deployment of custom OS on
bare-metal or virtualized infrastructure (video).

66. InfraKit Architecture
66
Group Controller
Metadata Exporter
Instance Plugin (T3)
Infrastructure API
Manager
Flavor Plugin (F2)
Spec Store
infrakit CLI
Leadership
Templates Playbooks
Event Publisher
Resource Controller
Application / Orchestration API
Dependency
Graph
Template
Processor
Node 1 (T1)
Node 1 (T3)
Node 1 (T1)
Node 1 (T1)
Node (T3)
Instance Plugin (T2)
Instance Plugin (T1)
Flavor Plugin (F1)
instance (T2)
instance (T2)
Volume (T2) Network (T1)
Application Definition/ Development
Orchestration & Management
Runtime
Provisioning
Infrastructure (Bare Metal/Cloud)
Metadata Plugin (M1)
Event Plugin (E1)
Event Plugin (E1)
Metadata Plugin (M1)
scale
drain
join
provision/
configure
destroy
http://169.254.169.254
● Active controllers
● Modular, plugin-based
● Defined SPI
● Customizable, contextual
CLI

67. InfraKit Deployment
67
CLI
API
Control Plane
• High availability, single leader
• Can share leader election / spec
storage with higher-level systems:
• Docker swarm mode
• etcd (k8s)
• As Docker or containerd / oci
containers
• Typically “embedded” in control plane
of higher systems as “system”
containers (e.g. LinuxKit image)

68. InfraKit Community: active and growing
• Made public at LinuxCon, Berlin in October, 2016
•1.5K Github stars, 140+ forks
•16 infrastructure providers
•4 maintainers, 4 companies (Docker, IBM, NTT, Axway)
•25 contributors total, 200+ members on slack
•460+ commits, 7 releases, ~50 commits / month
•Meetups: Moby Project Summit, April 20, 2017;
Next: June 19, 2017
68

69. InfraKit Community
69
source: https://www.openhub.net/p/infrakit

70. InfraKit - Why CNCF
• Aligned with CNCF goals
– Cloud-native: container packaged, micro-services oriented
– Dynamic, self-healing for cloud-native, distributed services
• Enhancing & complementary to CNCF projects
– Common infrastructure provisioning and automation
– Kubernetes: cluster autoscaler
– Prometheus: infrastructure monitoring & automated
remediation
70

71. Status in May 2017

72. Support more platforms
72
• Compute:
• Bare-metal: HP OneView, MAAS, RackHD
• Public cloud: AWS, GCP
• MacOS X (HyperKit); Docker containers
• Coming soon: Azure, IBM, Digital Ocean,
Packet, libvirt
• Other resource types
• AWS - vpc, subnets, gateways, etc.

73. Improve usability
73
• Templates
• Complex scripts and configuration in any format;
no more escape quotes in JSON
• Fetch templates from remote repositories
• Playbooks
• CLI - flags, prompts — config driven and
dynamic
• Share “playbooks” from remote repositories

74. Improve core system
74
• High Availability — Swarm Mode or etcd
• New Plugin types — Metadata and Events
• Metadata: cluster-wide sysfs and reflection
• Events - publish / subscribe
• Remote client access:
infrakit -H host:port to remote cluster

75. Use Cases
75
• Support container orchestration
• bootstrapping + day N management
• API for cluster autoscaling
• k8s, Docker Swarm Mode
• Bare-metal + GPU provisioning
• IoT — LinuxKit integration / custom kernel
deployment

76. Status September 2017

77. Infrakit Update - September, 2017
• Provision AWS spot instances
(672 @YujiOshima)
• Multi-Zone / Multi-Cloud / Multi-Tiered provisioning
(652, 671 @chungers, 668 @YujiOshima)
• Improved Kubernetes support
(676 @YujiOshima)
• Improved Terraform integration
(651, 663, 670 @kaufers)
• Docker Swarm Ingress controller (621 @chungers)

78. Example: build an autoscaling group
● Pick a plugin to create instances
● Add flavor plugin
● Embed config inside definition of a group.
ID: group/workers
Properties:
Instance:
Plugin: terraform
Properties:
// terraform config here
Flavor:
Plugin: kubernetes/worker
Properties:
// config add-on, etc.
terraform
kubernetes configs
Group RPC API
infrastructure API
Client

79. … across zones / clouds
● Wrap instance plugins with Selector
● Selector selects plugin to provision,
based on weights or spread evenly.
ID: group/workers
Properties:
Instance:
Plugin: selector/weighted
Properties:
aws-us-east/workers:
gcp-us-central/workers:
Options: - aws-us-east:80
- gcp-us-central:20
Flavor:
Plugin: kubernetes/worker
Properties:
// config add-on, etc.
aws-us-east
kubernetes configs
Group RPC API
Client
gcpaws
gcp-us-central
80% 20%

80. … with provisioning priorities
● Tiered selector is just another Instance
● Selects one option after another until
provisioning succeeds.
ID: group/workers
Properties:
Instance:
Plugin: selector/tiered
Properties:
Plugin: vsphere/on-prem-workers:
Properties: // ...
Plugin: aws/ec2-spot-instance:
Properties: // spot price...
Plugin: aws/ec2-instance:
Properties: // on-demand…
Flavor:
Plugin: kubernetes/worker ...
on-prem: vsphere
kubernetes configs
Group RPC API
Client
cloud: AWS spot
cloud: AWS on-demand

81. Get involved
https://github.com/docker/infrakit
dockercommunity.slack.com:
#infrakit

82. Learn More at OSS Summit
- Wednesday, September 13 • 4:00pm - 4:40pm
Unikernels: Where Are They Now? - Amir Chaudhry,
Docker
- Thursday, September 14 • 9:00am - 12:10pm
Tutorial: Docker Container Orchestration: Building
Clusters in Production - Bret Fisher, DevOps Sysadmin
and Docker Captain & Laura Frank, Codeship

83. Moby Summit at OSS NA
Thursday, September 14, 2017
“An open framework to assemble specialized
container systems without reinventing the
wheel.”
Tickets:
https://www.eventbrite.com/e/moby-summit-los-angeles-tickets-35930560273

84. Bella Center, Copenhagen
16-19 October, 2017
https://europe-2017.dockercon.com/
10% discount code: CaptainPhil

85. THANK YOU