1. © MIRANTIS 2013 PAGE 1© MIRANTIS 2013
Module 1:
OpenStack
Architecture
Overview
Mirantis, 2013

2. © MIRANTIS 2013 PAGE 2
Goals

3. © MIRANTIS 2013 PAGE 3
Goals
• Understand OpenStack purpose and use cases
Understand the OpenStack ecosystem
• Definition
• History
• Projects

4. © MIRANTIS 2013 PAGE 4
Goals
• Understand OpenStack purpose and use cases
Understand the OpenStack ecosystem
• Definition
• History
• Projects
BACKGROUND

5. © MIRANTIS 2013 PAGE 5
Goals
• Understand OpenStack purpose and use cases
Understand the OpenStack ecosystem
• Definition
• History
• Projects
• Understand OpenStack architecture
• Logical architecture
• Provision VM request flow
• Components details
BACKGROUND

6. © MIRANTIS 2013 PAGE 6
Goals
• Understand OpenStack purpose and use cases
Understand the OpenStack ecosystem
• Definition
• History
• Projects
• Understand OpenStack architecture
• Logical architecture
• Provision VM request flow
• Components details
BACKGROUND
THEORY

7. © MIRANTIS 2013 PAGE 7
What is OpenStack?
As described by the OpenStack Foundation:
"Open source software for building
private and public clouds“

8. © MIRANTIS 2013 PAGE 8
OpenStack as IaaS

9. © MIRANTIS 2013 PAGE 9
OpenStack Capabilities

10. © MIRANTIS 2013 PAGE 10
OpenStack Capabilities
• VMs on demand
• provisioning
• snapshotting

11. © MIRANTIS 2013 PAGE 11
OpenStack Capabilities
• VMs on demand
• provisioning
• snapshotting
• Volumes

12. © MIRANTIS 2013 PAGE 12
OpenStack Capabilities
• VMs on demand
• provisioning
• snapshotting
• Volumes
• Networks

13. © MIRANTIS 2013 PAGE 13
OpenStack Capabilities
• VMs on demand
• provisioning
• snapshotting
• Volumes
• Networks
• Object storage for VM images and arbitrary files

14. © MIRANTIS 2013 PAGE 14
OpenStack Capabilities
• VMs on demand
• provisioning
• snapshotting
• Volumes
• Networks
• Object storage for VM images and arbitrary files
• Multi-tenancy
• quotas for different tenants
• user can be associated with multiple tenants

15. © MIRANTIS 2013 PAGE 15
OpenStack History
Date Rel Projects Type Note
Jul 2010 N/A PoC
* Rackspace Hosting & NASA
joint launch
Oct 2010 Austin Nova, Swift PoC
Feb 2011 Bexar Nova, Glance, Swift PoC
Apr 2011 Cactus Nova, Glance, Swift PoC
** 6 month development
cycle starts
Sep 2011 Diablo Nova, Glance, Swift Prod
1st
production release
(Cactus) at Internap (10/27)
Apr 2012 Essex
Nova, Glance, Swift, Horizon,
Keystone
Prod
Common web UI and shared
authentication mechanism
added
Sep 2012 Folsom
Nova, Glance, Swift, Horizon,
Keystone, Quantum, Cinder, Oslo
Prod
OpenStack Foundation
Established
Apr 2013 Grizzly
Nova, Glance, Swift, Horizon,
Keystone, Quantum, Cinder, Oslo
Prod
Ceilometer and Heat
integration projects added
Oct 2013 Havana
Nova, Glance, Swift, Horizon,
Keystone, Quantum, Cinder, Oslo,
Heat, Ceilometer
Prod Coming Soon!
** Follows similar Ubuntu 6 month release cycle
* Pre-July 2010 is predicated by Rackspace Cloud Files project (Swift), NASA Nebula project (Nova)

16. © MIRANTIS 2013 PAGE 16
OpenStack Grizzly Projects
• Core Projects:
• Nova (Compute Service)
• Glance (Image Service)
• Quantum (Network Service)
• Cinder (Block Storage Service)
• Swift (Object Store Service)
• Common Projects:
• Keystone (Identity Service)
• Horizon (Dashboard)
• Library Projects:
• Oslo (Shared Infrastructure Code)
• Incubated Projects (Coming in Havana)
• Ceilometer (Metering/Monitoring)
• Heat (Orchestration)

17. © MIRANTIS 2013 PAGE 17
Each OpenStack Project

18. © MIRANTIS 2013 PAGE 18
Each OpenStack Project
• Is also a “top-level” OpenStack component

19. © MIRANTIS 2013 PAGE 19
Each OpenStack Project
• Is also a “top-level” OpenStack component
• Has an elected “Project Technical Lead” (PTL)

20. © MIRANTIS 2013 PAGE 20
Each OpenStack Project
• Is also a “top-level” OpenStack component
• Has an elected “Project Technical Lead” (PTL)
• Has separate developers and design teams

21. © MIRANTIS 2013 PAGE 21
Each OpenStack Project
• Is also a “top-level” OpenStack component
• Has an elected “Project Technical Lead” (PTL)
• Has separate developers and design teams
• Has a well defined public API
• With the exception of Horizon, which is the Web GUI, all
other projects have a RESTfull (JSON/HTTP) API

22. © MIRANTIS 2013 PAGE 22
Each OpenStack Project
• Is also a “top-level” OpenStack component
• Has an elected “Project Technical Lead” (PTL)
• Has separate developers and design teams
• Has a well defined public API
• With the exception of Horizon, which is the Web GUI, all
other projects have a RESTfull (JSON/HTTP) API
• Has a separate database and isolated
persistent layer

23. © MIRANTIS 2013 PAGE 23
OpenStack Projects: Begining (Cactus)

24. © MIRANTIS 2013 PAGE 24
OpenStack Projects: Now (Grizzly)

25. © MIRANTIS 2013 PAGE 25
Dev Trends
• Decoupling of features (e.g. Nova-volume
became Cinder)
• API-s to communicate
• Common generic API/Infrastructure (Oslo)
• Backends & drivers (everything’s pluggable)

26. © MIRANTIS 2013 PAGE 26
OpenStack Projects Relationships

27. © MIRANTIS 2013 PAGE 27
OpenStack Projects Relationships

28. © MIRANTIS 2013 PAGE 28
OpenStack Projects Relationships

29. © MIRANTIS 2013 PAGE 29
OpenStack Projects Relationships
MySQL
Database
Not a project, but
important to
understand the
relationship

30. © MIRANTIS 2013 PAGE 30
OpenStack Projects Relationships
MySQL
Database
Not a project, but
important to
understand the
relationship
RabbitMQ Not a project, but
important to
understand the
relationship

31. © MIRANTIS 2013 PAGE 31
Deployment – Pick up What
You Want
• The components can be mixed & matched
• Base:
• Nova
• Keystone
• Dashboard
• Glance
• Mutually exclusive scenarios
• Some components can conflict

32. © MIRANTIS 2013 PAGE 32
OpenStack: “Typical” Deployment Topology
Control Cluster
Horizon Keystone
GlanceQuantum
Cinder
Cloud Ctrl.
(nova)
Compute Cluster
compute
node
compute
node
compute
node
compute
node
compute
node
compute
node
Storage Cluster (Swift)
storage
node
storage
node
storage
node
storage
node
storage
node
storage
node

33. © MIRANTIS 2013 PAGE 33
OpenStack: “Typical” Deployment Topology
Control Cluster
Horizon Keystone
GlanceQuantum
Cinder
Cloud Ctrl.
(nova)
Compute Cluster
compute
node
compute
node
compute
node
compute
node
compute
node
compute
node
Storage Cluster (Swift)
storage
node
storage
node
storage
node
storage
node
storage
node
storage
node
Heavy CPU and
RAM

34. © MIRANTIS 2013 PAGE 34
OpenStack: “Typical” Deployment Topology
Control Cluster
Horizon Keystone
GlanceQuantum
Cinder
Cloud Ctrl.
(nova)
Compute Cluster
compute
node
compute
node
compute
node
compute
node
compute
node
compute
node
Storage Cluster (Swift)
storage
node
storage
node
storage
node
storage
node
storage
node
storage
node
Heavy CPU and
RAM Heavy Disk and I/O

35. © MIRANTIS 2013 PAGE 35
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
OpenStack Projects:
Communication Types
HTTP
AMQP
SQL
Native API
iSCSI
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
UI: Horizon or CLI

36. © MIRANTIS 2013 PAGE 36
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
OpenStack Projects:
Communication Types
HTTP
AMQP
SQL
Native API
iSCSI
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Every OpenStack service exposes access
to restful API via HTTP
UI: Horizon or CLI

37. © MIRANTIS 2013 PAGE 37
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
OpenStack Projects:
Communication Types
HTTP
AMQP
SQL
Native API
iSCSI
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Every OpenStack service exposes access
to restful API via HTTP
UI: Horizon or CLI
Each action treated as distributed
transaction, state built as MQ messages

38. © MIRANTIS 2013 PAGE 38
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
OpenStack Projects:
Communication Types
HTTP
AMQP
SQL
Native API
iSCSI
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Every OpenStack service exposes access
to restful API via HTTP
Each service updates it’s own DB with
state information as actions are performed
UI: Horizon or CLI
Each action treated as distributed
transaction, state built as MQ messages

39. © MIRANTIS 2013 PAGE 39
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
OpenStack Projects:
Communication Types
HTTP
AMQP
SQL
Native API
iSCSI
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Every OpenStack service exposes access
to restful API via HTTP
Each service updates it’s own DB with
state information as actions are performed
UI: Horizon or CLI
Each action treated as distributed
transaction, state built as MQ messages
Direct
access calls,
ex. Plugins,
NetApp,
Nicira, etc.

40. © MIRANTIS 2013 PAGE 40
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
OpenStack Projects:
Communication Types
HTTP
AMQP
SQL
Native API
iSCSI
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Cinder Block
storage
provided as
iSCSI storage
to VMs
Every OpenStack service exposes access
to restful API via HTTP
Each service updates it’s own DB with
state information as actions are performed
UI: Horizon or CLI
Each action treated as distributed
transaction, state built as MQ messages
Direct
access calls,
ex. Plugins,
NetApp,
Nicira, etc.

41. © MIRANTIS 2013 PAGE 41
OpenStack REST API
• OpenStack public API is a RESTful API

42. © MIRANTIS 2013 PAGE 42
OpenStack REST API
• OpenStack public API is a RESTful API
• REST stands for Representational State Transfer

43. © MIRANTIS 2013 PAGE 43
OpenStack REST API
• OpenStack public API is a RESTful API
• REST stands for Representational State Transfer
• REST is a stateless client/server protocol with a uniform
interface for accessing the object model

44. © MIRANTIS 2013 PAGE 44
OpenStack REST API
• OpenStack public API is a RESTful API
• REST stands for Representational State Transfer
• REST is a stateless client/server protocol with a uniform
interface for accessing the object model
• OpenStack RESTful API is implemented using HTTP
GET/PUT/POST/DELETE in combination with JSON for
data

45. © MIRANTIS 2013 PAGE 45
Part 1 Recap

46. © MIRANTIS 2013 PAGE 46
Part 1 Recap
• OpenStack – open source software for building
IaaS

47. © MIRANTIS 2013 PAGE 47
Part 1 Recap
• OpenStack – open source software for building
IaaS
• OpenStack release cycle is every 6 month

48. © MIRANTIS 2013 PAGE 48
Part 1 Recap
• OpenStack – open source software for building
IaaS
• OpenStack release cycle is every 6 month
• OpenStack is an umbrella over multiple
independent projects (components)

49. © MIRANTIS 2013 PAGE 49
Part 1 Recap
• OpenStack – open source software for building
IaaS
• OpenStack release cycle is every 6 month
• OpenStack is an umbrella over multiple
independent projects (components)
• All OpenStack Components talk RESTful API

50. © MIRANTIS 2013 PAGE 50
Part 1 Recap
• OpenStack – open source software for building
IaaS
• OpenStack release cycle is every 6 month
• OpenStack is an umbrella over multiple
independent projects (components)
• All OpenStack Components talk RESTful API
• Most OpenStack Components have dedicated DB
(SQL) and MQ (QP), some talk to 3rd party
components using their native APIs

51. © MIRANTIS 2013 PAGE 51
Use case: Provision VM
• Most common and complex process
• Interacts with all OpenStack components

52. © MIRANTIS 2013 PAGE 52
Initial State
Assumes Tenant is created,
provisioning quota is available, user
has an access to Horizon/CLICloud Operator, DevOp, etc.
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
UI: Horizon or CLI
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell

53. © MIRANTIS 2013 PAGE 53
Step 1: Request Provisioning
– From UI

54. © MIRANTIS 2013 PAGE 54
Step 1: Request Provisioning
– From UI
• Login to Horizon

55. © MIRANTIS 2013 PAGE 55
Step 1: Request Provisioning
– From UI
• Login to Horizon
• Specify params of VM
• VM Name
• Image (OS type)
• Flavor (specifies CPU, Memory, Disk)
• Network (required for Folsom or later)
• Optional (SSH Keys, Persistent volumes, comments, etc.)

56. © MIRANTIS 2013 PAGE 56
Step 1: Request Provisioning
– From UI
• Login to Horizon
• Specify params of VM
• VM Name
• Image (OS type)
• Flavor (specifies CPU, Memory, Disk)
• Network (required for Folsom or later)
• Optional (SSH Keys, Persistent volumes, comments, etc.)
• Hit "Create" button

57. © MIRANTIS 2013 PAGE 57
What is Horizon
"The OpenStack Dashboard
(Horizon) provides a
baseline user interface
for managing OpenStack services.“

58. © MIRANTIS 2013 PAGE 58
Horizon Notes

59. © MIRANTIS 2013 PAGE 59
Horizon Notes
• "Stateless“, no DB

60. © MIRANTIS 2013 PAGE 60
Horizon Notes
• "Stateless“, no DB
• Error handling is delegated to back-end

61. © MIRANTIS 2013 PAGE 61
Horizon Notes
• "Stateless“, no DB
• Error handling is delegated to back-end
• Doesn't support all API functions

62. © MIRANTIS 2013 PAGE 62
Horizon Notes
• "Stateless“, no DB
• Error handling is delegated to back-end
• Doesn't support all API functions
• Can use memcached or database to store
sessions

63. © MIRANTIS 2013 PAGE 63
Horizon Notes
• "Stateless“, no DB
• Error handling is delegated to back-end
• Doesn't support all API functions
• Can use memcached or database to store
sessions
• Gets updated via Nova API polling

64. © MIRANTIS 2013 PAGE 64
Horizon Internals
• Subprojects
• Horizon – generic Python Django libraries and
components to work with REST-based back-end / restful
web service
• Openstack Dashboard - web app itself, exposes UI for
OpenStack with styles, locale, etc.
• Dashboard for each component = Individual
nested Django app
• Easily modifiable
• Modularly developed

65. © MIRANTIS 2013 PAGE 65
Step 1: Request VM
Provisioning via UI/CLI
Cloud Operator, DevOp, etc.
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
UI: Horizon or CLI
User logs in to UI
Specifies VM params: name,
flavor, keys, etc. and hits
"Create" button
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell

66. © MIRANTIS 2013 PAGE 66
Step 1: Request Provisioning
– Under the Hood

67. © MIRANTIS 2013 PAGE 67
Step 1: Request Provisioning
– Under the Hood
• Form params are converted to POST data

68. © MIRANTIS 2013 PAGE 68
Step 1: Request Provisioning
– Under the Hood
• Form params are converted to POST data
• "Create" request initiate HTTP POST request to
back-end
• To Keystone if auth token is not cached – step 2

69. © MIRANTIS 2013 PAGE 69
What is Keystone?
"Keystone provides
Identity, Token, Catalog and
Policy services
for use specifically by projects in the
OpenStack family.“
Keystone was developed by the OpenStack community but is written as a “generic” authentication /
authorization mechanism for any 2 or more restful API services to communicate

70. © MIRANTIS 2013 PAGE 70
Keystone Architecture
Deploys with it’s own DB but can
also be integrated with LDAP or
other EAS
Contains user,
role, and
tenant data
Contains
temporary
tokens
Rule management
interface and rule-
based
authorization
Contains
endpoint
registry

71. © MIRANTIS 2013 PAGE 71
Keystone Data Model
• User: has account credentials, is associated with one or more
tenants
• Tenant: unit of ownership in OpenStack, contains one or more
users
• Role: a first-class piece of metadata associated with many
user-tenant pairs
• Token: identifying credential associated with a user or user
and tenant
• Extras: bucket of key-value metadata associated with a user-
tenant pair
• Rule: describes a set of requirements for performing an action

72. © MIRANTIS 2013 PAGE 72
Keystone Key Concept
• What service exposes
• http://myservice/instances/* - GET/POST/PUT
• http://myservice/images/* - GET
• How RBAC mapping works
• Role X in Tenant Y can do actions A,B,C
• User: GET/POST/PUT on instances, images
• Admin: GET/POST/PUT on tenants, users, quotas
• Each API Service has it’s own RBAC
enforcement through policy files

73. © MIRANTIS 2013 PAGE 73
Step 2: Validate Auth Data
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
UI: Horizon or CLI
Horizon sends HTTP request to
Keystone. Auth info is specified
in HTTP headers.
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell

74. © MIRANTIS 2013 PAGE 74
Step 2: Validate Auth Data
• Horizon sends HTTP request to Keystone
• Keystone parses HTTP header info and verifies
that
• The credentials are valid (Authentication)
• User-Tenant-Role mapping is valid (Access Control)
• The requested action is available for this user
(Authorization)

75. © MIRANTIS 2013 PAGE 75
Step 2: Validate Auth Data - Success
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
UI: Horizon or CLI
Keystone sends temporary token
back to Horizon via HTTP.
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell

76. © MIRANTIS 2013 PAGE 76
Step 1: Request Provisioning
– Under the Hood
• Form params are converted to POST data
• "Create" request initiate HTTP POST request to
back-end
• To Keystone if auth token is not cached – step 2

77. © MIRANTIS 2013 PAGE 77
Step 1: Request Provisioning
– Under the Hood
• Form params are converted to POST data
• "Create" request initiate HTTP POST request to
back-end
• To Keystone if auth token is not cached – step 2
• To Nova API if auth token hasn't expired yet – step 3

78. © MIRANTIS 2013 PAGE 78
Nova API
“Nova API is a
RESTful API web service
which is used to interact with Nova"

79. © MIRANTIS 2013 PAGE 79
Nova API Characteristics

80. © MIRANTIS 2013 PAGE 80
Nova API Characteristics
• Exposes REST API via HTTP

81. © MIRANTIS 2013 PAGE 81
Nova API Characteristics
• Exposes REST API via HTTP
• Provides system for managing multiple APIs on
different sub-domains
• EC2-compatible – Starting to be deprecated
• Compute API – all innovation happens here

82. © MIRANTIS 2013 PAGE 82
Nova API Characteristics
• Exposes REST API via HTTP
• Provides system for managing multiple APIs on
different sub-domains
• EC2-compatible – Starting to be deprecated
• Compute API – all innovation happens here
• The only "allowed" way to interact with Nova

83. © MIRANTIS 2013 PAGE 83
Nova API Characteristics
• Exposes REST API via HTTP
• Provides system for managing multiple APIs on
different sub-domains
• EC2-compatible – Starting to be deprecated
• Compute API – all innovation happens here
• The only "allowed" way to interact with Nova
• Stateless - HA-ready

84. © MIRANTIS 2013 PAGE 84
Nova API Clients
Active effort in the community to make
one CLI to “rule them all”, currently
multiple CLIs available
OpenStack dashboard is
currently the only “unified”
OpenStack API client

85. © MIRANTIS 2013 PAGE 85
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 3: Send API Request to Nova API
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
UI: Horizon or CLI
Horizon sends POST request to
Nova API (signed with given token).

86. © MIRANTIS 2013 PAGE 86
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 4: Validate API Token
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
UI: Horizon or CLI
Nova API sends HTTP request to
validate API token to Keystone.

87. © MIRANTIS 2013 PAGE 87
Step 4:
Validate Token – Keystone API

88. © MIRANTIS 2013 PAGE 88
Keystone /w PKI - Token
Validation
• User gets one-time-password on creation
• User uses it to establish a key-pair
• Public key is signed and stored on Keystone
• From this point user uses client certificate to login
• Nova API performs offline check of the validity of token using CA&Cert it has from Keystone
Keystone
Key
generate_cms_token(
meta, keystone_key)user/pass/tenant
signed_cms_token
signed_cms_token
Nova
CA&Cert from
Keystone
verify(signed_cms_toke
n, Cafile,certfile)

89. © MIRANTIS 2013 PAGE 89
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 4: Validate API Token - Sucess
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
UI: Horizon or CLI
Keystone validates API token and
sends HTTP response with token
acceptance/rejection info.

90. © MIRANTIS 2013 PAGE 90
Step 5:
Process API Request Process

91. © MIRANTIS 2013 PAGE 91
Step 5:
Process API Request Process
• Validate request params
• Typographical errors are verified on code level
• Cloud-related params are validated via DB requests

92. © MIRANTIS 2013 PAGE 92
Step 5:
Process API Request Process
• Validate request params
• Typographical errors are verified on code level
• Cloud-related params are validated via DB requests
• If request cannot be processed then throw an
exception

93. © MIRANTIS 2013 PAGE 93
Step 5:
Process API Request Process
• Validate request params
• Typographical errors are verified on code level
• Cloud-related params are validated via DB requests
• If request cannot be processed then throw an
exception
• If request can be processed
• Save initial state to the Database

94. © MIRANTIS 2013 PAGE 94
Nova Database
“Nova Database stores current
state of all objects in compute
cluster."

95. © MIRANTIS 2013 PAGE 95
Nova Database
• In theory can be any relational database
• Most of the deployments are done with MySQL or
PostgreSQL
• Nova API talks to DB via SQLAlchemy (python
ORM (Object Related Mapper))
• DB HA should be done via external tools (like
Galera or Multi-Master replication Model for
MySQL)

96. © MIRANTIS 2013 PAGE 96
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 5: Process API Request
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
UI: Horizon or CLI
Nova API parses request to
python object model and
validates it by fetching data
from Nova DB. If request is
valid, it saves initia db entry
about VM to the database.

97. © MIRANTIS 2013 PAGE 97
Step 5:
Process API Request Process
• Validate request params
• Typographical errors are verified on code level
• Cloud-related params are validated via DB requests
• If request cannot be processed then throw an
exception
• If request can be processed
• Save initial state to the database

98. © MIRANTIS 2013 PAGE 98
Step 5:
Process API Request Process
• Validate request params
• Typographical errors are verified on code level
• Cloud-related params are validated via DB requests
• If request cannot be processed then throw an
exception
• If request can be processed
• Save initial state to the database
• Send message with next actions to MQ – step 6

99. © MIRANTIS 2013 PAGE 99
Message Queue
"Message Queue is a unified way for
collaboration between nova
components."

100. © MIRANTIS 2013 PAGE 100
Messaging Process Example
• 2 modes:
• rpc.cast - don't wait for result (fire and forget)
• rpc.call - wait for result (when there is something to
return)

101. © MIRANTIS 2013 PAGE 101
Messaging Process Example
• 2 modes:
• rpc.cast - don't wait for result (fire and forget)
• rpc.call - wait for result (when there is something to
return)
Ex. Nova API Ex. Nova Scheduler

102. © MIRANTIS 2013 PAGE 102
Messaging Process Example
• 2 modes:
• rpc.cast - don't wait for result (fire and forget)
• rpc.call - wait for result (when there is something to
return)
Ex. Nova API Ex. Nova Scheduler

103. © MIRANTIS 2013 PAGE 103
Messaging Process Example
• 2 modes:
• rpc.cast - don't wait for result (fire and forget)
• rpc.call - wait for result (when there is something to
return)
Ex. Nova API Ex. Nova Scheduler

104. © MIRANTIS 2013 PAGE 104
Messaging Process Example
• 2 modes:
• rpc.cast - don't wait for result (fire and forget)
• rpc.call - wait for result (when there is something to
return)
Ex. Nova API Ex. Nova Scheduler

105. © MIRANTIS 2013 PAGE 105
Messaging Process Example
• 2 modes:
• rpc.cast - don't wait for result (fire and forget)
• rpc.call - wait for result (when there is something to
return)
Ex. Nova API Ex. Nova Scheduler

106. © MIRANTIS 2013 PAGE 106
OpenStack Messagings Notes

107. © MIRANTIS 2013 PAGE 107
OpenStack Messagings Notes
• Uses multiple queues within single RabbitMQ
instance
• Used by services to build machine state
• Each compute node has a queue for scheduling

108. © MIRANTIS 2013 PAGE 108
OpenStack Messagings Notes
• Uses multiple queues within single RabbitMQ
instance
• Used by services to build machine state
• Each compute node has a queue for scheduling
• Messages traffic is not intensive

109. © MIRANTIS 2013 PAGE 109
OpenStack Messagings Notes
• Uses multiple queues within single RabbitMQ
instance
• Used by services to build machine state
• Each compute node has a queue for scheduling
• Messages traffic is not intensive
• Doesn't send broadcast messages, e.g. for
monitoring, uses API polling instead

110. © MIRANTIS 2013 PAGE 110
OpenStack Messagings Notes
• Uses multiple queues within single RabbitMQ
instance
• Used by services to build machine state
• Each compute node has a queue for scheduling
• Messages traffic is not intensive
• Doesn't send broadcast messages, e.g. for
monitoring, uses API polling instead
• HA should be configured separately, e.g.
mirrored queues, not handled by OpenStack

111. © MIRANTIS 2013 PAGE 111
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 6a: Publish Provisioning Request
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Nova API makes rpc.call to
Scheduler. It publishes a
short message to scheduler
queue with VM info. UI: Horizon or CLI
Request has been validated, but no action has
been taken yet, i.e. which host, IP address, etc.

112. © MIRANTIS 2013 PAGE 112
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 7: Pick up Provisioning Request
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Scheduler picks up the
message from MQ.
UI: Horizon or CLI

113. © MIRANTIS 2013 PAGE 113
Nova Scheduler
“Nova Scheduler is a daemon, which
determines, on which compute host
the request should run.“
• Only provisioning time component, i.e. not like VMware’s DRS
• Typically co-located with the Cloud Controller

114. © MIRANTIS 2013 PAGE 114
VM Scheduling:
Typical Requirements
• provision VM to particular host
• provision VMs of the particular tenant to
isolated hosts
• provision all VMs on different hosts
• provision VMs to "higher density" hosts

115. © MIRANTIS 2013 PAGE 115
Nova Scheduler: Available
Schedulers
Scheduler Description Behavior
Chance Picks a host that is up Random
Simple
Picks a host that is up and
has the fewest running
instances
Least Loaded
Filter
Picks the best-suited host
which satisfies selected
filter
Custom JSON Filters
Multi (Deprecated, to be
replaced by cells)
A scheduler that holds
multiple sub-schedulers
Collection of filters
commonly used for
multi-site or customized
deployments

116. © MIRANTIS 2013 PAGE 116
Nova Scheduler: Filtering
Affinity, Anti-affinity,
etc.
Eliminate
inapplicable hosts

117. © MIRANTIS 2013 PAGE 117
Nova Scheduler: Filters
Filter Description
affinity Same host or different host
availability zone Least cost inside selected availability zone
core Least CPU core utilization
ram Only return hosts with sufficient RAM
json Allows simple JSON based grammar. Can be used to
build custom schedulers.
i/o filter out hosts with too many concurrent I/O
operations
compute capabilities match attributes with compute node's capabilities
(e.g. CPU arch.)

118. © MIRANTIS 2013 PAGE 118
Nova Scheduler: Filters
Filter Description
aggregate specs match the attributes for the instance with those
provided by aggregate
image properties find compute nodes with capabilities matching
image specification from glance
isolated host match given image with a group of compute nodes
trusted host (by Intel) finds only "attested" hosts
type find only compute nodes which do not run any
instances
… A lot more

119. © MIRANTIS 2013 PAGE 119
Nova Scheduler: Filters
• Filters are statically configured in nova.conf
• Multiple filters can be specified
• It is possible to create custom filter
• Inherit from BaseHostFilter class
• Override host_passes(self, host_state, filter_properties)

120. © MIRANTIS 2013 PAGE 120
Nova Scheduler: Filtering
RAM, CPU, etc.
Integer values

121. © MIRANTIS 2013 PAGE 121
Nova Scheduler: Weights and
Costs
• Cost - integer value
• Every compute host can have several cost
functions associated with it
• If no cost functions associated - use default
from nova.conf
• weight = sum(costi + weigth_fni)

122. © MIRANTIS 2013 PAGE 122
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 8a: Schedule Provisioning
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Scheduler fetches
information about the
whole cluster from
database, filters, and
selects compute node and
updates DB with its ID
UI: Horizon or CLI

123. © MIRANTIS 2013 PAGE 123
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 8b: Provision Scheduled
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Scheduler publishes
message to the compute
queue (based on host ID) to
trigger VM provisioning UI: Horizon or CLI

124. © MIRANTIS 2013 PAGE 124
VM Provisioning Algorithm
• Step 9 – Nova Compute gets message from MQ and asks
Nova Conductor for VM info from database
• Step 10 – Nova Compute queries Quantum (previously Nova-
Network in Essex) to allocate networking information
• Step 11 – Nova Compute queries Cinder to allocate volume
information (optional step for persistent data)
• Steps 12-14 – Nova Compute fetches VM image (base OS)
from Glance
• Step 15 – Nova Compute passes all information about VM (in
a single message) to Hypervisor and Hypervisor (KVM / Xen)
creates an instance

125. © MIRANTIS 2013 PAGE 125
Nova Compute
“Nova Compute is a worker
daemon, which primarily creates
and terminates VMs via
Hypervisor API."

126. © MIRANTIS 2013 PAGE 126
Nova Compute Drivers
Nova Compute
XCP
VM
VM
VMWare
VM
VM
HyperV
VM
VM
LPAR
VM
VM
libvirt
KVM
VM
VM
Xen
VM
VM
Qemu
VM
VM
LXC
VM
VM
Today only 1 hypervisor type per
cloud instance. Libvirt / KVM is most
common deployment
Maintained
by Citrix
Maintained by
VMWare
Maintained
by Microsoft
Maintained
by IBM
Native or
through libvirt
Bare
Metal
VM
VM
Experimental
at this point

127. © MIRANTIS 2013 PAGE 127
Nova Compute Drivers
(Continued)
• Functionality is not 100% similar
• Exact "run_instance" flow depends on driver
implementation
• Most of the features are developed and tested
on KVM

128. © MIRANTIS 2013 PAGE 128
Nova Compute Config
(nova.conf on each host)
• --libvirt_type
• Hypervisor being used. In this deployment ‘kvm’ is specified.
• --libvirt_uri
• URI to use for connection to hypervisor. In this deployment ‘qemu+tcp:///system’ is specified.
• --sql_connection
• Database connection string in SQLAlchemy format. This is used for connecting to state database (if
Nova Conductor is not used)
• --rabbit_host
• IP address for RabbitMQ host. Non-standard port also can be specified
• --glance_host
• IP address and port of Glance Image Service host. This is needed for streaming virtual boot images.
• --glance_api_server
• IP address and port of Glance API server. This is needed for getting virtual boot images meta-data.

129. © MIRANTIS 2013 PAGE 129
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 9a: Start VM Provisioning
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Nova Compute gets
message from MQ
UI: Horizon or CLI

130. © MIRANTIS 2013 PAGE 130
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 9b: Start VM Provisioning
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Nova Compute makes
rpc.call to Nova Conductor
for Information on VM from
DB
UI: Horizon or CLI

131. © MIRANTIS 2013 PAGE 131
Nova Conductor
“The Nova Conductor service is key
to completing no-db-compute"

132. © MIRANTIS 2013 PAGE 132
Nova Conductor Notes
controller node
DB
nova-conductor
compute node
nova-compute
rpc.call()

133. © MIRANTIS 2013 PAGE 133
Nova Conductor Notes
• Eliminites remote DB access (security)
controller node
DB
nova-conductor
compute node
nova-compute
rpc.call()

134. © MIRANTIS 2013 PAGE 134
Nova Conductor Notes
• Eliminites remote DB access (security)
• Horizontal scalability (performance)
controller node
DB
nova-conductor
compute node
nova-compute
rpc.call()

135. © MIRANTIS 2013 PAGE 135
Nova Conductor Notes
• Eliminites remote DB access (security)
• Horizontal scalability (performance)
• Hides DB implementation/schema from the Nova Compute (upgrades)
controller node
DB
nova-conductor
compute node
nova-compute
rpc.call()

136. © MIRANTIS 2013 PAGE 136
Nova Conductor Notes
• Eliminites remote DB access (security)
• Horizontal scalability (performance)
• Hides DB implementation/schema from the Nova Compute (upgrades)
• Possible offloading of long-running operations from other services, not just Nova Compute
controller node
DB
nova-conductor
compute node
nova-compute
rpc.call()

137. © MIRANTIS 2013 PAGE 137
Nova Conductor Notes
• Eliminites remote DB access (security)
• Horizontal scalability (performance)
• Hides DB implementation/schema from the Nova Compute (upgrades)
• Possible offloading of long-running operations from other services, not just Nova Compute
• Beneficial for operations that cross multiple compute nodes (migration, resizes)
controller node
DB
nova-conductor
compute node
nova-compute
rpc.call()

138. © MIRANTIS 2013 PAGE 138
Nova Conductor Notes
• Eliminites remote DB access (security)
• Horizontal scalability (performance)
• Hides DB implementation/schema from the Nova Compute (upgrades)
• Possible offloading of long-running operations from other services, not just Nova Compute
• Beneficial for operations that cross multiple compute nodes (migration, resizes)
• “This is just one (major) step along the path”
controller node
DB
nova-conductor
compute node
nova-compute
rpc.call()

139. © MIRANTIS 2013 PAGE 139
VM Provisioning Algorithm
• Step 9 – Nova Compute gets message from MQ and asks
Nova Conductor for VM info from Nova Conductor
• Step 10 – Nova Compute queries Quantum (previously Nova-
Network in Essex) to allocate networking information
• Step 11 – Nova Compute queries Cinder to allocate volume
information (optional step for persistent data)
• Steps 12-14 – Nova Compute fetches VM image (base OS)
from Glance
• Step 15 – Nova Compute passes all information about VM (in
a single message) to Hypervisor and Hypervisor (KVM / Xen)
creates an instance

140. © MIRANTIS 2013 PAGE 140
Quantum
"network-as-a-service"

141. © MIRANTIS 2013 PAGE 141
Quantum Notes

142. © MIRANTIS 2013 PAGE 142
Quantum Notes
• Provides a flexible API (POST / GET) for service providers or
their tenants to manage OpenStack network topologies
• Create networks, associate VMs, set routers, etc.

143. © MIRANTIS 2013 PAGE 143
Quantum Notes
• Provides a flexible API (POST / GET) for service providers or
their tenants to manage OpenStack network topologies
• Create networks, associate VMs, set routers, etc.
• Presents a logical API and a corresponding plug-in
architecture that separates the description of network
connectivity from its implementationion

144. © MIRANTIS 2013 PAGE 144
Quantum Notes
• Provides a flexible API (POST / GET) for service providers or
their tenants to manage OpenStack network topologies
• Create networks, associate VMs, set routers, etc.
• Presents a logical API and a corresponding plug-in
architecture that separates the description of network
connectivity from its implementationion
• API evolves independently of the compute API, allowing to
introduce more advanced network capabilities (e.g. QoS,
ACLs, etc.)

145. © MIRANTIS 2013 PAGE 145
Quantum Notes
• Provides a flexible API (POST / GET) for service providers or
their tenants to manage OpenStack network topologies
• Create networks, associate VMs, set routers, etc.
• Presents a logical API and a corresponding plug-in
architecture that separates the description of network
connectivity from its implementationion
• API evolves independently of the compute API, allowing to
introduce more advanced network capabilities (e.g. QoS,
ACLs, etc.)
• In Folsom/Grizzly one can choose to stay with nova-network
(Essex approach) or to go with Quantum

146. © MIRANTIS 2013 PAGE 146
Quantum Architecture -
“Bird's Eye" View
3rd Party plug-in,
networking data is stored
outside of OpenStack /
Quantum
Quantum native
functionality

147. © MIRANTIS 2013 PAGE 147
Network Configuration Flow
• Allocate MAC addresses
• Allocate IPs (for each network)
• Associate IP and MAC with VM (DB)
• Setup network - L2:
• configure L2 via a quantum plugin
• actual action can be variable, depending on the plugin used
(with OVS plugin the action is: plugging an instance into the
integration bridge on the hypervisor)
• Setup network - L3
• Update DHCP config
• Initialize gateway
Allocation during
cloud setup
Association and
Setup during VM
provisioning

148. © MIRANTIS 2013 PAGE 148
Available Quantum Plugins
• Linux Bridge
• OpenVSwitch (most common)
• Nicira NVP
• Cisco (UCS Blade + Nexus)
• Ryu OpenFlow controller
• NEC ProgrammableFlow Controller

149. © MIRANTIS 2013 PAGE 149
Step 10: Configure Network
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Nova Compute makes a call to
Quantum API to provision
network for the instance
UI: Horizon or CLI
Keystone
Keystone Server
Keystone DB
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell

150. © MIRANTIS 2013 PAGE 150
Step 10: Configure Network (Continued)
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Quantum configures IP, gateway,
DNS name, L2 connectivity, etc.
UI: Horizon or CLI
Keystone
Keystone Server
Keystone DB
Static
Dynamic
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell

151. © MIRANTIS 2013 PAGE 151
VM Provisioning Algorithm
• Step 9 – Nova Compute gets message from MQ and asks
Nova Conductor for VM info from database
• Step 10 – Nova Compute queries Quantum (previously Nova-
Network in Essex) to allocate networking information
• Step 11 – Nova Compute queries Cinder to allocate volume
information (optional step for persistent data)
• Steps 12-14 – Nova Compute fetches VM image (base OS)
from Glance
• Step 15 – Nova Compute passes all information about VM (in
a single message) to Hypervisor and Hypervisor (KVM / Xen)
creates an instance

152. © MIRANTIS 2013 PAGE 152
Cinder
"block storage as-a-service"

153. © MIRANTIS 2013 PAGE 153
Cinder Notes

154. © MIRANTIS 2013 PAGE 154
Cinder Notes
• Optional

155. © MIRANTIS 2013 PAGE 155
Cinder Notes
• Optional
• iSCSI solution which can plug into a number of
storage backends

156. © MIRANTIS 2013 PAGE 156
Cinder Notes
• Optional
• iSCSI solution which can plug into a number of
storage backends
• Volume can be attached only to 1 instance at a
time

157. © MIRANTIS 2013 PAGE 157
Cinder Notes
• Optional
• iSCSI solution which can plug into a number of
storage backends
• Volume can be attached only to 1 instance at a
time
• Persistent volumes keep their state
independent of instances

158. © MIRANTIS 2013 PAGE 158
Cinder Architecture

159. © MIRANTIS 2013 PAGE 159
Cinder Drivers
• iSCSI
• Fibre Channel
• Xen Storage Manager
• Nexenta
• NetApp
• Zadara VPSA
• SAN
• NFS (volumes as sparse files)
• RBD Ceph
• IBM Storwize / XIV
• HP 3PAR
• Coraid
• Huawei
• Scality SOFS
• GlusterFS
• LVM thin provisioning support
• Mirrored LVM
• XenAPINFS
• EMC VNX/VMAX arrays
• Solidfire

160. © MIRANTIS 2013 PAGE 160
Step 11: Request Volume
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
UI: Horizon or CLI
Keystone
Keystone Server
Keystone DB
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
It is assumed a volume is
already created. Nova
Compute contacts Cinder to
get volume data. Can also
attach volumes after VM is
built.

161. © MIRANTIS 2013 PAGE 161
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 11: Request volume (Continued)
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Nova Compute sets up
iSCSI initiator & instructs
the Hypervisor to mount
iSCSI vol. as a new block
device UI: Horizon or CLI
Keystone
Keystone Server
Keystone DB

162. © MIRANTIS 2013 PAGE 162
VM Provisioning Algorithm
• Step 9 – Nova Compute gets message from MQ and asks
Nova Conductor for VM info from database
• Step 10 – Nova Compute queries Quantum (previously Nova-
Network in Essex) to allocate networking information
• Step 11 – Nova Compute queries Cinder to allocate volume
information (optional step for persistent data)
• Steps 12-14 – Nova Compute fetches VM image (base OS)
from Glance
• Step 15 – Nova Compute passes all information about VM (in
a single message) to Hypervisor and Hypervisor (KVM / Xen)
creates an instance

163. © MIRANTIS 2013 PAGE 163
Glance
"The Glance project provides
services for discovering,
registering, and retrieving virtual
machine images."

164. © MIRANTIS 2013 PAGE 164
Glance Summary
• Images-as-a-Service
• Can use multiple back-ends for image storage
• Supports multiple image formats

165. © MIRANTIS 2013 PAGE 165
Glance Architecture

166. © MIRANTIS 2013 PAGE 166
Glance Capabilities

167. © MIRANTIS 2013 PAGE 167
Glance Capabilities
• CRUD images (Create, Read, Update, Delete)

168. © MIRANTIS 2013 PAGE 168
Glance Capabilities
• CRUD images (Create, Read, Update, Delete)
• Search images via filters
• name
• container format
• disk format
• size_min, size_max
• status

169. © MIRANTIS 2013 PAGE 169
Glance Capabilities
• CRUD images (Create, Read, Update, Delete)
• Search images via filters
• name
• container format
• disk format
• size_min, size_max
• status
• Caches images
• uses SQLite or FS that supports xattrs for caching
• queues images for prefetching
• prefetches images
• prunes images
• cleans invalid cache entries

170. © MIRANTIS 2013 PAGE 170
Glance Image Formats
Disk Format Description
raw This is an unstructured disk image format
vhd This is the VHD disk format, a common disk format used by virtual machine monitors from
VMWare, Xen, Microsoft, VirtualBox, and others
vmdk Another common disk format supported by many common virtual machine monitors
vdi A disk format supported by VirtualBox virtual machine monitor and the QEMU emulator
iso An archive format for the data contents of an optical disc (e.g. CDROM).
qcow2 A disk format supported by the QEMU emulator that can expand dynamically and supports Copy
on Write
aki This indicates what is stored in Glance is an Amazon kernel image
ari This indicates what is stored in Glance is an Amazon ramdisk image
ami This indicates what is stored in Glance is an Amazon machine image

171. © MIRANTIS 2013 PAGE 171
Custom Image Creation
• Get installation ISO
• Create VM (qemu-img create)
• Start VM and connect to it via VNC console
• Install image without LVM
• Create default iptables rules
• Install and configure cloud-init
• With cloud-init configure image
• Prepare image for OpenStack
• Extract root partition, kernel and ramdisk
• cleanup
• package

172. © MIRANTIS 2013 PAGE 172
Step 12: Request VM Image from Glance
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Nova Compute requests VM
image from Glance via Image ID
UI: Horizon or CLI
Keystone
Keystone Server
Keystone DB
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell

173. © MIRANTIS 2013 PAGE 173
Step 13: Get Image URI from Glance
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
If image with given image ID can be
found - return URI – HTTP Get URI
UI: Horizon or CLI
Keystone
Keystone Server
Keystone DB
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell

174. © MIRANTIS 2013 PAGE 174
Step 14: Download Image from Swift
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Nova Compute downloads image using URI,
given by Glance, from Swif
(or Glance's back-end)
UI: Horizon or CLI
Keystone
Keystone Server
Keystone DB
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell

175. © MIRANTIS 2013 PAGE 175
VM Provisioning Algorithm
• Step 9 – Nova Compute gets message from MQ and asks
Nova Conductor for VM info from database
• Step 10 – Nova Compute queries Quantum (previously Nova-
Network in Essex) to allocate networking information
• Step 11 – Nova Compute queries Cinder to allocate volume
information (optional step for persistent data)
• Steps 12-14 – Nova Compute fetches VM image (base OS)
from Glance
• Step 15 – Nova Compute passes all information about VM (in
a single message) to Hypervisor and Hypervisor (KVM / Xen)
creates an instance

176. © MIRANTIS 2013 PAGE 176
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 15: Start VM Rendering via Hypervisor
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Nova Compute fetches
information about VM from
DB, creates a command to
Hypervisor and delegates
VM rendering to Hypervisor. UI: Horizon or CLI
Keystone
Keystone Server
Keystone DB
In case of KVM / libvirtd this is
a single XML VM config file

177. © MIRANTIS 2013 PAGE 177
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 16: VM is UP
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
iSCSI communication
begins for volume
UI: Horizon or CLI
Keystone
Keystone Server
Keystone DB
Nova Compute sends a
message to Nova
Conductor to update
DB with VM state

178. © MIRANTIS 2013 PAGE 178
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 17: User is Happy
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Horizon polls Nova API for
VM status and power state,
which is taken from
Database.
UI: Horizon or CLI
Keystone
Keystone Server
Keystone DB

179. © MIRANTIS 2013 PAGE 179
Recap:

180. © MIRANTIS 2013 PAGE 180
Recap:
• Users logs into Horizon and initiates a VM create

181. © MIRANTIS 2013 PAGE 181
Recap:
• Users logs into Horizon and initiates a VM create
• Keystone authorizes

182. © MIRANTIS 2013 PAGE 182
Recap:
• Users logs into Horizon and initiates a VM create
• Keystone authorizes
• Nova initiates provisioning and saves state to DB

183. © MIRANTIS 2013 PAGE 183
Recap:
• Users logs into Horizon and initiates a VM create
• Keystone authorizes
• Nova initiates provisioning and saves state to DB
• Nova Scheduler finds appropriate host

184. © MIRANTIS 2013 PAGE 184
Recap:
• Users logs into Horizon and initiates a VM create
• Keystone authorizes
• Nova initiates provisioning and saves state to DB
• Nova Scheduler finds appropriate host
• Quantum configures networking

185. © MIRANTIS 2013 PAGE 185
Recap:
• Users logs into Horizon and initiates a VM create
• Keystone authorizes
• Nova initiates provisioning and saves state to DB
• Nova Scheduler finds appropriate host
• Quantum configures networking
• Cinder provides block device

186. © MIRANTIS 2013 PAGE 186
Recap:
• Users logs into Horizon and initiates a VM create
• Keystone authorizes
• Nova initiates provisioning and saves state to DB
• Nova Scheduler finds appropriate host
• Quantum configures networking
• Cinder provides block device
• Image URI is looked up through Glance

187. © MIRANTIS 2013 PAGE 187
Recap:
• Users logs into Horizon and initiates a VM create
• Keystone authorizes
• Nova initiates provisioning and saves state to DB
• Nova Scheduler finds appropriate host
• Quantum configures networking
• Cinder provides block device
• Image URI is looked up through Glance
• Image is retrieved via Swift

188. © MIRANTIS 2013 PAGE 188
Recap:
• Users logs into Horizon and initiates a VM create
• Keystone authorizes
• Nova initiates provisioning and saves state to DB
• Nova Scheduler finds appropriate host
• Quantum configures networking
• Cinder provides block device
• Image URI is looked up through Glance
• Image is retrieved via Swift
• VM is rendered