OpenStack Object Storage (code-named Swift) is open source software for creating redundant, scalable object storage using clusters of standardized servers to store petabytes of accessible data. It is not a file system or real-time data storage system, but rather a long-term storage system for a more permanent type of static data that can be retrieved, leveraged, and then updated if necessary. Primary examples of data that best fit this type of storage model are virtual machine images, photo storage, email storage and backup archiving. Having no central "brain" or master point of control provides greater scalability, redundancy and permanence. Swift Version 1.5 is set for release the week of June 11th. Get a clear understanding of the power of Swift, the changes in the current release and the features slated for the next major OpenStack release: Folsom. As a bonus, watch how easy it is to setup OpenStack Swift with Dell's "Crowbar - the Cloud Unboxer." Discussion that follows will include scalability considerations, monitoring, and management of the system.

1. Insider's Perspective:
Exploring OpenStack Swift – Essex Release
OpenStack Boston Meetup
June 21, 2012
Judd Maltin

3. OpenStack Timeline Fulsom
Essex “Platform for
Innovation”
Diablo “Production
Ready” Core Platform for
Workable Innovation
Cactus
Foundation Stable Foundation
Network as a
Community
Bexar Development Exposes Gaps Included in Service
Forming Solidify Ubuntu 12.04 Block Storage
Austin First Community
Public Working Incubated: Public Adoptoin
Code Prototypes Loses VMware Network & Block Multiple Scale
Formation
& HyperV Storage Deployments
2011 2012
Nov 2010 Dec Feb Apr Jun Aug Oct Dec Feb Apr Jun Aug
Nov 2010: Feb 2011: Apr 2011: Sep 2011: Mar 2012: Oct 2012:
Austin Bexar Cactus Diablo Essex Fulsom
Release Release Release Release Release Release
Oct 2010: Apr 2011: Oct 2011: Apr 2012:
Design Design Design Design
Summit Summit Summit Summit

4. A highly scalable, redundant,
unstructured data store designed to
store large amounts of data cheaply.

5. Why Swift?
Good use cases:
●Storing media libraries (photos, music, videos, etc.)
●Archiving video surveillance files
●Archiving phone call audio recordings
●Archiving un/compressed log files
●Archiving backups
●Storing and loading of OS Images, etc.
●Storing file populations that grow continuously on a practically infinite
basis.
●Storing small files (<50 KB). OpenStack Object Storage is great at this.
●Storing billions of files.
●Storing Petabytes (millions of Gigabytes) of data.

6. What Swift is (the 411)
● Written in Python
● SQLite, Rsync, Memcache, XFS
● HTTP/ReST API
● Logical Parts
● Accounts
● Containers
● Objects
● CDN Integration
● No single point of failure
● Last write wins

7. What Swift Ain't
Not a filesystem or block storage:
Does not use the typical POSIX filesystems semantics like
open(),read(),write(), seek() … rather HTTP actions like
PUT,GET,DELETE,POST.
Not a database:
Stores unstructured, file based data.
Only download, upload, retrieve data but does not process data.
Only basic tagging.
Barely even hierarchical:
Only one level of container depth. Vague tagging.

8. System Components

9. Example OpenStack
Object
To Load Balancers
Storage Hardware
Proxies
5 Zones
2 Proxies per 25
Storage Nodes
10 GigE to Proxies
1 GigE to
Storage Nodes
24 x 2TB Drives
per Storage Node
Example Large Scale Deployment -- Many Configs
Possible

10. Object Storage Key Features
ReST-based API Data distributed evenly
throughout system
Scalable to multiple
petabytes, billions of
objects
Account/Container/Object
structure (not file system, no
nesting) plus Replication (N
copies of accounts, containers,
objects)
No central
database
Hardware agnostic: standard
hardware, RAID not required

11. System Components
PYTHON PROCS:
Proxy Server: Authentication, ACLs, quotas, rate-limiting
Account Server: Handles listing of containers, stored as
SQLite DB
Container Server: Handles listing of objects, stored as
SQLite DB
Object Server: Blob storage server, metadata kept in xattrs,
data in binary format
Store your objects on XFS
Object location based on hash of name & timestamp

12. System Components (cont.)
●
The Ring: Mapping of names to entities (accounts, containers,
objects) on disk.
●
Stores data based on zones, devices, partitions, and replicas
●
Weights can be used to balance the distribution of partitions
●
Used by the Proxy Server for many background processes
●
Proxy Server: Request routing, exposes the public API
●
Replication: Keep the system consistent, handle failures
●
Updaters: Process failed or queued updates
●
Auditors: Verify integrity of objects, containers, and accounts
●
Deleters: Account Reaper, Replication

13. System Components (cont., cont.)
● Auth System: Completely pluggable. Temp Auth, Keystone, Roll-your-
own
●
Replication: (again?) replication works by crawling the proc's local
filesystems and querying the cluster for the file existing elsewhere.
Of course, it just compares MD5 sums.
●
DB Replication vs. Object Replication
●
Container to Container Synchronization: Spread your clusters
out, sync out-of-band

14. Swift Containers
https://swift.example.com/v1/bobsfishfry/tuna/tuna1.jpg
Container Based Auth:
●
X-Container-Read: accountname
●
X-Container-Write: accountname:username
●
X-Container-Read: referer:
The Container Rules:
●
Make lots of containers! Cheap!
●
No nested containers
●
No / in container names
●
<256 bytes (incl URL encoding)

15. Coming in Folsom (1.5)
* Versioned Objects!
* Large Object Support (with no need for client
support)
* Expiring Object support
* StatsD Support
* Logging is now middleware
* Amazon S3 compatibility is now external project
* DB Preallocation: great for spinning media, horrible
for SSDs

16. Coming Later!
Ring Builder Web Service
Upgraded WebOb (1.2) support
(unicode, etc.)
Drive Failure Detector and Remediation
Token Service

17. OpenStack Community Resources
● IRC (freenode)
● #openstack
● #openstack-dev
● #openstack-meeting
● Docs
● http://wiki.openstack.org
● http://docs.openstack.org
● http://swift.openstack.orga
● http://nova.openstack.org
● Twitter
● @openstack