Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc •...
Transcript of Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc •...
![Page 1: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/1.jpg)
Linux Clusters Institute:Ceph & Swift
Sam LinstonUtah Center for High Performance Computing
J.D. Maloney | Storage EngineerNational Center for Supercomputing Applications (NCSA)
Georgia Tech, August 15th – 18th 2017
![Page 2: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/2.jpg)
Introduction• The rise of object storage is showing itself these past few years• Driven by the massive growth of datasets and prominence of
large web-scale companies that need to store obscene amounts of data
• Object storage and Amazon S3 storage are closely recognized together for example• Many options are out there for object storage, way more than
just Ceph and Swift• These are the two most popular ones in the HPC space
• As research data sharing continues to grow and web portals to datasets are build, we need to build them on something
August 15th-18th 2017 2
![Page 3: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/3.jpg)
Ceph
August 15th-18th 2017 3
![Page 4: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/4.jpg)
Overview• Object/Block/Posix file system that is owned by Red Hat• It along side GlusterFS are in Red Hat Storage Server
product• Open Source, free to use, commercial support available
through Red Hat• Provides Object, Block, and Posix storage• Very popular in the cloud/virtualization space
• Can underpin things like Open Stack and Proxmox
• Runs on “commodity” hardware• Can be deployed over Ethernet or IB
• Ethernet is very popular with this FS
August 15th-18th 2017 4
![Page 5: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/5.jpg)
General High Level View
August 15th-18th 2017 5
Image Credit: ceph.com
![Page 6: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/6.jpg)
Deeper Architecture Views
August 15th-18th 2017 6
Image Credit: apprize.info
![Page 7: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/7.jpg)
Deeper Architecture Views
August 15th-18th 2017 7
Image Credit: ksperis.com
![Page 8: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/8.jpg)
A ”Jack of All Trades”• Offers all three kinds of storage• The LIBRADOS layer brokers all interfaces with the storage
system for great flexibility• Valuable in that one managed system can house
everything• Only mainstream file system besides Spectrum Scale that
can support all three types• Many only provide POSIX, or block/object
August 15th-18th 2017 8
![Page 9: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/9.jpg)
Data StructureStorage Blob• The configuration of the MONs
and OSDs result in a blob of storage
• Organized mass of storage• No structure• This very loose structure is what
allows Ceph to be the flexible and powerful solution it is
August 15th-18th 2017 9
Image Credit: Sam Linston
![Page 10: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/10.jpg)
Data StructurePools• A set of pools are created during system setup• Several pools for user information and several other utility pools• Pool for data payload
• Pool is configured with desired level of resiliency• Two resiliency methods
• Can be N way replication (2x or 3x data copies)• Can be K+M fragment erasure coding (newer, but stable feature)• Desired endurance of data and other environment characteristics determine
which type of data protection is best
August 15th-18th 2017 10
![Page 11: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/11.jpg)
Data StructurePlacement Groups• Ensures proper, balanced object disbursement and configured
redundancy• Number of PGs in a pool determined by
• Number of OSDs in cluster• Number of replicas or level of resiliency
August 15th-18th 2017 11
![Page 12: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/12.jpg)
Architecture ConsiderationsFailure Domains• Set of rules defining segregation of replicated instances or
distributed object fragments• Can be as coarse or granular as needed• Machine, rack, row, room, datacenter, etc
• Configure failure domains to minimize the chance of down time of the service
• How this is done is mostly dependent on environment and resources you have available
• Something to consider and plan for before going into production
August 15th-18th 2017 12
![Page 13: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/13.jpg)
Architecture ConsiderationsI/O Storms• What will the rebalance impact be if one server and its
disks fails• Chose smaller number of disks/server to reduce the
intensity of the I/O storm• As small as 12 to 1• As large as 60 to 1• Depends on tier, server power and bandwidth
• Take rebuild situations into consideration when deciding on backend network infrastructure so that recoveries don’t have a very noticeable impact on the quality of service
August 15th-18th 2017 13
![Page 14: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/14.jpg)
Architecture ConsiderationsSet a High Watermark• Never good to push file systems to high levels of filled
capacity, a high watermark should be determined• Rough target of ~75%-80% full• Must be room to rebalance to
• Too full and the ability to be resilient decreases• Once your system starts to approach your high watermark
start working on adding capacity and rebalance the data across systems
• Consider this in your design, how are you going to add on in the future when growth occurs
August 15th-18th 2017 14
![Page 15: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/15.jpg)
Architecture ConsiderationsPerformance• Dedicated Journal devices (usually SSDs)
• Ratio of Journal devices to non-journal devices• The more that share the same journal device, the more contention there is
going to be for quick writes• Network bandwidth on the backend
• 10G/40G/100G Ethernet• Backbone between switches
• Different tiers of performance• Consider a fast tier with SSD journals, or even an all SSD tier• Lower tier with fewer/lesser performing journal devices or no journal
devices at all
August 15th-18th 2017 15
![Page 16: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/16.jpg)
Ceph DeploymentCeph-Deploy Tool• Consolidates the deployment process under one command• Originally Ceph was installed using a suite of tools• Now the same tool is used for:
• Installation of Ceph software• Commissioning the various Ceph components (ADM, MON, RGW, CephFS,
RBD)• Prepare raw storage• Create OSDs
• Does not destroy OSDs• Convoluted process
August 15th-18th 2017 16
![Page 17: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/17.jpg)
Ceph DeploymentOther Tools• Ceph Tool
• Older tool• Still used to provision pools
• Query System
• RADOSGW-ADMIN Tool• Used for user creation and manipulation• Set quota• Query usage
• Custom Scripts• Many deployment and management scripts available• See what the community has provided already, don’t re-invent the wheel if
you don’t have to
August 15th-18th 2017 17
![Page 18: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/18.jpg)
Features Worth Highlighting• Supports Thin Provisioning
• Great for hosting virtual machine images
• Both the S3 Restful API and the Openstack Swift API are supported
• Erasure Encoding across disks• Much more efficient use of space than replication
• Snapshots and Clones• Large and growing community that is there to support
others getting into Ceph for the first time
August 15th-18th 2017 18
![Page 19: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/19.jpg)
Usage PatternsArchive/Backup• Very popular consumption pattern• Can be User or Admin managed• Space usage managed by quotas
• One drawback in management — No groups
• Access through many tools• rclone, Globus, s3cmd, posix mount (if enabled)
• Great for dumping large tar.gz bundled datasets into
August 15th-18th 2017 19
![Page 20: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/20.jpg)
Usage PatternsWEB• Objects can be accessed via a URL
• Similar to AWS• Resolved as https://<bucket_name>.hostname.domain.name
• Or https://hostname.domain.name/<bucket_name>
• Not easily browsed• Space is inherently flat• All objects are peers• Expresses the page as raw XML
• Objects must be made “public” to be URL accessible• Access can be set to expire
August 15th-18th 2017 20
![Page 21: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/21.jpg)
Usage PatternsS3 API• Popular in the web application space, the leading standard
that most object storage systems support (not just Ceph or Swift)
• Program directly to the API• Very scriptable, and is Restful
• Use other programmatic applications like Python Boto
August 15th-18th 2017 21
![Page 22: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/22.jpg)
User ExperienceBuckets and Objects• All object space is flat
• No tree structure or hierarchy
• To help users navigate a tree structure is represented through tools
• Structure is represented as buckets (analogous to directories) and objects
• One user cannot by default see another users buckets/objects
• Must be shared
August 15th-18th 2017 22
![Page 23: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/23.jpg)
User Experience
August 15th-18th 2017 23
Image Credit: Sam Linston
![Page 24: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/24.jpg)
Resourceshttp://docs.ceph.com/docs/luminous/http://ceph.com/resources/https://www.redhat.com/en/technologies/storage/ceph/resourceshttp://ceph.com/irc/
August 15th-18th 2017 24
![Page 25: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/25.jpg)
Ceph Questions Before Moving Along
August 15th-18th 2017 25
![Page 26: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/26.jpg)
Swift
August 15th-18th 2017 26
Image Credit: swiftstack.com
![Page 27: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/27.jpg)
Overview• Object file system that was developed around Open Stack
• Open stack use for object is the largest use case for this object storage solution
• Large user base in the cloud space with lots of companies backing the project
• Leverages a ring based approach, with hashes determining file placement across the system
• Uses two types of servers• Proxy Servers – Field API requests• Storage Servers – Dish out and take the data
August 15th-18th 2017 27
![Page 28: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/28.jpg)
How Is it Different• Unlike Ceph, only provides Object based storage
• Has a very different architecture• Narrower use focus• Scales in a different, but neat way
• Leverages proxy servers for data access instead of clients directly being in contact with the storage servers
• Has better multi-region support• Doesn’t rely on the Master-Slave concept that Ceph uses
• Assumes eventual consistency vs absolute consistency• Can be an issue in certain instances
August 15th-18th 2017 28
![Page 29: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/29.jpg)
Swift High Level Architecture
August 15th-18th 2017 29
Image Credit: linkedin.com
![Page 30: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/30.jpg)
Swift Lower Level Architecture
August 15th-18th 2017 30
Image Credit: redhat.com
![Page 31: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/31.jpg)
Ring Hashing
August 15th-18th 2017 31
Image Credit: ihong5.wordpress.com, redhat.com
![Page 32: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/32.jpg)
Benefits to the Ring Method
August 15th-18th 2017 32
• Allows the number of proxy servers to scale extremely well• All proxy servers are equal as the ring ensures they each
know where all the data lives• No need for coordinating between the proxy servers
• Ring hash ensures that there will be no collisions of objects• Ring is static (unless capacity is being added or removed)
• Ease of updating and rolling proxy servers as there are just less to serve data requests
• If proxy server update failure occurs, doesn’t impact the system in any other way as long as quorum is still available
![Page 33: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/33.jpg)
Downsides to the Ring Method
August 15th-18th 2017 33
• Increases the latency of data access to the clients• No matter the speed of the server, it will add latency to the request as it
has to broker the data exchange• Especially when working with smaller objects, latency increases are more
noticeable
• Streaming performance also effected• Same issue as the increased latency, proxy servers decrease the ability of
streaming performance of large files
• Eventual consistency can be problematic• Details on next slide
• Performance is now dependent on scaling two independent server groups
![Page 34: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/34.jpg)
Consistency Overview
August 15th-18th 2017 34
• What has to occur before a write to the system has been acknowledged back to the client machine
• Multiple operations occur when a write is done on the file system• When is it ok to tell the client the write is done?• The sooner the write is acknowledged, the sooner the next one can be
sent
• Absolute vs Eventual consistency each come with upsides and downsides
• Ceph and Swift handle this differently even though both are object stores
![Page 35: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/35.jpg)
Absolute Consistency
August 15th-18th 2017 35
• All replicas of the object being written need to be in place and completed before the write is acknowledged back to client
• More strict, but ensures the write completed and the object is in the correct state
• Prevents conflict writes to an object from happening at the same time on different replicas, and a collision occurring
• Becomes a performance bottleneck when replication across physical regions is involved
• Latency between sites will delay the write completion which thus increases the latency that the client sees
![Page 36: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/36.jpg)
Eventual Consistency
August 15th-18th 2017 36
• All replicas of the object being written do not need to be in place and completed before the write is acknowledged back to client
• Less strict, more flexible for architectures where more latency is involved
• Opens door to conflicting writes to an object from happening at the same time on different replicas
• Beneficial in multi-region deployments as usually multiple regions aren’t dealing with the same objects
• No increased latency from having a site over a WAN connection that could hurt performance
![Page 37: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/37.jpg)
Load Balancing
August 15th-18th 2017 37
• Load balancer is needed to distribute the API requests across the different proxy servers
• Clients aren’t explicitly aware of the specific architecture, count, or names of the servers backing the service
• Load balancer puts them in contact with random proxy server, client is handed data from the storage server holding data
![Page 38: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/38.jpg)
Architecture Considerations
August 15th-18th 2017 38
Proxy Servers• Design with high network throughput capacity, 10Gb or
greater speeds• Moderate RAM needs, 32GB/64GB sufficient• Good candidate for dense blade systems (4 node- 2U chassis
available from many vendors)• Doesn’t need much local disk• If access comes over public networks, use SSL to encrypt
traffic
![Page 39: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/39.jpg)
Architecture Considerations
August 15th-18th 2017 39
Storage Servers• Where the data actually lives, so high drive capacity is
desired• Consider your drive choice
• Request is single thread to single drive, so single object stream throughput is at most the performance of one drive
• Beware of SMR drives or consumer drives to keep performance up• Network performance important for feeding data to clients,
use 10Gb or faster• Beneficial to have two networks one for data access and another for cluster
activities such as replication, especially if you use lower speed networks
![Page 40: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/40.jpg)
Rate Limiting
August 15th-18th 2017 40
• Prevents a single container access from dominating the system and hurting performance of other users
• Can be down with bandwidth or IOPS• Rate limits can be applied not just to containers, but also to
certain users/accounts• Great for users who consume high I/O needs• Ensures fair access to the system for all
• Architecture Note:• Time synchronization across the proxy servers is important here, skew can
result in some rate limiting issues as the proxy servers track how much data is being access over given times
![Page 41: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/41.jpg)
Resources
August 15th-18th 2017 41
• https://docs.openstack.org/swift/latest/admin_guide.html• https://www.swiftstack.com/product/openstack-swift• https://wiki.openstack.org/wiki/Swift
![Page 42: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/42.jpg)
Acknowledgements
August 15th-18th 2017 42
• Thanks to Sam Linston from University of Utah for help with slide development content
• Members of the SET group at NCSA for slide review• Members of the steering committee for slide review
![Page 43: Linux Clusters Institute: Ceph & Swift · • Machine, rack, row, room, datacenter, etc • Configure failure domains to minimize the chance of down time of the service • How this](https://reader031.fdocuments.us/reader031/viewer/2022041014/5ec55e86419eb03a8221963e/html5/thumbnails/43.jpg)
Questions
August 15th-18th 2017 43