Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud...
Transcript of Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud...
![Page 1: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/1.jpg)
Cloud, IOT and Analytics
Baremetal with Apache CloudStackApacheCon Europe 2016
Jaydeep Marfatia
![Page 2: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/2.jpg)
© 2016 Accelerite. All Rights Reserved.2
Me
Director of Product ManagementCloud Products
Accelerite
Background
Project lead for open source projectXenMan/ConVirt
Co-Founder Convirture Corp.
Architect for Oracle 10g Enterprise Manager
![Page 3: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/3.jpg)
© 2016 Accelerite. All Rights Reserved.3
Apache CloudStack
• Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform
• Easy to deploy, turnkey solution that includes the entire "stack" of features most organizations want with an IaaS cloud
• Used by many enterprises for their private cloud, as well as for running large public clouds !
• CloudStack currently supports the most popular hypervisors: VMware, KVM, Citrix XenServer, Hyper-V and more
![Page 4: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/4.jpg)
© 2016 Accelerite. All Rights Reserved.4
Accelerite
• Enterprise Infrastructure software company
• Acquired CloudPlatform from Citrix early this year
• Commercial product based on CloudStack
• 50+ engineers and adding more
![Page 5: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/5.jpg)
© 2016 Accelerite. All Rights Reserved.5
Recent contributions by Accelerite
• Template upload from browser
• DHCP/DNS Offload
• LDAP integration
• CoreOS/Docker on CloudStack
• VPN enhancements
• VMWare Networking improvements
• Baremetal enhancements
• 200+ Bug fixes in the product
![Page 6: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/6.jpg)
CloudStackUser Interface
![Page 7: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/7.jpg)
© 2016 Accelerite. All Rights Reserved.7
User Console
![Page 8: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/8.jpg)
© 2016 Accelerite. All Rights Reserved.8
Admin Console
![Page 9: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/9.jpg)
© 2016 Accelerite. All Rights Reserved.9
Self-service VM Provisioning
Zone Template Compute Disk Affinity Network Launch
![Page 10: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/10.jpg)
© 2016 Accelerite. All Rights Reserved.10
Monitoring Cloud Infrastructure
© 2016 Accelerite. All rights reserved.
![Page 11: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/11.jpg)
Baremetal Team
![Page 12: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/12.jpg)
© 2016 Accelerite. All Rights Reserved.12
Team
• Harikrishna Patnala – Apache Committer, working on CloudStack 4+ years
Expertise : Baremetal and Virtual Router
• Jayapal Uradi – Apache Committer, 4+ years
Expertise : CloudStack Networking
• Suresh Sadhu – Quality Assurance of CloudStack, 5+ years
Expertise : All rounder
![Page 13: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/13.jpg)
Motivation for Baremetal
![Page 14: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/14.jpg)
© 2016 Accelerite. All Rights Reserved.14
Why Baremetal?
• Meet high performance computing requirements
• Hadoop, Spark, HPC
• Take advantage of specialized hardware
• GPUs, Blade Micro-servers
• Databases
• Virtual Desktop Infrastructure management
• Desired for Docker Containers
![Page 15: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/15.jpg)
Requirements
![Page 16: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/16.jpg)
© 2016 Accelerite. All Rights Reserved.16
Data Center
![Page 17: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/17.jpg)
© 2016 Accelerite. All Rights Reserved.17
Data Center
Servers
Switch
Rack
Secondary Storage
![Page 18: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/18.jpg)
© 2016 Accelerite. All Rights Reserved.18
Workload Partitioning
Hadoop
Switch
Secondary Storage
Kubernetes
Rack
![Page 19: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/19.jpg)
© 2016 Accelerite. All Rights Reserved.19
Multi-Tenancy
Tenant 1
Switch
Secondary Storage
Tenant 2
Rack
![Page 20: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/20.jpg)
© 2016 Accelerite. All Rights Reserved.20
Multi-Tenancy and Workload partitioning
Tenant 1
Switch
Secondary Storage
Tenant 2
Tenant 1 - DBRack
![Page 21: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/21.jpg)
© 2016 Accelerite. All Rights Reserved.21
Data Center – VMs and Baremetal
Tenant 1 Network
Rack
Tenant 2 Network
![Page 22: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/22.jpg)
© 2016 Accelerite. All Rights Reserved.22
Baremetal in CloudStack
• Ability to deploy Baremetal servers as easy as spinning up a VM
• Multi-tenant Baremetal orchestration with network isolation
• Flexibility of using mixed environments of VM and Baremetal
• DB tier on Baremetal and other tiers on VMs for a multi-tier application
• Extensible architecture for different flavors of hardware and network
elements
![Page 23: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/23.jpg)
What all do we need ?
![Page 24: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/24.jpg)
© 2016 Accelerite. All Rights Reserved.24
PXE server
• Basic Networking
• Flat network
• External PXE/DHCP/DNS Servers
• Advanced Networking
• Isolation using VLANs
• Additionally VR acts as PXE
• All VM networking services mapped to Baremetal
![Page 25: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/25.jpg)
© 2016 Accelerite. All Rights Reserved.
Advanced Networking
Network isolation with VLAN
Guest1
VM1
Guest1
VM2
Guest1
VM3Guest1
VR
VLAN2000
Management
Storage
Public VLAN99
Hypervisor 1 Hypervisor 2
Switch
Console, Storage VMs
Hypervisor 2
Hypervisor 2
Public
VLAN99
![Page 26: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/26.jpg)
© 2016 Accelerite. All Rights Reserved.
Advanced Networking
Network isolation with VLAN
Guest1
VM1
Guest1
VM2
Guest1
VM3Guest2
VM1
Guest2
VM2Guest2
VM3
Guest2
VRGuest1
VR
VLAN2000
VLAN2001
Management
Storage
Public VLAN99
Hypervisor 1 Hypervisor 2
Switch
Console, Storage VMs
Hypervisor 2
Hypervisor 2
Public
VLAN99
![Page 27: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/27.jpg)
© 2016 Accelerite. All Rights Reserved.
Advanced Networking
VLAN based isolation
Networking services:
• DHCP
• DNS
• Source NAT
• Static NAT
• Port Forwarding
• Load Balancing
• Firewall
• VPN
• PXE
![Page 28: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/28.jpg)
© 2016 Accelerite. All Rights Reserved.28
Templates
• PXE Server hosts the Installer Kernels and Kickstart files
• Template Registration is done by giving the URL from where OS Packages can be retrieved
PXE Server/VR
Kickstart, Installer Kernels
NFS
OS Package Repository
Baremetal Host
HTTP
TFTP
![Page 29: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/29.jpg)
© 2016 Accelerite. All Rights Reserved.29
Example Template URL
• ks=http_link_to_kickstartfile;kernel=nfs_path_to_pxe_kernel;initrd=nfs_path_to_pxe_initrd
• ks=http://10.147.28.7/export/home/hari/rhel65/ks1.ks;kernel=10.147.28.7:/export/home/hari/rhel65/RHEL65/images/pxeboot/vmlinuz;initrd=10.147.28.7:/export/home/hari/rhel65/RHEL65/images/pxeboot/initrd.img
![Page 30: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/30.jpg)
© 2016 Accelerite. All Rights Reserved.30
Service Offering
• Service offering allows administrator to determine size of the VM in
terms of CPU and memory
• e.g. small vm 2 cpu, 512 MB ram
• For baremetal, CPU and memory information is specified while adding
the host along with a tag and IPMI details
• e.g. host-44, 8 cpu, 32 GB ram –tag ssd_bm, ipmi info : ip, user, password
• Tagging scheme is used to map service offerings and baremetal host
• e.g. SSD baremetal medium – 8 cpu, 32 GB ram – tag ssd_bm
![Page 31: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/31.jpg)
© 2016 Accelerite. All Rights Reserved.31
Switch Configuration
• Switch credentials
• Switch topology information
• host – switch port mapping Switch
BaremetalNodes
MS NodeVR on Hypervisor
Template
Storage
{ "racks": [{ "l2Switch": { "ip": "10.147.28.11", "username": "admin",
"password": "password", "type": "Force10” },
"hosts":[{ "mac": "d4:ae:52:7a:a5:9c", "port": "tengigabitethernet:0/30"
}, { "mac": "d4:ae:52:7a:a5:9d", "port": "tengigabitethernet:0/31"
} ]
} ]}
![Page 32: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/32.jpg)
Provisioning Flow
![Page 33: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/33.jpg)
© 2016 Accelerite. All Rights Reserved.33
Management Server
Baremetal Provisioning Flow – Advanced Zone
Management Server
Physical Switch
Management ServerBaremetal
Hosts
VR(w/ PXE &
DHCP/DNS)
1. User Requests Baremetal Instance
2. Management Server configures PXEDHCP on VR and physical switch with VLANS
3. Baremetal host started with IPMI commandto PXE boot
4. Host gets DHCP
5. Host pulls PXE image
6. Image is pulled and installed
7. Host pulls user data from VR
![Page 34: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/34.jpg)
© 2016 Accelerite. All Rights Reserved.34
Windows Baremetal Provisioning (Approach)
• Windows Deployment server needs to be programmed
• Need to invoke powershell commands from Linux
• Agent to be present in WDS Server
• Using RPC/or Event Queue mechanism, send Powershell commands to WDS
• AD / DHCP is used for authentication and PXE booting
![Page 35: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/35.jpg)
© 2016 Accelerite. All Rights Reserved.35
Management Server
Windows Baremetal Provisioning Flow
Management Server
DHCP/DNSActive
Directory
Management ServerBaremetal
Hosts
WDS (w/ PXE)
1. User Requests Baremetal Instance
2. Management Server configures WDS Server with PXE, switch configured
3. Baremetal host started with IPMI commandto PXE boot
4. Host gets DHCP
5. Host pulls PXE image
6. Image is pulled and installed
7. Host pulls user data from VR
Switch
![Page 36: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/36.jpg)
© 2016 Accelerite. All Rights Reserved.36
KubernetesCluster
Baremetal Use Case for Container Deployment
1. Install CoreOS on Baremetal hosts
2. Setup networking – Basic or Isolated (program the switch)
3. Create k8s cluster using cloud-config sent via userdata service on VR
4. Use Kubernetes dashboard within CloudStack UI to deploy and manage containers
5. All CloudStack tenanting and isolation features apply
MgmtServer
Virtual Router
Node1
Node2
MasterDashboard
cloud-config
CoreOS Images
![Page 37: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/37.jpg)
Extensibility
![Page 38: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/38.jpg)
© 2016 Accelerite. All Rights Reserved.38
Network Switch Plugin<<interface>>
BaremetalSwitchBackend
• getSwitchBackendType()• prepareVlan()• removePortFromVlan()
Force10BaremetalSwitchBackend
• getSwitchBackendType() { return TYPE }• prepareVlan()• removePortFromVlan()
public static final String TYPE = "Force10"
![Page 39: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/39.jpg)
© 2016 Accelerite. All Rights Reserved.39
Network Switch Plugin
• Currently, a Dell Force10 switch is implemented
• To Extend to other types, implement following two methods as a plugin
Arista plugin Implementation Details:1.Implement AristaBaremetalSwitchBackend.java and implement the interface BaremetalSwitchBackend. For example interface refer to https://github.com/apache/CloudStack/blob/master/plugins/hypervisors/baremetal/src/com/cloud/baremetal/networkservice/Force10BaremetalSwitchBackend.java.2. Need to implement the following three functions with appropriate actions for Arista switches in the abovefile. (definition of interface is in https://github.com/apache/CloudStack/blob/master/plugins/hypervisors/baremetal/src/com/cloud/baremetal/networkservice/BaremetalSwitchBackend.java):public interface BaremetalSwitchBackend {
String getSwitchBackendType();void prepareVlan(BaremetalVlanStruct struct);void removePortFromVlan(BaremetalVlanStruct struct);
}3. Add AristaBaremetalSwitchBackend bean configuration into https://github.com/apache/CloudStack/blob/master/plugins/hypervisors/baremetal/resources/META-INF/CloudStack/core/spring-baremetal-core-context.xml.
![Page 40: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/40.jpg)
Looking forward
![Page 41: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/41.jpg)
© 2016 Accelerite. All Rights Reserved.41
Baremetal Projects : Contributions welcome
• WDS integration (WIP)
• Baremetal in VPC Network
• Add switch integration from different vendors
• NIC Bonding
• RAID
• UEFI Boot
• Redfish integration
• Auto-discovery of hosts and NIC details
![Page 42: Jaydeep Marfatia...•Highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform •Easy to deploy, turnkey solution that includes the entire "stack"](https://reader030.fdocuments.us/reader030/viewer/2022041109/5f0d10e57e708231d4388287/html5/thumbnails/42.jpg)
Q&A