Jargon, Protocols and Uniforms as Barriers to Effective Communications
Communications and Protocols
24
HEC FSIO Workshop August 9, 2011 Communications and Protocols Communications and Protocols Active Networks and Active Object Active Networks and Active Object Storage Storage John A. Chandy John A. Chandy Department of Electrical and Computer Engineering Department of Electrical and Computer Engineering Janardhan Singaraju, Ajith Thamarakuzhi, Cengiz Karakoyunlu, Orko Janardhan Singaraju, Ajith Thamarakuzhi, Cengiz Karakoyunlu, Orko Momin, Mike Runde, Paul Wortman Momin, Mike Runde, Paul Wortman
description
Communications and Protocols. Active Networks and Active Object Storage John A. Chandy Department of Electrical and Computer Engineering Janardhan Singaraju , Ajith Thamarakuzhi , Cengiz Karakoyunlu , Orko Momin , Mike Runde , Paul Wortman. Active Storage Networks. Active Disks - PowerPoint PPT Presentation
Transcript of Communications and Protocols
HEC FSIO Workshop
August 9, 2011
Communications and ProtocolsCommunications and Protocols
Active Networks and Active Object StorageActive Networks and Active Object Storage
John A. ChandyJohn A. ChandyDepartment of Electrical and Computer EngineeringDepartment of Electrical and Computer Engineering
Janardhan Singaraju, Ajith Thamarakuzhi, Cengiz Karakoyunlu, Orko Momin, Mike Runde, Janardhan Singaraju, Ajith Thamarakuzhi, Cengiz Karakoyunlu, Orko Momin, Mike Runde, Paul WortmanPaul Wortman
Active Storage NetworksActive Storage Networks
• Active DisksActive Disks
– Intelligence at the disk can distribute computation to parallel Intelligence at the disk can distribute computation to parallel disksdisks
– Process data in streamsProcess data in streams
– Disks only have local view of dataDisks only have local view of data
• Active Storage NetworkActive Storage Network
– Network has a global view of dataNetwork has a global view of data
– Distributed caching of file system metadata and dataDistributed caching of file system metadata and data
– Redundancy optimizationsRedundancy optimizations
Active Storage Networks
HEC FSIO Workshop
August 9, 2011
Active storage networksActive storage networks
1 /3 Gbps
1/3 Gbps
1/3 Gbps1 Gbps
• An ASN is comprised of a smart switch along with intelligence An ASN is comprised of a smart switch along with intelligence embedded in the I/O network. embedded in the I/O network.
• Network Switches have global view of the data and can Network Switches have global view of the data and can perform in-stream data reduction and transformation perform in-stream data reduction and transformation operations.operations.
• ASN can enhance storage node performance as well as ASN can enhance storage node performance as well as improve the computational performance of the parallel I/O improve the computational performance of the parallel I/O systems.systems.
Active Storage Networks
HEC FSIO Workshop
August 9, 2011
Network switch topologyNetwork switch topology
• 2-dilated flattened butterfly2-dilated flattened butterfly
Active Storage Networks
HEC FSIO Workshop
August 9, 2011
1
1
2 5
9
2
3
4
3
5
6
4
7
8
6
7
8
10
11
12
13
14
15
16
Hardware ImplementationHardware Implementation
• NetFPGA board from StanfordNetFPGA board from Stanford
• 4 GigE connects4 GigE connects
• 2 SATA connectors for node to node communication2 SATA connectors for node to node communication
• PCI bus for node to node communicationPCI bus for node to node communication
Active Storage Networks
HEC FSIO Workshop
August 9, 2011
Active Storage NetworksActive Storage Networks
• Application operationsApplication operations
– Reduction operations - min/max, k-means Reduction operations - min/max, k-means clustering, searchclustering, search
– Transformational operations – streaming, sort, Transformational operations – streaming, sort,
• File System OperationsFile System Operations
– LockingLocking
– Redundancy optimizationsRedundancy optimizations
Active Storage Networks
HEC FSIO Workshop
August 9, 2011
Parallelization techniquesParallelization techniques
• Functional units are re-used on reaching the Functional units are re-used on reaching the reconfigurable hardware area limits.reconfigurable hardware area limits.
• Data level parallelism by distributing the Data level parallelism by distributing the data to several functional units in several data to several functional units in several switch elements.switch elements.
• Functional level parallelism by distributing Functional level parallelism by distributing functions to several elements. functions to several elements.
Active Storage Networks
HEC FSIO Workshop
August 9, 2011
KK-means clustering-means clustering
Active Storage Networks
HEC FSIO Workshop
August 9, 2011
Runtime per iterationRuntime per iteration
Active Storage Networks
HEC FSIO Workshop
August 9, 2011
Data searchData search
Active Storage Networks
HEC FSIO Workshop
August 9, 2011
Kmin/KmaxKmin/Kmax
Active Storage Networks
HEC FSIO Workshop
August 9, 2011
Data sortData sort
Active Storage Networks
HEC FSIO Workshop
August 9, 2011
Redundancy optimizationsRedundancy optimizations
Active Storage Networks
HEC FSIO Workshop
August 9, 2011
• RAIDed files, parity calculated in switchRAIDed files, parity calculated in switch
File lockingFile locking
• Lock table in switchLock table in switch
Active Storage Networks
HEC FSIO Workshop
August 9, 2011
Active Storage Networks
HEC FSIO WorkshopAugust 9, 2011
Active Storage NetworksActive Storage Networks
• Lessons LearnedLessons Learned
– Hardware design is hardHardware design is hard
– HW Libraries can helpHW Libraries can help
– ASNs make most sense for reductionsASNs make most sense for reductions
– Storage systems optimizations show promiseStorage systems optimizations show promise
• What needs to get doneWhat needs to get done
– Better HW designBetter HW design
– Application and FS hooksApplication and FS hooks
– When to do ASN and when to do SW?When to do ASN and when to do SW?
Active Storage Networks
HEC FSIO WorkshopAugust 9, 2011
Active Object StorageActive Object Storage
• Active DisksActive Disks
– Intelligence at the disk can distribute computation to parallel disksIntelligence at the disk can distribute computation to parallel disks
• Active Object Storage for Parallel File SystemsActive Object Storage for Parallel File Systems
– Active Disks for OSDsActive Disks for OSDs
– Use Active Storage to improve parallel file system performanceUse Active Storage to improve parallel file system performance
– Use Active Storage to improve parallel file system reliabilityUse Active Storage to improve parallel file system reliability
– Application aware storage and autonomic storage using active OSDs.Application aware storage and autonomic storage using active OSDs.
Active DisksActive Disks
• Can we use OSDs to make Active Disks a reality?Can we use OSDs to make Active Disks a reality?
– Application-aware storageApplication-aware storage
• Object attributes can give hints to the diskObject attributes can give hints to the disk
• Application specificApplication specific
– Parallel File SystemsParallel File Systems
• Felix et al. added a filtering layer to Lustre to provide active Felix et al. added a filtering layer to Lustre to provide active processingprocessing
– T10 OSD?T10 OSD?
Active Storage Networks
HEC FSIO Workshop
August 9, 2011
Active Storage Networks
HEC FSIO WorkshopAugust 9, 2011
Active Disks using OSDActive Disks using OSD
• Previous ImplementationPrevious Implementation
– Based on disc-osdBased on disc-osd
– Object-oriented (Java)Object-oriented (Java)• Attach object types to storage objectsAttach object types to storage objects
• Define methods for object typesDefine methods for object types
• New ImplementationNew Implementation
– Based on osc-osd (supported by Panasas)Based on osc-osd (supported by Panasas)
– RPC - Call functions on OSD remotelyRPC - Call functions on OSD remotely
– Execute Engines – C, Java, Python, etc.Execute Engines – C, Java, Python, etc.
Active Storage Networks
HEC FSIO WorkshopAugust 9, 2011
Active Disks using OSDActive Disks using OSD
• How do you move code from client to target within How do you move code from client to target within OSD framework?OSD framework?
– Create an object with the codeCreate an object with the code
– Each function object has a special attribute that defines Each function object has a special attribute that defines the type of associated execute enginethe type of associated execute engine
– OSD can support multiple execute enginesOSD can support multiple execute engines
Active Storage Networks
HEC FSIO WorkshopAugust 9, 2011
Active Disks using OSDActive Disks using OSD
• How do you execute the method remotely within the OSD How do you execute the method remotely within the OSD framework?framework?
– New EXECUTE FUNCTION command so that we can New EXECUTE FUNCTION command so that we can invoke a functioninvoke a function
– We use the CDB continuation to specify the parametersWe use the CDB continuation to specify the parameters
– Results (if any) returned directly or written to a new objectResults (if any) returned directly or written to a new object
From T10/08-185r5 changes to OSD-2From T10/08-185r5 changes to OSD-2
Active Storage Networks
HEC FSIO WorkshopAugust 9, 2011
Active Disks using OSDActive Disks using OSD
Client
OSD
iSCSIOSD Target C Engine
Java Engine
Function Object
.so, .class
Active Storage Networks
HEC FSIO WorkshopAugust 9, 2011
Active Disks using OSDActive Disks using OSD
• Status:Status:
– C and Java engines completeC and Java engines complete
– Python engine soonPython engine soon
– OrangeFS support for OSDsOrangeFS support for OSDs
SummarySummary
• Active storage networksActive storage networks
– Improves performance of computation kernelsImproves performance of computation kernels
– Useful in parallel file system optimizationsUseful in parallel file system optimizations
• Active storage for improved file system Active storage for improved file system performanceperformance
• Acknowledgements: NSF CCF-0621448, CCF-Acknowledgements: NSF CCF-0621448, CCF-093787093787
Active Storage Networks
HEC FSIO Workshop
August 9, 2011
Communication and ProtocolsCommunication and Protocols
Active Storage Networks
HEC FSIO Workshop
August 9, 2011
• Coherence schemesCoherence schemes
• Scalable abstractions for scientific dataScalable abstractions for scientific data
• Scalable replication, relocation, failure detection, and fault toleranceScalable replication, relocation, failure detection, and fault tolerance
• Topology aware storage layoutTopology aware storage layout
• Wide area storage access protocolsWide area storage access protocols
• Cloud storage?Cloud storage?
• Inter-stack communication?Inter-stack communication?
• Memory hierarchy?Memory hierarchy?
