HopsFS & ePipe - GitHub Pages · 2018-10-03 · HopsFS & ePipe 1 Distributed Computing and...

Mahmoud IsmailKTH

HopsFS & ePipe

Distributed Computing and Analytics Workshop, September 26th 2018

�2Problem: Data layer to store millions of these images and their annotations

At Scale

Open Images Dataset

At Scale

Open Images DatasetOpen Images DatasetOpen Images DatasetOpen Images DatasetOpen Images DatasetOpen Images DatasetOpen Images DatasetOpen Images DatasetOpen Images DatasetOpen Images DatasetOpen Images DatasetDataset X

Requirements

•Reading/Writing millions of images with high throughput

•Attaching annotations to each image, and then searching using these annotations

Hadoop Software Stack

HDFS Architecture

DataNodes

HDFS Architecture

DataNodes

NameNode

HDFS Architecture

DataNodes

HDFS Client

NameNode

HDFS Architecture

DataNodes

HDFS Client

NameNode

HDFS Architecture

DataNodes

HDFS Client

Where can I save the file?

NameNode

HDFS Architecture

DataNodes

HDFS Client

Where can I save the file?

DataNodes Addresses

NameNode

HDFS Architecture

DataNodes

HDFS Client

NameNode

HDFS Architecture

DataNodes

HDFS Client

NameNode

HDFS Architecture

DataNodes

HDFS Client

NameNode

HDFS Architecture

DataNodes

HDFS Client

File Blocks Mappings

File System Metadata

NameNode

HDFS Architecture

DataNodes

File1 Blk1 ! DN1, Blk2 ! DN5, Blk3 ! DN3

NameNode

HDFS Architecture

DataNodes

File1 Blk1 ! DN1, Blk2 ! DN5, Blk3 ! DN3 File2 Blk1 ! DN1, Blk2 ! DN4File3 Blk1 ! DN1, Blk2 ! DN2, Blk3 ! DN3File4 Blk1 ! DN100File5 Blk1 ! DN4, Blk2 ! DN2, Blk3 ! DN9

… … … …FileN Blk1 ! DN2, Blk2 ! DN8

NameNode

HDFS Performance at Scale

DataNode

NameNode

DataNode

NameNode

DataNode

NameNode

DataNode

NameNode

HDFS Limitations

• Namespace size upper bound: ~ 500 million files

• At most 70-80 thousands file system operations / sec

HopsFS

HopsFS Architecture

�1111

NameNodeFile Blocks Mappings

File4 Blk1 ! DN100

File2 Blk1 ! DN1, Blk2 ! DN4

FileN Blk1 ! DN2, Blk2 ! DN8

HopsFS Architecture

�1111

NameNode File Blocks Mappings

File1 Metadata

File2 Metadata

File3 Metadata

File4 Metadata

File5 Metadata

File6 Metadata

File7 Metadata

File8 Metadata

HopsFS Architecture

�1111

NameNode File Blocks Mappings

File1 Metadata

File2 Metadata

File3 Metadata

File4 Metadata

File5 Metadata

File6 Metadata

File7 Metadata

File8 Metadata

Distributed Database

HopsFS Architecture

�1111

NameNode

File1 Metadata

File2 Metadata

File3 Metadata

File4 Metadata

File5 Metadata

File6 Metadata

File7 Metadata

File8 Metadata

HopsFS Architecture

�1111

NameNode

File1 Metadata

File2 Metadata

File3 Metadata

File4 Metadata

File5 Metadata

File6 Metadata

File7 Metadata

File8 Metadata

HopsFS Architecture

�1111

NameNode

File1 Metadata

File2 Metadata

File3 Metadata

File4 Metadata

File5 Metadata

File6 Metadata

File7 Metadata

File8 Metadata

HopsFS Architecture

�1111

NameNode

File1 Metadata

File2 Metadata

File3 Metadata

File4 Metadata

File5 Metadata

File6 Metadata

File7 Metadata

File8 Metadata

HopsFS Architecture

�1111

NameNode

File1 Metadata

File2 Metadata

File3 Metadata

File4 Metadata

File5 Metadata

File6 Metadata

File7 Metadata

File8 Metadata

HopsFS Architecture

�1111

NameNode

File1 Metadata

File2 Metadata

File3 Metadata

File4 Metadata

File5 Metadata

File6 Metadata

File7 Metadata

File8 Metadata

HopsFS Scalability

• 16X-37X the throughput of HDFS

• 37 times more files than HDFS

• 10 times lower latency

Scale Challenge Winner (2017)Hops

Integration with NVMe

https://cloud.google.com/compute/docs/disks/performance

HDFS (and S3) are designed around large blocks (optimized to overcome slow random I/O on disks), while new NVMe hardware supports fast random disk I/O (and potentially small blocks sizes)

Small files

100 KB

512 KB

256 MB

128 GB

File Size

a. File Size Distribution

Yahoo HDFS File DistributionSpotify HDFS File Distribution

LC HopsFS File Distribution At Yahoo! and Spotify ≈20% of the files are less than 4 KB. Logical Clocks’ HopsFS cluster ≈68% of the files are less than 4 KB

Small files

100 KB

512 KB

256 MB

128 GB

File Size

a. File Size Distribution

Yahoo HDFS File DistributionSpotify HDFS File Distribution

LC HopsFS File Distribution At Yahoo! and Spotify ≈20% of the files are less than 4 KB. Logical Clocks’ HopsFS cluster ≈68% of the files are less than 4 KB

100 KB

512 KB

256 MB

128 GB

File Size

b. File Operations Distribution

Spotify HDFS File Ops DistributionLC HopsFS File Ops Distribution At Spotify, and Logical

Clocks ≈ 42% and ≈18% of all the file system operations are performed on files less than 16 KB files

Size Matters

Small Files performance in HopsFS

Open Images dataset

83.5% of the files in the dataset are ⩽ 64 KB.

Open Images Dataset

Requirements

•Reading/Writing millions of images with high throughput

• Attaching annotations to each image, and then searching using these annotations

Attaching Extended Metadata

Foreign key

attach /images/1.jpeg ’1 cat and 1 guitar’

Foreign key

Free text search?

HopsFS | ElasticSearch

HopsFS

HopsFS ElasticSearch

1.jpeg

dog [1.jpeg,……]

1.jpeg

dog [1.jpeg,……]

Get All images that has a dog

1.jpeg

dog [1.jpeg,……]

1 Dog1 Cat and 1 Guitar

1.jpeg

dog [1.jpeg,……]

1.jpeg

dog [1.jpeg,……]

1.jpeg

dog [1.jpeg,……]

Store X

1.jpeg

dog [1.jpeg,……]

Store X

HopsFS

NDB ePipe

HopsFS

NDB ePipe

HopsFS

Log fs changes

HopsFS

Log fs changes

ChangeStream ePipe

HopsFS

Log fs changes

ChangeStream ePipe

HopsFS

Log fs changes

ChangeStream ePipe

ElasticSearch

HopsFS

Log fs changes

ChangeStream ePipe

ElasticSearch

Store X

Store Y

HopsFS

Log fs changes

ChangeStream ePipe

ElasticSearch

Store X

Store Y

HopsFS

Log fs changes

ChangeStream ePipe

ElasticSearch

Store X

Store Y

HopsFS

Log fs changes

ChangeStream ePipe

ElasticSearch

Store X

Store Y

HopsFS

NDBePipe

HopsFS

Create f1

Append f1

HopsFS

Create f1

Append f1

Epoch1

Create f1Append f1

HopsFS

Create f1

Append f1

Epoch1

Create f1Append f1

HopsFS

Create f1

Append f1Create f2Delete f1

Epoch1

Create f1Append f1

HopsFS

Create f1

Epoch1Epoch2

Create f1Append f1

Create f2Delete f1

HopsFS

Create f1

Epoch1Epoch2

Create f1Append f1

Create f2Delete f1

HopsFS

Create f1

Epoch1Epoch2

Create f1Append f1

Create f2Delete f1

Delete f2

HopsFS

Create f1

Epoch1Epoch2Epoch3

Create f1Append f1

Create f2Delete f1

Delete f2

HopsFS

Create f1

Epoch1Epoch2Epoch3

Create f1Append f1

Create f2Delete f1

Delete f2

Order?Order?

HopsFS

Create f1

Epoch1Epoch2Epoch3

Create f1Append f1

Create f2Delete f1

Delete f2

Order?Order?

HopsFS

Create f1

Epoch1Epoch2Epoch3

Create f1Append f1

Create f2Delete f1

Delete f2

Order?Order?

NDB Ordering Properties

• Property 1: epochs are totally ordered.

• Property 2: Changes within the same transaction happen in the same epoch.

• Property 3: Changes on files are ordered only if they are in different epochs, that is, no ordering is guaranteed within the same epoch

Adding version numbers

HopsFS

Create f1

Epoch1Epoch2Epoch3

Create f1Append f1

Create f2Delete f1

Delete f2

Order?Order?

HopsFS

Create f1

Epoch1Epoch2Epoch3

Create f1Append f1

Create f2Delete f1

Delete f2

Order?Order?

HopsFS

Create f1

Epoch1Epoch2Epoch3

Create f1Append f1

Create f2Delete f1

Delete f2

Order?Order?

, 1, 2

HopsFS

Create f1

Epoch1Epoch2Epoch3

Create f1Append f1

Create f2Delete f1

Delete f2

Order?Order?

, 1, 2

HopsFS

Create f1

Epoch1Epoch2Epoch3

Create f1Append f1

Create f2Delete f1

Delete f2

Order?Order?

, 1, 2

, 1, 3

, 1, 2

, 1, 3

HopsFS

Create f1

Epoch1Epoch2Epoch3

Create f1Append f1

Create f2Delete f1

Delete f2

Order?Order?

, 1, 2

, 1, 3, 2

, 1, 2

, 1, 3

ePipe Ordering Properties

•Property 4 & 5: Version number ensures serializability of changes on the same file/directory within epochs.

•Property 6: The order of changes for different files/directories within the same epoch doesn't matter.

• Low replication lag (~100msec)

• High throughput

Requirements

• Reading/Writing millions of images with high throughput

• Attaching annotations to each image, and then searching using these annotations

Questions?

HopsFS & ePipe - GitHub Pages · 2018-10-03 · HopsFS & ePipe 1 Distributed Computing and...

Documents

Transcript of HopsFS & ePipe - GitHub Pages · 2018-10-03 · HopsFS & ePipe 1 Distributed Computing and...

Seeing Metadata: Representing Digital Collection Metadata

Working with Metadata in ArcGIS Aleta Vienneau. Working with metadata in ArcGIS View metadata Edit metadata Set your metadata style Leverage metadata.

Expert Review Big Metadata, Smart Metadata, and …microblogging.infodocs.eu/wp-content/uploads/2017/12/...Big metadata, smart metadata, and metadata capital are presented as part

Metadata AIXM 5.1 Seminar 12 – 13 December 2011. Metadata 2 What is metadata? Why is metadata relevant to us? How is metadata used in AIXM?

Discovery Metadata -Metadata Standards vs. New Requirements

Working with Metadata in ArcGIS · Esri UC2013 . Technical Workshop . Metadata in ArcGIS • ArcGIS metadata goals • Editing metadata • Setting your metadata style • Leveraging

Metadata Repositories for Interoperable/Shareable Metadata.

What is metadata? Book metadata in product information files Book metadata in digital content files Book metadata carriers Who creates and uses metadata?

Metadata Sandboxing | Education Metadata Meetup

04 Metadata and Metadata Management

Metadata & CMDI CLARIN Component Metadata Infrastructure

From Metadata to MetaDATA

arXiv:1606.01588v2 [cs.DC] 22 Feb 2017 · HopsFS: Scaling Hierarchical File System Metadata Using NewSQL Databases Salman Niazi, Mahmoud Ismail, Seif Haridi, Jim Dowling KTH - Royal

Data Management: Documentation & Metadata Metadata (Structured Documentation)

Metadata @ CSC Michael Moeller Metadata Specialist.

HopsFS: Scaling Hierarchical File System Metadata Using ... - HopsFS: Scaling Hierarchic… · HopsFS: Scaling Hierarchical File System Metadata Using NewSQL Databases Salman Niazi,

Oy Metadata Content j of Metadata. Discovery Access Understanding Levels of Metadata joy of Metadata Metadata Standards Why standards Which standards.

HopsFS: Scaling Hierarchical File System Metadata … Scaling Hierarchical File System Metadata Using NewSQL Databases Salman Niazi, Mahmoud Ismail, Seif Haridi, Jim Dowling KTH -

Metadata lecture 3, metadata schemes

WMO Core Metadata Profile version 1 - WIS browsingwis.wmo.int/2010/metadata/version_1-2/WMO Core Metadata Profile v1... · metadata, i.e. the overall management of the metadata resources.