6th dCache WS | Daniel Becker| 18 April 2012 | 1

Daniel Becker6th dCache workshop, Zeuthen, April 18, 2012

The HTTP Federation

6th dCache WS | Daniel Becker| 18 April 2012 | 2

Content

• WLCG federated storage• The catalogue synchronization solution• Redirection using http federation• Redirection on error

6th dCache WS | Daniel Becker| 18 April 2012 | 3

How is WLCG federating storage

LFC

SE

Catalogues

StorageElements

ROOT

Client

store

register

NON-ATOMIC

Problem:Often results in dangling references(and “dark” data)

6th dCache WS | Daniel Becker| 18 April 2012 | 4

Suggested solution: CAT-SYNCEM

I IN

FSO

-RI-

26

16

11

SE1 SEnSE2

SE or exp. catalogue subscribes to the relevant topics (e.g. “Changes”)

SE sends to the appropriate topics (e.g. “Lost”)

Subscribes to the relevant topics (e.g. “Lost”)

SE Sends to the appropriate topics (e.g. “Changes”)

CentralCatalogue

Othercatalogue/SEe.g. ATLASBroker(s)

Stolen fromFabrizio

Propagates “Lost File” events to catalogues

6th dCache WS | Daniel Becker| 18 April 2012 | 5

Good for now

• Is a sufficient solution for now• But there might be better ones

6th dCache WS | Daniel Becker| 18 April 2012 | 6

Our Goal

LFC

storage element

storage element

storage elementROOT

WebDAV

Provide transparent access to data through catalogues, using standard protocols : http(s), WebDAV

Redirection from catalogues to the final data source doesn’t require intermediate steps by the user but is part of the protocol.

6th dCache WS | Daniel Becker| 18 April 2012 | 7

How does it work

LFC

‘dir’ requestOr GET LFN

File list orSURL

SE

User Clicks

Request SURL

Return TURL

Quick Hack Mapping Service

Redirect

Head NodeGET TURL

RedirectGET TURL

Data Pool

DATA

FILE FOUND

LFCRedirect SURL

SE-1get request

redirect

Instead of File Not Found

SE-2get request

DATA

FILE NOT FOUND

NEW

6th dCache WS | Daniel Becker| 18 April 2012 | 8

NEW: Redirection on error

• Global Catalogue• Provides clients with encoded alternatives.• Storage Elements interpret the provided

alternatives.• And create a redirection

6th dCache WS | Daniel Becker| 18 April 2012 | 9

Redirection on error

• Transparent for the client

• Provides feedback to redirectors

• Utilizes HTTP standards only

6th dCache WS | Daniel Becker| 18 April 2012 | 10

Redirection on error

https://<se-replica-1>/sfn/replica/1?rid=1&r1=3,https://<se-replica-3>/sfn/replica/3&r2=5,https://<se-replica-5>/sfn/replica/5&r3=-1,https://<catalog>/lfn/of/file

6th dCache WS | Daniel Becker| 18 April 2012 | 11

Redirection on error

https://<se-replica-3>/sfn/replica/3?rid=3&notfound=1&r1=5,https://<se-replica-5>/sfn/replica/5&r2=-1,https://<catalog>/lfn/of/file

6th dCache WS | Daniel Becker| 18 April 2012 | 12

Redirection on error

https://<se-replica-5>/sfn/replica/5?rid=5&forbidden=3&notfound=1&r1=-1,https://<catalog>/lfn/of/file

6th dCache WS | Daniel Becker| 18 April 2012 | 13

Redirection on error

https://<catalog>/lfn/of/file?forbidden=3&notfound=1,5

6th dCache WS | Daniel Becker| 18 April 2012 | 14

Next Step

GET LFN

Redirect SURL

SE

Request SURL

Return TURL

Quick Hack

Redirect

GET TURL

RedirectGET TURL

Mapping service

Head Node

Data Pool

DATA

LFC

SE SE

BestMatchEnginePortal

CandidateCollection

Engine

Query SE’s for filee.g. http GET (header)

6th dCache WS | Daniel Becker| 18 April 2012 | 15

DEMO @ Hepix (Thursday)

BestMatchEnginePortal

CandidateCollection

Engine

dCache@DESY

DPM@Taipei

Telecom cloud @Germany

GEOIP

6th dCache WS | Daniel Becker| 18 April 2012 | 16

Thanks