Europython 2008: DataPortability and Python

Post on 23-Jan-2015

4.527 views 3 download

description

This is the presentation I gave at EuroPython 2008. It introduces people to the concepts of Data Portability and shows some standards which can be of use in that context and explains how to use this in Python.Last but not least I introduce my project pydataportability.

Transcript of Europython 2008: DataPortability and Python

DataPortabilityand Python

Christian Scholz

http://mrtopf.de

twitter.com/mrtopf

Plone

Python

virtual worlds

DataPortabilityhttp://dataportability.org

Video by Michael Picksmashcut-media.com

Policy

Technology

the ingredients

http://openid.com

Single-Sign-On

http://openidenabled.com

YADIS

?http://mrtopf.myopenid.com

?http://mrtopf.myopenid.com

?http://mrtopf.myopenid.com

X-XRDS-Location: http://mrtopf.myopenid.com/xrds

<?xml version="1.0" encoding="UTF-8"?><xrds:XRDS xmlns:xrds="xri://$xrds" xmlns:openid="http://openid.net/xmlns/1.0" xmlns="xri://$xrd*($v*2.0)"> <XRD>

<Service priority="0"> <Type>http://specs.openid.net/auth/2.0/signon</Type> <Type>http://openid.net/sreg/1.0</Type> <Type>http://openid.net/extensions/sreg/1.1</Type> <Type>http://schemas.openid.net/pape/policies/2007/06/phishing-resistant</Type> <Type>http://openid.net/srv/ax/1.0</Type> <URI>http://www.myopenid.com/server</URI> <LocalID>http://mrtopf.myopenid.com/</LocalID> </Service> <Service priority="1"> <Type>http://openid.net/signon/1.1</Type> <Type>http://openid.net/sreg/1.0</Type> <Type>http://openid.net/extensions/sreg/1.1</Type> <Type>http://schemas.openid.net/pape/policies/2007/06/phishing-resistant</Type> <Type>http://openid.net/srv/ax/1.0</Type> <URI>http://www.myopenid.com/server</URI> <openid:Delegate>http://mrtopf.myopenid.com/</openid:Delegate> </Service> <Service priority="2"> <Type>http://openid.net/signon/1.0</Type> <Type>http://openid.net/sreg/1.0</Type> <Type>http://openid.net/extensions/sreg/1.1</Type> <Type>http://schemas.openid.net/pape/policies/2007/06/phishing-resistant</Type> <Type>http://openid.net/srv/ax/1.0</Type> <URI>http://www.myopenid.com/server</URI> <openid:Delegate>http://mrtopf.myopenid.com/</openid:Delegate> </Service> </XRD></xrds:XRDS>

X-XRDS-Location: http://mrtopf.myopenid.com/xrds

X-XRDS-Location: http://mrtopf.myopenid.com/xrds

<Service priority="0"> <Type>http://specs.openid.net/auth/2.0/signon</Type> <Type>http://openid.net/sreg/1.0</Type> <Type>http://openid.net/extensions/sreg/1.1</Type> <Type>http://schemas.openid.net/pape/policies/2007/06/phishing-resistant </Type> <Type>http://openid.net/srv/ax/1.0</Type> <URI>http://www.myopenid.com/server</URI> <LocalID>http://mrtopf.myopenid.com/</LocalID></Service>

XRDSeXtensible Resource Descriptor Sequence

http://oauth.net

Authorization

Service A Service B

Service A Service B

Service A Service B

Service A Service B

Service A Service B

Redirect Redirect

Service A Service B

Redirect Redirect

Token

OAuth Discoveryhttp://oauth.net/discovery/

XRDS-Simplehttp://xrds-simple.net

The discovery protocol provides a way for Consumers to receive

the required configuration data from Service Providers in a

machine-readable format. It creates an extendable framework

for communicating current and future services.

http://foaf-project.org

Friend of a FriendFOAF

foaf:Person

„Christian Scholz“

foaf:name

„male“

foaf:gender

foaf:Person

„Christian Scholz“

foaf:name

„male“

foaf:gender

foaf:Person

„Mary“

foaf:name

foaf:Person

„Christian Scholz“

foaf:name

„male“

foaf:gender

foaf:knows

foaf:Person

„Mary“

foaf:name

<rdf:RDF

xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"

xmlns:foaf="http://xmlns.com/foaf/0.1/">

<foaf:Person rdf:ID="me"> <foaf:name>Christian Scholz</foaf:name>

<foaf:knows>

<foaf:Person> <foaf:name>Mary</foaf:name>

</foaf:Person>

</foaf:knows> </foaf:Person>

</rdf:RDF>

RDFResource Description Framework

http://rdflib.net/

http://microformats.org

Example: XHTML Friends Network (XFN)

Christian Marymet,friend

http://mrtopf.de http://mary.com

Example: XHTML Friends Network (XFN)

Christian Marymet,friend

http://mrtopf.de http://mary.com

<a href=“http://mary.com“>Mary</a>

Example: XHTML Friends Network (XFN)

Christian Marymet,friend

http://mrtopf.de http://mary.com

<a href=“http://mary.com“ rel=“met, friend“>Mary</a>

Example: XHTML Friends Network (XFN)

Christian Christianme

http://mrtopf.de http://seesmic.com/mrtopf

Example: XHTML Friends Network (XFN)

Christian Christianme

http://mrtopf.de http://seesmic.com/mrtopf

<a href=“http://seesmic.com/mrtopf“ rel=“me“>Video</a>

<DIV class="intro vcard" style=""> <A class="url uid" href="/mrtopf/"> <IMG class="user photo" src="2278_large.jpg"/> </A> <H1 class="fn nickname">Christian S.</H1> <P class="description adr">36-year-old gentleman from

<SPAN class="locality">Aachen, Germany</SPAN><SPAN class="country-name">DE</SPAN>

</P> <P class="description note">

Tao Takashi in Second Lifemrtopf on the internet, IRC, TwitterPython/Zope/Plone programmer

</P></DIV>

Example: hCard (VCard)

Example: hCard (VCard)

<DIV class="intro vcard" style=""> <A class="url uid" href="/mrtopf/"> <IMG class="user photo" src="2278_large.jpg"/> </A> <H1 class="fn nickname">Christian S.</H1> <P class="description adr">36-year-old gentleman from

<SPAN class="locality">Aachen, Germany</SPAN><SPAN class="country-name">DE</SPAN>

</P> <P class="description note">

Tao Takashi in Second Lifemrtopf on the internet, IRC, TwitterPython/Zope/Plone programmer

</P></DIV>

in Python

Almost Universal Microformats Parserhttp://code.google.com/p/aump/

pydataportabilityhttp://pydataportability.net/

whereare my

services

?

Service Catalogue

Service Catalogue

http://xrds-simple.net

<XRDS xmlns="xri://$xrds"> <XRD xmlns:simple="http://xrds-simple.net/core/1.0" xmlns="xri://$XRD*($v*2.0)" version="2.0">

<Type>xri://$xrds*simple</Type>

<Service priority="10"> <Type>http://specs.example.com/wish_list/1.0</Type> <URI simple:httpMethod="GET">http://books.example.com/wishlist</URI> <LocalID>jane</LocalID> </Service>

<Service priority="20"> <Type>http://specs.example.com/wish_list/1.0</Type> <URI simple:httpMethod="GET" priority="10">https://dvds.example.org/lists/wishes</URI> <URI simple:httpMethod="GET" priority="20">http://dvds.example.org/lists/wishes</URI> <LocalID>janedoe</LocalID> </Service>

</XRD></XRDS>

<Service priority="10">

<Type>http://specs.example.com/wish_list/1.0</Type>

<URI simple:httpMethod="GET">

http://books.example.com/wishlist</URI>

<LocalID>jane</LocalID>

</Service>

Weaving itall together!

Weaving itall together!

goals

goals

Access

goals

Access

Discovery

goals

Access

Discovery

Conversion

from pydataportability.microformats.base.interfaces

from pydataportability.microformats.base.interfaces import IHTMLParser

from pydataportability.microformats.base.interfaces import IHTMLParser

parser = getUtility(IHTMLParser,name="beautifulsoup")()

from pydataportability.microformats.base.interfaces import IHTMLParser

parser = getUtility(IHTMLParser,name="beautifulsoup")()mf = parser.fromURL("http://twitter.com/mrtopf")

from pydataportability.microformats.base.interfaces import IHTMLParser

parser = getUtility(IHTMLParser,name="beautifulsoup")()mf = parser.fromURL("http://twitter.com/mrtopf")mf.parse()

from pydataportability.microformats.base.interfaces import IHTMLParser

parser = getUtility(IHTMLParser,name="beautifulsoup")()mf = parser.fromURL("http://twitter.com/mrtopf")mf.parse()

for name,result in mf.microformats.items():

from pydataportability.microformats.base.interfaces import IHTMLParser

parser = getUtility(IHTMLParser,name="beautifulsoup")()mf = parser.fromURL("http://twitter.com/mrtopf")mf.parse()

for name,result in mf.microformats.items():print "Parser: %s" %name

from pydataportability.microformats.base.interfaces import IHTMLParser

parser = getUtility(IHTMLParser,name="beautifulsoup")()mf = parser.fromURL("http://twitter.com/mrtopf")mf.parse()

for name,result in mf.microformats.items():print "Parser: %s" %name

for r in result:

from pydataportability.microformats.base.interfaces import IHTMLParser

parser = getUtility(IHTMLParser,name="beautifulsoup")()mf = parser.fromURL("http://twitter.com/mrtopf")mf.parse()

for name,result in mf.microformats.items():print "Parser: %s" %name

for r in result: print r

Parser: xfnXFN Relationships for http://mrtopf.de/connect: meXFN Relationships for http://twitter.com/jack: contactXFN Relationships for http://twitter.com/biz: contact...

Parser: hcardVCard for Christian Scholz (fn=Christian Scholz)VCard for None (fn=None, url=http://twitter.com/jack) VCard for None (fn=None, url=http://twitter.com/biz) ...

<XRDS xmlns="xri://$xrds"> <XRD xmlns:simple="http://xrds-simple.net/core/1.0" xmlns="xri://$XRD*($v*2.0)" version="2.0">

<Type>xri://$xrds*simple</Type>

<Service priority="10"> <Type>http://specs.example.com/wish_list/1.0</Type> <URI simple:httpMethod="GET">http://books.example.com/wishlist</URI> <LocalID>jane</LocalID> </Service>

<Service priority="20"> <Type>http://specs.example.com/wish_list/1.0</Type> <URI simple:httpMethod="GET" priority="10">https://dvds.example.org/lists/wishes</URI> <URI simple:httpMethod="GET" priority="20">http://dvds.example.org/lists/wishes</URI> <LocalID>janedoe</LocalID> </Service>

</XRD></XRDS>

<XRDS xmlns="xri://$xrds"> <XRD xmlns:simple="http://xrds-simple.net/core/1.0" xmlns="xri://$XRD*($v*2.0)" version="2.0">

<Type>xri://$xrds*simple</Type>

<Service priority="10"> <Type>http://specs.example.com/wish_list/1.0</Type> <URI simple:httpMethod="GET">http://books.example.com/wishlist</URI> <LocalID>jane</LocalID> </Service>

<Service priority="20"> <Type>http://specs.example.com/wish_list/1.0</Type> <URI simple:httpMethod="GET" priority="10">https://dvds.example.org/lists/wishes</URI> <URI simple:httpMethod="GET" priority="20">http://dvds.example.org/lists/wishes</URI> <LocalID>janedoe</LocalID> </Service>

</XRD></XRDS>

from pydataportability.xrds.parser import XRDSParser

from pydataportability.xrds.parser import XRDSParser

fp = urllib2.urlopen(url)

from pydataportability.xrds.parser import XRDSParser

fp = urllib2.urlopen(url)p = XRDSParser(fp)

from pydataportability.xrds.parser import XRDSParser

fp = urllib2.urlopen(url)p = XRDSParser(fp)

for s in p.services:

from pydataportability.xrds.parser import XRDSParser

fp = urllib2.urlopen(url)p = XRDSParser(fp)

for s in p.services:print "Type:",s.type

from pydataportability.xrds.parser import XRDSParser

fp = urllib2.urlopen(url)p = XRDSParser(fp)

for s in p.services:print "Type:",s.typeprint "Prio:",s.priority

from pydataportability.xrds.parser import XRDSParser

fp = urllib2.urlopen(url)p = XRDSParser(fp)

for s in p.services:print "Type:",s.typeprint "Prio:",s.priorityprint "LocalID:",s.localid

from pydataportability.xrds.parser import XRDSParser

fp = urllib2.urlopen(url)p = XRDSParser(fp)

for s in p.services:print "Type:",s.typeprint "Prio:",s.priorityprint "LocalID:",s.localid for uri in s.uris:

from pydataportability.xrds.parser import XRDSParser

fp = urllib2.urlopen(url)p = XRDSParser(fp)

for s in p.services:print "Type:",s.typeprint "Prio:",s.priorityprint "LocalID:",s.localid for uri in s.uris: print " ",uri.uri

Type: http://specs.example.com/wish_list/1.0Prio: 10LocalID: jane http://books.example.com/wishlist

Type: http://specs.example.com/wish_list/1.0Prio: 20LocalID: janedoe https://dvds.example.org/lists/wishes http://dvds.example.org/lists/wishes

XRDS

HTML

XRDS

hCard

XFN

HTML

XRDS

hCard

XFN

XRDS

FOAF

HTML

XRDS

hCard

XFN

XRDS

FOAF

meta/header

HTML

from pydataportability.discovery import discover

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

>>> services

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

>>> services[<Service type 'xfn'>, <Service type 'hcard'>, <Service type 'hcard'>]

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

>>> services[<Service type 'xfn'>, <Service type 'hcard'>, <Service type 'hcard'>]

>>> services[0].service_type

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

>>> services[<Service type 'xfn'>, <Service type 'hcard'>, <Service type 'hcard'>]

>>> services[0].service_type'http://gmpg.org/xfn/11'

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

>>> services[<Service type 'xfn'>, <Service type 'hcard'>, <Service type 'hcard'>]

>>> services[0].service_type'http://gmpg.org/xfn/11'

>>> services['http://gmpg.org/xfn/11']

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

>>> services[<Service type 'xfn'>, <Service type 'hcard'>, <Service type 'hcard'>]

>>> services[0].service_type'http://gmpg.org/xfn/11'

>>> services['http://gmpg.org/xfn/11'][<Service type 'xfn'>]

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

>>> services[<Service type 'xfn'>, <Service type 'hcard'>, <Service type 'hcard'>]

>>> services[0].service_type'http://gmpg.org/xfn/11'

>>> services['http://gmpg.org/xfn/11'][<Service type 'xfn'>]

>>> services[0].payload

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

>>> services[<Service type 'xfn'>, <Service type 'hcard'>, <Service type 'hcard'>]

>>> services[0].service_type'http://gmpg.org/xfn/11'

>>> services['http://gmpg.org/xfn/11'][<Service type 'xfn'>]

>>> services[0].payload[<XFNRelationships at ...>, ...]

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

>>> services[<Service type 'xfn'>, <Service type 'hcard'>, <Service type 'hcard'>]

>>> services[0].service_type'http://gmpg.org/xfn/11'

>>> services['http://gmpg.org/xfn/11'][<Service type 'xfn'>]

>>> services[0].payload[<XFNRelationships at ...>, ...]

>>> services[2].loaded

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

>>> services[<Service type 'xfn'>, <Service type 'hcard'>, <Service type 'hcard'>]

>>> services[0].service_type'http://gmpg.org/xfn/11'

>>> services['http://gmpg.org/xfn/11'][<Service type 'xfn'>]

>>> services[0].payload[<XFNRelationships at ...>, ...]

>>> services[2].loadedFalse

from pydataportability.discovery import discoverservices = discover('http://twitter.com/mrtopf')

>>> len(services)3

>>> services[<Service type 'xfn'>, <Service type 'hcard'>, <Service type 'hcard'>]

>>> services[0].service_type'http://gmpg.org/xfn/11'

>>> services['http://gmpg.org/xfn/11'][<Service type 'xfn'>]

>>> services[0].payload[<XFNRelationships at ...>, ...]

>>> services[2].loadedFalse

>>> services[2].load()

status?

status?

supports hCard/XFN

status?

supports hCard/XFN

supports XRDS

status?

supports hCard/XFN

supports XRDS

supports service discovery

Christian Scholzcs@comlounge.net

Christian Scholz

thanks!cs@comlounge.net