Design of an Evaluation Framework for Open Web Data Applications

7/29/2019 Design of an Evaluation Framework for Open Web Data Applications

1/48

Mo#va#on

DataontheWeb

Someeyecatchingopenerillustra#nggrowthandordiversityofwebdata

25/05/12 1Stefan Dietze

DesignofanEvalua#onFramework

forOpenWebDataApplica#ons

HendrikDrachsler,SlaviStoyanov

(TheOpenUniversityoftheNetherlands,NL):WP2Lead

25 February 2013


2/48

LinkedUpVision

2

Adopting open Web data, educational and cross-domain resources to

provide educational services up to university degree-level in open anddistance education.

1) A domain-specific

(educational) set ofcriteria

2) A generic set of criteria

for the evaluation of openWeb data applications.

There is a substantial risk that the evaluation criteria are not

adequately specified."

At what point are the criteria adequately specified?When the stakeholders agree with the criteria.

25 February 2013Hendrik Drachsler


3/48

DevelopmentoftheEvalua#onFramework

3

P1: Initialisation P2: Establishment

and Evaluation

P3: Exit and

Sustainability

M0-M6: Preparation M7-M18: Competition cycle M18-M24: Finalising

Literature review

Cognitive Mapping

Group Concept

Mapping

Documentation

Dissemination

EF proposal Expertvalidation

Final

releaseof EF

Competition

Revie

w ofEF

Refinementof EF

Newversio

n

Draft

3x

Practical experiences

and refinement



4/48

DevelopmentoftheEvalua#onFramework



5/48

LinkedUp



6/48

GroupConceptMapping

6

Group Concept Mapping resembles thePost-it notes problem solving technique

and Delphi method

GCM involves participants in a fewsimple activities (generating, sorting

and rating of ideas) that most people are

used to.

GCM is different in two substantial ways:

1. Robust analysis (MDS and HCA)

GCM takes up the original participants contribution and then quantitatively

aggregate it to show their collective view (as thematic clusters)

2. VisualisationGCM presents the results from the analysis as conceptual maps and other graphical

representations (pattern matching and go-zones).



7/48

GroupConceptMapping

25 February 2013 7

innovations in way network is delivered (investigate) corporate/structural alignment assist in the development of non-traditional partnerships (Rehab with the

Medicine Community) expand investigation and knowledge of PSN'S/PSO's continue STHCS sponsored forums on public health issues (medicine

managed care forum) inventory assets of all participating agencies (providers, Venn Diagrams) access additional funds for telemedicine expansion better utilization of current technological bridge continued support by STHCS to member facilities expand and encourage utilization of interface programs to strengthen the

viability and to improve the health care delivery system (ie teleconference)

discussion with CCHN

...organize the

issues...

brainstorm

Work quicklyand effectivelyunder pressure

49

Organize the workwhen directions are

not specific.39

Decide how tomanage multiple

tasks.20 Manage resources effectively.

4

sort

12

34

5

12

34

5

3Sc

anamu

ltitud

eofin

forma

tiona

nd

decid

ewha

tisim

porta

nt.

12

34

5

12

34

5

12

34

5

12

34

5

12

34

5

1Ma

nageti

meeff

ective

ly

2Ma

nagere

source

seffec

tively.

3Sc

anam

ultitud

eofin

formatio

n

andde

cidew

hatisimp

ortant.

4De

cideh

owto

manag

emulti

pletasks.

5Or

ganize

thewo

rkwhen

direction

saren

otspec

ific.

1

Manag

etime

effective

ly

Ratin

gShe

et

rateHendrik Drachsler


8/48

GroupConceptMapping



9/48

GroupConceptMapping



10/48

GroupConceptMapping



11/48

GroupConceptMapping

11






Work quickly andeffectively under

pressure49

Organize the workwhen directions are not

specific.39



4

sort

12

34

5

12

34

5

3Sc

anam

ultitud

eofin

forma

tiona

nd

decide

whatisi

mport

ant.

12

34

5

12

34

5

12

34

5

1 2

34

5

12

34

5

1Mana

getim

eeffec

tively

2Ma

nager

esourc

eseff

ective

ly.

3Scan

amulti

tudeo

finfor

mation

and

decide

whati

simportan

t.

4De

cideh

owtoma

nagem

ultiple

tasks

.

5Orga

nizeth

ework

when

direction

saren

otspec

ific.

1Ma

nageti

meeffective

ly

Ratin

gShe

et

rate

organize

Management

Financing

Regionalization

STHCS as model

Community & Consumer Views

Information ServicesTechnology

map the issues...



12/48

GroupConceptMapping

12

ManagementFinancing

Regionalization

Mission & Ideology


Information Services Technology



managed care forum) inventory assets of all participating agencies (providers, Venn Diagrams) access additional funds for telemedicine expansion better utilization of current technological bridge continued support by STHCS to member facilities expand and encourage utilization of interface programs to strengthen theviability and to improve the health care delivery system (ie teleconference) discussion with CCHN


pressure49


specific.39



4

sort

12

34

5

12

34

5

3Sc

anam

ultitud

eofin

forma

tiona

nd

decide

whatisi

mport

ant.

12

34

5

12

34

5

12

34

5

1 2

34

5

12

34

5

1Mana

getim

eeffec

tively

2Ma

nager

esourc

eseff

ective

ly.

3Scan

amulti

tudeo

finfor

mation

and

decide

whati

simportan

t.

4De

cideh

owtoma

nagem

ultiple

tasks

.

5Orga

nizeth

ework

when

direction

saren

otspec

ific.

1Ma

nageti

meeffective

ly

Ratin

gShe

et

rate

organize

Management

Financing

Regionalization

STHCS as model



map

...prioritize the issues...



13/48

GroupConceptMapping

13

WP2 Progress

Invited 122 external experts 56 experts contributed 212 indicators for the

evaluation framework

After cleaning -> 108 indicators remained 26 experts sorted on similarity in meaning 26 experts rated on priority and

applicability



14/48

GroupConceptMapping

14

A point map



15/48

GroupConceptMapping

15

A cluster map 15



16/48

GroupConceptMapping

16

A cluster map 6



17/48

GroupConceptMapping

17

Clusters labels



18/48

GroupConceptMapping

18

Rating Map Priority



19/48

GroupConceptMapping

19

Rating Map Applicability



20/48

GroupConceptMapping



21/48

GroupConceptMapping



22/48


This silde is available at:http://www.slideshare.com/Drachsler

Email: [email protected]

Skype: celstec-hendrik.drachsler

Blogging at: http://www.drachsler.de

Twittering at: http://twitter.com/HDrachsler

Many thanks for your attention!


23/48

Practical experiences and refinement


Competitions

Reviewof EF

Refinement

of EF

Newversion

Draft

3x


24/48

Networkofsupportingorganisations

(see 3.2Spreadingexcellence,exploitingresults,disseminatingknowledge)

Dissemination(events, training)

Datasharinginitiatives

Communitybuilding&clustering

Technologytransfer

Cashpriceawards&consulting

Initialisation

Personal

data

Webdata

LinkedUp

submissiondata

LinkedUp Support Actions

LinkedUp ChallengeEnvironment

LinkedUp EvaluationFramework

Methods and TestCases

LinkedUp DataTestbed

Competitor ranking list

P

PS

F

EI

B

OC

C

S

E

T

3stagesoftheLinkedUp competition

Stage1-Initialisation

Stage2

Lowestrequirementslevelforparticipation

Initalprototypesandmockups,useofdatatestbedrequired

10to20projectsareexpected

Stage3

Mediumrequirementslevelforparticipation

Workingprototypes,minimumamount of

datasources,cleartargetusergroup


Stage4

Deploymentinreal-world usecases

Sustainabletechnologies,reaching out

to criticalamountofusers,


Participation

criteria

Challengeandevalua#onframeworkaimedat:

Technologicalleapinrobustness/scalabilityof(Big)dataintegraontechnologies(dataanalycs,mining,storage,analysis)

Real-worldusecasescenario:paradigmshiIinWeb-basededucaonbyexploingopenWebdata

LinkedUpinanutshell

LinkedUpinanutshell


Challenge


25/48

ManagementFinancing

Regionalization

Mission & Ideology









pressure49


specific.39



4

sort

12

34

5

12

34

5

3Sc

anam

ultitud

eofin

forma

tiona

nd

decide

whatisi

mport

ant.

12

34

5

12

34

5

12

34

5

12

34

5

12

34

5

1Mana

getim

eeffec

tively

2Ma

nager

esourc

eseff

ective

ly.

3Scan

amulti

tudeo

finfor

mation

and

decide

whati

simportan

t.

4De

cideh

owtoma

nage

multip

letask

s.

5Orga

nizeth

ework

when

direction

saren

otspec

ific.

1Ma

nageti

meeffective

ly

Ratin

gShe

et

rate

brainstorm

organize

Management

Financing

Regionalization

STHCS as model



mapr = .72

Counties 1 & 2 County 3

4.23

3.55

4.4

3.56


Management

Information Services

Regionalization

Technology

Financing

Mission & Ideology

Technology


Information Services

Management

Regionalization

Financing

Mission & Ideology

...examine consensus...

prioritize


26/48

Cogni#veMapping


1. Literature review of suitable evaluation approaches and criteria

2. Review of comprising initiatives such as LinkedEducation, MULCE, E3FPLEand

the SIG dataTEL


27/48

r = .72

4.23

3.55

4.4

3.56







pressure49


specific.39



4

sort

12

34

5

12

34

5

3Sc

anam

ultitud

eofin

forma

tiona

nd

decide

whatisi

mport

ant.

12

34

5

12

34

5

12

34

5

12

34

5

1 2

34

5

1Mana

getim

eeffec

tively

2Ma

nager

esourc

eseff

ective

ly.

3Scan

amulti

tudeo

finfor

mation

and

decide

whati

simportan

t.

4De

cideh

owtoma

nage

multip

letask

s.

5Orga

nizeth

ework

when

direction

saren

otspec

ific.

1Ma

nageti

meeffective

ly

Ratin

gShe

et

rate

brainstorm

organize

Management

Financing

Regionalization

STHCS as model



map

ManagementFinancing

Regionalization

Mission & Ideology



prioritize

2.471.24.8

1.8

5

Importance

Feasibility

2.61

10172225

3334 37

45

62

7576

and drill back down todetails for prioritizing action


28/48

LinkedUpinanutshell

LinkedUpinanutshell

Networkofsupportingorganisations

(see 3.2Spreadingexcellence,exploitingresults,disseminatingknowledge)

Dissemination(events, training)

Datasharinginitiatives

Communitybuilding&clustering

Technologytransfer

Cashpriceawards&consulting

Initialisation

Personal

data

Webdata

LinkedUp

submissiondata

LinkedUp Support Actions

LinkedUp ChallengeEnvironment

LinkedUp EvaluationFramework

Methods and TestCases

LinkedUp DataTestbed

Competitor ranking list

P

PS

F

EI

B

OC

C

S

E

T

3stagesoftheLinkedUp competition

Stage1-Initialisation

Stage2

Lowestrequirementslevelforparticipation

Initalprototypesandmockups,useofdatatestbedrequired


Stage3

Mediumrequirementslevelforparticipation

Workingprototypes,minimumamount of

datasources,cleartargetusergroup


Stage4

Deploymentinreal-world usecases

Sustainabletechnologies,reaching out

to criticalamountofusers,


Participation

criteria


Challenge

Webdata

LinkedOpendata(30+billionstatements)

GeneralWebdata(OA-PMHfeeds,

webmetadataetc)

Datamanagementtools

Storage Analycs Mining ntegraon,mapping

Educa#onaldata&resources

OERmetadata OpenLearn OpenCourseware Ariadne iTunesU EUprojectresults


29/48


Dataintegraon,Webtechnologies&evaluaon

DisseminaonandexploitaonofopenWebdata

Educaonaltechnologies,(meta)dataandresources

Scienficexcellencein

threestrategicareas:

LinkedUpconsor#um

Dataintegraon,Webtechnologoies,evaluaon


30/48

L3SResearchCenter,LeibnizUniversity,DE

LeadinginstuteinWebscience&datatechnologiesaswellastechnology-enhancedlearning

StrongexperienceincoordinangECR&Dprojects

KMI,TheOpenUniversity,UK

LeadingR&DinstuteinareasrelatedtoLinkedUp Worldslargestdistanceuniversity(over200.000

students)

LinkedUpconsor#um

Dataintegraon,Webtechnologoies,evaluaon


31/48

LinkedUpconsor#um

Educaonaltechnologies,data&resources






students)

CELSTEC,TheOpenUniversity,NL

R&Dinstuteineducaonaltechnologiesandpartofthelargestdistanceuniversityinthenetherlands

ExactLearningSolu#ons,IT

SMEineducaonaltechnologiesandserviceswithlong-standingexperiencein(EC-funded)R&Dprojects

Elsevier,NL

Leadingscienfic&educaonalpublisher nnovaveresearchonthefutureofpublishing&

extensiveexperienceindatacompeons


32/48

LinkedUpconsor#um

Disseminaon&exploitaonofWebdata






students)

CELSTEC,TheOpenUniversity,NL

R&Dinstuteineducaonaltechnologiesandpartofthelargestdistanceuniversityinthenetherlands

ExactLearningSolu#ons,IT

SMEineducaonaltechnologiesandserviceswithlong-standingexperiencein(EC-funded)R&Dprojects

Elsevier,NL

Leadingscienfic&educaonalpublisher nnovaveresearchonthefutureofpublishing&

extensiveexperienceindatacompeons

TheOpenKnowledgeFounda#on,UK

Not-forprofitorganisaontopromoteopenknowledgeanddata;globalnetwork

Hostofkeyevents(OKCon)andpla_orms(egCKAN)


33/48

ECPOKKAM:Webenty

idenficaon&discovery

ECBPNmEducator:

ntegraonofeducaonal

resourcesbasedonLOD

ECSTREPLUSA:Semanc

Webtechnologiesfor

sharingofOER

ECNoESTELLAR:

educaonalWeb

technologiesnetwork

OpenScout:promoonof

useofopeneducaonal

content

LinkedUpconsor#um

Collaboraons:R&Dprojects



34/48

ECPOKKAM:Webenty

idenficaon&discovery

ECBPNmEducator:

ntegraonofeducaonal

resourcesbasedonLOD

ECSTREPLUSA:Semanc

WebtechnologiesforsharingofOER

ECNoESTELLAR:

educaonalWeb

technologiesnetwork

OpenScout:promoonof

useofopeneducaonal

content

LinkedUpconsor#um

Collaboraons:R&Dprojects&events/iniaves


LLE:LinkedLearning(LinkedDatafor

Educaon)workshopseries

LALD:LearningAnalycsandLinked

Dataworkshop(series)

LinkedEducaonh`p://linkededucaon.org

LinkedUniversies

h`p://linkeduniversies.org

JointspecialissuesrelatedtoLinkedUp

(SemancWebJournalandLE)

EuropeanAssociaonforTechnology-enhancedLearning(EATEL)


35/48

LinkedUp

OtherrelatediniavesfromLinkedUppartners

Large-scalechallenges&compe##ons

OpenDataChallenge(h`p://opendatachallenge.org/):Europes

largestopendatacompeon,430

submissionsfrom24memberstates

ElsevierGrandChallenge(h`p://www.elseviergrandchallenge.com):

communicaonofscienficinformaon.

Seman#cWebChallenge(h`p://challenge.semancweb.org/)large-scaleSemancWebdataapplicaons

Seman#cWebServiceChallenge(h`p://sws-challenge.org):evaluaonof

semancwebservicetechnologies



36/48

LinkedUp











Webdatadissemina#onandevents

TheOpenKnowledgeConference(OKCon):annualopenknowledgeconferencerunby

OKFN

DataTELthemeteam:gatheringofopendatawithineducaon(OUNL)

OpenGovernmentDataCamp:h`p://ogdcamp.org/

OpenDataHandboookh`p://opendatahandbook.org/:livingonlinemanualforbasicconceptsofopendata

Topicalworkinggroupsandhackdays,egh`p://okfn.org/wg/



37/48

LinkedUp











Webdatadissemina#onandevents

TheOpenKnowledgeConference(OKCon):annualopenknowledgeconferencerunby

OKFN

DataTELthemeteam:gatheringofopendatawithineducaon(OUNL)

OpenGovernmentDataCamp:h`p://ogdcamp.org/

OpenDataHandboookh`p://opendatahandbook.org/:livingonlinemanualforbasicconceptsofopendata

Topicalworkinggroupsandhackdays,egh`p://okfn.org/wg/

Datacatalogues&(educa#onal)datasets

CKAN:CKANistheComprehensiveKnowledgeArchiveNetwork,themostimportantregistryofopenknowledgedatasets(hostedandmanagedbyOKFN).

LUCERO,h`p://data.open.ac.uk:firstextensiveLinkedDatauniversitydataset,approachadoptedbymanyuniversies

acrossEurope

mEducatorLinkedEduca#onalresources:oneoffirstOERdatasetsinLinkedDatacloud(LUH,OUUK)



38/48

LinkedUp

Goals

Objective2Evaluation

Frameworkfor

OpenWeb

Data

Applications

Objective1

OpenWeb

DataSuccess

Stories

EducationalWebdata&

technologies

createevaluate

support

create

demonstrate

Objective3Technology

Transferinthe

Education

Sector

demonstrate

support

support

evaluatesupportcreate

demonstrate/support

evaluate



39/48

LinkedUp

Goals=>tangibleoutcomes


Frameworkfor

OpenWeb

Data

Applications

Objective1

OpenWeb

DataSuccess

Stories

EducationalWebdata&

technologies

createevaluate

support

create

demonstrate


Transferinthe

Education

Sector

demonstrate

support

support


demonstrate/support

evaluate

Compe##onframework&community Evalua#onframeworkforlarge-scale

Webdataapplicaons

(metrics,methods,benchmarks)

Large-scaledatatestbedofevaluateddatasets

Cross-domainandapplicaondomain-specificaspects



40/48

LinkedUp



Frameworkfor

OpenWeb

Data

Applications

Objective1

OpenWeb

DataSuccess

Stories

EducationalWebdata&

technologies

createevaluate

support

create

demonstrate


Transferinthe

Education

Sector

demonstrate

support

support


demonstrate/support

evaluate

Highlyinnova#ve,evaluatedapplica#onsoflarge-scaleWebdata

Opencompeon(LinkedUpChallenge)offersincenveandsupporttosteer

submissionstowardstechnologicalleap

Educaonalscenario:(a)challengingvisionand(b)real-worldscenarioand

requirements



41/48

LinkedUp



Frameworkfor

OpenWeb

Data

Applications

Objective1

OpenWeb

DataSuccess

Stories

EducationalWebdata&

technologies

createevaluate

support

create

demonstrate


Transferinthe

Education

Sector

demonstrate

support

support


demonstrate/support

evaluate

Technologytransfer,increaseincollabora#onandawareness(bestprac#ces,clusters/communi#es,

events)

TransferofinnovaveR&Dresults Significantincreaseinawarenessabout

openWebdataanddataintegraon

methodstoproducescalableandrobust

informaonsystemsofglobalrelevance.



42/48

International(outside Europe)

LinkedUp

Exploitaon,disseminaon,sustainability

PersistentLinkedUpNetwork

CommonwealthofLearning,COL(Canada)

AthabascaUniversity(Canada)

SURFNL(Netherlands)

UniversitdeFribourg-Departmentof

ComputerScienceeXascalenfolabGroup

DemocritusUniversityofThrace

AristoteleUniversityofThessaloniki

(Greece)

nstuteforEducaonalTechnologies

NaonalResearchCounciloftaly

ClamMessinaServiceandResearchCentre

Eurix

(extensiblecommunityofindustrialandacademicins#tu#ons)


43/48

LinkedUp



Clustering

Jointclusteringacvieswithrelatedorganisaons(LinkedUp

Network)

andEC-fundedR&Dprojects,suchas

LOD2 ARCOMEM SEALS,etc.

Dissemina#onevents&plaYorms

Showcases&tutorialscollocatedwithrelevantconferences

(WWW,SWC,ESWC,CDE,LAKetc)

Systemdemonstraons Topicalhackdays LLE,LALD,DataTELworkshopseries

(established,persistentandgrowing

communies)

OpenKnowledgeConferenceOKCON LinkedEducaon.org,

LinkedUniversies.org


44/48

LinkedUp



Clustering

Jointclusteringacvieswithrelatedorganisaons(LinkedUp

Network)

andEC-fundedR&Dprojects,suchas

LOD2 ARCOMEM SEALS,etc.

Dissemina#onevents&plaYorms

Showcases&tutorialscollocatedwithrelevantconferences

(WWW,SWC,ESWC,CDE,LAKetc)

Systemdemonstraons Topicalhackdays LLE,LALD,DataTELworkshopseries

(established,persistentandgrowing

communies)

OpenKnowledgeConferenceOKCON LinkedEducaon.org,

LinkedUniversies.org

Standardisa#on

Parcipaon/supportofstandardisaonofschemasandtechnologiesthroughworking

groups(egh`p://okfn.org/wg/)

Datacatalogues(egCKAN)andcommunitydataportals(egh`p://bibsoup.net/)

Standardisaoniniavesandworkinggroups(egCreaveCommonsLRM)

Viraldissemina#onchannels SharingofpublicaonsviaMendeley,

ResearchGate,CiteULike,

Academia.edu

Adversementofslides,showcasesanddemovideosonSlideshare,

Youtube,Videolectures.net,Vimeo

SocialnetworkchannelssuchasTwi`er,Linkedn

SourcecodesharingviaSourceForge Useofopenlicensingschemes(CC)


45/48

LinkedUp

Workplan:milestones

List and schedule of milestones

Milestone

no.

Milestone name WPs. L e a d

participant

Expected

Date

Means of verification

MS1 Competition and evaluationframework established, challenge

designed and initiated, data

testbed provided.

WP1,WP2,

WP3,

WP4,

WP6

LUH M6 Challenge design developed (D1.1delivered), competition and evaluation

framework established; initial calls

published, initial dissemination actions

conducted. D3.1a delivered; Critical

amount of data consumers reached.

MS2 Initial cluster and communityestablished

WP1,

WP4

OKFN M6 D4.1 delivered. Initial dissemination andclustering events organised. Minimum

amount of subscribers to LinkedUp

newsletters and websites.

MS3 Challenge submissions obtainedand selected.

WP1,WP2,

WP3

OUNL M12 D2.3a Evaluation results LinkedUpchallenge (first stage) delivered. Initialversion of handbook.

MS4 Applications developed andevaluated.

WP1,

WP2,

WP3,

WP5

OUNL M18 D2.3 (a,b) are delivered, describingevaluation results. Critical amount of

technologies were evaluated and

deployed (stage 3).

MS5 Applications deployed andevaluated in wide network of

institutions; evaluation framework

finalised and published.

WP1,

WP2,

WP3,

WP5

OUNL M24 D1.3 and D2.3(a,b,c) are delivered,describing evaluation results. Critical

amount of technologies were evaluated

and deployed (stage 3). The evaluation

framework has been published andrefined (D2.2b).

MS6 Impact achieved; exit andsustainability strategy

implemented.

WP4,

WP5

ELS M24 Impact has been achieved. Exit andsustainability plan has been developed

(D5.2b) and implemented. Impact report

delivered. Showcases published.


Stage1

Stage2

Stage3

Stage4


46/48

LinkedUp

Managementstructure

IvanaMarenzi(LUH)&ProfWolfgangNejdl(LUH)

DrStefanDietze(LUH)



47/48

LinkedUp

Managementstructure

DrHendrikDrachsler

(OUNL)

DrEelcoHerder

(LUH)

DrMathieudAquin

(OUUK)

DrLauraJames

(OKFN)

DrElisabe`aParodi

(ELS)

DrStefanDietze

(LUH)



48/48

LinkedUp

Nextsteps

Startdate

Prospecvestartdate:1November(1Octoberfeasibleaswell)

Ongoingprepara#onstoenablequickstart

Staffing Staffplanningalreadyinplace(Excelsheettobepresented) Jobopeningspublished(whereapplicable)

Challengedesign,community&clusters(MS1,MS2) UsingestablishedchannelsandeventsofLinkedUppartners

(LinkedUpNetwork)

Website(aheadofprojectstartdate)

Challengedesignandinialisaon:inialcallsexpectedbyFebruary2013

MS1 - Competition &

Evaluation Frameworkestablished, challenge

designed and initiated,data testbed.

D1.1, D1.2, D2.1,D2.2.1, D3.1.1, D3.2.1,D6.1.1

MS2 - Initial clusterand communityestablished

D4.2, D4.3, D4.4.1

(M6)

D4.1

(M3)

Design of an Evaluation Framework for Open Web Data Applications

Documents

Transcript of Design of an Evaluation Framework for Open Web Data Applications