NeSS Data Exchange 2.0

Timeline for NeSS Data Exchange

• Started as a pilot called Cafeteria in 2006• Rejuvenated by joint initiative with CLG 2007• Beta April 2008• Release 1.0 December 2008• Release 1.1 April 2009• Release 1.2 November 2009• Release 2.0 February 2010• Eventually to be replaced by WDP API

Cafeteria

• Pilot system built in 2006• Web service layer over NeSS database• Generic NeSS-ML output XML• No “discovery”• Used by iGather (Scisys)• Left on hold until decision to commence

NeSS Interoperability project which was part-funded by CLG

4

CLG’s Future Vision: The “Data Web”

Places dB

DCSF DCLG ONS AuditCommission

LA

LSP

LA

LSP

LA

LSP

Internet

Hubs

LA

LSP

CentralServices

Reg Obs

MethodsSystem

toSystem

GSi

OGDs Inspectorates

Author : M Stone v 1.6 30/01/08

NDE’s Role

• To supply CLG’s Data Interchange Hub• To support other Local Information Systems• National usage secondary consideration• Web 2.0 apps also secondary consideration• Service can be used live - “hunter”• Or to accumulate data - “gatherer”

Hunter Gatherer

NESS DATASTORE

NeSS Data Exchange

Client Application

NESS DATASTORE

NeSS Data Exchange

Data Collector ‘Module’

LIS Datastore

Client Application

Solution A: ‘Hunter’ Solution B: ‘Gatherer’

Web Site vs. Web Service

Browser Web Site Data Access Component

DB

Browser

SpreadSheet

Other

AJAX

ExcelMacro

Java.Net etc.

NDEWebService

DataAccessComponent

DB

Fig. 1 Web Site

Fig. 2 Web Service

XML Formats

• Looked at SDMX – too complex, problems with geography, output hard to consume

• Also considered keeping NeSS-ML but it was considered too abstract and generic to be easily consumed

• Went with LGDX (was PIE) to be consistent with CLG hub – also simple and rigid so easy to consume. Only downer is it cannot support cross-tabs.

• Retained NeSS-ML for internal use and potential transform to other formats in future.

Discovery

• Half of NDE is the Discovery Service• This allows you to ask questions of the NeSS

database. Often this will be a string of queries…. • Get list of subjects• User selects “Health”• Get datasets for subject “Health”• User selects “General Health UV20”• Get variables for dataset UV20• User clicks i button next to “Not Good Health”• Get metadata for this variable

Delivery

• The other half is the Delivery Service• This delivers the data payload as a self-

documenting chunk of XML. • Initially this required a complex request to be

built up from discovery calls.• Simplified in Release 1.1• Initially only LDGX format• SDMX and RDF added in Release 1.2

Clients

• ONS supplied sample clients to show how the service can be used with different languages

• Some interest from commercial companies to supply “middleware”

• Access initially via SOAP only (REST-like version to follow)

• Authenticated access via manual registration only. Valuable at first but not seen as so important now.

Sample Client (Excel and SVG)

Pros and Cons of NDE

• POWERFUL – everything the NeSS website can do you can do with NDE, including useful geographical functions

• COMPLEX – big manual, many operations, understanding of NeSS database required

• RELIABLE – 99.6% uptime• INCOMPATIBLE – with some newer systems,

notably Adobe Flex and Ruby

Where are We Now?

• NDE has had generated only moderate interest so far, and there are currently no live applications using it as far as we know (though SASPAC beta looks promising).

• Versions 1.2 and 2.0 look to redress some of the “Cons” and provide alternative output formats (corporate requirement, not from NeSS)

• Small budget for improvements but committed to continued support for foreseeable future.

• NDE is proving to be a useful toe in the water for the WDP API that will eventually replace it.

What’s New?

• VERSION 1.2• Adobe Flex support (+demo)• SDMX output• RDF output (experimental)

• VERSION 2.0• Lite Service

Adobe Flex

• Premier Data Visualisation Tool• NDE 1.1 does not have the correct

configuration to allow Flex applications to call it.

• Changes made to 1.2 to make it work.• Demonstration “dashboard” application

supplied.• Uses SOAP service, will supply a REST-like

version with 2.0

Client Screenshot

SDMX

• International Data Interchange Format• Being adopted by ONS, NOMIS and many

overseas organisations such as Eurostat• Can comprehensively model all data• Flexible and Complex• Generated on-the-fly by NDE• Message contains definitions and data• Harder to consume than LGDX

Sample Output sdmx_sample2.xml

RDF

• Building block for the Semantic Web• Triples consisting of subject, predicate and

object• Format favoured for Tim Berners-Lee’s UK

Public Data Project• Generated on-the-fly by NDE• Experimental format based on SCOVO• Could supply complete geographic

hierarchies like OS Parishes

RDF Example

• Alan Pardew (subject) is manager of (predicate) Southampton FC (object).

• Markus Liebherr (subject) is chairman of (predicate) Southampton FC (object).

• Using this method everything on the www is interlinked

• Good for metadata• Possibly too verbose for large datasets –

NeSS has about 10,000,000,000 triples.Sample file ness_transformed_to_rdf.xml

Lite Service

• Simplified, fewer operations

• REST or SOAP• Low security• Optional registration• Aimed at web 2.0

and other lightweight users

• Possibility for reduced functionality

Simplified Operations

• Hide some of the complexities of the NeSS database

• Cut down the perceived number of operations by adding a few more!

• Split into “Core” and “Advanced” operations• Rationalise the area search into a single

operation• Make the user documentation less daunting

SOAP Web Service

• The Simple Object Access Protocol• Standard for web services for many years• Usually operates via an HTTP POST • Operations defined via a WSDL document• Many languages such as Java and VB can

use the WSDL to create objects• SOAP can also be used for AJAX

applications via XMLHTTP with raw XML but it is clunky

REST Web Service (HTTP GET)

• Not RESTful by the textbook definition, but calls are made via a simple URL, e.g.

• http://www.neighbourhood.statistics.gov.uk/interop/NeSSLiteDiscoBindingPort/GetAreaChildren?Areaid=565457

• Can only accept simple parameters (strings and numbers)

• No auto-generation of objects from WSDL• Preferred to SOAP for many modern web

platforms

Security: Full Service

Discovery Endpoint

Delivery Endpoint

SOAP HeaderWS-SecurityUserID Password

SOAP Body

SOAP Body

SOAP HeaderWS-SecurityUserID Password

BasicAuthentication

BasicAuthentication

NeSS Login Module

BOM

XRender

OPERATION

OPERATION

Request

Request

HTTPS

HTTPS

Security: Lite Service

URI

URI

Lite Discovery Endpoint

Lite Delivery Endpoint

OPERATION

OPERATION

BOM XRender

Request

Request

HTTPS

HTTPS

NDE Lite - Other

• Despite no need to register, users can still sign up for notifications and tech support

• There will be no loss of functionality except for the complex getDataCube function

• Current plan is to run NDE Lite as a separate application in parallel with the current service.

• Turn off old service once users have had plenty of time to migrate