Distribute Query Processing over Streaming and Stored Data

A Semantically Enabled Service Architecture for Mashups over Streaming and Stored DataAlasdair J G GrayUniversity of Manchester

Extended Semantic Web Conference 2011How I wished wed picked a shorter title 0Overview of the TalkMotivation: Estuarine FloodingSemantic Sensor Web SemSorWebRequirementsArchitectureSemantic property documentsDemo application: sample mashupJune 20111Semantic service architecture for mashups A. J. G. GraySemantic property documents provide the glue that connects the services in the architecture1Motivation: Estuarine FloodingThe SolentStrait separating the Isle of Wight from English mainlandBusy shipping channel and portsComplex tidal and wave patternsTwo high tides2Semantic service architecture for mashups A. J. G. GrayJune 2011

Strait of water separating Isle of Wight from English mainland

Two high tides -> increased opportunities for getting ships in and out -> better for business2The SolentJune 20113

Images: http://www.wikipedia.com/

Semantic service architecture for mashups A. J. G. GrayDiversity of use:IndustryLivingRecreationConservation areas3Estuarine FloodingFinancial implicationsDamageLoss of businessPersonal factorsEmotional impact

Flood predictionLocationsSeverityRequires correlatingSea-state dataWeather forecastsDetails of sea defencesResponse PlanningEvacuation routesPersonnel deploymentRequires more dataTraffic reportsShippingJune 20114

Image: http://www.metro.co.uk/Semantic service architecture for mashups A. J. G. GrayEnvironmental decision support systemsFlood emergency response4

Flood defences data (database)

Meteorological forecasts

Flood Detection and ResponseDetect overtopping events in the Solent region sea-level > sea-defenceSea-level: sensorsDefence heights: databases

Provide contextual informationWeb feedsOther sources: maps, models5June 2011Real-time sensor data

Wave,Wind,Tide

Semantic service architecture for mashups A. J. G. GrayOther sources:Maps, models,

Overtopping: a wave or tide exceeds the height of the sea defence

Sensor data provides current sea-state conditionsNational Flood and Coastal Defences Database (NFCDD) provides height of sea walls, etc

Contextual DataWeather feed provides predicted wind speed and direction, contextual streaming dataMaps -> contextual visual data

Report data in a form understandable to the user, ontology5Accurate characterisation of conditions that define an eventCorrelation of data of differing modalitiesIntegrating data from heterogeneous data modelsDiscovery of relevant data sourcesPresentation and control of informationJune 20116Sensor Web RequirementsProvide flood risk details of overtopping events in the Solent region with high wind speed observationsProvide flood risk details of overtopping events in the Solent region with high wind speed observationsProvide flood risk details of overtopping events in the Solent region with high wind speed observationsProvide flood risk details of overtopping events in the Solent region with high wind speed observationsProvide flood risk details of overtopping events in the Solent region with high wind speed observationsProvide flood risk details of overtopping events in the Solent region with high wind speed observationsSemantic service architecture for mashups A. J. G. GrayR1: Describe data of interest, system discovers where data is located and how to connect itR2: Identifying overtopping events requires streaming and stored dataR3: Autonomous, pre-existing data sources with own views/expression of dataR4: Identify sources on spatiotemporal and thematic contentR5: GUI and mashups -> evolving needs through a flooding event

Three fold solution:Ontologies for common vocabularyService based architectureSemantic property documents6ApplicationServicesSemantic RegistrySemantic IntegratorData SourceConnectivity BridgeApplicationsConcrete ResourceSemSorWeb ArchitectureInterfacesService MetadataRegistrationDiscoveryIntegrationQueryData AccessSubscriptionNotificationJune 20117Semantic Property DocumentApplication Tier consists of Web Services. The architecture does not talk about applications.Applications will use the architectureConnectivity Bridge part of the service that virtualises a resourceService management are capabilities of a service not a separate service!

Following points remain true:Any service can call any other service!Any service can be called by an applicationMiddleware adds value by using Semantics7Reconcile TerminologySSNSWEETServiceCoastal DefencesOrdnance SurveyAdditional RegionsRoleDOLCE UltraLiteSchemaFOAFUpperExternalSSG4Env infrastructureFlood domainJune 20118Semantic service architecture for mashups A. J. G. GrayProvides common terminology

Reuse of standard ontologies: SSN, DOLCE, SWEET, FOAF, Ordnance SurveyBottom tier domain specific8Sea-State Sensor Data: Channel Coastal Observatory

43 sensors deployed around UK coastMeasuringTides (7)Waves (24)Weather conditions (12)On and off shoreBespoke hardwareFixed functionalityFixed data rateCentral data centreJune 20119

Semantic service architecture for mashups A. J. G. GrayMeasuring: tides, waves, weather.Sense, store and forward, no processing

Data processing only available once the data has reached the data centre

Related data sources: streaming and stored (databases)9Publishing CCO Sensor DataService DescriptionStreaming data serviceDataset DescriptionSpatiotemporal data coverageThematic data coverageTide heightDataset schema descriptionenvdata_sndownpier_tide (ts:int, Observed:float, Tz:float, Hs:float, HMax:float, Tp:float)

[2005, NOW];

envdata_SandownPier_Tide

HMaxJune 2011Semantic service architecture for mashups A. J. G. Gray10Declarations against ontology networkPublished by data sourceStored in registryUsed as connecting glue10Environment Agency: Flood Defences Database (NFCDD)Flood defence detailsType of defenceWallSand banksLocationHeightJune 201111

Semantic service architecture for mashups A. J. G. GrayPublishing NFCDD DataRelational databaselocations (id:int, latitude:decimal, longitude:decimal,location:string, storm_threshold:decimal) Service DescriptionStored data serviceDataset DescriptionSpatiotemporal data coverageThematic data coverageSea defencesJune 2011Semantic service architecture for mashups A. J. G. Gray1213

Flood Web ApplicationSmashing it all togetherSevere Weather Alert!Gale force winds, Boscombe Bay

Flood response manager Large number of data sourcesNeed intuitive application to manage complexity13Relevant Data SourcesSensor/Streaming DataSea-state sensor dataCCOPort authorityWaveNetShippingAISTraffic AlertsRSS feed

Contextual DataBase MapsFlood defense detailsLand usageWeather forecastsFlooding forecasts

June 201114Semantic service architecture for mashups A. J. G. Gray

User Login: Locate Relevant DataUser logs in selecting:RoleRegionTaskValues parameterise registry lookupsSelecting role, region, and data type affects what data is retrieved from the registry15Initial DisplayJune 2011Semantic service architecture for mashups A. J. G. Gray16

Inform user of potential overtopping events.

Pose query to integrated data source.Integrating Sensed and Stored DataWeb ApplicationApplication ServicesIntegratorDQPCCO-WSCCO-StoredGET http:///geojson?interval=xxx&resource=yyy&query=zzzSPARQLExecuteFactory(integrator, query)GenericQueryFactory(snee, pull, query)EPREPRURLJSONGET URLSPARQLResultSetGetStreamItem(int:, )WebRowSetSQLExecute(cco, query)GetStreamItem(cco:, )WebRowSetGetStreamItem(snee:pull:, )WebRowSetGetStreamItem(cco:, )WebRowSetGetStreamItem(snee:pull:, )WebRowSetGetStreamItem(cco:, )WebRowSetJSONGET URLSPARQLResultSetGetStreamItem(int:, )Steps:Pose queryQuery execution commencesApplication polls for data no data yetContinuous query evaluationApplication periodically polls for data

Web application forces Application Services to set up a query to the Integrator.Integrator returns an endpoint reference to the resource for retrieving answers.Application Services returns a URL from which the latest values can be retrievedApplication periodically polls Application Services using the URL for new data which is returned as a JSON array

When the integrator receives the SPARQLExecuteFactory call it initiates a query call to the SNEE-WS, for the example query shown this integrated data from the streaming and stored sources. Periodically the integrator makes request for data items from the last item seen.

When SNEE-WS receives the GenericQueryFactory call, it makes a suitable query call to the stored data service to retrieve the static data. It also starts pulling the relevant streams from the CCO-WS. The GetStreamItem calls from SNEE-WS to CCO-WS are done on a per relevant stream basis and periodically repeated based on the rate of the specific data stream. The SQLExecute call is repeated, probably once a day.

The integrator periodically pulls data from the SNEE-WS. Hopefully based on the declaration of the rate of the answer stream.

Finally, the Web application periodically makes calls to the Application Services using the URL provided in the initial interaction which then pulls the data from the integrator resource.

Note, the periodicity of the polling calls Integrator->SNEE-WS and SNEE-WS->CCO-WS are not determined by responses to service calls. Data is cached at the various intermediary services to make it more instantly available.17Sensor Data

June 2011Semantic service architecture for mashups A. J. G. Gray18Conclusions: Requirements revisitedAccurate characterisation of conditions that define an eventDeclarative queriesCorrelation of data of differing modalitiesQuery evaluation over streaming and storedIntegrating data from heterogeneous data modelsOntology-based access to streaming and stored dataDiscovery of relevant data sourcesSemantic registryPresentation and control of informationApplication services to support, e.g. smash-upsJune 201119Semantic service architecture for mashups A. J. G. GrayAcknowledgementsFor more information: http://www.semsorgrid4env.eu/Demo application: http://www.semsorgrid4env.eu/services/dynamic-demo

June 2011Semantic service architecture for mashups A. J. G. Gray20

Seven partners from: Greece, Spain, England

Manchester focused on data processing and architecture Southampton focused on the rapid application development and channel coastal observatory.NKUA focused on discoveryEMU and Deimos use case driversTechIdeas: technical infrastructure

20Application TierMiddleware TierData TierApplicationServicesSemantic RegistrySemantic IntegratorData SourceConnectivity BridgeApplicationsConcrete ResourceSemSorWeb ArchitectureJune 201121Semantic Property DocumentApplication Tier consists of Web Services. The architecture does not talk about applications.Applications will use the architectureConnectivity Bridge part of the service that virtualises a resourceService management are capabilities of a service not a separate service!

Following points remain true:Any service can call any other service!Any service can be called by an applicationMiddleware adds value by using Semantics21