ESA Sentinels Data Hub Service
Transcript of ESA Sentinels Data Hub Service
Serco Business
Copernicus Sentinels Data Hub Service
Collaborative Ground Segment WorkshopESRIN, 4-5 December 2018
Serco Business
Data Access Operational StatusOutline
• Data Hub Service Scope• ColHub Operational History• Service utilisation statistics• Service Availability• Service Architectural overview• Main issues and resolutions• Short Term Outlook
Serco Business
Ops Centre2
Ops Centre3
Ops Centre1
Data Access
ReportingS-5P PDGS
S-3 PDGS
S-2 PDGS
S-1 PDGS
DHuS Service
Data Hub Service Scope
Serco Business
Data Hub Service ScopeData Access Services
Copernicus Open Access Hub
Collaborative Data Hub
Copernicus Services Data Hub
International Access Hub
DIAS Access Hub
PrimaryServices
Commissioning Data Hub
Expert Users Data Hub
Pre-operational Data Hub
Temporary Data Access Hub
Demo Hub
Africa Cast Hub
Data Access Services
AncillaryServices
Serco Business
Data Hub Service Operation highlights
Serco Business
Collaborative Data Hub Service Timeline
2015 20192016 2017 2018
26-Jan-15Node 2
18-Apr-16Node 1
14-Mar-18Node 3
Jan-15S1A NTC
Sep-15S1A NRT
May-16S2A
Sep-16S1B
Nov-17S3A SRAL Oct-18
S3A OLCI
Oct-17S2B
DHRColl
2
Coll
9 1
DHRColl
13 5
DHRColl
18 7
DHRColl
15 6
Use
rsG
row
thD
ata
Pub
licat
ions
Acc
ess
Nod
es
Serco Business
0
200
400
600
800
1000
1200
2015
-03
2015
-04
2015
-05
2015
-06
2015
-07
2015
-08
2015
-09
2015
-10
2015
-11
2015
-12
2016
-01
2016
-02
2016
-03
2016
-04
2016
-05
2016
-06
2016
-07
2016
-08
2016
-09
2016
-10
2016
-11
2016
-12
2017
-01
2017
-02
2017
-03
2017
-04
2017
-05
2017
-06
2017
-07
2017
-08
2017
-09
2017
-10
2017
-11
2017
-12
2018
-01
2018
-02
2018
-03
2018
-04
2018
-05
2018
-06
2018
-07
2018
-08
2018
-09
2018
-10
2018
-11
Collaborative Data Hub Service Utilisation Timeline
2015 20192016 2017 2018
26-Jan-15Node 2
18-Apr-16Node 1
14-Mar-18Node 3
Jan-15S1A NTC
Sep-15S1A NRT
May-16S2A
Sep-16S1B
Nov-17S3A SRAL Oct-18
S3A OLCI
Oct-17S2B
[TiB]
DHR
Coll
Downloaded Volumes of Data per month
13 PiB
15 PiB
TotalsSince2015
Serco Business
Collaborative Data Hub Service Availability
Fire in Greece
Hosting site Maintenance
Serco Business
Collaborative Data Hub Service High level Setup
Ops Centre2
Ops Centre1
Sentinel-1PDGS
Sentinel-2PDGS
Sentinel-3PDGS
Ops Centre3
ColHubNode 2
ColHubNode 1
ColHub Node 3
S1A, S1B NRT+NTC
S1A, S1BNRT
S2A, S2BS3A
L1, L2NRT+ STC+ NTC
S1A, S1B NRT+NTC
S2A, S2B
S3A NRT+STC+NTC
S1A, S1B NTC
S2A, S2B
S3A NRT+STC+NTC
S-1(A+B) NRT+NTCS-2 (A+B)S-3A SRAL OLCI
2 weeks 3 weeks1 month
Data Offer
Rolling Policy
Collections S1 NRT Collection S1 NRT Collection S1 NRT Collection
Downl. Quota 10 10 10
S-1(A+B) NRT+NTCS-2 (A+B)S-3A SRAL OLCI
S-1(A+B) NRT+NTCS-2 (A+B)S-3A SRAL OLCI
Serco Business
Data fetching Data routing Backend Ingestion Frontend
publicationDissemination
Architectural layout at centre 1
S1
S2
A NTC B NTCA+B NRT
A L2A B L1CA L1C B L2A Repack.
OLCIA NTC B NRTA NRT B NTC Repro.
SLSTRA NTC B NRTA NRT B NTC Repro.
SRALA NTC B NRT+STCA NRT+STC B NTC Repro.
SYNA SYN B SYN B VGTA VGT
S3
(27)
Col Hub1
Sci Hub
API Hub
Cop Hub
DIAS Hub
(16+1)(12)
= Download manager instance
= Integrity check and compression
= DHuS software instance
= Data routing and circulation
= Data synchronization
Open Access Hub
Collaborative Data Hub
Copernicus ServicesData Hub
DIAS Access Hub
S1
S2
S3
Data Synch
(4)
Monitoring and Reporting ElementsPo
stG
res
Influ
xDB
Gra
fana
= Long Term Data warehouse (DWH)
= Real Time Database (RTDB)
= Analytics Platform Server (APS)
= Sysma Agent (SA)
Web application for the visualization of the metrics stored in the RTDB
MyS
QL,
Py
hton Metrics Collector serving both the DWH and the
RTDB. With caching capability
Stores the metrics used for reporting and data analysis with a long term data preservation (> 4 years).
Stores the metrics used for real-time monitoring and alarming with a short term data preservation (4 months).
Bash
, PL
SQ
L,TW
iki
= Automatic Weekly Reporting (AWR) Scripts for the automatic extraction of metrics from the DWH and web application for the visualization of the reports.
= Alarming System
Bosu
n
Application for triggering alarms notification based on thresholds set on metrics stored in the RTDB
= Sentinels Data Dashboard Web application for reporting metrics to the service stakeholders. Based on the data stored in the DWH
Serco Business
Main Issues and Resolution
• Sentinel-3 quicklook and thumbnail synchronization issue
• Database performances (optimization and maintenance)
• Data Gaps filling
Serco Business
Short Term Outlook
• Publication of Sentinel-3B products• Publication of Sentinel-2 Level-2 Global• Sentinel-2 Level-2 processing on demand dedicated presentation• Access to offline data from Long Term Archives dedicated presentation
Serco Business
Conclusions