Overall System Architecture of Big Data of Wind Power Based on IoT_20161...
19
1 Overall System Architecture of Big Data of Wind Power Based on IoT [email protected] 2016.10 Wind Power Big Data and IoT Forum Huang Yuan
Transcript of Overall System Architecture of Big Data of Wind Power Based on IoT_20161...
1
Overall System Architecture of Big Data of Wind Power Based on IoT
[email protected] 2016.10
Wind Power Big Data and IoT Forum
Huang Yuan
2
Summary
Overall Architecture of Big Data Platform
Based on IoT
Agenda
Industry Challenges and
Technology Trends
Project “Big Data Plat form of Wind Power”
2
1 3
4
Industry Challenges and Technology Trends- Distributed Energy Assets Life Cycle Management & System
• Industry Challenges• Distributed Energy Assets System is the
foundation to face the revolution of Wind Power Industry Assets Life Cycle Management.
• Powerful and accurate data acquisition & analysis capability based on IoT & Big Data is the core to establish Distributed Energy Assets System.
• Data completeness, consistency, timeliness and validity are the prime challenges in all information systems, especially for the Distributed System.
• Technology Trends• IoT technology could solve the issues of full
sources and reliable data acquisition & efficient and secure transmission.
• Big Data technology provides the ability to process and analyze massive data (TB/PB level, structured/non-structured data).
3
Goals
System
Technology
Assets Life
Cycle Management
Distributed Energy Management System
Big Data
IoT
Data
Project “Big Data Platform of Wind Power”- To Face the Challenges of Distributed Energy Assets Life Cycle Management
4
Project Goals
} Industrial standard – unified data model for wind turbine I/O, status and fault to >20 Manufacturers and >90 Types of Wind Turbine.
} Reference architecture and validation for Wind Power big data platform based on IoT, big data and cloud computing technologies.
} Big data analytics models for accurate prediction, diagnosis, maintenance to achieve effective decision, better efficiency and lower cost.
} Pilot in Longyuan to prove the business value of the platform: better profitability and management maturity.
Project Duration
} Two Years, 2016-2017.
Project Background
} The 1st state-funded big data platform for Wind Power, invested by National Energy Administration, China Guodian Group and China Longyuan Power Group.
Project “Big Data Platform of Wind Power”- China Longyuan Power Group, the world‘s Largest Wind Farm Operator
5
15GW+ Capacity of Wind Power
11,000+ Wind Turbines
160+ Wind Plants in China, Canada and South Africa
Accumulated Wind Power Generation
126 Billion KWH
Accumulated 130 Million Tones of CO2 Saving
Business and Citizenship
World-wide Partnership
The world’s largest wind farm operator, funded in 1993
5
Overall Architecture of Big Data PlatformInformation Standardization • Data Acquisition • Transmission • Big Data Platform/Application • Security Architecture
Requirements
Design Principles• Standardization• Safety &
Compliance• Reliability• Manageability• Self-adaption• …
Big Data Platform# of sensor points：10M or more# of data increase：400TB/year or more # of data for analytics: 1PByte or more# of prediction models deployed: 100+
Big Data Application# Support for world-wide wind plant management and users# of Applications: 64 or morefailure prediction/operation optimization/asset management/visualization analysis
Data Acquisition# of manufacturers: 20 or more# of Wind Turbine types: 90 or more# of Wind Turbine: 10K+# of sensors per Turbine: x00-1K+# of data collection frequency: 1Hz or higher
Data Transmission# High security, High availability and Support for Hybrid cloud
6
Design Principles• Standardization• Safety &
Compliance• Reliability• Manageability• Self-adaption• …
Overall Architecture of Big Data Platform- Information Standardization
Information Standardization
Data Acquisition
Data Transmission
Big Data Platform
Big Data Application
7
Information standardizati
on is the foundation
of information
system
Standard name and definition of
wind turbine states
Standard name and definition of
wind turbine sensor points
Standard name and definition of
wind turbine fault types
Secu
rity
Arc
hite
ctur
e
Info
rmat
ion
Stan
dar
diz
atio
n
Overall Architecture of Big Data Platform- Full Data Acquisition
Big Data Acquisition
8
• Wind turbine• Transformer substation• Anemometer tower• Wind Power prediction• Vibration monitoring• AVC/AGC• Smart grid and meter• Voice/Video …
Real Time Device/System Data
• Macro political policy• Macro economic policy• Social Network• …
External Data
• Finance• Infrastructure projects• HR• …
ERP System Data
Design Principles• Standardization• Safety &
Compliance• Reliability• Manageability• Self-adaption• …
Data Acquisition
Data Transmission
Big Data Platform
Big Data Application
Secu
rity
Arc
hite
ctur
e
Info
rmat
ion
Stan
dar
diz
atio
n
Overall Architecture of Big Data Platform- Data Transmission --- Function Architecture
Data Transmission
9
HA/Efficient/SecureNetwork
Resume broken transferCompression/Encryption/…
Uniform mobile device access
Completed Transmission
Solution
Design Principles• Standardization• Safety &
Compliance• Reliability• Manageability• Self-adaption• …
Data Acquisition
Data Transmission
Big Data Platform
Big Data Application
Secu
rity
Arc
hite
ctur
e
Info
rmat
ion
Stan
dar
diz
atio
n
Wind Plant Network
Regional Network
Global Center Network
Office IT Network
Overall Architecture of Big Data Platform- Data Transmission --- Technology Architecture
Data Acquisition
Data Transmission
Big Data Platform
Big Data Application
Secu
rity
Arc
hite
ctur
e
Info
rmat
ion
Stan
dar
diz
atio
n
10
Overall Architecture of Big Data Platform- Big Data Platform --- Function Architecture
Big Data Platform
11
Data ETLExtract-Transform-Load
Data AccessRTDB/Kafka/Flume/…
Data ManagementHDFS/RDB/Meta Data/…
Data ProcessingMapReduce/SPARK/Storm/…
Data Interface/ToolsODBC/JDBC/R/SPSS/…
Design Principles• Standardization• Safety &
Compliance• Reliability• Manageability• Self-adaption• …
Data Acquisition
Data Transmission
Big Data Platform
Big Data Application
Secu
rity
Arc
hite
ctur
e
Info
rmat
ion
Stan
dar
diz
atio
n
12
-SaaSBI/ERP/Prediciton/Visualization
- PaaSHadoop/RTDB/RDB/SPSS/…
-IaaS-Engine Room/Server/Netwrok/…
Overall Architecture of Big Data Platform- Big Data Platform ---Technology Architecture
Data Acquisition
Data Transmission
Big Data Platform
Big Data Application
Secu
rity
Arc
hite
ctur
e
Info
rmat
ion
Stan
dar
diz
atio
n
Overall Architecture of Big Data Platform- Big Data Application
Big Data Application
13
BIERPOA…
EBA/Failure Prediction/OperationOptimization/…
MobileAPPs/
VisualizationApps/
…
Asset Life Cycle
Management
Design Principles• Standardization• Safety &
Compliance• Reliability• Manageability• Self-adaption• …
Data Acquisition
Data Transmission
Big Data Platform
Big Data Application
Secu
rity
Arc
hite
ctur
e
Info
rmat
ion
Stan
dar
diz
atio
n
14
Big Data System
SourceDatabase
Big Data Analytic Platform
Longyuan Power Group Big Data Analytic
Platform
Customer Analytic Model
LongyuanPowerGroup
Business Personnel
Customer Business
Personnel
Third Party
Business Personnel
Longyuan Power Group Internal
Application System
PaaS
Ap
plication A
ccess G
ateway
Developm
ent &
Test Platform
Business Service B
us
API
IaaS
ApplicationServer
(Middleware)
Cloud Management Platform
Compute
Store
Network/Message
Management
Resource Management
UserManagement
SystemMonitoring
Service Management
Security
ObjectiveStorage
BlockStorage
ImageService
Third Party Big
Data Analytic Platform
1
23
1
4
5
6
Application Boundary Security
2 User Identity and Access Control
3 Big Data Platform Data and Management Security
5 Storage Encryption Machine and Access Control
6 Transmission Encryption
4 Memory and Image Isolation and Erasure
Overall Architecture of Big Data Platform- Security Architecture
Secu
rity
Arc
hite
ctur
e
15
Cloud Monitoring
Application vulnerability
Application vulnerability
Security Event Monitoring
Cloud Auditing
Database Maintenance
Cloud Service
EmergencyResponse
API Interface
Monitoring &Warning
Defense
Log Collection
Reinforcement
Cloud Resource Platform
Cloud Security Operation Platform
User Control
Authorization Management
Platform Security Management View
Big Data Analysis
Business Security Management View
Cloud Defense
VFWDefense DDOS
VPN
Network Server Application
Cloud Fortress Machine
Management
Cloud Control Platform
Cloud Resource SchedulingCloud Platform Management
Cloud
Com
pliance
Availability
Database vulnerability
KnowledgeDatabase
AntivirusWAFAnti-
Tamper
Web Log
MeasurementService
SecurityReinforcement
CloudCompliance
Overall Architecture of Big Data Platform- Security Architecture
Secu
rity
Arc
hite
ctur
e
Field Gateway
Device Connectivity
Analytics & Operationalized Insights
Presentation & Business Connectivity
Overall Architecture of Big Data Platform- Infrastructure Based on IoT --- Self Hosted Mode
Batch AnalyticsHadoop Spark, Mapreduce, R
Hot Path AnalyticsHadoop Storm, Spark Stream
Presentation &
Business Applications
Mobile BI, ERP, SCMFront-End Cluster and ETL
Field Gateway
Field Gateway
Management
Dedicated Network,
Self-hosted Dedicated
Data Center
Strength Weakness
q Higher CAPEX for infrastructure
q High OPEX for infrastructure and dedication operation team
q Construction and deployment Time
q High security requirements for dedicated infrastructure
q Real-time monitoring and remote control
q All of the data and systems in the user's local
Data AcquisitionIntelligent Gateway/
RTDB/…
Data TransmissionSDH/MSTP/
MPLS-VPN/…
Data CenterBig Data Platform: Hadoop/RDB/ERP/BI/…
Big Data Application: EBA/Prediction/Optimization/…
16
Field Gateway
Device Connectivity
Analytics & Operationalized Insights
Presentation & Business Connectivity
Batch Analytics & VisualizationsAzure Hdinsight, Azure Machine Learning
Hot Path AnalyticsAzure Stream Analytics, Azure HDInsight Storm
Presentation &
Business Connectivity
App Service, Websites
Hot Path Business Logic
Micro-services(Docker and Service Fabric)IOT Hub
Field Gateway
Management
Platform-as-a-Service
in Public Cloud
Strength Weakness
q Weak for higher security requirements for dedicated infrastructure(network/data center)
q Weak for real-time control
q All of the data and systems at the far end
q Lower CAPEX for infrastructure
q Lower OPEX for infrastructure
q Pay-as-go and less deployment time
Overall Architecture of Big Data Platform- Infrastructure Based on IoT --- Cloud Mode
Data AcquisitionIntelligent Gateway/
IoT Edge /…
Data TransmissionInternet /4G/…
LPWA(NBIoT/LoRa/SigFox…)
Data CenterIoT Hub + Cloud Service (IaaS + PaaS)
17
Summary
18
1
2
3
IoT technology helps us to improve the capability to build deeper and more accurate digital modeling for the real world and achieve more closely interconnection between people, things and services.
Big Data technology provides the capability of analysis and data mining for massive data generated by IoT. It helps us reveal the nature and connections of things and will have great support for digitization of Energy Management System, even for Intelligence Transformation.
Project “Big Data Platform of Wind Power” is under building, technical exchanging with experts and cooperations from parties who have interested in it are both very welcomed.
19
Q&A
