Technical Note on Common Services, Intermediate …...H2020 – EINFRA – 2015 – 1 Page 1 of 37...
Transcript of Technical Note on Common Services, Intermediate …...H2020 – EINFRA – 2015 – 1 Page 1 of 37...
H2020 – EINFRA – 2015 – 1 Page 1 of 37
Technical Note on Common Services,
Intermediate Version
Workpackage 5 VRE Infrastructure and Services Design and Development
Task 5.2 Common Services
Author (s) Ugo Di Giammatteo
Serena Avolio
Sergio de Gioia
ACS
ACS
ACS
Reviewer (s) Fulvio Marelli
Simone Mantovani
T2
MEEO
Approver (s) Pedro Gonçalves
Cristiano Silvagni
T2
ESA
Authorizer Mirko Albani ESA
Document Identifier EVER-EST DEL WP5-D5.2
Dissemination Level Public
Status Approved by the EC
Version 1.1
Date of issue 14 December 2018
H2020 – EINFRA – 2015 – 1 Page 2 of 37
Document Log
Date Author Changes Version Status
27/6/2016 Fulvio Marelli TOC 0.1 Draft
03/08/2016 Ugo Di Giammatteo Revised TOC 0.2 Draft
14/10/2016 Serena Avolio Enriched TOC 0.3 Draft
18/10/2016 Serena Avolio Section 2,3,4 (now 5) 0.4 Draft
27/10/2016 Serena Avolio Section 4 0.5 Draft
28/10/2016
Ugo Di giammatteo
Sergio de Gioia Issue for revision 0.6 Draft
23/11/2016
Serena Avolio
Sergio de Gioia
Implementation of changes from reviewers, annexes and some figures added
0.7 Draft
25/11/2016 Serena Avolio Implementation of minor changes from reviewers
0.8 Draft
08/12/2016 Serena Avolio Implementation of changes from Approvers
0.9 Draft
12/12/2016 Serena Avolio Implementation of final comments.
1.0 Draft to be approved by the EC
14/12/2018 Cristiano Silvagni Document status 1.1 Approved by the EC
H2020 – EINFRA – 2015 – 1 Page 3 of 37
Table of contents
1 Introduction ........................................................................................................................................... 9
1.1 Purpose and scope ........................................................................................................................... 9
1.2 Who shall read this document .......................................................................................................... 9
1.3 System context ................................................................................................................................ 9
1.4 Services list...................................................................................................................................... 9
2 WSO2 IS ............................................................................................................................................... 11
2.1 Services offered overview .............................................................................................................. 11
2.2 Design elements ............................................................................................................................ 11
2.3 Customisation ............................................................................................................................... 12
2.4 Installation guide ........................................................................................................................... 13
2.5 Using the service – quick overview ................................................................................................. 13
2.5.1 Getting started ............................................................................................................................... 16
2.6 Service public API’s ........................................................................................................................ 16
2.6.1 The logic ......................................................................................................................................... 17
2.6.2 APIs ................................................................................................................................................. 18
2.6.3 Troubleshooting ............................................................................................................................. 22
3 WSO2 ESB ............................................................................................................................................ 23
3.1 Services offered overview .............................................................................................................. 23
3.2 Design elements ............................................................................................................................ 23
3.3 Customisation ............................................................................................................................... 24
3.4 Installation guide ........................................................................................................................... 24
3.5 Using the service – quick overview ................................................................................................. 24
3.6 Service public API’s ........................................................................................................................ 24
3.6.1 Troubleshooting ............................................................................................................................. 24
4 WSO2 DAS ........................................................................................................................................... 26
4.1 Services offered overview .............................................................................................................. 26
4.2 Design elements ............................................................................................................................ 26
4.3 Customisation ............................................................................................................................... 27
4.4 Installation guide ........................................................................................................................... 28
4.5 Using the service – quick overview ................................................................................................. 28
4.6 Service public API’s ........................................................................................................................ 28
4.6.1 The logic ......................................................................................................................................... 28
4.6.2 DAS APIs for EVER-EST system ....................................................................................................... 28
4.6.3 Troubleshooting ............................................................................................................................. 29
5 SeaFile ................................................................................................................................................. 31
5.1 Services offered overview .............................................................................................................. 31
5.2 Design elements ............................................................................................................................ 31
5.3 Customisation ............................................................................................................................... 31
H2020 – EINFRA – 2015 – 1 Page 4 of 37
5.4 Installation guide ........................................................................................................................... 31
5.5 Using the service – quick overview ................................................................................................. 31
5.6 Service public API’s ........................................................................................................................ 32
5.6.1 Troubleshooting ............................................................................................................................. 33
Annex A Installation Guides ................................................................................................................... 34
A.1. WSO2 IS Installation guide ............................................................................................................. 34
A.1.1 Software prerequisites ................................................................................................................... 34
A.1.2 Hardware prerequisites.................................................................................................................. 34
A.1.3 Uninstallation procedure ............................................................................................................... 34
A.2 WSO2 ESB Installation guide .......................................................................................................... 35
A.2.1 Software prerequisites ................................................................................................................... 35
A.2.2 Hardware prerequisites.................................................................................................................. 35
A.2.3 Uninstallation procedure ............................................................................................................... 35
A.3 WSO2 DAS Installation Guide ......................................................................................................... 36
A.3.1 Software prerequisites ................................................................................................................... 36
A.3.2 Hardware prerequisites.................................................................................................................. 37
A.3.3 Uninstallation procedure ............................................................................................................... 37
H2020 – EINFRA – 2015 – 1 Page 5 of 37
List of Figures
Figure 2-1 WSO2 IS architectural diagram in the orange box. The Service Providers box shows as the other EVER-
EST components interacts with the IS................................................................................................................. 12
Figure 2-2 Configuration of a service provider with the different possible authentication protocols....................... 13
Figure 2-3 VRE Portal, highlighted the login button that redirects to EVER-EST IS .................................................... 14
Figure 2-4 Login page of the EVER-EST IS .................................................................................................................... 15
Figure 2-5 Consent page on the EVER-EST IS .............................................................................................................. 15
Figure 2-6 User is authenticated and redirected back to the portal .......................................................................... 16
Figure 2-3Exposed WSDL ............................................................................................................................................. 17
Figure 2-3 Authentication Process and API protection ............................................................................................... 18
Figure 3-1The Service BUS........................................................................................................................................... 24
Figure 4-1The WSO2 DAS architecture. ...................................................................................................................... 27
Figure 4-2How to create an event receiver in DAS ..................................................................................................... 27
Figure 5-1Seafile Web Interface .................................................................................................................................. 32
Figure 5-2Seafile desktop application ......................................................................................................................... 32
List of Tables
Table 1 Operation name and its endpoint for SOAP request in version 1.1 ............................................................... 18
Table 2 Example of a validate action request with SOAP in version 1.1 ..................................................................... 19
Table 3 Operation name and its endpoint for SOAP request in version 1.2 ............................................................... 19
Table 4 Example of a validate action request with SOAP in version 1.2 ..................................................................... 20
Table 5 Examples of responses to a validate action request. The three different possibles statuses. ...................... 21
Table 6 Description of the required fileds to configure an event receiver in DAS ..................................................... 28
Table 7 Description of the APIs provided by DAS. ...................................................................................................... 29
Table 8 The parameters used in the different APIs listed and explained. .................................................................. 29
Table 9 The possibles error codes from the APIs ........................................................................................................ 29
Table 10 WSO2 IS software prerequisites ................................................................................................................... 34
Table 11 WSO2 IS hardware prerequisites ................................................................................................................. 34
Table 12 WSO2 ESB software prerequisites ................................................................................................................ 35
Table 13 WSO2 ESB hadware prerequisites ................................................................................................................ 35
Table 14 WSO2 DAS software prerequisites ............................................................................................................... 36
Table 15 WSO2 ESB software prerequisites ................................................................................................................ 36
Table 16 WSO2 DAS hardware prerequisites .............................................................................................................. 37
Table 17 WSO2 ESB hardware prerequisites .............................................................................................................. 37
H2020 – EINFRA – 2015 – 1 Page 6 of 37
Definitions and Acronyms
Acronym Description
ABAC Attribute Based Access Control
ACL Access Control List
AJAX Asynchronous JavaScript and XML
AMQP Advanced Message Queuing Protocol
API Application Programming Interface
BAPI Business Application Programming Interface
CMS Content Management System
COTS Commercial) Off-the-Shelf
CRL Certificate Revocation List
CRUD Create read Update Delete
CSV Comma-Separated Values
DAS Data Analytic Server
DOI Digital Object Identifier
E-PDSC Extended-Preservation Dataset Content
EAI Enterprise Application Integration
EDA Event Driven Architecture
EDI Electronic Data Interchange
EO Earth Observation
ERP Enterprise Resource Planning
ES Earth Science
ESA European Space Agency
ESB Enterprise Service Bus
EVAT Expert-user Visual Analysis Tool
EVER-EST European Virtual Environment for Research - Earth Science Themes
FTP File Transfer Protocol
FTPS FTP over SSL
GIF Graphics Interchange Format
GUI Graphical User Interface
HTML Hypertext Mark-up Language
HTTP Hypertext Transfer Protocol
HTTPS HTTP over TLS, HTTP over SSL, and HTTP Secure
IDE Integrated Development Environment
IIOP Internet Inter-ORB Protocol
IMAP Internet Message Access Protocol
H2020 – EINFRA – 2015 – 1 Page 7 of 37
IO Input Output
IPR Intellectual Property Rights
IS Identity Server
ISO International Organization for Standardization
IT Information Technology
IWA Integrated Windows Authentication
JIT Just In Time
JMX Java Management Extension
JPEG Joint Photographic Experts Group
JSON JS Object Notation
LDAP Lightweight Directory Access Protocol
LGPL Lesser General Public License
MEA Multi-sensor Evolution Analysis
MLLP Minimum Lower Layer Protocol
MOM Message Oriented Middleware
MQTT MQ Telemetry Transport
OAGIS Open Applications Group Integration Specification
OASIS Organization for the Advancement of Structured Information Standards
OCSP Online Certificate Status Protocol
OIDC Open ID Connect
OGC Open Geospatial Consortium
OPI Operator Panel Interface
PBAC Policy Based Access Control
PDSC Preservation Dataset Content
PNG Portable Network Graphics
PSNC Poznań Supercomputing and Networking Center
POP Point of presence
RBAC Role Based Access Control
RDF Resource Description Framework
REST Representational State Transfer
RO Research Objects
RODL Research Objects Digital Library
RSS Rich Site Summary
SFTP Secure File Transfer Protocol
SLA Service Level Agreement
SMTP Simple Mail Transfer Protocol
SPARQL Protocol and RDF Query Language
H2020 – EINFRA – 2015 – 1 Page 8 of 37
SQL Structured Query Language
SSO Single Sign-On
SVG Scalable Vector Graphics
TCP Transmission Control Protocol
UDP User Datagram Protocol
URI Uniform Resource Identifier
URL Uniform Resource Locator
VAC Visual Analysis Client
VRC Virtual Research Community
VRE Virtual Research Environment
WCF Windows Communication Foundation
WCPS Web Coverage Processing Service
WCS Web Coverage Service
WebCGM Web Computer Graphics Metafile
WFMS Workflow Management Systems
WFS Web Feature Service
WMS Web Map Service
WSDL Web Services Description Language
XML EXtensible Mark-up Language
Applicable Documents
Document ID Document Title
EVER-EST DEL WP5-D5.1 VRE Architecture and Interfaces Definition
Reference Documents
Document ID Document Title
EVER-EST DEL WP5-D5.1 VRE Architecture and Interfaces Definition
EVER-EST DEL WP5-D5.4 Technical Note on e-Research application services, Intermediate version
EVER-EST DEL WP5-D5.3 Technical Note on digital information and e-collaboration services
H2020 – EINFRA – 2015 – 1 Page 9 of 37
1 Introduction This Technical Note is the first deliverable of Task 5.2, whose objective is to design and implement the common services that will be used in the VRE.
This document takes into account the overall design sketched in the DoA, refined during the first 5 months of the project and reported in EVER-EST DEL WP5-D5.1. It takes also into account the subsequent work of analysis that allowed defining the detailed interfaces between main components the EVER-EST system. This interface definition has been carried out during dedicated technical meetings, plenary project meetings and numerous bi-lateral meetings and calls.
1.1 Purpose and scope
The purpose of this document is to describe the design of the Common Services that will be integrated and used within the first version of the EVER-EST system.
The scope comprises: the identification of all services that belong to this category, the grouping of these services according to the relevant components and the definition of interfaces between them and with services of other groups.
1.2 Who shall read this document
This document has been thought mainly to provide implementation and configuration information about the Common Services to the technical partners in the consortium. It contains also some information about the utilisation of the Common Services that can be useful for the VRC members.
1.3 System context
The Common Services category represents the family of services, which are used by different elements of the EVER-EST infrastructure. They refer to the category of services that will guarantee the correct functioning between various infrastructure components and between the infrastructure and the final users.
One of the most important goals of the common services is to provide a web services architecture satisfying the Service Oriented Architecture (SOA) principles. One of the SOA core concepts the loose coupling that means reducing the assumptions that two parties (components, applications, services, programs, users) make about each other when they exchange information. Following this concept the EVER-EST system is tolerant to changes because the parties are not tightly coupled to each other, it is also easily extensible, and, most important for a Virtual Research Environment, it allows the implemented infrastructure to be extended beyond the initial Earth Science communities to other scientific domains.
The SOA approach is achieved using middleware and interoperability standards. The common functionalities that have been identified to ease the integration of new/existing components are listed in the following paragraph.
1.4 Services list
A Server Component can offer multiple services.
The following services are offered to the EVER-EST users/components:
1. Authentication 2. Authorization 3. Account Provisioning
H2020 – EINFRA – 2015 – 1 Page 10 of 37
4. Identity Administration 5. Security 6. Message routing 7. Message transformation 8. Auditing 9. Data Discovery 10. Data Access 11. Data Storage
The first five Common Services are offered by the WSO2 Identity Server (IS). The component offering the message routing and transformation is the WSO2 Enterprise Service Bus (ESB). The components offering the Auditing service are the WSO2 Data Analytic Server (DAS)together with the WSO2 ESB. The personal Data Storage and Data Discovery services are provided by the SeaFile component, the Data Discovery and Data Access for what concerns EO data are guaranteed by the use of the Open Search specifications, OpendData protocol and OGC standards for discovering metadata on the different catalogues. The identified components that are going to be detailed in this document are:
1. WSO2 IS 2. WSO2 ESB 3. WSO2 DAS 4. SeaFile
The involved protocol, standard or specifications are the following:
1. OpenSearch Specification 2. OpenData protocol 3. OGC Standards
and they have been already detailed in the EVER-EST DEL WP5-D5.1 and in the EVER-EST DEL WP5-D5.3.
H2020 – EINFRA – 2015 – 1 Page 11 of 37
2 WSO2 IS WSO2 IS provides secure Identity Management for the EVER-EST Web Application services and APIs by managing identity and entitlements of the users securely and efficiently.
2.1 Services offered overview
Among the common services the ones offered by the IS for EVER-EST are briefly detailed hereafter:
1. Authentication, the process of identifying an individual. Authentication merely ensures that the individual is who he or she claims to be.
2. Authorization, the process of granting or denying access to a network resource. The authorization allows the user access to various resources based on the user's identity.
3. Account Provisioning, the process of providing users with access to data and technology resources. The process implies that the access rights and privileges are monitored and tracked to ensure the security of an enterprise's resources.
4. Identity Administration, the process of creating new and modifying or deleting existing identities as well as managing the security entitlements associated with those identities.
5. Security, the process to provide secure access to resources based on standards and model for security
All these functionalities are centralized to support the coordinated usage of the different EVER-EST components and allow their interoperability.In EVER-EST there are multiple applications that require authentication, users should be able to log in at one place and still have seamless access to all the other applications. The IS takes care of transformation of protocols and tokens.. Once the user provides its credentials it's automatically authenticated on all application enabling a Single Sign On (SSO) scenario between multiple heterogeneous authentication protocols.
The different components authenticated by the IS can choose to expose a System for Cross Domain Identity Management (SCIM) endpoint for the users provisioning/de-provisioning functionality that can be consumed by the IS. Alternatively the components can implement an own Just In Time (JIT) user provisioning to complete, for their necessity, the user information exposed for them by the IS. Currently the VRE portal and ROHUB Portal have implemented their own JIT provisioning.
2.2 Design elements
WSO2 IS is a COTS configured and extended for the EVER-EST environment. It enables customization and extension through its componentized architecture.
The WSO2 Identity Server is used directly by the administrator users, through its Management Console. Apart from such users, the Identity Server is mainly used in EVER-EST as an identity provider for the other components, services and applications.
The following diagram depicts inside the orange box the entire architecture of the Identity Server to show how its parts allow the integration of the existing and future EVER-EST components.
H2020 – EINFRA – 2015 – 1 Page 12 of 37
Figure 2-1 WSO2 IS architectural diagram in the orange box. The Service Providers box shows as the other EVER-
EST components interacts with the IS.
The IS works using inbound auhtenticators. The inbound authenticators parse incoming requests and construct corresponding responses for all the supported protocols. Among the supported authentication protocols there are the authentication protocols used by the others EVER-EST sub-systems, in particular Open Id Connect and SAML 2.0.
2.3 Customisation
The IS needs to be configured and customized to integrate the different EVER-EST components, applications and services.
The various EVER-EST components need to be registered in the IS as service provider and linked to the involved authentication protocol. The Open ID Connect and the SAML 2.0 authentication protocols are involved in the integration of the current EVER-EST components. The following figure highlights one of the configuration steps needed to register one of the EVER-EST components on the IS.
H2020 – EINFRA – 2015 – 1 Page 13 of 37
Figure 2-2 Configuration of a service provider with the different possible authentication protocols.
In addition to the registration processes, the login pages and the other pages like error, notification screens, and the redirection page for SSO needs to be customized for the EVER-EST project.
2.4 Installation guide
The installation / uninstallation procedures of the IS on the dedicated machine are summarised in Annex A.1 of this document.
2.5 Using the service – quick overview
Whenever an application is asked to deliver a service for which the user needs to be authenticated, it redirects the user to the login page of the EVER-EST IS.
Soon after the user logins to EVER-EST, the application that triggered the process receives a certified user identity issued by the identity provider as an automatic means to log the user into the application as well. Next the user is redirected back to the application where the service, requested initially, can now be delivered.
The following figures provides an example of the different phases of the authentication process from the VRE Portal. In the first step the VRE portal is asked to deliver a service for which the user needs to be authenticated, it
H2020 – EINFRA – 2015 – 1 Page 14 of 37
redirects the user to the login page of the EVER-EST IS. There the user is asked to enter its credentials. From then on, if credentials entered were valid, the user is authenticated and in this working session it will not be asked for its credentials any more till it logs out or the session times out. Soon after the user login to EVER-EST, the application (VRE Portal) that triggered such a process receives a certified user identity issued by the IS as an automatic means to log the user into the application as well. Next the user is redirected back to the application where the service, requested initially, can now be delivered.
Figure 2-3 VRE Portal, highlighted the login button that redirects to EVER-EST IS
H2020 – EINFRA – 2015 – 1 Page 15 of 37
Figure 2-4 Login page of the EVER-EST IS
Figure 2-5 Consent page on the EVER-EST IS
H2020 – EINFRA – 2015 – 1 Page 16 of 37
Figure 2-6 User is authenticated and redirected back to the portal
2.5.1 Getting started
To start using the services offered by the IS the generic EVER-EST application needs to be registered in the IS.
The registration process of an Open Id Connect (OIDC) relying party has the following steps:
• Registration of the RPs in the IS (redirect URL and Relying Party name needed).
• Communication of ClientId and secret to the Relying Party.
The registration process of a Service provider using the SAML 2.0 authentication protocol has the following steps:
• Registration of the SP in the IS.
• IS/SP Metadata exchanging.
2.6 Service public API’s The WSO2 Identity Server provides several web services APIs for managing identities. Among them only the APIs exposed to the EVER-EST components are documented within this deliverable. The APIs exposed in the EVER-EST system have the aim to protect some of the subsystem resources. The protection is offered via OAuth2.0 authorization protocol. The OAuth protocol is used to authorize websites or applications to access their resources on other applications. OAuth provides to clients a ‘secure delegated access’ to server resources. In particular OAuth2.0 provides also a specific authorization flow. The EVER-EST components that need to validate the access to their resources can be provided of credentials for using the APIs offered by the validation web service provided by the WSO2 IS. The WSDL of the web service is exposed at URL:https://sso.everest.psnc.pl/services/OAuth2TokenValidationService?wsdl
H2020 – EINFRA – 2015 – 1 Page 17 of 37
Figure 2-7Exposed WSDL
The only operation to use for the purpose to validate the OAuth2.0 access token is the “validate”.
2.6.1 The logic This API provides a secure method to protect resources. In the EVER-EST system the ROHUB APIs are the resources that need protection. When an API consumer, in our concern the VRE Portal, want to access these resources the API owner, in our concern the ROHUB, following the OAuth2.0 specification and using the exposed API can validate the access to the resources. Following the OAuth2.0 specification, the API owner (Resource owner) owns the responsibility to prevent unauthorized users from accessing an API (or a Resource in general). For that to be possible the specification prescribes that API consumers send a valid OAuth2.0 access token along with every API call. Such access token must be sent as the HTTPS request parameter. Once the API Server gets the call it needs to validate the access token against the OAuth2.0 Authorization server, before processing the request and replying to the consumer API. The following figure provides an overview of the process, it contains the authentication process, steps 1-8, and the protected API call, steps 9-12.
H2020 – EINFRA – 2015 – 1 Page 18 of 37
Figure 2-8 Authentication Process and API protection
2.6.2 APIs
The OAuth2.0 access token validation web service is provided by WSO2 IS. The only operation to use for the purpose is “validate”.
Operation URL for SOAP v1.1
Soap Action:
urn:validate
https://sso.everest.psnc.pl/services/OAuth2TokenValidationService.OAuth2TokenValidationServic
eHttpsSoap11Endpoint/
Table 1 Operation name and its endpoint for SOAP request in version 1.1
Type REQUEST SAMPLE
SOAP
1.1
POST
https://sso.everest.psnc.pl/services/OAuth2TokenValidationService.OAuth2TokenValidationService
H2020 – EINFRA – 2015 – 1 Page 19 of 37
HttpsSoap11Endpoint/ HTTP/1.1
Accept-Encoding: gzip,deflate
Content-Type: text/xml;charset=UTF-8
SOAPAction: "urn:validate"
Content-Length: 579
Host: sso.everest.psnc.pl
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1.1 (java 1.5)
Cookie: JSESSIONID=76D417C89BB22FEC0AB76F446B1EE175
Cookie2: $Version=1
Authorization: Basic T0F1dGhUb2tlblZlcmlmaWVyOjgkenVeJ3o=
<soapenv:Envelopexmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"xmlns:xsd="htt
p://org.apache.axis2/xsd" xmlns:xsd1="http://dto.oauth2.identity.carbon.wso2.org/xsd">
<soapenv:Header/>
<soapenv:Body>
<xsd:validate>
<xsd:validationReqDTO>
<xsd1:accessToken>
<xsd1:identifier>636489a7-8481-33fa-82cc-c5700e277106</xsd1:identifier>
<xsd1:tokenType>Bearer</xsd1:tokenType>
</xsd1:accessToken>
</xsd:validationReqDTO>
</xsd:validate>
</soapenv:Body>
</soapenv:Envelope>
Table 2 Example of a validate action request with SOAP in version 1.1
Operation URL for SOAP v1.2
Soap Action:
urn:validate
https://sso.everest.psnc.pl/services/OAuth2TokenValidationService.O
Auth2TokenValidationServiceHttpsSoap12Endpoint/
Table 3 Operation name and its endpoint for SOAP request in version 1.2
Type REQUEST SAMPLE
SOAP
1.2
POST
https://sso.everest.psnc.pl/services/OAuth2TokenValidationService.OAuth2TokenValidationService
HttpsSoap12Endpoint/ HTTP/1.1
Accept-Encoding: gzip,deflate
H2020 – EINFRA – 2015 – 1 Page 20 of 37
Content-Type: application/soap+xml;charset=UTF-8;action="urn:validate"
Content-Length: 559
Host: sso.everest.psnc.pl
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1.1 (java 1.5)
Cookie: JSESSIONID=6EEE2C8538225FB26984056094A47A22
Cookie2: $Version=1
Authorization: Basic T0F1dGhUb2tlblZlcmlmaWVyOjgkenVeJ3o=
<soap:Envelopexmlns:soap="http://www.w3.org/2003/05/soap-
envelope"xmlns:xsd="http://org.apache.axis2/xsd"
xmlns:xsd1="http://dto.oauth2.identity.carbon.wso2.org/xsd">
<soap:Header/>
<soap:Body>
<xsd:validate>
<xsd:validationReqDTO>
<xsd1:accessToken>
<xsd1:identifier>536489a7-8481-33fa-82cc-c5700e277106</xsd1:identifier>
<xsd1:tokenType>Bearer</xsd1:tokenType>
</xsd1:accessToken>
</xsd:validationReqDTO>
</xsd:validate>
</soap:Body>
</soap:Envelope>
Table 4 Example of a validate action request with SOAP in version 1.2
Status Response
Expire
d
<soapenv:Envelopexmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:validateResponsexmlns:ns="http://org.apache.axis2/xsd">
<ns:returnxsi:type="ax2354:OAuth2TokenValidationResponseDTO"
xmlns:ax2354="http://dto.oauth2.identity.carbon.wso2.org/xsd"xmlns:xsi="http://www.w3.org/20
01/XMLSchema-instance">
<ax2354:authorizationContextTokenxsi:nil="true"/>
<ax2354:authorizedUserxsi:nil="true"/>
<ax2354:errorMsg>Access token expired</ax2354:errorMsg>
<ax2354:expiryTime>0</ax2354:expiryTime>
<ax2354:scopexsi:nil="true"/>
<ax2354:valid>false</ax2354:valid>
H2020 – EINFRA – 2015 – 1 Page 21 of 37
</ns:return>
</ns:validateResponse>
</soapenv:Body>
</soapenv:Envelope>
Valid <soapenv:Envelopexmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:validateResponsexmlns:ns="http://org.apache.axis2/xsd">
<ns:returnxsi:type="ax2354:OAuth2TokenValidationResponseDTO"
xmlns:ax2354="http://dto.oauth2.identity.carbon.wso2.org/xsd"xmlns:xsi="http://www.w3.org/20
01/XMLSchema-instance">
<ax2354:authorizationContextTokenxsi:nil="true"/>
<ax2354:authorizedUser>[email protected]</ax2354:authorizedUser>
<ax2354:errorMsgxsi:nil="true"/>
<ax2354:expiryTime>3280</ax2354:expiryTime>
<ax2354:scope>openid</ax2354:scope>
<ax2354:valid>true</ax2354:valid>
</ns:return>
</ns:validateResponse>
</soapenv:Body>
</soapenv:Envelope>
Invali
d
<soapenv:Envelopexmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:validateResponsexmlns:ns="http://org.apache.axis2/xsd">
<ns:returnxsi:type="ax2354:OAuth2TokenValidationResponseDTO"
xmlns:ax2354="http://dto.oauth2.identity.carbon.wso2.org/xsd"xmlns:xsi="http://www.w3.org/20
01/XMLSchema-instance">
<ax2354:authorizationContextTokenxsi:nil="true"/>
<ax2354:authorizedUserxsi:nil="true"/>
<ax2354:errorMsg>Invalid access token</ax2354:errorMsg>
<ax2354:expiryTime>0</ax2354:expiryTime>
<ax2354:scopexsi:nil="true"/>
<ax2354:valid>false</ax2354:valid>
</ns:return>
</ns:validateResponse>
</soapenv:Body>
</soapenv:Envelope>
Table 5 Examples of responses to a validate action request. The three different possibles statuses.
H2020 – EINFRA – 2015 – 1 Page 22 of 37
2.6.3 Troubleshooting A properly configured logging system is used in identifying errors and security threats. There are several ways logs of a running IS instance can be viewed.
• Through the Management Console. The administrator of the IS can access to the Management consol of the Identity Server and monitor the different actions in which IS has been involved. The monitor menu includes a list of features focused on providing logs and statistics related to monitoring the IS. The components of the monitor menu have the aim to monitor system statistics, system logs and trace the SOAP requests. The system statistics page shows certain statistics related to the WSO2 IS instance. These include free memory, request count, server name, server start time, system up time, active services, total memory, average response time, minimum response time, and maximum response time. The system logs page displays information regarding the log files of the current product. Furthermore, it has a feature that allows the administrator to view and download log files according to their preference. The SOAP Tracer shows SOAP messages, SOAP message requests, and SOAP message responses.
• If the Management console is not accessible troubleshooting can be performed through the log files that are stored in the <PRODUCT_HOME>/repository/logs folder. This folder contains current logs in a log file with a date stamp. Older logs are archived in the wso2carbon.log file.
• If the Management console is not accessible troubleshooting can be also performed through the command prompt/shell terminal that opens when running the script to start the server.
H2020 – EINFRA – 2015 – 1 Page 23 of 37
3 WSO2 ESB WSO2 Enterprise Service Bus provides, when it is needed, the bus for the messages exchanged between service provider and service consumer.
3.1 Services offered overview
Among the common services the ones offered by the ESB and used by the EVER-EST system are briefly detailed here:
1. Message routing, the process of routing requests to correct service provider. 2. Message transformation, the process of transforming message formats between consumer and
provider.
These functionalities are centralized in EVER-EST to integrate the RoHub functionalities with the VRE portal.
The ROHub functionalities are offered to the end user of the VRE Portal using the mediation of the ESB that acts as a middleware.
The ESB performs a transformation of format, serves the ROHub output formats in a format easier to manage from the VRE Portal (xml to json), it routes the different requests from the VRE portal to the correct RoHub endpoint (REST API endpoint or SPARQL endpoint). The ESB also performs a mediation between VRE Portal and ROHub mapping a single call from VRE Portal into N calls to RoHub sending most of them in parallel.
3.2 Design elements
WSO2 ESB is a COTS configurable to work in the EVER-EST environment.
The role of ESB in the EVER-EST service oriented architecture is that of an enabler of communications among systems and components both currently involved in VRE and, taking in due care extensibility concern, upcoming in future evolutions of VRE. Such enablement consists in leveraging extensive connectivity technologies supported by WSO2 ESB thus making it possible to integrate very heterogeneous components. At the same time the use of ESB on VRE has been inspired by a low-intrusion philosophy, whereby components endowed with a mature level of communications technology are allowed to interoperate directly. The detailed analysis of existing components carried out during the design phase has thus reduced the need for ESB as a communication enabler just between the exposed RoHub functionalities and the VRE portal. The purpose of using ESB is twofold, concerning Design and Performance issues. About the design the ESB allows to streamline the usage of ROHUB services, repurposing them to the EVER-EST user-action semantics (Service Façade pattern). About the performance the ESB allows to gain in user-perceived speed of processing, splitting complex user actions commanded by the VRE portal into smaller units taken care of by the ESB asynchronously and concurrently (Mediation pattern).
H2020 – EINFRA – 2015 – 1 Page 24 of 37
Figure 3-1The Service BUS
3.3 Customisation
The ESB needs to be configured and customized to integrate the different EVER-EST components, applications and services. The major configuration is the message mediation. This functionality can be any combination of advanced mediations such as transformation and content-based routing. The configurations have been done to allow format management, routing, mediation and trasformation.
For the format management the transformations configured allow to transform the output format from ROHUB, XML, in JSON format for the VRE portal.
About the routing, the configuration done allows to redirect the requests among the different ROHUB backend (ROHUB REST API endpoint or ROHUB SPARQL endpoint) depending on the request from the VRE portal
About the mediation, the ESB is configured to do a mediation mapping one call on VRE to several calls on ROHUB.
Concerning the transformation the ESB sanitizes, following the ROHUB specifications, the filters coming from the VRE portal before passing them to RO.
3.4 Installation guide
The installation / uninstallation procedures of the ESB on the dedicated machine are summarised in Annex A.2 of this document.
3.5 Using the service – quick overview
The use of the service is completely hidden to the final user. The VRE portal will use the Façade API exposed by the ESB.
3.6 Service public API’s
The service public API’s are described in the EVER-EST DEL WP5-D5.3.
3.6.1 Troubleshooting A properly configured logging system is used in identifying errors and security threats. There are several ways logs of a running ESB instance can be viewed.
H2020 – EINFRA – 2015 – 1 Page 25 of 37
• Through the Management Console. The administrator of the ESB can access to the Management consol of the ESB and monitor the different actions in which ESB has been involved. The monitor menu includes a list of features to monitor and manage the server runtime. It is possible to see mediation tracing, mediation statistics and system logs. You can monitor messages that are mediated through the ESB using the Mediation Tracer in the management console. Mediation statistics allow a server administrator to collect and view runtime statistical information from sequences, proxy services, and endpoints through the ESB management console.
• If the Management console is not accessible troubleshooting can be performed through the log files that are stored in the <PRODUCT_HOME>/repository/logs folder. This folder contains current logs in a log file with a date stamp. Older logs are archived in the wso2carbon.log file.
• If the Management console is not accessible troubleshooting can be also performed through the command prompt/shell terminal that opens when running the script to start the server.
H2020 – EINFRA – 2015 – 1 Page 26 of 37
4 WSO2 DAS WSO2 DAS and WSO2 ESB, working in synergy with each other, offer data analytics functionalities. WSO2 ESB is the component that mainly will be used to send event to WSO2 DAS for processing. The role of the ESB in this case is that of a data collector from the various systems that make up EVER-EST infrastructure.
4.1 Services offered overview
The common service offered by the conjunction of DAS and ESB for EVER-EST is the auditing service.
This service, can show who has accessed the system and what operations he or she has performed during a given period of time. Auditing is useful both for maintaining security and for monitoring system performances.
4.2 Design elements
The WSO2 DAS and ESB are COTS configurable to work in the EVER-EST environment. The WSO2 DAS is a platform where different kinds of data analytics can be performed:
• Real-time: thanks to an embedded CEP engine “WSO2 Siddhi”, data received by DAS can be processed on the fly, with or without persistence.
• Batch: with the support of Apache Spark, scheduled batch processing can be implemented on data received and persisted on DAS.
• Interactive: with the support of indexing and searching power of Apache Lucene, users are able, on demand, to query data or monitor activities.
• Predictive: thanks to the embedded WSO2 Machine Learner, using persisted data as source dataset for the learning, new data sent to DAS are used to predict future events.
The WSO2 ESB is the component that will be mainly used to send event to WSO2 DAS for processing. The ESB plays, in this case, a role of a data collector from the various systems that made up EVER-EST infrastructure. Data will be collected by the ESB mainly through listening on log files and data repositories while such data are produced or as a second option (for systems that do not have a data repository shareable for the purpose) through receiving real-time data feeds via HTTP directly from the systems. Once ESB gets the data it sends such collected information to WSO2 Data Analytics Server where data analysis can take place.
H2020 – EINFRA – 2015 – 1 Page 27 of 37
Figure 4-1The WSO2 DAS architecture.
4.3 Customisation
The DAS and ESB need to be configured and customized to integrate the different EVER-EST components, applications and services.
The aggregation of data from the different components can be done configuring an event receiver as in the following figure.
Figure 4-2How to create an event receiver in DAS
H2020 – EINFRA – 2015 – 1 Page 28 of 37
Section Description
To The event stream from which the event receiver will fetch the events for processing.
Mapping
Configuration
Specific properties of the selected input event adapter.
From The event stream from which the event receiver will fetch the events for processing
Adapter
Properties
The format of the message that is received. You can configure custom mappings on the selected
format via advanced settings.
Table 6 Description of the required fileds to configure an event receiver in DAS
4.4 Installation guide
The installation / uninstallation procedures of the DAS on the dedicated machine are summarised in Annex A.3 of this document.
4.5 Using the service – quick overview
After collecting data, and performing various analysis on them to produce meaningful information, the final usage of the service is to present this information. WSO2 DAS queries the analysed information and shows it in various UIs. There are different ways to query and present analysed information, the ones selected for EVER-EST are:
1. Visualizing Results in WSO2 DAS, access the WSO2 DAS Analytics Dashboard, create new dashboard and new gadget or visualize the existing ones.
2. Communicating Results Through REST API, WSO2 DAS stores the received events and processed data in its underlying data storage system, where that data can be retrieved using the standards. Analytics REST API allows external parties to query this data.
4.6 Service public API’s
The interface of the analytics data service as exposed as REST APIs.
4.6.1 The logic
Analytics REST API allows external parties to query this data.
4.6.2 DAS APIs for EVER-EST system
The REST APIs are secured with basic authentication and all the Response are in JSON format.
Function REST API
Checking if a table exists GET /analytics/table_exists?table={tableName}
Checking if the Given Records Store Supports
Pagination
GET /analytics/pagination/<recordstore-name>
Drilling down through hierarchical categories POST /analytics/facets
Getting the record count of a table GET /analytics/tables/{tableName}/recordcount
Getting the search record count of a table POST /analytics/search_count
H2020 – EINFRA – 2015 – 1 Page 29 of 37
Listing all tables GET /analytics/tables
Retrieving Aggregated Values of Given Records POST /analytics/aggregates
Retrieving All Record Stores GET /analytics/recordstores
Retrieving matching records through a drill down
search
POST /analytics/drilldown
Retrieving records of a table GET
/analytics/tables/{tableName}/{from}/{to}/{start}/{count}
Retrieving the count of matching records through a
drill down search
POST/analytics/drillDownScoreCount
Searching records of a table POST /analytics/search
Table 7 Description of the APIs provided by DAS.
Params Value
{tableName} The name of the table that you want to find in the DAS server.
{from} The starting time to retrieve records from. This is an optional
parameter.
{to} The ending time up to which records should be retrieved. This is
an optional parameter.
{start} The paginated index from value. This is an optional parameter.
{count} The paginated records count to be read. This is an optional
parameter.
{timeout} The timeout in seconds after waiting until the process completes.
{recordstore-name} The name of the record store for which you need to carry out a
check.
Table 8 The parameters used in the different APIs listed and explained.
Status Response
success 20*
error 40*/50*
Table 9 The possibles error codes from the APIs
4.6.3 Troubleshooting A properly configured logging system is used in identifying errors and security threats. There are several ways logs of a running DAS instance can be viewed.
• Through the Management Console. The administrator of the DAS can access to the Management console of the DAS and monitors the different actions in which DAS has been involved. The monitor menu includes a list of features to monitor and manage the server runtime. From the Monitor menu it is possible monitor events statistics and trace events. Event Statistics is an important feature which helps monitoring purposes of events.
H2020 – EINFRA – 2015 – 1 Page 30 of 37
This presents realtime requests and responses vs time for all the incoming and outgoing topics. DAS Event Tracer provides huge functionality to trace the event in each and every component. Event tracer will help to check the event when travels along the components.
• If the Management console is not accessible troubleshooting can be performed through the log files that are stored in the <PRODUCT_HOME>/repository/logs folder. This folder contains current logs in a log file with a date stamp. Older logs are archived in the wso2carbon.log file.
• If the Management console is not accessible troubleshooting can also be performed through the command prompt/shell terminal that opens when running the script to start the server.
H2020 – EINFRA – 2015 – 1 Page 31 of 37
5 SeaFile SeaFile1 offers a repository for the personal data storage. EVER-EST tools and services need to be able to reference and access these resources.
5.1 Services offered overview
Users need a space where they can store complementary or auxiliary resources (e.g., documents, scripts, papers, intermediate results, etc.)
5.2 Design elements
All data, files, documents, scripts, uploaded in Seafile are available through a Web link, so that they can be referenced, particularly within a research object. More in general, EVER-EST applications and services can reference and access the stored resources. The solution is economic and scalable, and the component is able to increase its storage capacity seamlessly.
Seafile guarantees, among other features, file synchronisation, version control, public link sharing, desktop client and web API.
5.3 Customisation
A Seafile instance for EVER-EST is accessible at https://box.everest.psnc.pl/. This instance of Seafile is integrated with the EVER-EST Identity Service Provider by means of the SAML 2.0 authentication protocol. Possible future configurations/customisation are definition of user quotas or increasing of the initial size (5TB).
5.4 Installation guide
For a complete guide of Seafile Server, including service installation and uninstallation procedures:
https://www.gitbook.com/book/freeplant/seafile-server-manual/details
5.5 Using the service – quick overview
To start to use Seafile functionalities as EVER-EST user you need to be logged in the EVER-EST system.
SeaFile Web Services will be accessed in an authenticated session, the web interface will appear as in the following figure.
1 https://www.seafile.com/en/home/
H2020 – EINFRA – 2015 – 1 Page 32 of 37
Figure 5-1Seafile Web Interface
The SeaFile desktop application can be downloaded to have the complete functionalities offered by the server component. It guarantee a reliable file Syncing, high performance, and easy upgrade. The client application will appear as in the following figure.
Figure 5-2Seafile desktop application
5.6 Service public API’s
The other services, applications and components can access Seafile's functions remotely via REST calls. All API calls must be authenticated. The offered APIs are WEB, python and PHP respectively well documented at the following link:
https://manual.seafile.com/develop/web_api_v2.1.html
H2020 – EINFRA – 2015 – 1 Page 33 of 37
https://manual.seafile.com/develop/python_api.html
https://github.com/rene-s/Seafile-PHP-SDK
5.6.1 Troubleshooting
If the installation did not finish successfully, the following file can be verified for errors: /root/[installer_name]_installation.log.
H2020 – EINFRA – 2015 – 1 Page 34 of 37
Annex A Installation Guides
A.1. WSO2 IS Installation guide
1. On the machine dedicated to IS:Download the Identity Server2 2. Extract the archive file to a dedicated directory for the Identity Server, which will hereafter be referred
to as <IS_HOME>. 3. Set your JAVA_HOME environment variable to point to the directory where the Java Development Kit
(JDK) is installed on the computer. 4. Set your system properties when the server starts 5. Installing as a service 6. Install the Apache Web Server to put in front of the IS 7. Configure the Apache Web Server to be a proxy server for the IS
A.1.1 Software prerequisites
Application Purpose and Version
Oracle Java
SE
Development
Kit (JDK)
To launch the product as each product is a Java application.
Supported version: JDK 7 or 8.Oracle and IBM JRE 1.7 are also supported when running.
Web Browser To access the product's Management Console. The Web Browser must be JavaScript
enabled to take full advantage of the Management console
Table 10 WSO2 IS software prerequisites
A.1.2 Hardware prerequisites
Memory • ~ 2 GB minimum
~ 512 MB heap size. This is generally sufficient to process typical SOAP messages but
the requirements vary with larger message sizes and the number of messages processed
concurrently.
Disk • ~ 1 GB, excluding space allocated for log files and databases.
Table 11 WSO2 IS hardware prerequisites
A.1.3 Uninstallation procedure
1. Uninstall the service 2. Delete system properties 3. Remove the <IS_HOME>
2 http://wso2.com/products/identity-server/#download. At the time of writing this document, version 5.2 is available.
H2020 – EINFRA – 2015 – 1 Page 35 of 37
4. Remove the Apache Web Server configuration for the IS
A.2 WSO2 ESB Installation guide
On the machine dedicated to ESB:
1. Download the Enterprise Service Bus3 2. Extract the archive file to a dedicated directory for the Identity Server, which will hereafter be referred
to as <PRODUCT_HOME>. 3. Set your JAVA_HOME environment variable to point to the directory where the Java Development Kit
(JDK) is installed on the computer. 4. Set your system properties when the server starts 5. Installing as a service
A.2.1 Software prerequisites
Application Purpose and Version
Oracle Java
SE
Development
Kit (JDK)
To launch the product as each product is a Java application.
Supported version: JDK 7 or 8.Oracle and IBM JRE 1.7 are also supported when running
(not building).
Web Browser To access the product's Management Console. The Web Browser must be JavaScript
enabled to take full advantage of the Management console
Table 12 WSO2 ESB software prerequisites
A.2.2 Hardware prerequisites
Memory • ~ 2 GB minimum
• ~ 1 GB heap size. This is generally sufficient to process typical SOAP messages but the
requirements vary with larger message sizes and the number of messages processed concurrently.
Disk • ~ 1 GB, excluding space allocated for log files and databases.
Table 13 WSO2 ESB hadware prerequisites
A.2.3 Uninstallation procedure
On the machine dedicated to ESB:
1. Uninstall the service 2. Delete system properties
4 http://wso2.com/products/enterprise-service-bus At the time of writing this document, version 5.0 is available
H2020 – EINFRA – 2015 – 1 Page 36 of 37
3. Remove the <PRODUCT_HOME>
A.3 WSO2 DAS Installation Guide
On the machine dedicated to DAS:
1. Download the Data Analytics Server4. 2. Extract the archive file to a dedicated directory for the Identity Server, which will hereafter be referred
to as <DAS_HOME>. 3. Set your JAVA_HOME environment variable to point to the directory where the Java Development Kit
(JDK) is installed on the computer. 4. Set your system properties when the server starts 5. Installing as a service
A.3.1 Software prerequisites
On the machine dedicated to DAS:
Application Purpose and Version
Oracle Java
SE
Development
Kit (JDK)
To launch the product as each product is a Java application.
Supported version: JDK 7 or 8.Oracle and IBM JRE 1.7 are also supported when running
(not building).
Web Browser To access the product's Management Console. The Web Browser must be JavaScript
enabled to take full advantage of the Management console
JDBC-
compliant
Connector
for Java
Required as a standardized database driver for Java platforms and development.
Supported Version: 1.7.0 or later
Table 14 WSO2 DAS software prerequisites
Application Purpose and Version
Oracle Java
SE
Development
Kit (JDK)
To launch the product as each product is a Java application.
Supported version: JDK 7 or 8.Oracle and IBM JRE 1.7 are also supported when running
(not building).
Web Browser To access the product's Management Console. The Web Browser must be JavaScript
enabled to take full advantage of the Management console
Table 15 WSO2 ESB software prerequisites
4 https://docs.wso2.com/display/DAS310/Downloading+the+Product At the time of writing this document, version 3.1 is available
H2020 – EINFRA – 2015 – 1 Page 37 of 37
A.3.2 Hardware prerequisites
On the machine dedicated to DAS:
Memory • ~ 8 GB minimum
• ~ 4 GB minimum
Disk • ~ 1 GB, excluding space allocated for log files and databases.
Table 16 WSO2 DAS hardware prerequisites
On the machine dedicated to ESB:
Memory • ~ 2 GB minimum
• ~ 1 GB heap size. This is generally sufficient to process typical SOAP messages but the
requirements vary with larger message sizes and the number of messages processed concurrently.
Disk • ~ 1 GB, excluding space allocated for log files and databases.
Table 17 WSO2 ESB hardware prerequisites
A.3.3 Uninstallation procedure
On the machine dedicated to DAS:
1. Uninstall the service 2. Delete system properties 3. Remove the <DAS_HOME>