A. Dworak BE-CO-IN, CERN. Agenda 228th June 2012 Sum up of the previous report Middleware...
Transcript of A. Dworak BE-CO-IN, CERN. Agenda 228th June 2012 Sum up of the previous report Middleware...
![Page 1: A. Dworak BE-CO-IN, CERN. Agenda 228th June 2012 Sum up of the previous report Middleware prototyping Transport Serialization Design concepts.](https://reader036.fdocuments.us/reader036/viewer/2022062314/56649e315503460f94b223bc/html5/thumbnails/1.jpg)
THE NEW CMW-RDAMOTIVATION, DESIGN AND TECHNICAL CHOICES
BE/CO TC28TH JUNE 2012, GENEVA
A. Dworak
BE-CO-IN, CERN
![Page 2: A. Dworak BE-CO-IN, CERN. Agenda 228th June 2012 Sum up of the previous report Middleware prototyping Transport Serialization Design concepts.](https://reader036.fdocuments.us/reader036/viewer/2022062314/56649e315503460f94b223bc/html5/thumbnails/2.jpg)
2
Agenda
28th June 2012
Sum up of the previous report Middleware prototyping
Transport Serialization
Design concepts Keep proven solutions Modularity Data Model Transport Client Server
Work progress
Andrzej Dworak, The new CMW-RDA
![Page 3: A. Dworak BE-CO-IN, CERN. Agenda 228th June 2012 Sum up of the previous report Middleware prototyping Transport Serialization Design concepts.](https://reader036.fdocuments.us/reader036/viewer/2022062314/56649e315503460f94b223bc/html5/thumbnails/3.jpg)
Andrzej Dworak, The new CMW-RDA 3
Previous reports
28th June 2012
“Middleware Trends and Market Leaders 2011” 14/10/2011, ICALEPCS 2011 20/10/2011, BE-CO-IN TC
![Page 4: A. Dworak BE-CO-IN, CERN. Agenda 228th June 2012 Sum up of the previous report Middleware prototyping Transport Serialization Design concepts.](https://reader036.fdocuments.us/reader036/viewer/2022062314/56649e315503460f94b223bc/html5/thumbnails/4.jpg)
Andrzej Dworak, The new CMW-RDA 4
Previous reports
28th June 2012
“Middleware Trends and Market Leaders 2011” 14/10/2011, ICALEPCS 2011 20/10/2011, BE-CO-IN TC
Requirements gathering, user interviews, internal reviewsShortcomings of the system, missing functionality
![Page 5: A. Dworak BE-CO-IN, CERN. Agenda 228th June 2012 Sum up of the previous report Middleware prototyping Transport Serialization Design concepts.](https://reader036.fdocuments.us/reader036/viewer/2022062314/56649e315503460f94b223bc/html5/thumbnails/5.jpg)
Andrzej Dworak, The new CMW-RDA 5
Previous reports
28th June 2012
“Middleware Trends and Market Leaders 2011” 14/10/2011, ICALEPCS 2011 20/10/2011, BE-CO-IN TC
CORBA: C++ and Java implementations differ Shrinking community Lack of new releases and bug fixes Expensive on resources Complex error prone API Problems with asynchronous and
non-blocking communication No support for pub/sub Missing support for versioning or
flexible interfaces (hard to upgrade)
Current design: Monolithic (hard to maintain, extend
and test) Not scalable (poor parallelization) Not optimal design makes it
unnecessarily heavy and slow Lack of QoS, message priorities Missing server-side management of
client connections After all the years and fixes it is BIG
Ma
jor
pro
ble
ms
with
CO
RB
A
and
cu
rre
nt c
ode
ba
se u
sin
g it:
Fro
m u
sers
’ p
ersp
ectiv
e
Service instability Client may influence Server Fails when overloaded Does not scale Impossible to optimize the usage
Unnecessarily limited (e.g. connection #), slow and heavy.
Needs workarounds (e.g. Proxy) Unfriendly API Lack of full diagnostics
===========================================================
![Page 6: A. Dworak BE-CO-IN, CERN. Agenda 228th June 2012 Sum up of the previous report Middleware prototyping Transport Serialization Design concepts.](https://reader036.fdocuments.us/reader036/viewer/2022062314/56649e315503460f94b223bc/html5/thumbnails/6.jpg)
Andrzej Dworak, The new CMW-RDA 6
Previous reports
28th June 2012
“Middleware Trends and Market Leaders 2011” 14/10/2011, ICALEPCS 2011 20/10/2011, BE-CO-IN TC
![Page 7: A. Dworak BE-CO-IN, CERN. Agenda 228th June 2012 Sum up of the previous report Middleware prototyping Transport Serialization Design concepts.](https://reader036.fdocuments.us/reader036/viewer/2022062314/56649e315503460f94b223bc/html5/thumbnails/7.jpg)
Andrzej Dworak, The new CMW-RDA 7
Previous reports
28th June 2012
“Middleware Trends and Market Leaders 2011” 14/10/2011, ICALEPCS 2011 20/10/2011, BE-CO-IN TC
![Page 8: A. Dworak BE-CO-IN, CERN. Agenda 228th June 2012 Sum up of the previous report Middleware prototyping Transport Serialization Design concepts.](https://reader036.fdocuments.us/reader036/viewer/2022062314/56649e315503460f94b223bc/html5/thumbnails/8.jpg)
Andrzej Dworak, The new CMW-RDA 8
Previous reports
28th June 2012
“Middleware Trends and Market Leaders 2011” 14/10/2011, ICALEPCS 2011 20/10/2011, BE-CO-IN TC
IceYAMI4
Presented so far
![Page 9: A. Dworak BE-CO-IN, CERN. Agenda 228th June 2012 Sum up of the previous report Middleware prototyping Transport Serialization Design concepts.](https://reader036.fdocuments.us/reader036/viewer/2022062314/56649e315503460f94b223bc/html5/thumbnails/9.jpg)
Andrzej Dworak, The new CMW-RDA 9
Middleware prototyping
28th June 2012
More testsScalabilityCorrectnessError handlingExceptional situations
Prototyping of the system parts
IceYAMI4
USE
![Page 10: A. Dworak BE-CO-IN, CERN. Agenda 228th June 2012 Sum up of the previous report Middleware prototyping Transport Serialization Design concepts.](https://reader036.fdocuments.us/reader036/viewer/2022062314/56649e315503460f94b223bc/html5/thumbnails/10.jpg)
Andrzej Dworak, The new CMW-RDA 10
Serialization Prototyping
28th June 2012
Necessary to transport data between different architectures Currently self-describing serialization
Based on CORBA (on top of IDL) Stay with self-describing serialization, chose better library
Used so far in RDA No need to manage IDLs Easier to upgrade or change the protocol Need to encode type description
Slightly slower Slightly bigger output
Review of +10 serialization libraries Testing and prototyping with:
Google Protocol Buffers MessagePack Apache Avro Compared with CORBA
MessagePackUSE
![Page 11: A. Dworak BE-CO-IN, CERN. Agenda 228th June 2012 Sum up of the previous report Middleware prototyping Transport Serialization Design concepts.](https://reader036.fdocuments.us/reader036/viewer/2022062314/56649e315503460f94b223bc/html5/thumbnails/11.jpg)
Andrzej Dworak, The new CMW-RDA 11
More details at:
28th June 2012
MiddlewareGeneral review and tests
http://wikis/display/MW/CMW+ReviewPresentation of the review and test results
http://wikis/download/attachments/49218519/FRBHMULT05_talk.pdf?version=1&modificationDate=1328608532000
More details on the 1st step also in A. Dworak “Middleware trends and market leaders 2011”
More details on the 2nd step also in A. Dworak “The new CERN Controls Middleware”
SerializationPresentation with review details and test results
http://wikis/download/attachments/49218519/Self-describing+Serialization.pdf?version=1&modificationDate=1328608647000
![Page 12: A. Dworak BE-CO-IN, CERN. Agenda 228th June 2012 Sum up of the previous report Middleware prototyping Transport Serialization Design concepts.](https://reader036.fdocuments.us/reader036/viewer/2022062314/56649e315503460f94b223bc/html5/thumbnails/12.jpg)
Andrzej Dworak, The new CMW-RDA 12
Design concepts – proven concepts
28th June 2012
Keep the proven concepts: Device/property model Communication through get, set and subscribe/publish
calls Properties represented as dynamic Data objects Self-describing serialization Authorize and authenticate user access
![Page 13: A. Dworak BE-CO-IN, CERN. Agenda 228th June 2012 Sum up of the previous report Middleware prototyping Transport Serialization Design concepts.](https://reader036.fdocuments.us/reader036/viewer/2022062314/56649e315503460f94b223bc/html5/thumbnails/13.jpg)
Andrzej Dworak, The new CMW-RDA 13
Design concepts – modularity
28th June 2012
Less coupling between system parts Easier to test and maintain
dependency injectionmocking
Will allow to exchange parts of the system (e.g. transport = ZeroMQ or serialization)
Will help during the migration process (old RDA to new RDA)
![Page 14: A. Dworak BE-CO-IN, CERN. Agenda 228th June 2012 Sum up of the previous report Middleware prototyping Transport Serialization Design concepts.](https://reader036.fdocuments.us/reader036/viewer/2022062314/56649e315503460f94b223bc/html5/thumbnails/14.jpg)
Andrzej Dworak, The new CMW-RDA 14
Design concepts – transport
28th June 2012
Transport executed asynchronously and independently of the user code execution
Separate channels for different types of communicationAdministrativeGet and SetSubscriptions
Optimization Internal and through user API
Intelligent buffering on the transport level Immutable data allows better and safer parallelization 0-copy (when sending, for receiving not 0 but also optimized)
Extended diagnostics
![Page 15: A. Dworak BE-CO-IN, CERN. Agenda 228th June 2012 Sum up of the previous report Middleware prototyping Transport Serialization Design concepts.](https://reader036.fdocuments.us/reader036/viewer/2022062314/56649e315503460f94b223bc/html5/thumbnails/15.jpg)
Andrzej Dworak, The new CMW-RDA 15
Design concepts – data model
28th June 2012
Property as a set of named fields with dynamic types Current type set extended to support:
Fixed length integer types (C++ int64_t, int32_t, int16_t, int8_t)Multidimensional arraysNested Data objects
Management of data ownershipAllows 0-copy for arrays and nested data objects
Context class with direct support forFiltersRBAC tokens
![Page 16: A. Dworak BE-CO-IN, CERN. Agenda 228th June 2012 Sum up of the previous report Middleware prototyping Transport Serialization Design concepts.](https://reader036.fdocuments.us/reader036/viewer/2022062314/56649e315503460f94b223bc/html5/thumbnails/16.jpg)
Andrzej Dworak, The new CMW-RDA 16
Design concepts – client
28th June 2012
Can get, set and monitor values of properties Uses predefined policies which will handle communication events
and errors Connect/reconnect policy Subscribe policy
Underlying communication is always non-blocking and asynchronous
For convenience API allows to perform the calls as Asynchronous Synchronous Buffered asynchronous Buffered synchronous
RBAC role may guarantee a client a higher priority Client may be managed on-line (similarly to servers and cmw-
admin)
![Page 17: A. Dworak BE-CO-IN, CERN. Agenda 228th June 2012 Sum up of the previous report Middleware prototyping Transport Serialization Design concepts.](https://reader036.fdocuments.us/reader036/viewer/2022062314/56649e315503460f94b223bc/html5/thumbnails/17.jpg)
Andrzej Dworak, The new CMW-RDA 17
Design concepts – server
28th June 2012
Asynchronous execution of implemented get, set and (un)subscribe handlers
Non blocking data publishingMay set delivery policy (normal/background)Message priority and QoS
Extended management (also on-line via CMW Admin)Turn on/off get, set, publishing(Un)block client / low-priority clientsDisconnect client / low-priority clientsSend commands to clients
Ping/hello Disconnect request “Failover to” request