Data integration in real time (pvmanager, graphene and all of that) Gabriele Carcassi.

Data integration in real time(pvmanager, graphene and all of that)

Gabriele Carcassi

The problem

• Different domains (v3, v4, DISCS, CS-Studio, …) produce and consume data

• How do we integrate that data?– How does CS-Studio gets CA and PVA data?– How do we integrate archive data with live data?– How do web services get CA data?– How does v4 get web service data?– How does MATLAB gets PVA and web service data?– How does CS-Studio gets results from a script?– ….

Objective of the talk

• Discuss strategically how to reshape pvmanager/epics-util/graphene to achieve that goal

• Discuss tactically what our priorities are– Efficient use of the resources we have

• Please, interrupt!

HIGH LEVEL VIEW

Pluggabledatasources

Source rateDesired rateClient

Rate throttling:<= request<= the client can handle

Caching orqueuing


Formulalanguage



Caching orqueuing

Graphene


Formulalanguage



Caching orqueuing

Pluggableservices

Serviceregistry

Services(command/response)

Datasources(publish/subscribe)

Graphene

Status

• The framework around pvmanager has been growing in functionality– Originally started as small layer to handle caching/queueing– Now includes access to services (jdbc, pva, exec, web, …),

extensible data manipulation language, graphs (graphene), …– Used outisde CS-Studio (embedded in Vasu’s Proteus, Xihui’s

WebPDA)• Increasing contributions

– CosyLab contributing pva support, 3 UMich students working on graphene, ITER planning to contribute archived data integration, FRIB contributing array manipulation functions

Current problems• Focus dominated by CS-Studio

– Other use cases always left in the backburner (e.g. rebroadcast through pva, support for other languages, MATLAB)

– No active effort into an overall strategy• Where does one ask for support?

– Unclear mailing list– pvmanager and graphene mailing list exist but they are not used. Updates

on pvmanager were posted on cs-studio mailing list. Some reports are in CS-Studio github issues. Support for everything is private e-mail.

• Not well thought out process of contribution– Contributors often have to redo the CS-Studio integration themselves just to

test their changes– Split into 3 repositories causes problems

• Trying to step back and reassess

Possible way forward• Time to start to focus on this layer on its own merits!

– Focus more equally on CS-Studio, v4, DISCS, web access and other uses; facilitate independent integration and contributions

• Create a Data Integration In Real Time (diirt) top project– pvmanager is not a good name for something that does services and other data sources– Single mailing list for all pvmanager/vtype/epics-util/graphene development and

support– Single github issue tracker, will move all repositories to github

• More transparent release planning for diirt– Gather requirements from CS-Studio, v4 and DISCS more equally. A contact from each?

Eric/Greg/Vasu• Increase effectiveness of contributions

– Branch of CS-Studio with development version• Contributor can push changes in pvmanager repository, and get CS-Studio update for free

– Independent framework for load/integration/normalization tests• Contributor focuses his testing only up to diirt integration

Links to subprojects

Summary ofcurrent issues

Build status

Links to github

Automaticallycommittedfrom mainpvmanagerrepo

Download tolatest dev build

For example

diirtautomatic integration

Matej

Kunal

Updates pva services

Adds a formula function

Greg

Eric

Get cs-studio devversion and test

Future goal

diirtautomatic integration

Matej, Kunal,UMICH students,ITER, …

cs-studio dev

web gateway dev

pva gateway dev

Contribute to the data integration framework:automatically contribute to a varied set of applications!

TESTING

Integration test framework

• We need a way to run integration tests– test IOCs -> CAJ -> pvmanager– Including disconnects due to power cycle and network

downtime– Corner cases (e.g. different type at reconnect)– Ability to check server state (e.g. number of monitors open)– Ability to drop in and run the tests in production

environment (to check specific versions of EPICS and network configurations)

• Start a script on the server side, start a script on the client side, come back in 15 minutes

public final void run() throws Exception { // Start first IOC init("typeChange1"); // Connect pv addReader(PVManager.read(channel("double-to-i32")), TimeDuration.ofHertz(50)); pause(1000); // Stop current IOC and start the second restart("typeChange2"); pause(2000); }

public final void verify(Log log) { // Expect 3 connection notifications: connect/disconnect/connect log.matchConnections("double-to-i32", true, false, true); // Expect 3 value notifications. Match value/alarm, but not the timestamp log.matchValues("double-to-i32", ALL_EXCEPT_TIME, newVDouble(0.0, newAlarm(AlarmSeverity.INVALID, "UDF_ALARM"), newTime(Timestamp.of(631152000, 0), null, false), displayNone()), newVDouble(0.0, newAlarm(AlarmSeverity.UNDEFINED, "Disconnected"), newTime(Timestamp.of(631152000, 0), null, false), displayNone()), newVInt(0, newAlarm(AlarmSeverity.INVALID, "UDF_ALARM"), newTime(Timestamp.of(631152000, 0), null, false), displayNone())); }

Integration test framework• Covered

– Simple reboot: connect pv, ioc down, ioc up, only 1 monitor open– Simple network outage: connect, network down, network up, only 1 monitor open– Multiple reboots: connect pv, ioc cycle 10 times– Type change: connect double pv, ioc cycle, pv become integer– Constant pv: conect to double/int/string/enum that do not change– Slow changing pv: conect to double updating at 1 Hz (same rate received)– Fast changing pv: conect to double updating at 100 Hz (reduced rate received)– Alarm changing pv: conect to double updating at 1 Hz for alarm only– Write pv: change value for double/int

• Not yet covered– Add all remaining types for disconnection test– Add all types for type change– Add all types for slow changing pvs– Add all types for fast changing pvs– Add all types for alarm changing pvs– Add all types for write pvs– Add metadata changes– Add access control changes– Add multiple reader on a single pv (only 1 monitor open)– Add nanosec out of range for time– Old RTYP handling


• Martin Konrad took what I had and modified for his own purposes– Testing archiver, ca gateway, …– We diverged a little bit: need to re-align

• Andrej Babic is tasked to improve testing for CS-Studio• My hope is that these efforts can be aligned into one

– FLINT: framework for load/integration/normalization tests• Load: large number of pvs, large data size• Integration: different IOC versions, different OS, different network

configurations• Normalization: data comes out looking the same, predictable event count

and order

– Work with multiple protocols (pva and ca at least)


• Still not enough: testing on my machine does not guarantee anything– We have found that performance profile changes significantly from

machine to machine (even with same OS)• Active scanning does not produce noticeable load on my machine on any

os (Win/SL/Debian) while it does on other

– We have found that some Java network libraries behave not up to spec on different machine• Currently, I have to disable my wireless every time I run tests with ca or pva

• We need:– Set of automated tools that run tests and gather performance data– Set of people that actually run those tests pre-emptively on their

HW/OS combination

OVERALL ARCHITECTURE

pvmanager core

pva fileca

Producer/Consumer

Processing

Visualization

vTypes Data Definition

Aggregation

formula

graphene

Data integration in real time (Diirt) architecture

CS-Studio pva gateway web gatewayMATLABPython

http pva execjdbc

Command/response

cf

arch-app

Historic data

ch-arch

Data serialization/conversion

csv

png/jpgNTTypes

HDF5

datasource support services support

… …

archiver support…

Client tools Servers

DIIRT CONSUMERS

CS-Studio

• pvmanager now serves data to most applications in CS-Studio– Allows formulas in BOY– Allows access to services in BOY

• Set of common features for pvmanager configuration in eclipse product

• Graphene widget continued improvements

CS-Studio

• Areas of possible improvements in CS-Studio– Better integration in BOY (e.g. allow color widget

to be a formula)– Take advantage of pause/resume– Single point to configure color/fonts/time

format/… for all applications• Areas of possible improvements in diirt– Performance optimization (passive scanning)– Improve graphene widgets

python

• pvmanager and other diirt components are currently implemented in Java and there is a desire to access the functionality in python

• Different options:– Jython– Re-implement the whole stack in Python– Re-implement the whole stack in C++– Implement a thin library in python that uses an

embedded JVM– Others?

python – Jython

• No starter• No access to all the library that make people

want to use python in the first place

python – python implementation

• Benefits– Have all the functionality in pure python– Portability across python platforms

• Costs/challenges– Lack of well-spec’d memory and threading model?– Keeping Java and python implementations in line– Solves python, does not solve anything else

python – C++ implementation

• Benefits– Have all the functionality in C++, which makes it possible to

support other languages (such as python)• Costs/challenges– Threading and memory support OS dependent? C++11

may solve this– Probably can’t rely on immutable objects as in Java

implementation– Significantly harder to write and support: very few can do

this– Keeping Java and C++ implementations in line

python – embedded JVM

• Benefits– Leverages all the work already done– Versions keep in synch– Probably the cheapest

• Costs/challenges– Is it actually doable, with reasonable

performance? Different techniques to investigate (JNI, Py4J, javacpp, …)

– Can we keep the interface reasonable? Maybe we can start with this, and re-implement later?

– Contributions to the core are limited to Java (but maybe this is not so bad)

Py4j

diirt

CA PVA

Python apps

MATLAB

• I’d be interested to at least write a simple example of use in Matlab– At least to know if it’s possible and how easy

it is• Is this something v4 or others are

interested in?• My current showstoppers:– I don’t have MATLAB and it costs few k$– I don’t really know how to use MATLAB– I don’t really know what to do with it (need

use cases)

MATLAB

diirt

CA PVA

Web gateway using web sockets

• Desire to have web tools on top of CA• Web sockets now allow updates from servers• Couple of projects are already starting to use

pvmanager within a web server• Should have general purpose web gateway,

powered by diirt and websockets?– Standardize JSON serialization of vTypes? XML

(InfoSet, Fast InfoSet, EXI)?– Maybe a thin javascript client library as well? Re-

implement WebCA? Binding for widgets (d3js)?– Should this support services as well? Provides

access of pva services to web clients

Web server

diirt

CA PVA

Web

app

s

Mob

ile a

pps

Pva gateway on top of pvmanager

• We have talked about this forever!• Create a pva server that re-

broadcasts what gets from pvmanager– Protocol translation (ca -> pva)– Data aggregation and processing– Could also work for services (web

services -> pva)• ChannelFinder could be a good candidate

pva server

diirt

CA Web services

pva

clie

nts

pva

serv

ices

DIIRT PRODUCERS AND INTEGRATION

Historic data

• Desire to integrate archived data and live data to provide a single client resident cache for real-time processing– Graphs in time– Computation (histrograms,

correlation, fits)– Data export (to csv, HDF5, …)

• ITER is looking into contributing in this space

diirt

CA Archived data

pva

gate

way

CS-S

tudi

o

Functions

• Better support for disconnect/alarm/time• Adding element-wise array operations– arraySum(VNumberArray, Vnumber)– arrayDiv(VNumberArray, VNumberArray)

• Adding experimental DFT

Support for nd-arrays

• VNumberArray is already n-dimensional• Was extended to include cell boundaries• Boundaries used

by histogram plot• Formula function

that constructs thefull object from ahistogram record

Support for nd-arrays

• Students working on intensity graph• Hopefully will be ready

by end of May– Support for

non-homogenouscells as well

v4 integration

• Cosylab has been contributing pva support for– Datasources (already integrated)– Services (in late-March release)

• Other things we already stated we need– Extract the VTypes <-> NTTypes

conversion/wrapping library

<?xml version="1.0" encoding="UTF-8"?><pvAccessRPCService ver="1" name="pvRPCServiceSample" description="A test pvRPC service"> <channelName>testChannel</channelName> <methods> <method name="sum" description="sum the two numbers"> <structureid>uri:ev4:nt/2012/pwd:NTXY</structureid> <argument name="a" description="first number" type="VDouble"/> <argument name="b" description="second number" type="VDouble"/> <result name="c" description="sum result" type="VDouble"/> </method> <method name="concate" description="concate two strings"> <argument name="a" description="first string" type="VString"/> <argument name="b" description="second string" type="VString"/> <result name="c" description="concate result" type="VString"/> </method> <method name="getTable" description="get table as a standalone result"> <result standalone="true" description="table result" type="VTable"/> </method> <method name="getImage" description="get image as a standalone result"> <result standalone="true" description="image result" type="VImage"/> </method> <method name="multipyString" description="multiply the string"> <argument name="string" description="string to multiply" type="VString"/> <argument name="times" description="how many times to multiply" type="VInt"/> <result name="c" description="multiply result" type="VString"/> </method> <method name="getTimeInMiliseconds" description="get time in milliseconds"> <result name="c" description="time result" type="VDouble"/> </method> <method name="nop" description="method without arguments and no result"> </method> </methods></pvAccessRPCService>

v4 service example

<?xml version="1.0" encoding="UTF-8"?><pvAccessRPCService ver="1" name="pvRPCServiceSample" description="A test pvRPC service"> <channelName>testChannel</channelName> <methods> <method name="sum" description="sum the two numbers"> <structureid>uri:ev4:nt/2012/pwd:NTXY</structureid> <argument name="a" description="first number" type="VDouble"/> <argument name="b" description="second number" type="VDouble"/> <result name="c" description="sum result" type="VDouble"/> </method> <method name="concate" description="concate two strings"> <argument name="a" description="first string" type="VString"/> <argument name="b" description="second string" type="VString"/> <result name="c" description="concate result" type="VString"/> </method> <method name="getTable" description="get table as a standalone result"> <result standalone="true" description="table result" type="VTable"/> </method> <method name="getImage" description="get image as a standalone result"> <result standalone="true" description="image result" type="VImage"/> </method> <method name="multipyString" description="multiply the string"> <argument name="string" description="string to multiply" type="VString"/> <argument name="times" description="how many times to multiply" type="VInt"/> <result name="c" description="multiply result" type="VString"/> </method> <method name="getTimeInMiliseconds" description="get time in milliseconds"> <result name="c" description="time result" type="VDouble"/> </method> <method name="nop" description="method without arguments and no result"> </method> </methods></pvAccessRPCService>

v4 service example

Best part:I have no idea how itworks! Cosylabsupports it andCS-Studio picks it up.

DISCS integration

• Proteus is already embedding pvmanager for ca data access– Could benefit from standard web gateway

• We should more actively try DISCS service integration through diirt (and CS-Studio)– Maybe standard web service bindings to JSON, so

we can (like JDBC and PVA) only require to write an XML file?

diirt service interfacepublic abstract class ServiceMethod {

public String getName(); public String getDescription();

public Map<String, Class<?>> getArgumentTypes(); public Map<String, String> getArgumentDescriptions();

public final Map<String, Class<?>> getResultTypes(); public Map<String, String> getResultDescriptions(); public abstract void executeMethod( Map<String, Object> arguments, WriteFunction<Map<String, Object>> callback, WriteFunction<Exception> errorCallback);}

Goals for diirt services

• Not general purpose! Some services in some cases– Have parameters that can be expressed by vTypes (e.g.

VString and VNumbers)– Can return data that can be expressed in a vTable,

vNumericArray, …– These can be digested by general purpose clients

• For example, channel finder– Will need a specific UI to manage tags and properties– Can export into a table the result of a query

• For the general purpose only we use diirt services

Where to draw the line?

• What do I use for creating a Java UI client? diirt!

• What do I use to access services? diirt!• What do I use to do protocol conversion? diirt!• What should I use to toast my bread? diirt!• What should I give my spouse for Christmas?

diirt!


• All data can be accessed in diirt != all data access should use diirt

• I am fully committed to the first, I am really skeptical about the second– I don’t know all possible use cases– Software design is about tradeoffs– Waste of resources


• For services, I think it’s already clear: only calls that can be used by “general purpose clients”– Is the client specific to your service? (e.g. log entry

for logbook, property management for channelfinder)

– Are VTypes unreasonable representations for request and response?

– Is the data useful only in the context of its service?• Services integrated in diirt should be the partial

“view” that is “general purpose”


• For datasources (CA and PVA specifically)• Some clients that will never use diirt because they need direct access to

the protocol API:– Debugging tools– IOC

• Some clients should use something like diirt because they are just going to re-implement it:– UI tools– General purpose clients– Web socket gateway

• Some things are in between, and I think it depends on different trade-offs– PVA gateway

• Do you want to faithfully forward the precise semantic of the pva protocol?• Do you want make available data from other sources and do aggregation?


• There is a problem of architecture decency:– An interface can’t be both general purpose and give access to

everything protocol specific• If you expose all the options of all the possible protocol

implementations…– Current: ca:// pva:// file://

• Formulas - e.g. =`pvArray1` + columnOf(‘pvTable’, “column2”)

– Planned: http:// jmx://– Talked about: jms:// snmp://

• I think we just end up with an unusable mess– Each new feature needs to be somehow understood by all clients

and all datasource implementation


• There is a problem of resource allocation:– We already have a CA and PVA library and multiple

people working on them– Suppose we implement CA GET CALLBACK in

pvmanager: what is it that we can then do that we couldn’t before? What is the cost to all clients and all data providers that now has to support it?

– Doing things through diirt by itself is not a value


• I think “Data integration in real time” identifies the scope pretty nicely: it has value if it’s a new way to get/combine/publish data– Are you creating an application that is protocol or service specific?– Do you need to access functionality (modes of communications

and/or data types) that depends on a specific service or protocol?– Is your application running in its own process (or with others in a

framework like Eclipse)? Is anybody going to configure which datasources/services it access?

• The more you answer yes, the more you should consider going to the original API directly


• We can still “kludge” in some things– 99% is no, but there is that _one_ thing

• E.g. CA PUT CALLBACK is supported by adding some JSON after the channel name

• Could support file locks as JSON after the filename

But it will never be like native protocol support• We can’t sacrifice the general purpose client– Transparent access to different datasources, services,

file format, …

because that’s the whole point of diirt!

Data integration in real time (pvmanager, graphene and all of that) Gabriele Carcassi.

Documents

Transcript of Data integration in real time (pvmanager, graphene and all of that) Gabriele Carcassi.