Antonio Romero Marin, Aurelien Fernandes, Jose Rolland Lopez De Coca, Nikolay Tsvetkov, Zereyakob...

APEX Experience from the Accelerators Controls and Operations

Data Management in the Beams Department

Antonio Romero Marin, Aurelien Fernandes, Jose Rolland Lopez De Coca, Nikolay Tsvetkov, Zereyakob Makonnen,

Zory Zaharieva

BE-CO

Database Developers Forum – APEX

20.05.2014

APEX Experience from the Accelerators Controls and Operations Data Management 20.05.2014 22

Contents

• Introduction to the Controls environment and some of the services

using APEX

• Interesting features for developers

• APEX_COMMON Framework

• Plugins & custom components, look and feel features, etc.

• Best Practices

• Conclusion

Z.Zaharieva 22.04.2013 Controls Configuration Service - LS1 Plans EDMS doc: 1281984 2


Introduction to the Accelerators Data Management

• Complex and challenging area

• Federation of various databases, e.g.• Accelerators Controls Configuration• Diagnostics and Monitoring (DIAMON)• Alarms (LASER)• Accelerators Entities and Signals Naming• Logging and Measurements, etc.

• Strategy• Relational DB technology – Oracle DBs for all technical data• Data-driven applications and APIs - Oracle technology stack

• Reliable database services and user interfaces (24/7/365)• On-line usage of database services for the accelerators Control and

Operation


APEX Development throughout the years

• Starting with HTMLDB• beginning of 2005, HTMLDB v.1.6

• Need for rapid application development (Agile programming)• fast prototyping and short time to production deployment

• quickly respond to new user requirements

• Ready to use templates, widgets and page components• professional look-and-feel with minimum effort on design – concentrate on the

application itself

• The nature of the interfaces to be implemented• web-interfaces, database-centric• APEX supports client-side scripting for more dynamic / interactive / feature-rich

interfaces


First Trials of APEX

• Data browsing interfaces and portal for the Accelerators Entities and Signals Naming Database

• The Naming DB – dictionary for the accelerators complexes• Equipment types• Equipment positions in the accelerators• Settings and observables (signals)

• Read-only interfaces to the data• LHC Equipment catalogue• SPS equipment catalogue• PS equipment and layout components catalogue• Signals Simple Extraction Interface• Signals Experts Extraction Interface• Naming Domain Portal


Accelerators Naming DB Interfaces

• Data-browsing Interfaces (window-on-data) – development from scratch• low level of complexity• no transactions• some business logic to get the data out• user authentication and simple user authorization• easy extraction of data into XML/CSV files


Portal Pages and News feeds

• Simple portal pages

• Access to other related interfaces, documentation, help pages

• Dynamically constructed using data stored in tables

• News, RSS feeds, News Archive, etc.


History Log Browsers

• History logging in our dbs• All data modifications are recorded (to know who did what and when)

• History Log Browser objectives• Access the history log for any table in the database for which history is

enabled • Provide simple search capabilities for the history logs• Provide a summary for the changes of a given record based on the PK

• Completely dynamic configuration• List of tables, list of columns per table, primary keys for tables (USER_

tables)• Dynamic PL/SQL embedded in APEX regions

• Portable solution• Deployed to several of the databases, which are using the same mechanism

for keeping history without any changes to the interface components


History Log Browsers


• The heart of the Controls System – the basis for the Configuration Management of all Controls System components in the entire accelerator complex

• Provides data for all configuration items and their relationships, required for the correct functioning of the Controls System

• All controls devices (>105,000) and parameters (> 2,000,000)

• Hardware and software configuration of all Front-End Computers

(>3,500), Drivers generation

• The Accelerators Timing System, Operators Consoles, etc.

• Service with 30 years of history• Supporting the requirements of the entire accelerator complex – LHC, SPS and PS

complex

Controls Configuration Service


Controls Configuration Data Browser

• An existing PL/SQL OWA application with ~ 100 reports• existing PL/SQL code base in the CCDB• a complete re-engineering of the application was undertaken in 2006• possibility to reuse some of the PL/SQL code• ‘thick’ database layer – PL/SQL packages in the database

• The first big scale data browsing application to use APEX

• Nowadays >220 reports used by a user community of >300 people - thanks to• the ease of APEX reports development• Introduction of APEX user interactive reports


Alarms (LASER) Database

• LASER (Alarms) - capturing, storing and notification of anomalies for the whole accelerator chain and technical infrastructure

• Alarms DB - 3 different database areas• Pre-defined alarm definitions• User configurations for the alarms displays• Time-stamped run-time alarms events

• First APEX data editing interfaces - 2010• fast development• evolution of APEX• suite of Data Management Tools (7 Editors)• developed from scratch• give users the possibility to explore their

data and maintain it


Controls Configuration Data Editors

• Data-manipulation interfaces• complexity of the applications due to transaction management• complex business logic to handle modifications of the data• stringent data access rights – e.g. specific authorization modules, virtual

private db, etc.

• Existing web-deployed, rich clients, using Oracle ADF

• Renovation of the existing interfaces and providing new ones with APEX since beginning of 2011


Controls Configuration Data Editors

• The first relatively big scale and complex editing applications to be developed / migrated to APEX

• So far - 15 Data Editors (> 100 forms), ~ 250 user community

• 4 editors remaining to be renovated


Internal developers applications

• Due to the ease of development, the developers are using APEX to create applications for managing some of the developers specific processes

• Admin Editor for Service Interventions

• DB Schema Change Management Editor

• APEX _COMMON Metadata Editor


APEX in the Controls Environment nowadays

• Specific reporting tools• used for CCS, Laser, Diamon, Accelerators Naming, Measurements • > 300 reports

• Generic reporting modules and applications for managing the developers tasks• History Browser; Schema Change Management; Admin Interventions Config, etc.

• Portals and news pages with RSS feeds

• Data editors• used for CCS, Laser, Diamon, Layout, R2E, SM18 Data Analysis • > 25 editors (> 200 editing forms)


Using APEX for the accelerators data management

• Flexible environment for development• Allows you to do a lot of things• It is up to the developers to provide a structured approach to application

design

• Complete integration with our databases (Oracle, PL/SQL)

• Provides a lot of features and components ‘out of the box’• Fast and easy development

• We can do even better • Custom libraries and components

• Best practices, guidelines and coding standards

Speed and Scalability


APEX_COMMON framework

• APEX built-in tabular forms

• APEX manual tabular forms

• APEX_COMMON framework

• Examples and demo

• Evolution of the framework



Tabular forms – what is it?

• Grid to edit multiple rows or records at once


Built-in tabular forms in APEX

• Very easy creation with APEX wizard

• Provide built-in functionality for insert/update/delete

• Other built-in features (validations, partial refresh, CSV export…)


Limitations of the APEX built-in tabular forms

• Really easy to create, but there are important limitations

• Just ONE tabular form per page

• No Parent-Child tabular forms

• Conditional rendering is not possible

• Does not keep changes across page submits


APEX manual tabular forms

• APEX report using the APEX_ITEM package

• Usually using also APEX collections

• temporary storage of data within a user session

• Require more work than normal tabular forms but overcomes the

built-in tabular form limitations

• It also adds new features and possibilities

• e.g. use of interactive reports


Client-side

Server-side

Manual tabular forms – workflow processing

• Steps to process a manual tabular form - using collections

1. Create and load data into collection from the table

2. Create query using APEX_ITEM package

3. Update the collection

4. Commit changes in the DB

APEX Collection

Report on Collection

Tables

3

4

Pag

e R

end

erin

g

Pag

e Pro

cessing

1

2


APEX manual tabular forms – creation process

• You have to define many components for each collection

• APEX global processes, JavaScript functions, PLSQL procedures, etc. …

• Error prone

• Collection columns names like c002, page item names like f01

• If you want to use AJAX it adds more complexity

• Difficult to maintain the page and to follow its evolution, specially

when there are multiple developers


Our environment

• Large number of applications and pages

• Need of rapid development

• Many parent-child tabular forms in the same application page

• Custom logic that APEX standard components do not cover

• e.g. conditional item rendering

So we created APEX_COMMON to make those tasks easier


What is APEX_COMMON?

• A framework to ease the development of manual tabular forms Creation

Manipulation

Maintenance

• Modify behaviour of multiple pages in a systematic and fast way• Reusability of components

• Set of generic components (JavaScript libraries, PL\SQL packages,

APEX processes, etc.)

• Generates code for the manual tabular form

• Use standard APEX libraries


APEX_COMMON Architecture Overview

APEX Application

AJAX On Demand processes and POST submits

APEX_COMMONmetadata

APEX_COMMON

PL/SQL API

Database User Schema

APEX_COMMON JavaScript API


• DB side• Metadata in tables for the manual tabular form definitions• APEX_COMMON PL/SQL package

• create collections• manipulate collections• commit changes in DB target tables

• Global APEX Application processes• Interfacing with the db

Architecture and Components


• Client side

• Global APEX dynamic actions (functions)

• Global JavaScript library

• Generic functions to interact with the data (upload data from page to collections, duplicate rows in tabular form…).

• Other generic features (highlight rows)

Architecture and Components

APEX Application

APEX_COMMON JavaScript API


• Tool to support APEX_COMMON

• Declare tabular forms definitions

• Automatic code generation

APEX COMMON CCDB Tool


Examples

and

DEMO


• Front-End Computer configuration form (Crates and Modules) • multiple child forms

Controls Configuration Hardware Editor


• Hardware Type Properties• custom logic

Controls Configuration Hardware Editor


• Virtual Classes form• 3 level parent-child forms

Controls Devices Editor


Create a manual tabular form

1. Without APEX_COMMONS

2. With APEX_COMMON

Demo


Evolution of the APEX_COMMON

• During the last 9 months since the creation of the APEX COMMON

• Optimizing performance when using collections and large data sets• save only the changes in the collections

• Management of APEX collection• 18 => 48 columns per manual tabular form

• Shared code in Page 0• Highlighting, Check all, Duplicate, Add Row…


Features under development – part1

• Editable Interactive Report• All common dynamic actions in Page 0 (e.g. load collection,

update collection)=> Less code in the pages


• Summarize user actions before commit

• More auto generation=> Fill information from database dictionary

• Capture page exit event and warning user about pending modification

• Optimization of server calls=> JSON manipulation in PL/SQL and JavaScript

Features under development – part2


APEX_COMMON Summary

• Multiple Tabular Form per page• APEX 5.0 will NOT have support for multiple tabular forms• APEX 5.1 will introduce a new region type that allows it

• It is in the statement of direction but… will it really be there?

• Low cost of code evolution and maintenance(code standardization)

• Easy to introduce custom logic in the pages

• It is a portable solution – you can use it• APEX_COMMON Wiki tutorial

• https://wikis/display/config/Create+manual+tabular+form+with+APEX_COMMON+package

https://wikis/display/config/Create+manual+tabular+form+with+APEX_COMMON+package

https://wikis/display/config/Create+manual+tabular+form+with+APEX_COMMON+package


Interesting Features for APEX Developers

• Cascading LOVs in tabular form

• Plugins

• Advanced manual tabular form with dynamic number of columns

• Bulk data loading features



• What do we gain from them?• make the application more user friendly• prevent the users of making mistakes

• Why is it interesting to present them today?• they are not built-in APEX components• could be useful for other APEX developers

Custom made components for APEX


Cascading LOVs in tabular form

• What does it mean cascading List Of Values?• the selection of the second LOV depends on the first one


• How does it look in our data editors?



• How do we do it ?• JavaScript• jQuery• JSON• OnDemand PL/SQL process

• Find the Wiki page on the following link: https://wikis.cern.ch/display/config/Create+a+cascading+LOV+Parent+columns+in+a+Tabular+Form


https://wikis.cern.ch/display/config/Create+a+cascading+LOV+Parent+columns+in+a+Tabular+Form




Plugins – usage and observations about them

• The most interesting and often used ones• Custom pop-up plugins for selection• Save before exit plugin

• Advantages of the custom pop-up plugins• possibility of quick search in multiple columns• shows structured data based on a custom SQL query• possibility of multiple columns update • doesn’t open new browser window

• Find “Custom Pop-up plugin for selection” Wiki pagehttps://wikis.cern.ch/pages/viewpage.action?pageId=70681028

https://wikis.cern.ch/pages/viewpage.action?pageId=70681028

https://wikis.cern.ch/pages/viewpage.action?pageId=70681028


Custom pop-up plugin for data selection

APEX provided one


Custom selection pop-up


• Transactional model in our data editors• auto-commit is not an option• the user decides when to commit or rollback the

changes

• Why do we need to check for changes?• prevent losing data modifications• ignore changes in the search region

• Find “Save Before Exit” Wiki page https://wikis.cern.ch/display/config/Prompt+user+when+leaving+or+closing+a+page+without+saving+the+form

Save before exit plugin

https://wikis.cern.ch/display/config/Prompt+user+when+leaving+or+closing+a+page+without+saving+the+form

https://wikis.cern.ch/display/config/Prompt+user+when+leaving+or+closing+a+page+without+saving+the+form


Save before exit plugin


Save before exit


• What was the challenge?• columns generated dynamically based on data stored in

a table

• How do we do it?• build SQL dynamically using pivot in order to transform

data stored into table rows and map them to columns • use “Generic Column Names” tabular form option• save the data in the DB using unpivot

Advanced manual tabular form with dynamic number of columns


Dynamically generated columns in a tabular form

Table1 Devices – names of devices stored in rows

Table2 LogicalEvents – stored in rows and presented in columns.Devices from the same class have the same number of LogicalEvents

Table3 Event Configuration – data stored in rows, based on the device and LogicalEvent


Bulk data loading features

• What is bulk data loading?

• Who benefits from providing this functionality?• users are able to import big sets of data using CSV files• support team reduces the number of requests

• We improved the standard APEX component• to provide more structured information• to add different highlighting for all the actions• to show result details for all updated and inserted rows

• Find the Wiki page on the following link https://wikis.cern.ch/display/config/CSV+data+import+-+APEX+Data+Loading+wizard

https://wikis.cern.ch/display/config/CSV+data+import+-+APEX+Data+Loading+wizard

https://wikis.cern.ch/display/config/CSV+data+import+-+APEX+Data+Loading+wizard


Interesting Features for APEX Developers

• From static to dynamic contents

• Look and feel features

• Collaborative development

• Functional testing

• The good, the bad and the ugly

• Best Practices



How to move from static to dynamic contents(1)


How to move from static to dynamic contents(2)

• Dynamic Action • Region refresh, Set value, Show, Hide, Clear• Handling of evens

• Click/doubleClick• Load/Unload page• KeyPress/MouseButtonPress• Change• …

• Report partial refresh (Page Partial Refresh - PPR)


How to move from static to dynamic contents(3)• JQuery (APEX > 4.0)

• Standard APEX API• Dynamic call to application process

// create ondemand requestvar req = new apex.ajax.ondemand(‘apex_application_process_name',function() { if(p.readyState == 1||p.readyState == 2||p.readyState == 3) { //process has not yet finish… } else if(p.readyState == 4) {//process has finish….. } else {return false;}});// send arrays (tabular form columns)req.ajax.AddArrayItems(apex.jQuery('[name=f03]'),1);// submit the ajax processreq._get();

• Region refresh$('#report_static_id').trigger('apexrefresh');

• Set and get Item value$s(‘P2_ITEM_ID’,4) , $v(‘P2_ITEM_ID’)


Look and feel features(1)

• How to keep the Tabular form features and reduce the number of columns for very large tables/forms• show the data in a different way

https://wikis.cern.ch/display/config/How+to+customize+your+Tabular+Form+column+disposition





• Space optimization on a page


Collaborative development useful features

• Use of a base Application• Authentication Scheme• Authorization Scheme• Theme• Navigation bar• Plugins

• Process for common management of objects• JS files, Images, workspace theme

• Page Lock feature


Functional testing for web interface

• Investigated into testing framework Watir WebDriver

• It uses Selenium WebDriver

• Higher level API than Selenium WebDriver

• Used by industry leading companies

• Open source

• Integration with Bamboo (standard BE-CO Continuous integration testing framework) (https://www.atlassian.com/software/bamboo )

https://www.atlassian.com/software/bamboo




The good, the bad and the ugly• Advisor

• Utilities-> Advisor

• Advisor does not check the application process

• Passing parameter via URL • (:) must be encoded before sending it

• Two tabs browser problems • no standard solution provided in APEX yet for this known

web applications issue• not possible do define the scope of page item


Some Best Practices

• Use aliases at application / page level

• Making the application portable

• Use of APEX-provided substitution strings - &APP_ID., &APP_ALIAS., etc.

• Custom-defined substitution strings – application definition level

• Reduce at the minimum the code in the APEX pages

• Move the PL/SQL code into DB packages.

• Best Practices For PL/SQL Development in Oracle Application Express (Steven Feuerstein)

http://www.toadworld.com/cfs-file.ashx/__key/communityserver-wikis-components-files/00-00-00-00-03/plsql-in-apex-best-practices.pdf








Contents

• Introduction to the Controls Environment and services using APEX

• APEX Commons Framework

• Interesting features for developers

• Plugins & custom components, look and feel features, etc.

• Best Practices

• Conclusion



Using APEX for the accelerators complex

• APEX has evolved a lot during the last 9 years since we started using it in the domain of the Accelerators Data Management

• Could be used from small to relatively large/complex applications• Has proven itself able to fulfill the stringent application needs of the

accelerators data management

2005

2014


Using APEX for the accelerators complex

• Quality assurance rules and guidelines in order to ensure scalability• APEX guidelines – workspace structure, version control, naming

conventions, etc.

• PL/SQL and SQL guidelines

• coding standards and best practices

• Providing common components – customized objects and additional facilities for even faster development

• Uniformity within the applications

• Faster development

• Easy maintenance


Questions?

Thank you for your attention!

[email protected] [email protected] [email protected] [email protected] [email protected] [email protected]

mailto:[email protected]






Antonio Romero Marin, Aurelien Fernandes, Jose Rolland Lopez De Coca, Nikolay Tsvetkov, Zereyakob...

Documents

Transcript of Antonio Romero Marin, Aurelien Fernandes, Jose Rolland Lopez De Coca, Nikolay Tsvetkov, Zereyakob...