New logging and notification system for the Offline DropBox
16
New logging and notification system for the Offline DropBox Martynas Mitrulevi č ius Thursday, 26 April 2012 AlCaDB/PdmV meeting Vilnius University, Lithuania
description
New logging and notification system for the Offline DropBox. AlCaDB/ PdmV meeting. Thursday, 26 April 2012 . Martynas Mitrulevi č ius. Vilnius University, Lithuania. Outline. What is Offline DropBox Motivation Internship goals Authentication SQLite upload Meta data upload - PowerPoint PPT Presentation
Transcript of New logging and notification system for the Offline DropBox
New logging and notification system for the Offline DropBox
Martynas Mitrulevičius
Thursday, 26 April 2012
AlCaDB/PdmV meeting
Vilnius University, Lithuania
Martynas Mitrulevičius 2
Outline• What is Offline DropBox• Motivation• Internship goals• Authentication• SQLite upload• Meta data upload• Meta data structure• Upload workflow• Upload script• Offline DropBox DB schema• Online DropBox DB schema• Workflow in Online DropBox• References
April 26, 2012
Martynas Mitrulevičius 3
What is Offline DropBox
The Offline DropBox service:• is implemented using web application and
supports the automatic export of calibration constants.
• simplifies tasks for end-users in case they need to upload calibration resulting from an Offline analysis into the condition databases.
• is used to facilitate the automation of alignment and calibration workflows.
April 26, 2012
Martynas Mitrulevičius 4
Motivation• The current application is working reliably and smoothly,
allowing for calibration constants to be exported into production DB, but some consolidation from the development and the operation team is needed, and some features were also requested by end-users
• In the Offline part of the DropBox the notification system for the exportation of calibration constants is not implemented.
• In the online process, only the logging coming from the CMS Framework libraries (known as PopCon monitoring) is implemented.o Tails of the log file are exported for checking by expertso http://cms-conddb.cern.ch/popcon
April 26, 2012
Martynas Mitrulevičius 5
Internship goals• Get acquainted to technologies and frameworks
used at CERN and in the CMS experiment.• Create PdmV database.• Create Offline DropBox server with new logging
and notification system.o Create a Web page allowing for calibration dataset upload.o Create CLI for the automatic exportation of calibration constants.
April 26, 2012
Martynas Mitrulevičius 6
Authentication
April 26, 2012
• Authentication using CERN username and password• Only for members of cms-cond-dropbox• CERN Authentication SOAP WebServices for user authentication
Martynas Mitrulevičius 7
SQLite upload
April 26, 2012
• Necessary tables in SQLite file:• ORA_DB• ORA_NAMING_SERVICE• ORA_C_COND_IOVSEQUENCE• ORA_C_COND_IOVSEQU_A0
Martynas Mitrulevičius 8
Meta data upload
April 26, 2012
Martynas Mitrulevičius 9
Meta data structuredestDB connectionstring tag tagname inputtag inputtagname since begin Timetype timeSpec IOVCheck checker DuplicateTagHLT hltTag DuplicateTagEXPRESS expressTag DuplicateTagPROMPT promptTag DuplicateTagPCL pclTag usertext userlogchecksum sqlitechecksumusername user
April 26, 2012
Martynas Mitrulevičius 10
Upload workflow
April 26, 2012
Server tasks:1. Authenticates user.2. Gets SQLite file.3. Checks if SQLite is not corrupted and has required content.4. Prepares pre-filled form.5. Gets and validates metadata from the user.6. Creates Tarball from SQLite and meta data.7. Moves the file into Offline DropBox repository and adds new record
to DB.
OfflineDropBox
Web browser ServerSQLite file
Tarball file
Meta dataPre-filled meta data form
Filled meta data form
Login
Database
Martynas Mitrulevičius 11
Upload script• Command:
o python dropbox_upl.py -u username -s sqlitefile.db -t metadata.txt
• How it works:o Server authenticates the user.o Script uploads SQLite file.o Server checks if the file is not corrupted and has required
content.o Server returns the form.o Script reads meta data file, fills the form and upload it.o Moves file into Offline DropBox repository and acknowledges
the file transfer to the user and to the notification system.
April 26, 2012
Martynas Mitrulevičius 12
Offline DropBox DB schema
April 26, 2012
Martynas Mitrulevičius 13
Online DropBox DB schema
April 26, 2012
Martynas Mitrulevičius 14
Workflow in Online DropBox
1. Downloads Tarball file and extract it.2. Checks the SQLite file checksum.3. Moves to the DropBox directory.4. Process the file.5. Exports calibration constants with
synchronization if needed.6. Duplicates payloads if requested.All these steps are logged in the logging schema.
April 26, 2012
Martynas Mitrulevičius 15
References• Upload file:
o http://m-dropbox.cern.ch/uploadSQLiteFile• Get tarball list:
o http://m-dropbox.cern.ch/getTarList• Get tarball file:
o http://m-dropbox.cern.ch/getTarFile?file=FILENAME• Project files:
o https://svnweb.cern.ch/cern/wsvn/cmsconddbweb/OfflineDropBox
• Documentation:o https://twiki.cern.ch/twiki/bin/edit/CMS/OfflineDropBox
April 26, 2012
Martynas Mitrulevičius 16
Thank you
April 26, 2012
