CMS data challenges. The nature of the problem. What is GMA ? And what is R-GMA ? Performance...
-
Upload
neal-greer -
Category
Documents
-
view
213 -
download
0
Transcript of CMS data challenges. The nature of the problem. What is GMA ? And what is R-GMA ? Performance...
![Page 1: CMS data challenges. The nature of the problem. What is GMA ? And what is R-GMA ? Performance test description Performance test results Conclusions.](https://reader035.fdocuments.us/reader035/viewer/2022070414/5697c0131a28abf838ccca86/html5/thumbnails/1.jpg)
CMS data challenges. The nature of the problem.
What is GMA ? And what is R-GMA ? Performance test description Performance test results Conclusions
![Page 2: CMS data challenges. The nature of the problem. What is GMA ? And what is R-GMA ? Performance test description Performance test results Conclusions.](https://reader035.fdocuments.us/reader035/viewer/2022070414/5697c0131a28abf838ccca86/html5/thumbnails/2.jpg)
As part of the preparations for data taking CMS is performing DATA CHALLENGES.
Large number of simulated events to optimise detectors and prepare software Enormous processing requirements
BUTeach event is independent of all the others
each event can be generated on a machine without any interaction with any other
![Page 3: CMS data challenges. The nature of the problem. What is GMA ? And what is R-GMA ? Performance test description Performance test results Conclusions.](https://reader035.fdocuments.us/reader035/viewer/2022070414/5697c0131a28abf838ccca86/html5/thumbnails/3.jpg)
Work split between farms.
How to handle the book-keeping ?
a data-base automaticallyupdated
Implemented via a job wrapper BOSSOutput to <stdout> and <stderr> is intercepted and the information is recorded in a mySQL production database.Event generation and job accounting decoupled
![Page 4: CMS data challenges. The nature of the problem. What is GMA ? And what is R-GMA ? Performance test description Performance test results Conclusions.](https://reader035.fdocuments.us/reader035/viewer/2022070414/5697c0131a28abf838ccca86/html5/thumbnails/4.jpg)
DatabaseMachine
SubmissionMachine
UI
WorkerNode (WN) WN
WN
WN
WNWN
WN
WN
WN
![Page 5: CMS data challenges. The nature of the problem. What is GMA ? And what is R-GMA ? Performance test description Performance test results Conclusions.](https://reader035.fdocuments.us/reader035/viewer/2022070414/5697c0131a28abf838ccca86/html5/thumbnails/5.jpg)
DatabaseMachine
SubmissionMachine
UI
![Page 6: CMS data challenges. The nature of the problem. What is GMA ? And what is R-GMA ? Performance test description Performance test results Conclusions.](https://reader035.fdocuments.us/reader035/viewer/2022070414/5697c0131a28abf838ccca86/html5/thumbnails/6.jpg)
Producer
Consumer
Registry (Directory services)
register producer
locate producer
address of producer
data
data
data
data
Ask fordata
data
data
data
data
![Page 7: CMS data challenges. The nature of the problem. What is GMA ? And what is R-GMA ? Performance test description Performance test results Conclusions.](https://reader035.fdocuments.us/reader035/viewer/2022070414/5697c0131a28abf838ccca86/html5/thumbnails/7.jpg)
Developed for E(uropean) D(ata) G(rid)
Extends the GMA in two important ways
1. Introduces a time stamp on the data.
2. A relational implementation
3. Hides the registry behind the API
Can be used for information and monitoring
Each Virtual Organisation appears to have one RDBMS
![Page 8: CMS data challenges. The nature of the problem. What is GMA ? And what is R-GMA ? Performance test description Performance test results Conclusions.](https://reader035.fdocuments.us/reader035/viewer/2022070414/5697c0131a28abf838ccca86/html5/thumbnails/8.jpg)
The user interface to R-GMA is via SQL statements(not all SQL statements and structures are supported)
Information is advertised via a table createInformation is published via insertInformation is read via select … from
table
The first read request registers the consumer as interested in this data.
Relational queries are supported
NOTE : sql is the interface – it should not be supposed an actual database lies behind it.
![Page 9: CMS data challenges. The nature of the problem. What is GMA ? And what is R-GMA ? Performance test description Performance test results Conclusions.](https://reader035.fdocuments.us/reader035/viewer/2022070414/5697c0131a28abf838ccca86/html5/thumbnails/9.jpg)
R-GMA can be dropped into the framework with very little disruption
1. Set up calls for mySQL are replaced by those for R-GMA producers
2. An archiver (joint consumer/producer) runs on a single machine which collects the data from all the running jobs and writes it to a local database (and possible republishes it).
The data can then be queried either by direct mySQL calls or via R-GMA consumer (a distributed database has been created)
![Page 10: CMS data challenges. The nature of the problem. What is GMA ? And what is R-GMA ? Performance test description Performance test results Conclusions.](https://reader035.fdocuments.us/reader035/viewer/2022070414/5697c0131a28abf838ccca86/html5/thumbnails/10.jpg)
DatabaseBOSS
LAN Connection
R-GMAR-GMAR-GMAR-GMAR-GMAR-GMAR-GMAR-GMA
WAN Connection
![Page 11: CMS data challenges. The nature of the problem. What is GMA ? And what is R-GMA ? Performance test description Performance test results Conclusions.](https://reader035.fdocuments.us/reader035/viewer/2022070414/5697c0131a28abf838ccca86/html5/thumbnails/11.jpg)
The architecture of GMA clearly provides a putative solution to the wide area monitoring problem.
BUTDoes a specific implementation provide a
practical solution
Before entrusting CMS production to R-GMA, we must be confident that it will perform.
What load will it fail at and why ?
![Page 12: CMS data challenges. The nature of the problem. What is GMA ? And what is R-GMA ? Performance test description Performance test results Conclusions.](https://reader035.fdocuments.us/reader035/viewer/2022070414/5697c0131a28abf838ccca86/html5/thumbnails/12.jpg)
<Message length> 35 chars.
![Page 13: CMS data challenges. The nature of the problem. What is GMA ? And what is R-GMA ? Performance test description Performance test results Conclusions.](https://reader035.fdocuments.us/reader035/viewer/2022070414/5697c0131a28abf838ccca86/html5/thumbnails/13.jpg)
Multi-threaded jobeach thread produces messages. Length 35 chars, suitable distribution.
Threads starting time distribution can be altered.One machine delivers the R-GMA load of a farm.
R-GMA servlet
R-GMAconsumer
![Page 14: CMS data challenges. The nature of the problem. What is GMA ? And what is R-GMA ? Performance test description Performance test results Conclusions.](https://reader035.fdocuments.us/reader035/viewer/2022070414/5697c0131a28abf838ccca86/html5/thumbnails/14.jpg)
One machine per grid cluster providing loads of greater than the cluster
R-GMAconsumer
R-GMA servlet
R-GMA servlet
R-GMA servlet
R-GMA servlet
![Page 15: CMS data challenges. The nature of the problem. What is GMA ? And what is R-GMA ? Performance test description Performance test results Conclusions.](https://reader035.fdocuments.us/reader035/viewer/2022070414/5697c0131a28abf838ccca86/html5/thumbnails/15.jpg)
R-GMA can survive loads of around 20% of the current CMS requirements and does provides a grid method for monitoring. An overload of a factor 2 jobs causes problems after about five minutes running.
We believe these instabilities are soluble.
When production starts in earnest we will compare reality with our model.