Towards scalable, semantic-based virtualized storage resources provisioning
description
Transcript of Towards scalable, semantic-based virtualized storage resources provisioning
EUROPEAN UNION
Polish InfrastructurePolish Infrastructurefor Supporting Computational Sciencefor Supporting Computational Science
in the European Research Spacein the European Research Space
Towards scalable, semantic-based Towards scalable, semantic-based virtualized storage resources virtualized storage resources
provisioningprovisioning
Kornel Skałkowski, Renata Słota, Kornel Skałkowski, Renata Słota, Dariusz KrólDariusz Król, , Michał Orzechowski, Bartosz Kryza, Jacek KitowskiMichał Orzechowski, Bartosz Kryza, Jacek Kitowski
ACC Cyfronet AGH, Krakow, PolandACC Cyfronet AGH, Krakow, Poland
KU KDM 2012 : fifth ACC Cyfronet AGH users' conference : Zakopane, March 07–09, 2012
2
OutlineOutline
Introduction The QStorMan toolkit overview The QStorMan toolkit architecture QStorMan usage Recent improvements Current status of QStorMan Test results Future Work
IntroductionIntroduction
Data intensive applications and the 4th science paradigm Resources virtualization becomes ubiquitous Storage resources virtualization is often provided by cluster file systems like
Lustre IT infrastructure users expect more and more computing and storage power
as well as an appropriate QoS level
3
The QStorMan toolkitThe QStorMan toolkit
Main goal is to provide virtualized storage resources with QoS warrianties for data intensive applications
Users can define QoS requirements concerning storage resources on three levels: application, user, virtual organization
Currently we support the following non-functional requirements: Average Read/Write transfer rate, Current Read/Write transfer rate, Free capacity, Result cachability – dedicated for application, which generates a large number
of small files. The toolkit consists of three components:
Knowledge base (GOM) which stores semantic descriptions concerning storage resources and synchronizes the descriptions with a grid middleware
Dedicated monitoring service (SMED) which performs continuous, real-time monitoring of virtualized storage resources with semantic support
Intelligent resources matching service (SES) which combines information obtained from the GOM and SMED services as well as advanced semantic support in order to perfectly match a virtualized resource from the resources mesh
4
The QStorMan toolkit architectureThe QStorMan toolkit architecture
5
QStorMan usageQStorMan usage
1.Using system C library (libses-wrapper):declare your non-functional requirements in the GOM knowledge base export LD_PRELOAD=<path_to_libses_wrapper_librart>
2. Using C++ programming library (libses):
#include <LustreManager.h>
#include <StoragePolicyFactory.h>
using namespace lustre_api_library;
LustreManager manager;
StoragePolicy policy;
policy.setAverageReadTransferRate(50);
policy.setCapacity(100);
int descriptor = manager.createFile(„nazwa_pliku.dat”, &policy);
6
Recent improvementsRecent improvements
General purpose of the improvements is to provide a scalable, fully semantic-based solution for efficient provisioning of virtualized storage resources
SMED improvements: Utilization of the enhanced C2MS storage resources semantic model for
description of high-level QoS parameters Application of semanatic reasoners on the monitoring level
SES improvements Cache mechanism on demand – supporting large number of files generation Automatic registration of users in knowledge base – decrease required
administration effort GOM improvements
Security enhancements Scripts for administration
7
The QStorMan toolkit current statusThe QStorMan toolkit current status
Test installation is running at ACC Cyfronet AGH from over 1 year now A lot of tests were performed and no major bugs were found We have passed operational and security audits in PL-Grid succesfully We now waiting for official deployment in ACC Cyfronet, PCSS Poznan, TASK
Gdansk, and ICM Warsaw Official tutorials, workshops and other material are on the way Integrated with QoSCosGrid middleware from PCSS
We are willing to cooperate with anyone, who would like to test QStorMan in practice with an exisiting data intensive application
8
Test descriptionTest description
Synthetic testThe toolkit evaluation was performed by simulation of 8 users which were executing their applications on the Grid infrastructure3 users used the QStorMan toolkit during the applications execution, the others used plain Lustre file systemEvery user periodically saved and read a 60 GB file with random sleep periods between the succeeding operations (10 reads and 10 writes)Users started their applications with random delays in order to simulate real conditions in a Grid environment
Test with real user’s applicationSimulation of sound wave propagation inside human headOut-of-core computationsNo source code modifications5 instances of application running in parallel in order to generate enough load for storage system
9
Synthethic test resultsSynthethic test results
10
• 12% speedup between two fastest applications• 26% speedup on average (~7:20 h vs ~10 h)• No source code modification
Real user’s application test resultReal user’s application test result
11
• 15% speedup on average • Running on production infrastructure• No source code modification
Future workFuture work
Support for domain-oriented virtualized computing environments Implementation of new storage resources selection strategies Orientation toward Cloud computing environments
Dissemination and exploitation among possible users
12