CloudStack Metering – Working with the Usage Data
-
Upload
shapeblue -
Category
Technology
-
view
7.716 -
download
3
Transcript of CloudStack Metering – Working with the Usage Data
CloudStack Metering Working with the Usage Data
Tariq IqbalSenior Consultant
[email protected]: @TariqIqbal_ @ShapeBlue
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
Involved with CloudStack before donation to Apache Built and deployed CloudStack/CloudPlatform based
clouds for Enterprises and Service Providers globally Specialise in integrating CloudStack with Business
Support and Operational Support systems, which include: CloudPortals (including Payment Gateways, Fraud Control
solutions) Ticketing Systems CRM/ERP Systems Monitoring Solutions
About Me
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
“ShapeBlue are expert builders of public & private clouds. They are the leading global independent CloudStack / CloudPlatform
integrator & consultancy”
About ShapeBlue
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
CloudStack metering - What and how? CloudStack configuration to enable Usage capture Usage Data generation process Methods of accessing the Usage Data Real World Use Cases
Overview
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
Organisations need to be able to meter their cloud's consumption and monetize the cloud offerings by charging for usage
Majority of organisations already have existing billing or business support systems and just need access to CloudStack Usage Data
Organisations tend to have different billing criteria and charging models requiring a comprehensive integrated solution
Metering Requirements
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
CloudStack Usage?
CloudStack Metering
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
CloudStack does not offer any integrated usage monitoring or billing/chargeback capability
Root Admin can view System Capacity
Domain Admin and Users can view VM status
Metering in CloudStack today
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
CloudStack includes a Usage Server that creates summary usage records for the various resources consumed in CloudStack
The Usage Server is an optional, separately-installed component called ‘cloudstack-usage’
The Usage Server runs at least once per day. It can be configured to run multiple times per day and it’s behaviour is controlled by the following global configuration settings:
•
Cloud Usage Server
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
As CloudStack resources are created, consumed and destroyed, appropriate Event records are created in the ‘cloud’ database table: usage_events
There are 110 different Events: (51 shown)
CloudStack Events
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
The Usage Server schedules a job at the time specified by usage.stats.job.exec.time parameter. This job queries the ‘cloud’ database table: usage_events and generates summary usage records in the ‘cloud_usage’ database table: cloud_usage as follows:1. CloudStack events have been logged into the cloud.usage_event table2. cloudstack_usage job gets a list of the latest usage events3. Inserts the latest usage events into the cloud_usage.usage_events table4. Helper tables populated from cloud_usage.usage_event table after
parsing5. cloud_usage.cloud_usage table populated with aggregation range wise
data. Helper table data is used here
Usage Record Generation
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
The Usage Server collects 14 different types of resource usage:
Usage Types
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
The Usage Records show the amount of resources consumed by guest instances. The Usage Record format for Allocated & Running VMs is shown below: The other 7 Usage Record formats are:
Network Usage IP Address Disk Volume Template, ISO & Snapshot Load Balancer & Port Fwd Network Offering VPN User
Usage Records
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
Usage Record Example
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
Accessing Usage Data
CloudStack Metering
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
The usage records can be accessed through the CloudStack API
This is best done using the authentication port 8080 and requires both the API Key and Secret Key and for the API request to be signed
CloudStack provides the following Usage API calls with Root Admin access and supports XML and JSON response formats listUsageTypes – list of Usage Types accessible generateUsageRecords - asynchronous usage record
generation listUsageRecords - provides Usage records for a date
range
CloudStack API
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
The base command is:http://<HOST>:<8080>/client/api?command=listUsageRecords&startdate=yyyy-MM-dd&enddate=yyyy-MM-dd&signature
Append the various options:
ListUsageRecords API
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
<listusagerecordsresponse> <count>1</count> <usagerecord> <account>user5</account> <accountid>10004</accountid> <domainid>1</domainid> <zoneid>1</zoneid> <description>i-3-4-WC running time (Service Offering: 1) (Template: 3)</description> <usage>2.95288 Hrs</usage> <usagetype>1</usagetype> <rawusage>2.95288</rawusage> <virtualmachineid>4</virtualmachineid> <name>i-3-4-WC</name> <offeringid>1</offeringid> <templateid>3</templateid> <usageid>245554</usageid> <type>XenServer</type> <startdate>2009-09-15T00:00:00-0700</startdate> <enddate>2009-09-18T16:14:26-0700</enddate> </usagerecord>
</listusagerecordsresponse>
ListUsageRecords Response
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
CloudMonkey is a CLI tool for CloudStack and can be used as an interactive shell or from within a shell script to call the API
It can output the Usage Data in both JSON and Tabular form to a file or pipe it to another application for further processing
Filtering can be used to limit the result set It supports argument passing and shell automation The host, port, apikey and secretkey can be
configured on setup
CloudMonkey CLI
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
CloudMonkey list usagerecordsThe command to retrieve Usage Records is:
cloudmonkey list usagerecords domainid=7ded1404-d7fc-11e2-a70f-080027cfaf0b startdate=2013-06-01 enddate=2013-06-23 accountid=2
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
The usage records can be also accessed by directly querying the cloud_usage.cloud_usage MySQL table in CloudStack
Usage Data can be exported in a CSV format from a SELECT query
The SQL query can become complex if the ID fields in the data need to be de-referenced
Typically multiple passes of the query results may be required to extract the required information
SQL
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
Once you are able to access the Usage Data, you will need to design the mapping between the fields in the CloudStack Usage Data and the fields of the target system.
Considerations: Do you actually need all the Usage Data for all Resources? Any UUID values of the CloudStack resources should also
be imported into the target system for reference purposes Decide which system will be the master for the user
account data and if synchronisation is required
Mediation
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
Real World Use Cases
CloudStack Metering
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
MS Excel can be used to analyse CloudStack Usage Records The usage records are imported from the cloud_usage.
cloud_usage table into MS Excel via an MySQL ODBC connection
Either PivotTables or reports can be used to present the data.
This is convenient and great for integration testing between CloudStackand external systems.
MS Excel
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
Citrix CloudPortal Business Manager tightly integrates with the cloud_usage database on CloudStack/CloudPlatform.
The Usage Data is cross referenced against the utility pricing and subscription pricing in CPBM.
Scheduled Billing and Invoicing jobs generate the invoices in XML or PDF based on the tenants billing cycle.
Citrix CloudPortal
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
splunk> is an operational intelligence tool that was initially used for collecting and indexing infrastructure logs
We then connected splunk> to the CloudStack Usage database and automated the collection and indexing of Usage Data
Setup feature-rich dashboards and the capability to drill down intothe Usage Data for BI reporting
splunk>
@ShapeBlue #CloudStack #CCC13
CloudStack Collaboration Conference 2013
CloudStack Developers Guide, API Documentation & Wiki: http://cloudstack.apache.org/
CloudMonkey https://cwiki.apache.org/CLOUDSTACK/cloudstack-cloudm
onkey-cli.html How to Use MS Excel to Analyze CloudStack Usage
Records: http://support.citrix.com/article/CTX132030
CloudStack Mailing Lists (Users, Development, Marketing): http://cloudstack.apache.org/mailing-lists.html
Further Information
CloudStack Metering Working with the Usage Data
Tariq IqbalSenior Consultant
[email protected]: @TariqIqbal_ @ShapeBlue