Design and Implementation of a NAS Engineering

Management Object Database(NEMO)

Mark SonnenfeldRobert Kimmel

Storage AdministratorsCisco Systems

Company Overview

• Cisco founded in 1984 – currently over 70,000 employees• Q4 FY10 Revenue: $10.8 billion • Primarily perceived as a “router” or “switch” manufacturer • Products now diversified as a result of many acquisitions, and

include voice (e.g. IP phones) and video (e.g. Telepresence)• Consumer products now include Linksys, WebEx,, Flip Video• Unified Computing System (UCS) is a data center platform that

unites compute, network, storage access, and virtualization into a cohesive system

Technology Environment

• Manage over 2 PB of NAS storage at 22 engineering sites globally, including US, Europe, and Asia

• NAS infrastructure consists of both NetApp and EMC file servers – 120 filers globally

• Over 20,000 engineering clients running Linux (Redhat), Solaris, and Windows operating systems

• Clients are primarily engineers who require workspace, project space, build repositories, homedirs, tools/binaries, TFTP, etc.

• Majority of data center running VMware on UCS hardware (ESX host) with NAS backend, which utilizes snapshot and dedup technologies.

Unmanageable Storage!

• 29,000 filesystems across 120 file servers• 13.71 billion files/inodes• 1035 new filesystems per month• 4 storage administrators• 7,250 filesystems per person per year

= 8.6 mins per filesystem per person per year (Hope you have a fast network!)

Problem – Data Management

• No automated tools/scripts to create and manage the data• No accountability, ownership, contacts for the data• No auditing, history tracking for the data• No defined services, categories, tiers for the data• No capacity management or planning for growth• No performance management or tiering structure• No breakdown of data by department or site, therefore no way

to achieve a true chargeback model

= No time to manage the data

Solution - NEMO

• All decisions are based on some sort of data, regardless if it is a human or automated process.

• NEMO (NAS Engineering Management Objects) streamlines storage data (using backend database) to help make the correct decision quickly.

• The data is also accessible to tools and scripts to automate various operations.

• By adhering to strict standards and policies, services are formulated based on clients needs using a simple API –allowing complex environments to be more easily managed.

Solution – NEMO (cont’d)

• Within our storage methodology, all storage exports are considered storage objects.

• Each object is an individual entity, with a BU contact and department number.

• Each object has its own business rules based on the requestor’s requirements. These rules are defined by a category type (Project, TFTP, Workspace, etc.).

• Business rules may include items such as storage capacity, export restrictions, and tiering level.

Storage Administrator Methodology• Gathers data• Determines course of action• Implements change• Verifies results

Storage Methodology

NEMO Methodology• Pulls data from database• Makes a decision based on standards• Makes the change• Confirms changes were successful• Records the change back to database

NET RESULT = NO HUMAN RESOURCES REQUIRED

NEMO ComponentsFour core modules define NEMO as a collection of

objects pertaining to NAS storage.

1. Data collection API’s2. Database of NAS information3. Web interface4. Tools

API's WEB

interface

DB Tools

Data Collection

• The collection agents are a series of API Perl scripts, which query the filers from any API interface, gather the information, and populate the backend database for further use.

• Currently NEMO pulls the majority of its data from the servers via SSH, but SNMP can also be utilized.

• Initial data collection performed from a master trusted host, which spawns processes to local management hosts. These local hosts query the NAS devices and write data into the DB.

• The entire process takes about an hour and collects data from over 120 file servers.

• The database is a simple schema containing all storage and server information, including a log of the changes applied down to a single object’s level.

Data Collection (Cont’d)

Database Tables

DISK_INFO

PK DISK_SET_ID

SERVER_IDAGGR_IDTRAD_VOL_IDFILESYSTEM_NAMESERVER_NAMEDISK_TYPEDISK_SPEEDRECORD_DATEDISKCOUNT

SQL ExamplesGet available aggregate space by tier and location

select TIER_LEVEL, LOCATION, sum(AVAIL)"SUMAVAIL"

FROM NASTEAM.AGGR, NASTEAM.SERVER_INFOWHERE NASTEAM.AGGR.RECORD_DATE > SYSDATE -2 ANDSERVER_INFO.SERVER_NAME = NASTEAM.AGGR.SERVER_NAMEGROUP BY LOCATION,TIER_LEVELORDER BY LOCATION,TIER_LEVEL

TIER_LEVEL LOCATION SUMAVAIL1 DALLAS 6575181 MIAMI 146817062 NEW YORK 780628231 SAN DIEGO 01 BOSTON 8844163 CHICAGO 92809321 PARIS 114671 LONDON 10063621 ATLANTA 3205872

SQL Examples (Cont’d)Find anyone who increased their workspace from 32 GB

to 64 GB within the last 90 days

Select SERVERNAME,LOCATION,PROJECT_NAME,CREATOR,PROJ_SIZE,CREATE_DATE,ENTRY_DATE,NOTESFROM NASTEAM.PROJECT_SPACE, NASTEAM.PROJECT_NOTESWHERE PROJECT_SPACE.PROJECT_ID = PROJECT_NOTES.PROJECT_IDAND CATAGORY=‘WORKSPACE'AND ENTRY_DATE > SYSDATE-90AND NOTES like '%Script found a change in size from 32768 to 65536%'

SERVERNAME LOCATION PROJECT_NAME CREATOR PROJ_SIZE CREATE_DATE ENTRY_DATEN O T E S

snw-filer1 /vol/vol1 s n w u s e r 1 Found 6 5 5 3 6 05-06-2008 05-10-2010

Script found a change in size from 32768 to 65536

snw-filer02 /vol/vol3 s n w u s e r 4 snwroot 6 5 5 3 6 08-26-2008 05-10-2010

Script found a change in size from 32768 to 65536

snw-filer5 /vol/vol6 s n w u s e r 6 admin2 6 5 5 3 6 09-15-2008 03-16-2010

Script found a change in size from 32768 to 65536

snw-filer10 /vol/vol10 s n w u s e r 8 admin4 6 5 5 3 6 10-02-2008 02-13-2010

Script found a change in size from 32768 to 65536

Web Interface

• A Perl cgi generates the HTML/EXTJS webpage.• Uses AJAX to submit data, and XML to pull data.• AJAX calls a simple cgi to submit data to the

database.• XML is generated from a Perl cgi interfacing with

the database.

Web Interface

On submit this form generates a SQL query. In this example it creates

“select * FROM NASTEAM.PROJECT_SPACE WHERE PROJECT_SPACE.project_name LIKE '%rkimmel%‘ AND (PROJECT_SPACE.ACTIVE = 'YES') ORDER BY SERVERNAME,LOCATION,PROJECT_NAME”

Web Interface (Cont’d)The SQL is embedded in the URL, which invokes a cgi, which generates XML.

http://wwwin-ehs.cisco.com/smt/hvs/xml/generic-sql-xml.cgi?sql_query=select+*+FROM+NASTEAM.PROJECT_SPACE+WHERE+PROJECT_SPACE.project_name+LIKE+%27%25rkimmel%25%27++AND+%28PROJECT_SPACE.ACTIVE+%3D+%27YES%27%29+ORDER+BY+SERVERNAME%2CLOCATION%2CPROJECT_NAME+

Web Interface (Cont’d)

This XML is then read via EXTJS XML reader functions and generates thefollowing dynamic webpage and chart.

Web Interface (Cont’d)

Tools

Automated tools or scripts read and write data to the backend NEMO database

• Daily, weekly, or quarterly storage reports• Filesystem creation and resizing• Automatic client notification for exceeding user quota• Reclaiming underutilized storage space• Purging of old data• Growth forecasting based on trending/historical data

Tools (Cont’d)~> sudo NAS-create -n snw_test -s 100 -c "san jose"

Valid Categories are ARCHIVE,CACHE,CCP,DUMP,PROJECT,TFTP,VBS,WORKGROUPWhat Category of space is needed [PROJECT]:

The default tier for PROJECT is 2What tier of storage would you like to use [2]:Userid of the requester []: snwuserDepartment id of project (defaults to requester department number) [020202020]:

Storage requires SNW manager approval for 100GB of spaceUserid of the Approver []: snwuser_mgrRemedy Case number []: HD10101010Description []: SNW project space

Please wait..looking for space .....Server to use (NONE to EXIT) [snw-filer1]: Filesystem to create tree on [/vol/vol1]: Please wait examining filesystem....

NFS Exports [-sec=sys,rw=snw,root=trusted_host]: Did you want it windows accessible (Y/N) [N]: Y

Tools (Cont’d)Please wait, verifying space...... ......................................Space Verified

Please Verify the following information is correct--

Project Name : snw_testProject Size : 100 GBStorage Tier : 2Server locations : snw-filer1:/vol/vol1NFS export line : -sec=sys,rw=snw,root=trusted_hostRequester : snwuser (SNW User)Approver : snwuser_mgr (SNW Manager)Department number : 020202020Department name : SNW DEPTCase Number : HD10101010Platform : NetAppWindows Access : YES (/vol/vol0/links/projects)Windows Path : \\snw-filer1\projects\snw_testDescription : SNW project space

** Warning -- It will make changes to the filer now!

Are these settings Correct? (Y/N) [N]:

Q and A