T9 Test Automation

5/2/2013 11:15:00 AM

Flintstones or Jetsons? Jump Start

Your Virtual Test Lab

Presented by:

David Silk

Verisign, Inc.

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

David Silk

A passionate technologist with more than fifteen years of software development experience, David Silk is a systems assurance manager focused on building virtual lab infrastructure automation and test automation frameworks. David has been an engineer and manager in both configuration management and systems assurance. For the past ten years at Verisign, a technology leader in Internet infrastructure, he has played a lead role in implementing an automated system using virtualization to deliver software with greatly improved efficiency and increased savings on hardware investments. David’s zeal for automated solutions to drive software development practices has played an integral role in his success.

4/16/2013

1

Flintstones or Jetsons?

Jumpstart your Virtual Test Lab

David Silk, Verisign Inc.

StarEast 2013

* Flintstones and Jetsons are © Warner Bros. Animation & Cartoon Network Studios, originally created by Hanna-Barbera Productions, Inc.

2 Verisign Public

Introductions

• Background

• Developer, QA

• Config Mgt

• Software/Systems Ops

• Engineering Manager

• Systems Assurance

• Test Automation

• Virtual Lab Management

• Shared Infrastructure Dev

• Tech guy:

4/16/2013

2

3 Verisign Public

Introductions

4

Expectations

4/16/2013

3

5

Break down barriers,

automate manual steps,

accelerate your process.

6 Verisign Public

The Quality Virtualization Evolution

Virtualize

Automate Environments

Inventory Accountability

Application Deployment

Test Execution

4/16/2013

4

7 Verisign Public

Sleek, modern, cool.

Why Flintstones and Jetsons?

Clunky, manual, slow.

Jetsons�

Flintstones…

8 Verisign Public

The Scenario

• Consider this:

• A new release is coming, we need a test environment, but:

• We won’t finish the previous release until 2 weeks after the first test

cycle of the new release will start:

• Our other environment is for production support,

we can’t touch it:

• It takes at least a week to create and

shake out a new environment:

• Historically, we built time in our schedules

(days! weeks!!!) for the new environment

to be created and shaken out.

• Just like software builds are

consistent, repeatable events:

• Test environments should be created by

consistent, repeatable events.

1

2

3

4/16/2013

5

9 Verisign Public

The Quality Virtualization Evolution

Virtualize

Automate Environments

Inventory Accountability

Application Deployment

Test Execution

10 Verisign Public

Virtualization Technologies

• The advent of virtualization:

• combined with high-powered

servers with multi-core

processors, has made test

environments much more agile:

• Functional test environments

don’t need to scale like

production – perfect fit for virtualization.

• We can run many virtual servers on a single physical server,

providing scalability of environments to support more parallel

development and test efforts.

• Time and cost for new environments becomes bearable for any

project.

• BUT: do I need a team of highly trained experts to manage the

virtualization stuff?

• Not really. Its an operating system running cool software. We are

already pretty good at handling that stuff!

4/16/2013

6

11 Verisign Public

Virtualization Technologies (continued)

• So, where do you start?

• Pick a Virtualization technology vendor:

• VMWare – this is what we use.

• Openstack – Open Source Software

• Microsoft, KVM, Xen: and others:

• Pick one that fits best with your environment.

• We use a lot of servers running RedHat Enterprise Linux, Solaris,

FreeBSD operating systems. VMWare fits well.

• Next Steps: Hardware

• Pick a server with:

• Multiple multi-core processors – today 4 x hex-core procs

• Lots of RAM – 128 Gb+

• Storage – can get tricky, a shared storage system is best

• Using local server storage can work, make sure it uses RAID

12 Verisign Public

Takeaway #1

Virtualization and cloud computing are here and have

tremendous value. Embrace the technology.

Don’t plan a big roll out with a long lead time – make it agile and

get started in small increments today.

4/16/2013

7

13 Verisign Public

The Quality Virtualization Evolution

Virtualize

Automate Environments

Inventory Accountability

Application Deployment

Test Execution

14

Automation is key to making

virtualization manageable.

4/16/2013

8

15 Verisign Public

Getting Started

• Backstory:

• We started small.

• Manually installed operating systems.

• Requested IP addresses from network team.

• Manually installed a bunch of dependencies.

• Firewall rules.

• We started to grow:major support effort.

• We needed to automate to keep up.

• The Virtual QA Lab (VQL) was born.

• Dedicated focus on managing the virtual environments

16 Verisign Public

Automation Checklist for a Virtual Machine

• Automating the creation of a VM:

� Make use of the hypervisor command

line interface

� Script out the creation process

� Create a VM container in the hypervisor

� Includes virtual disk, memory,

# of CPUs to assign

� Use a boot ISO to start the VM and initiate the OS installation

� Assign an IP address

� Use a reserved pool of IP addresses

� Install software dependencies and infrastructure

� Monitoring, backup, antivirus, & other runtime dependencies

� E.g. Nagios, Java, Weblogic

� Setup user accounts

� Automate the creation of the required user accounts for the system

4/16/2013

9

17 Verisign Public

Advanced Automation Tools

• CMDB – authoritative inventory of all systems

• PXE booting – pre-execution environment

• Install an OS over the network

• Cobbler - http://cobbler.github.com/

• Account Management

• Centralized solution for managing

credentials, e.g. Kerberos, LDAP, AD

• Software package repository

• Enables automation to quickly deploy.

E.g. Yum on Linux

• Your IT or operations teams might already have

solutions for these – collaborate!

18 Verisign Public

Creating Whole Ecosystems

• Software systems have many moving parts

• Various application servers, Databases, third party apps

• The benefits of Virtualization

• VM templates/cloning – start from

the same state every time.

• Test Beds

• A combination of multiple unique

components in an application

environment all necessary

to run the app end-to-end.

• Application Env Profiles

• Combination of VM templates that

make up a whole test bed or env.

• Databases - big data wrinkles:

4/16/2013

10

19 Verisign Public

Making it work @ Verisign

• Virtualization standard

• VMWare an early standard

• Unique environment profiles

• QA environment churn

• Established standards:

• OS images existed, TADDs

• No end-to-end automation solution to start with:

• VMs were still experimental when we started

• Needed to bring together a system to bridge our evolving

systems.

• A Verisign Formation

• An automation platform developed internally

• Pluggable architecture, made VM automation a snap!

20 Verisign Public

Overview of Automation @ Verisign

Verisign Formation: VM Automation System

VMWare ESXi:Hypervisor

Cobbler:OS

Provisioning

Vipor:Orchestration Tool (Fabric)

Puppet:Infrastructure automation

Jira: Ticketing System

Tranquility:Account Mgt

ServiceNow: CMDB

Yum:RPM

Repository

Jenkins/ TeamCity/ Bamboo:CI Tool

4/16/2013

11

21 Verisign Public

Formation Snippet

• Developed @ Verisign

• Dedicated automation team

• Drives the automation

via a workflow that is

easily customized to

any environment.

• Adapter system allows

us to integrate with

other systems.

• You can use OSS or

Freeware tools to get

started.

22 Verisign Public

Takeaway #2

Automated, repeatable processes enable QA analysts and

engineers to focus more on the quality of the application

and how it meets the business needs.

Focus efforts to automate the environment setup steps that have to

be done every time before testing the application can begin.

4/16/2013

12

23 Verisign Public

The Quality Virtualization Evolution

Virtualize

Automate Environments

Inventory Accountability

Application Deployment

Test Execution

24

Knowledge is power.

Metrics are a catalyst.

4/16/2013

13

25 Verisign Public

Accountability is important

• VMs are easily created, easily decommissioned.

• Encourage recycling!

• Be good stewards of your assets.

• Keep an inventory online

• Assign owners for every VM created

• Quotas for each team/owner

• Limit the amount of resources each group

can consume

• Everyone has a budget of virtual disk/memory/CPUs

• Poll machines for idle time, last login

• Send reports to managers

• Automation should update the inventory

• Have a stakeholders meeting!

• State of the Virtualization Address

26 Verisign Public

Generate Metrics about your inventory�

4/16/2013

14

27 Verisign Public

Tracking Change Activity

• Jira tracks all requests

• Automation reads from

Jira, updates tickets with

progress.

• Some Virtual QA Lab

requests are for updates,

modifications.

• Automating all one-off

tasks may not be

appropriate.

• Each request is evaluated,

automation is constantly

evolving.

28 Verisign Public

Takeaway #3

Be a good steward of the investment you made in the

virtualized environment with an accurate inventory.

Use your existing toolsets to track changes and be accountable to

getting the most out of your investment.

4/16/2013

15

29 Verisign Public

The Quality Virtualization Evolution

Virtualize

Automate Environments

Inventory Accountability

Application Deployment

Test Execution

30

Software can be complex, but

deployments shouldn’t be.

4/16/2013

16

31 Verisign Public

Orchestrating Complex Deployments

• Challenges of software systems deployments

• Multiple dependent steps, specific order of execution

• Configuration information – system, server & application

• Lots of tools in the market, commercial & open source

• Verisign deployments

• Large databases, application

servers, gateways

• Network silos & multiple sites

• Tight timelines for

deployments

• DevOps & Continuous

Delivery

32 Verisign Public

Orchestration Scenarios

• Two scenarios:

• Upgrade an existing application

• Many dependent steps, order is

important.

• Most new software releases are

upgrades.

• Build a server from the ground up

• Can be bare metal (physical) or virtual

• New OS, all necessary tools plus your application

• Both approaches have valid use cases

• Live deployments must be upgraded.

• Backup and failover strategies keep you running

• If a server or device fails, a new one must replace it

• Business Continuity Planning

• Solving this problem can contribute to your BCP strategy!

4/16/2013

17

33 Verisign Public

Takeaway #4

Get both Dev and IT Ops working together to make

software deployments a consistent, repeatable process.

Embracing DevOps concepts of continuous delivery will help

application environment management even if you don’t plan to

push rapid updates to production.

34 Verisign Public

The Quality Virtualization Evolution

Virtualize

Automate Environments

Inventory Accountability

Application Deployment

Test Execution

4/16/2013

18

35

Are your tests always

running? If not, why not?

36 Verisign Public

Test Automation & the Build Server

• Continuous Integration

• Build often, test often.

• Dev teams and QA teams work

towards the same goal – fewer

bugs.

• You don’t have to be “agile” to

do this!

• Test Automation

• Trigger regression tests

with every build

• For every release candidate,

automated execution validates

the app before anyone starts testing.

• Tune your tests to be efficient!

Develop

BuildDeploy

Test

4/16/2013

19

37 Verisign Public

Takeaway #5

Integrate your test automation in the build system for

maximum effect.

Integrating your test automation into the build process provides

regular validation and it exposes development to the test results.

38 Verisign Public

Final thoughts�

• 640K ought to be enough for anybody.

• Microsoft Chairman Bill Gates,1981

• If you tried to read every document on the web, then for

each day's effort you would be a year further behind in

your goal.

• Anonymous

• There might be new technology, but technological

progress itself was nothing new - and over the years it

had not destroyed jobs, but created them.

• Margaret Thatcher

4/16/2013

20

39

Q&A

Thank You

© 2011 VeriSign, Inc. All rights reserved. VERISIGN and other trademarks, service marks, and designs

are registered or unregistered trademarks of VeriSign, Inc. and its subsidiaries in the United States

and in foreign countries. All other trademarks are property of their respective owners.