What You Need to Know

Brought to you by the experts at Centercode

Hardware Beta Tests

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

2

About the SeriesDifferent types of products require different beta testing tactics. Our What You Need to Know series is designed to walk readers through all of the different types of beta tests and give actionable advice to make your specific beta test a success.

AudienceThis chapter is intended for quality, marketing, project, and product managers who’ve been tasked with running a beta test for a hardware product. If you’re an experienced beta manager, this resource may provide additional useful tips to help further improve your hardware beta processes.

GoalsThis chapter explores the unique challenges of managing a hardware beta test, and provides practical advice for handling and overcoming those challenges.

For a comprehensive overview on beta testing, be sure to start with our first chapter on What You Need to Know About Beta Management before reading this chapter on what makes hardware tests unique and challenging.

Introduction

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

3

• We’ve been focused on delivering great Beta Test Management Solutions since 2001.

• Our offerings include the Centercode beta test management platform that’s been used to run thousands of beta tests globally; fully outsourced managed beta tests; Betabound, our community of more than 90,000 highly profiled test candidates; and our beta management library with a variety of free resources (like this one) all aimed at helping to improve beta programs of all shapes and sizes.

• Feel free to contact us with feedback or questions at info@centercode.com.

• Our clients run the gamut of products, markets, and company sizes, including a variety of companies that produce great hardware products:

About Us

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

4

Table of ContentsIntroduction 2

About Centercode 3

What Makes Hardware Tests Unique 5

How Beta Testing Benefits Hardware 6

Keys to a Successful Hardware Test:Planning 7Shipping and Tracking 8

International Shipping 9

Failed Units 9

Beta Unit Cannibalism 10

Mark your Beta Units 10

Recruit Targeted Testers 11

Kickoff 12Welcome Letter 13

Out of the Box Experience 14

Beta Unit Accessories 14

Managing 15Support Plan 16

Bug Reports 16

Firmware Update 17

The Testing Experience 17

Usability Feedback 18

Triage Process 18

Closing 19Letting Testers Keep the Beta Unit 20

Retrieval of Beta Units 20

When to Distribute Rewards 21

Processing Returned Units 21

Saying Thank You 22

Hardware Beta Test Checklist 23

How Centercode Can Help 24

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

5

Hardware products bring a few unique characteristics to the table that you’ll need to consider and plan for throughout the beta testing process.

• Limited Units • Hardware beta tester team sizes are generally determined by the number of pre-release units available and allocated to beta. Since you’ll have a limited number of units to distribute, you’ll need to be sure to maximize the use of each and every unit during your beta phase.

• Shipping and Logistics • Physical units add a lot of complications to a beta test, because you need to deal with getting the beta units where they need to be quickly and efficiently, both before and after the test.

• Confidentiality • Sending pre-release versions of your product into the wild is nerve racking. It’s much easier for a hardware beta unit to disappear than a software beta product. You have to take precautions to ensure your units don’t fall into the wrong hands.

• Short Time Frames • Hardware beta tests are generally condensed due to the time necessary to build and receive the beta version of the hardware, not to mention the time it takes to ship the units to and from testers. Unlike software, which can be patched throughout a beta, hardware betas are generally limited to one or two phases of beta testing due to the limitations of working with physical units.

• Varied Technological Ecosystems • Few hardware products exist in a vacuum. It’s likely that your product needs to perform in a variety of environments in conjunction with a range of other technology products, which can be tough to cover during a beta test without careful planning.

What Makes Hardware Tests Unique

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

6

Given the complexity of modern technology, effective beta testing is crucial to the success of every hardware product. Here are some of the key ways beta testing can benefit your product’s development.

• Beta testing will gauge your product’s “out of the box” experience • With any physical product, the initial customer experience is key. New customers can be easily overwhelmed and confused when they first interact with a product, and may end up returning the product before they even give it a chance. Beta testing can help you gauge your customers’ early experiences with your product, allowing you to identify where you can improve and plan for unforeseen support requirements.

• It puts the installation experience to the test • Installing a hardware product can be a big hurdle for many non-tech savvy customers, especially since many hardware products must integrate with existing technology (such as a home theater system or smartphone). A beta test can help you find the key issues with installation and integration, allowing you to mitigate them or prepare to support them.

• You’ll receive feedback on usability, ergonomics, and physical features • Beta testing is an opportunity to test far beyond the basic functionality of a product. Real customer feedback on the physical aspects of the product, such as intuitiveness and comfort, can be invaluable.

• It will test your product’s real-world performance • Hardware products rarely live in a vacuum. They’re generally built to function within an environment of existing adjacent products. Often, the success of the product is based on its performance in these diverse environments. While some of this may (and should) be simulated in a test lab, real customers will introduce new variables you may not have considered.

How Beta Testing Benefits Hardware

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

7

PlanningKickoffManagingClosing

{Keys to Running a Successful Hardware Beta Test

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

8

When it comes to running a successful a hardware beta test, thorough planning is crucial. Since hardware beta tests involve handling the logistics and challenges of physical units, along with the difficulties of maximizing small groups of testers in a tight time frame, you need to have a solid strategy in place long before your test begins. Here are some best practices we use when planning a hardware test to ensure everything goes smoothly from day one.

Planning Your Hardware Test

Create a process for shipping and tracking your hardware unitsWhen distributing beta units, you want to get them to your testers as quickly as possible, while still keeping a close eye on where they are. Create a process for tracking which units went to which testers, who needed replacement units, as well as who returned units. We recommend using a beta management tool to track this information, rather than an Excel spreadsheet, so that you can connect the tracking information to specific users and their feedback later. Also, be sure to communicate these processes clearly to testers, so they know what to expect. We suggest using overnight or expedited shipping when sending beta units to your testers, so they can start providing feedback right away.Pl

anni

ng

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

9

Plan

ning

Consider the additional complexity of international shippingIf you plan on testing internationally, other shipping hurdles may come up. Some countries have customs, taxes, duties, and regulations that can make shipping beta units a headache. For example, the packaging may need to be in another language or you may need to include a particular adapter. Additionally, you must be careful how you identify your shipment, as there are trigger words that may cause products to get stuck in customs. Do your research beforehand so you’re prepared if you decide to include international testers.

Plan for failed unitsYour limited supply of beta units is compounded by the fact that some of your pre-release units will inevitably fail. You have two options here. You can reserve some units as replacements (we recommend 10-20% of your total pool, based on quality) and switch out failed units as they break. You’ll need to set up a process for replacing broken units beforehand, so that you aren’t scrambling when the first tester reports a broken unit. Another option is to distribute all of your units, but accept that some of your testers won’t be able to test. In this case we suggest rewarding these users even though they didn’t contribute any feedback, seeing as the circumstances were out of their control.

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

10

Anticipate beta unit cannibalismJust prior to ramping up your beta, it’s common for other groups at your company to try to cannibalize pre-release units for purposes such as trade shows, additional quality testing, and sales demos. While at times this may be unavoidable (requiring you to adjust your goals and/or time frame), this is best handled by promoting beta testing as a “first class citizen” against these other options, since fewer units will mean a less effective beta test, greatly increasing the risk of a botched launch (which nobody wants to be responsible for).

Mark beta units and record their serial numbersIn order to differentiate your beta units from production units, physically mark them in some way before shipping them to your testers. Some companies go as far as to make beta units a different color or use secret markings the tester won’t notice. The goal here is to clearly identify the unit as a beta unit if it ends up in unauthorized hands. Just make sure the identification doesn’t detract from the user experience. Also, each of your beta units should have some sort of serial number. Make a list of these numbers before sending the units out and then note which tester received each unit. This will make it easier to account for all of your units at the end of your test and could allow you to disable or track a unit if it’s stolen while in the field.

Plan

ning

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

11

Recruit targeted testersSince you’ll probably be working with limited units, you’ll need to maximize the feedback you receive out of every unit. The best way to do this is with a highly focused tester selection strategy that will help you only recruit candidates that closely meet your target markets (both demographic and technical), while also demonstrating signs of being great participants. Plan on gathering specific information about each of your potential testers, the technology they own, and why they’re interested in testing for you. The best way to do this is through a qualification survey. This will allow you to build a pool of potential applicants. From there you can use the information you’ve gathered to select the best testers to maximize your limited hardware units.

Free Resource: Beta Tester Recruitment KitRecruiting quality beta testers to test your hardware product is crucial. You’ll need to maximize each and every beta unit that you have available, so you want to ensure that you’re placing them into the right hands. For more advice on recruiting great testers, download our Beta Tester Recruitment Kit. This resource includes a whitepaper full of best practices and supplemental worksheets to help you organize and execute a successful recruitment for your hardware test.

Plan

ning

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

12

Keys to Running a Successful Hardware Beta Test

PlanningKickoffManagingClosing

{Keys to Running a Successful Hardware Beta Test

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

13

During kickoff, your goal is to make sure your testers have everything they need to begin providing quality feedback right away. This will help prevent delays and reduce the time your team spends supporting testers during the early days of the test. This will also help to make sure you collect valuable information about the issues your customers might face during their initial interactions with the product. These best practices will help get your test period off to a good start and take advantage of your beta testers’ intitial excitement to try out your product.

Kicking Off Your Hardware TestKi

ckof

f Include a welcome letter Since you’ll have your testers’ attention when they open your package, it’s a great opportunity to remind them of a couple of important things they’ll need to know during the beta test. Include a welcome letter with the product thanking the individual tester for their participation and reminding them of their responsibilities. You can give them directions for their initial use of the product and remind them to give feedback on the unboxing and installation experience. If appropriate, this is also a great place to remind them that they’ll need to return the product (and all of the other materials in the package) at the end of the test. This letter helps to make the tester feel like they are part of something important and should take their role in your beta test seriously.

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

14

Kick

off

Make sure the out of the box experience is completeTry and get the beta unit to be as close to the final experience as possible, with the right packaging, documentation, and accessories. Beta testing provides a unique and valuable opportunity for customers to open and experience a product in their own homes, just as if they’d brought it home from the store. They can then provide insightful feedback on that experience, so you want to get it as close as possible to the real thing.

Include all ancillary parts and accessoriesMany products have (and need) accessories like chargers or cables. If your product requires additional parts to function, you have two options. You can either send the part with the product to each tester, or only recruit testers that already have the necessary pieces of equipment. If you have international testers, be sure to include any necessary adapters they may need to use the product and its accessories properly.

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

15

Keys to Running a Successful Hardware Beta Test

PlanningKickoffManagingClosing

{Keys to Running a Successful Hardware Beta Test

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

16

Once your beta test is up and running, you want to collect as much relevant feedback as you can to improve your product before launch. This section will help you put the right pieces in place to make sure you’re getting all the data you need during your hardware test.

Managing Your Hardware Test

Have a support process in place Since hardware beta tests involve having a limited group of testers using complex products in varied environments, they’re going to run into trouble. Make sure you have a support process in place to help your testers get back to testing quickly. As a bonus, this will be a good opportunity for you to test your product’s support processes and documentation so you can make improvements prior to launch.

Collect detailed bug report formsWhen a hardware product breaks, it can be difficult to tell what caused the failure. When a product fails during a beta test, it’s important to get details from the tester about the circumstances of the failure. What were you doing? Were their other devices involved? This way you have a better chance of recreating the problem. Make sure you have a comprehensive form for testers to fill out when they run into a problem.

Man

agin

g

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

17

Establish and test your firmware update process Updating your product’s firmware while it’s in the field is incredibly difficult. There are even cases where you may need to collect all the units and start the test over. However, these cases can sometimes be a good opportunity for you to assess how you’d like future customers to handle updates to your product, and whether or not this process is something that you can simply walk your customers through. Use these opportunities to test your support team’s ability to assist a customer through an update.

Create a great testing experienceSince you have a limited number of testers, it’s vital that you maximize the participation of each and every tester. Focus some time and effort on making the testing experience fun, pleasant, and rewarding for your testers. This will keep them engaged and participating throughout your test. This includes small gestures (such as thanking testers for each piece of feedback) and large ones (such as giving out a prize for the best bug report). For more ideas on building a rewarding testing experience, download our free Beta Tester Incentives Kit.

Man

agin

g

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

18

Solicit feedback about comfort, usability, and appearanceWith hardware products, the user experience is about much more than whether or not your product works. You’ll want to ask testers about the comfort, usability, and appearance of your product. These questions will result in highly subjective answers, which can be difficult to quantify. To alleviate this, create surveys that ask testers to rate the physical aspects of your product on a numbered scale, and then look for trends in the data. If one or two testers thinks that the default volume of your speakers is too high, then it may not be a problem. If the majority of the testers agree, however, you may want to reconsider the issue. You can also use discussion forums to allow testers to voice their opinions about the look and feel of the product in a less structured way.

Triage the feedback you receiveA lot of feedback is going to come in during your beta test, and you often don’t have the time or desire to fix every problem before launch. Triage the feedback you receive so you know what’s critical (the product crashes on installation), what may require some additional support or return costs (the enter button doesn’t work on 5% of units), and what can wait until the next generation of the product (many of the testers thought the product looked too boxy and the edges should be more rounded).

Man

agin

g

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

19

Keys to Running a Successful Hardware Beta Test

PlanningKickoffManagingClosing

{Keys to Running a Successful Hardware Beta Test

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

20

Hardware products can be tricky to handle once the testing phase is over, since testers have a physical unit in their possession that you may (or may not) need them to return. Here are some tips to help you close your tests effectively and leave your testers happy.

Closing Your Hardware Test

Give your testers the product, if possibleYou need to give your testers some sort of reward at the end of the test to thank them for their contribution. If possible, let them have a production version of the product as that reward. This gives them what you already know they want and is often an easy reward for you to provide. If that’s not possible, you can consider just letting them keep the beta unit instead. This will make your job easier because you don’t have to deal with the time and expense of collecting them. Plus, your testers have already signed NDAs, so you know they’ll keep quiet about your product. However, this all depends on how stable your beta units are and whether they’re reliable enough for your testers to use long term.

Retrieve your beta units, if necessaryIf you need the units returned, you must set this expectation upfront. Make sure that testers understand that the beta unit (and all of its packaging) must be returned and make this process as simple and easy as you can for them. Include return shipping labels in the initial package to the tester (testers should never incur shipping costs for participating in your beta), with clear instructions.

Clos

ing

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

21

Hold onto rewards until you get your beta units backOnly distribute the testers’ rewards after you’ve received the beta unit back from them. It can be tempting to send rewards out immediately at the close of a test. However, once a tester receives their reward, they will be less inclined to return the beta unit. If the tester knows right from the start that they must return the beta unit before they’ll receive their incentive, then they’ll much more likely to do so in a timely manner. You’ll also want to remind them that they need to return all of the packaging and beta materials (instructions, manuals, accessories, etc.) as well, so that you don’t have those materials in the field after your test is over.

Process the returned units Track which testers have returned their unit (using the tracking numbers from the return shipping labels you provided to testers and the serial numbers you wrote down at the start of your test). This way, you know which testers have earned their reward and which testers you still need to follow up with. This will also be important so you can connect specific beta units with the bugs they encountered during the test, so that your development team can fix and regress the issues before launch.

Clos

ing

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

22

Free Resource: Beta Tester Incentives KitBuilding your strategy for motivating and rewarding your testers involves many different considerations. You not only have to find ways to keep your testers consistently engaged, but you also need to decide whether or not your hardware product itself will make a realistic incentive for your testers. For more advice on building a cohesive incentives strategy, download our Beta Test Incentives Kit. This kit also includes a sophisticated beta incentives calculator to help you determine the specific incentives budget for your beta test.

Don't forget to say thank youNo matter what incentive you choose to give your testers at the end of your beta, also consider calling your testers or sending them each a personal thank you note after the test is over. Showing that you value their contribution goes a long way toward turning your beta testers into brand evangelists, which can be a valuable asset during the launch of your product and future beta tests. An email is a good start, but a signed thank you letter from a member of your team will make a much bigger impact on your testers. Since hardware tests are usually relatively small, this is an easy and profoundly meaningful way to close out your experience with your testers.

Clos

ing

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

23

Planning F Determine the number of available beta units

F Create a process for shipping and tracking

F Plan for international shipping (if necessary)

F Establish a process for failed units

F Build a plan to handle beta unit cannibalization

F Outline tester requirements and pool breakdown

F Mark devices as beta units

F Record beta units serial numbers

Kickoff F Draft welcome letters and initial instructions

F Gather everything for the “out of box” experience

F Include all necessary product accessories

Managing F Prepare support processes and documentation

F Design a detailed bug report form

F Determine your firmware update process

F Build a plan to create a great testing experience

F Create product physical feature assessment surveys

F Establish feedback management and triage process

Closing F Decide what to give testers as an incentive

F Communicate the unit retrieval process to testers

F Establish process for tracking and returns

F Distribute incentives to testers

F Call to thank testers and/or send thank you notes

Hardware Beta Success Checklist

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

24

We’ve put countless hardware products to the test using our beta testing solutions. They’re designed to help you create a powerful and cohesive beta program, resulting in a more successful hardware product.

Our Beta Management Platform • The Centercode beta test management platform is a complete system for managing beta tests from start to finish. It includes everything you need to recruit targeted testers, ship and track units, distribute incentives, and much more. Schedule a demo to learn more.

Our Managed Beta Tests • Our highly skilled Managed Betas Team has the experience and processes to run hardware tests in less time with better results. Find out more and request a free beta plan.

Our Beta Testers • Betabound is our panel of over 90,000 prequalified beta testers, eager to test your hardware product. This community is available to companies of all types looking to recruit great testers.

Our Beta Management Library • For more advice on handling all the challenges of building your beta program, visit our beta management library and our blog.

Schedule a demo to learn more!

How Centercode Can Help

WWW.CENTERCODE.COM

WHAT YOU NEED TO KNOW • HARDWARE BETA TESTING

25

www.centercode.com • 800-705-6540 • info@centercode.com • @centercode