Songkick Product Discovery FOWA (Michelle You)

37
PRODUCT MICHELLE YOU CPO & CO-FOUNDER DISCOVERY @WRECKINGBALL37
  • date post

    17-Sep-2014
  • Category

    Technology

  • view

    25
  • download

    4

description

How we do product discovery at Songkick. My talk at FOWA 2013.

Transcript of Songkick Product Discovery FOWA (Michelle You)

Page 1: Songkick Product Discovery FOWA (Michelle You)

PRODUCT

MICHELLE YOU CPO & CO-FOUNDER

DISCOVERY@WRECKINGBALL37

Page 2: Songkick Product Discovery FOWA (Michelle You)

Ian, Pete, and I started Songkick in 2007. Songkick makes sure you never miss concerts. You track all the artists you want to see live by, for example, scanning your Spotify playlists, and then whenever any of those bands announce a tour date in your city, we send you a concert alert.

Page 3: Songkick Product Discovery FOWA (Michelle You)

• 2nd-largest live music website after Ticketmaster • 8 million fans per month using Songkick

We’re now the second-largest live music website in the world after Ticketmaster. Every month, over 8 million fans come to Songkick to find out when their favorite artists are coming to town. !We’ve also been experimenting with a new platform we’re calling Detour, which enables fans to pledge in advance to buy tickets for an artist. They come to Detour, tell us which bands they want to see live and how much they’re willing to pay. They enter their credit card to reserve their ticket, but aren’t charged until the concert is confirmed. Since last year, we’ve been running a series of experiments to understand the potential in this platform. We’ve confirmed 45 concerts that have been created as a result of fans pledging.

Page 4: Songkick Product Discovery FOWA (Michelle You)

BUSINESS GOALS

PRODUCT MAGIC

USER NEEDS

TECHNICAL FEASIBILITY

My role at Songkick is to make sure we’re building the right things, and building them as quickly as possible. These things we make, whether apps, features, or entirely new products have to A) solve a real problem for the people who use them and B) achieve the company’s goals, whether that’s revenues, sign-ups, or whatever other metric is of vital importance to your business; and C) also be something we can feasibly build in a timely manner. This is typically what a product manager does, but even if you don’t have someone with that title on your team, someone is taking on this responsibility, whether they’re your CTO, CEO, designer, or project manager. Someone is that glue.

Page 5: Songkick Product Discovery FOWA (Michelle You)

Time is the enemy

As a startup, you’re constantly under extreme existential threat. Time is your enemy and your company’s imminent death is always around the corner. You need to show growth, hit targets, get to that next benchmark of success as quickly as possible. !You don’t have time to waste on building things no one wants or things that won’t achieve your business goals. But you also aren’t God or Steve Jobs, so you can’t assume you have perfect knowledge of what people want in the first place. !So you have to learn this as fast as possible. That’s what product discovery is. !So far so Lean Startup. !But how do you go about actually validating your hypotheses? How do you actually figure out what the right thing to build is? How do you do product discovery?

Page 6: Songkick Product Discovery FOWA (Michelle You)

GOLDEN RULE OF PRODUCT DISCOVERY

01 SPEED & QUANTITY

OF LEARNING

First golden rule of product discovery: Speed and quantity of learning.

Page 7: Songkick Product Discovery FOWA (Michelle You)

Tom Chi: Google X

Assume your ideas have a 5% chance of success • By the time you try 20

things, you have a 64% chance of success.

• By the time you try 50 things, you have a 90% chance of success.

Tom Chi of Google X gave an incredible, life-changing talk at a conference last year where he schooled the attendees, including me, on the simple mathematical reality of product discovery. !No matter how brilliant a product designer you are, the harsh truth is that maybe 5% of your ideas will succeed. If you assume that each idea you have has a 5% chance of success, by the time you try 20 things, your chance of success goes up to 64%, by the time you try 50 things, it goes up to 94%. So you need to try as many things as fast as possible. If you only get to idea #10 two years into your company’s life with 2 months of runway left, you’re shit out of luck. !So your #1 priority is to make sure you’re learning as fast as possible. It’s not just about a million A/B tests, it’s about learning what works, what doesn’t and why your users are behaving the way they do. !That means you’re looking for the cheapest, fastest way to get feedback on an idea, to validate an idea, to answer a question. To learn. Once you’ve learned and proven a hypothesis, you can be confident it’s worth investing time in building something—that what you build will actually be used, solve a problem, and help your business goals. !That’s the difference between product discovery vs. product delivery. Product discovery should be measured in hours and days, while product delivery is measured days and weeks.

Page 8: Songkick Product Discovery FOWA (Michelle You)

Product trio Product manager Design lead Tech lead

So let me talk you through how we do product discovery at Songkick. Let’s first start with who is involved in product discovery, the dream team. At Songkick we have what we call a product triumvirate. I learned this from Marty Cagan of SVPG, one of the best product thinkers around. !The product triumvirate consists of your product manager, your design lead, and your tech lead. Together, they represent the 3 ingredients to product success: the business, the user, and the engineering to make it all possible. The three of them are the BFFs responsible for product discovery. They should be meeting and talking every day about product discovery. At Songkick, we block out an hour every day after stand-up to talk about what we’re learning and doing regarding product discovery. Having this regular time to catch up on what we’ve learned and what we want to learn next has been really important to keep things moving. Remember what I said about speed being everything? !The three of us meet every day to ask questions and figure out the fastest way to answer them. These questions form the foundation of our hypotheses.

Page 9: Songkick Product Discovery FOWA (Michelle You)

GOLDEN RULE OF PRODUCT DISCOVERY

02 FRAME THE QUESTION

That leads me to the next golden rule of product discovery: frame the question.

Page 10: Songkick Product Discovery FOWA (Michelle You)

Asking the right questions

UI What do they see?

Interaction How will people actually interact with it?

Feature/project How will this help your product solve the problem?

Product proposition How will your product solve the problem?

Customer discovery What problem are you solving and for whom?

I like to think about these questions in terms of levels of the product. Depending on which level you’re exploring in, your questions will be different and will require different type of feedback and a different type of discovery. !At the bottom, deepest level of the product—the most important level—are your customers. Who are your customers? What do they have in common? And what is their biggest problem? How do they currently solve that problem? For example, at Songkick, we help fans go to more concerts. We promise that they’ll never miss their favorite bands live. Our customer is the lapsed concertgoer who has a busy life and doesn’t have time to put in the work of scouring listings and consuming music news. They maybe go to 2 concerts a year when they can be bothered, but ends up going to twice as many concerts after they sign up to Songkick. We learned early that our target customer isn’t the avid concertgoer who’s already going to 2 concerts a week because they’re already doing all the hard work to find out when concerts are happening. We can’t help them as much. !Then, you have the product you’ve created or will create to solve the problem. Will it work? Will it help people? Is it doing a good job? How do you know? How do we solve the problem? We help fans track their favorite artists and then send concert alerts to them via email or push notification when new concerts are announced for those artists. We aggregate all the ticket options for the concert so they can quickly and easily buy tickets. !What features help fans do this? One example for us is an iTunes import to track artists from your iTunes library. This feature sits alongside other ways of getting your taste as quickly & easily as possible, such as importing your Facebook likes or artists in your Spotify playlists. Again, if you don’t know how your product is solving a real problem, you won’t know if your feature will be useful or not. !At the interaction level, we have the specific ways people need to interact with your product in order to perform successfully complete their actions. These are the screens, buttons, gestures, taps, flows they need to experience to achieve their goals.

Page 11: Songkick Product Discovery FOWA (Michelle You)

GOLDEN RULE OF PRODUCT DISCOVERY

03 THE CHEAPEST,

FASTEST FEEDBACK

Frame the question first, then find the cheapest, fastest way to get answers to that question. Hint: it’s not always code and it’s not always A/B tests and it’s not always quantitative. !We start with the product triumvirate deciding on what questions we have and then as a trio, we come up with the fastest way to answer that question. !Remember what I said before about speed & quantity being everything? There’s usually a faster, cheaper way to get feedback, so keep digging.

Page 12: Songkick Product Discovery FOWA (Michelle You)

Toolkit: quantitative and qualitative• Analytics • A/B tests • Real-time behavior • In-app feedback • Surveys

• Mock-up and click tests • Landing page tests • Remote usability testing • User research: in person

and over Skype

Here are various tools in our Toolkit of feedback, quantitative and qualitative - Analytics: to learn what people are doing right now with your product. We use Google Analytics and Mixpanel - Real-time user behavior: CrayzEgg, Ghostrec - Landing page tests: Unbounce - In-app feedback: Qualaroo and Olark - Survey tools: Qualaroo & Polldaddy - Mock-up and click tests: VerifyApp, FiveSecondTest - Remote usability testing: Usertesting.com and WhatUsersDo - User research: in person and over Skype <— my favorite!

Page 13: Songkick Product Discovery FOWA (Michelle You)

EXAMPLE: CUSTOMER DISCOVERY

UI What do they see?

Interaction How will people actually interact with it?

Feature/project How will this help your product solve the problem?

Product proposition How will your product solve the problem?

Customer discovery What problem are you solving and for whom?

I’ll walk you through a few layers of product discovery and the methods of feedback we’ve used at Songkick. !Example 1: Customer discovery Who has the problem? What do they have in common? What is their problem and how do they solve it right now? In Y Combinator’s words, will you be making something people want? !In the lean startup world, this is customer discovery. In the user-centered design world, this is contextual inquiry. No matter what it’s called, it’s the foundation of making great products. You’re discovering your customers and their problems, often before a single wireframe is sketched, before any code is written. There is no substitute for face-to-face empathy when it comes to customer discovery. You can’t A/B test your way into understanding what problems people have. !Let’s take a look at Detour. Detour allows fans to pledge to buy tickets for artists before a tour date is confirmed. This real demand signal in the form of a saved, pre-authorized credit card then allows artists to plan their tour schedules more accurately. !But before Detour the product was launched, we conducted dozens of interviews with artists, managers, promoters, and agents to understand how tours are planned. Until that point in time, Songkick was only a product for fans. We understood the fan pain points very well and had years of research under our belt. But we knew very little about the other side of the equation: the artists. !So we met with all these different roles in the industry and asked them the same set of questions to start mapping out an understanding of the process artists go through when planning tours and the common pain points they share. Questions like, Tell me about the last tour you were involved in... What was your biggest frustration? !

Page 14: Songkick Product Discovery FOWA (Michelle You)

• Document what you’re trying to learn. • Write an interview script to learn those things. Use it for

every interview. • Ask open-ended questions & listen. • Ask people to tell you stories about specific experiences. • Don’t ask people to predict future behavior. • Pair interview & transcribe.

CUSTOMER DISCOVERY TIPS

Here’s what I’ve learned about conducting interviews with customers: !- Start out with what you’re trying to learn. Document and agree on those questions with whoever has an interest in the research. - Design a script of questions that will help you get answers to those questions. Having the same set of questions that you ask consistently will help patterns emerge among your customers. As you hear answers from dozens of people to the same questions, you’ll start to recognize the same answers and spot patterns. - Ask people to tell you specific stories about past experiences. Don’t ask people to generalize. Ask “Tell me about a time this happened to you.” Don’t ask “Do you usually.....?” People are unreliable at self-reporting behavior. - When possible, conduct these interviews with someone else who can also observe & take notes. It’s helpful to have someone to discuss what you thought you heard.

Page 15: Songkick Product Discovery FOWA (Michelle You)

TAKE THE TIME TO SYNTHESIZE INSIGHTS

After each hour-long interview, we sat down and synthesized everything we learned. We would take our raw transcript and pull out observations. We created affinity diagrams of our observations until we we had synthesized the raw transcripts into digestible insights. I call this processing raw broccoli into Doritos. It takes a long time and a lot of debate, but the insights that come out of the other side are then the most important observations you want to share with your team. If you just dumped your pages of transcripts onto people to read for themselves, they’d never do it, and they wouldn’t know how it should inform their day to day. !This process took us about one month from start to finish. We interviewed dozens of people. We were doing customer discovery from scratch, learning about a brand new customer and their problems.

Page 16: Songkick Product Discovery FOWA (Michelle You)

Through this process, one of the things we learned is that the availability of streaming music services such as YouTube and Soundcloud meant that artists were growing an international fanbase faster than ever before. But tour planning hadn’t yet adjusted to these new realities, so while artists knew they had some fans in Brazil, Mexico, Singapore, Indonesia, they weren’t sure how many and whether there were enough to plan a tour date. And there was a lot of waste in the system. It felt like a product such as Detour that exposed this demand in a reliable signal in the form of a pledge--a pre-ordered, guaranteed ticket--could help solve this problem.

Page 17: Songkick Product Discovery FOWA (Michelle You)

SONGKICK TOUR DATA 2008

Our own data backs this up. This is all the tour routes for all artists in 2008.

Page 18: Songkick Product Discovery FOWA (Michelle You)

SONGKICK TOUR DATA 2012

And these are all the tour routes in 2012. You can see the pink lines are much more dense overall, showing that there are more artists on the road than ever before, as well as a ton more tours going to Asia, South America, and Australia.

Page 19: Songkick Product Discovery FOWA (Michelle You)

EXAMPLE: PRODUCT PROPOSITION

UI What do they see?

Interaction How will people actually interact with it?

Feature/project How will this help your product solve the problem?

Product proposition How will your product solve the problem?

Customer discovery What problem are you solving and for whom?

So with Detour, we identified customers on both sides of the market: fans and artists, and had an idea for a product that could help solve their problem: a way for fans to pledge to buy tickets before the tour is planned.

Page 20: Songkick Product Discovery FOWA (Michelle You)

Once we’ve done the hard work to identify the customer and the problem they’re having, we still may not be sure exactly what proposition will resonate with them. We’ll write a few different propositions with an email collection box using a tool like Unbounce, and send traffic to each version to see which proposition converts best. This helps give a signal as to what is resonating most strongly. It won’t tell you exactly what the product should be, but it will help point you in the right direction.

Page 21: Songkick Product Discovery FOWA (Michelle You)

EXAMPLE: FEATURE

UI What do they see?

Interaction How will people actually interact with it?

Feature/project How will this help your product solve the problem?

Product proposition How will your product solve the problem?

Customer discovery What problem are you solving and for whom?

I’ll now talk you through the various ways we went through redesigning a feature we thought could be improved. !On Detour, the critical conversion rate for the product was the % of fans who saved their credit card. Without this saved credit card, a pledge was as trivial as a like or a listen, it didn’t help the artist forecast actual demand. So that conversion rate was the metric we wanted to improve. !We saw that our conversion rate to saving a credit card was lower than we thought it should be, so we came up with some questions. Remember what I said about framing the question? !- What step in the funnel are people dropping off? - Why aren’t people saving their credit card? - When is the right time to ask people for their credit card? - Is this a usability issue or a desirability issue? Do people understand why they need to save their credit card or do they just not want to? !Often, a lot of your discovery work will be about this distinction between usability and desirability. It’s easy to figure out whether your product is usable: whether people understand what they should do, what steps they need to take, and why. If the low conversion is not an issue with usability and understanding, but rather with desirability -- that’s a much harder problem to solve. !For each of these questions, the product triumvirate: the PM, TL, and design lead sat down and came up with ways to get feedback on those questions.

Page 22: Songkick Product Discovery FOWA (Michelle You)

LOVE YOUR FUNNEL

Let’s take the first question: Where are people dropping off? !To do this, it’s important that your product trio share an understanding of the steps people can take before they convert. Your funnel. Even for a seemingly simple site, your funnel will have many entry points and exit points. We diagrammed our funnel to saving credit card and it looked something like this.

Page 23: Songkick Product Discovery FOWA (Michelle You)

Once you have that funnel diagrammed, it’s important to make sure you’re measuring every step in the funnel. You have to do the boring, anal work of defining the events & properties that will be triggered when someone performs an action. It’s painstaking stuff if you’re not part of a big company with a business intelligence team, but when you see how much the data can teach you, it’ll be worth it. !We took this flow diagram and screenshot every step and put it on a wall. Then, as a product trio, we’ll sit down and write post-it notes of what we want to learn at each step of the funnel: How many people click that button? Are they logged-in or logged out? How many people saw an error message? Did they get to that page by clicking on the photo or the text, since both links go to the same place? What template are they on when they click that button? Are they connected to Twitter and Facebook? Are they on Android? !It’s easy to go overboard, so you should step back and ask how the information will help you make better decisions. This process can take hours when you’re doing it for the first time and are defing your categories for the first time, but once you do it, the data keeps on giving! !If you take the time to add the right analytics, you can answer so many questions without writing any code, like What page are people converting on? How many pages do they visit before they convert? How much time does it take them between steps in the funnel? What is the optimal time to remind someone about a concert? This is all learning. !Once we analyzed our funnel and had dropoff rates for each step, we knew what was happening, but not why.

Page 24: Songkick Product Discovery FOWA (Michelle You)

That’s when we started getting qualitative feedback and usability testing the product. !At Songkick, we’ve built our own usability lab in-house. We knocked down a wall in our meeting room and created a two-way mirror, where on the one side, the researcher and participant sit and look at prototypes, where on the other side of the wall, we can watch what’s happening, see a screenshare of what the user is seeing. It was cheap and meant that we didn’t have to rent a lab whenever we wanted to do usability testing. !Through usability testing, we learned that people didn’t understand that they needed to save their credit card. They didn’t realize that it was a step they needed to take. They would click on a “pledge” button, sit back, and think they were done. !Pledging to buy a ticket for a concert that doesn’t exist yet is such a weird and novel experience, they didn’t know what to expect.

Page 25: Songkick Product Discovery FOWA (Michelle You)

EXAMPLE: INTERACTIONS

UI What do they see?

Interaction How will people actually interact with it?

Feature/project How will this help your product solve the problem?

Product proposition How will your product solve the problem?

Customer discovery What problem are you solving and for whom?

The next step was to quickly generate a lot of ideas around how we could solve this problem. People didn’t understand that they need to save their card, so how could we design the interaction better to help users along?

Page 26: Songkick Product Discovery FOWA (Michelle You)

At Songkick, we often use cross-functional sessions to generate a lot of ideas. Our design director, Gideon, likes to say that everyone at Songkick is a designer, and I agree with him. Everyone from our customer support team to our CEO shapes our users’ understanding of who we are and what we do. So we do a lot of collaborative design studios, using a lot of post-its and Sharpies. The medium of pen & paper is fast and cheap and the appropriate level of fidelity when you’re coming up with new ideas.

Page 27: Songkick Product Discovery FOWA (Michelle You)

We’ve even gone as far as to prototype with cardboard to get our idea across.

Page 28: Songkick Product Discovery FOWA (Michelle You)

• Involve people outside product team. • Frame the problem clearly. • Give people time to come up with ideas solo

before going into group critique. • Synthesize the best ideas and prototype

different options. • This can be done in an afternoon.

DESIGN STUDIO TIPS

What we’ve learned about conducting these design studios: - Frame the problem very clearly at the start of the session. - Give people time to generate ideas solo, before going into a group critique. This allows for a wide varieties to emerge before you narrow down to the best ones. !Remember what I said before about speed and quantity? Design studios are great for slamming out a quantity of options. Then you have to give feedback and narrow in on the best ideas you want to prototype first. At Songkick, the design lead will take the output of our session and synthesize those ideas into prototypes we want to test with our users.

Page 29: Songkick Product Discovery FOWA (Michelle You)

• Start with your hypothesis: “If… then… because…”

• How many people do you need? • Record results. • Take the test down.

A/B TESTS

With this specific interaction of saving your credit card, we then A/B tested different versions of the interaction, asking people for their credit card at different steps, in modal pop-ups vs. full pages to determine which version would convert best. !When we A/B test an idea with real code, we always run a kick-off where we document our hypothesis for that variant. It comes in the form of an “if then because” statement. I learned this from the Mind the Product blog. !If we ask you to save your credit card in a pop-up after you click a pledge button, then you’ll be more likely to save your credit card, because it’s clear that’s the next step in the process. !It’s really important for the product trio to discuss this hypothesis and agree on it, because otherwise you’ll each have a different interpretation of what an A/B test result is telling you. !We have a checklist kick-off doc that includes a set of questions we run through at the start of any experiment to make sure we don’t miss any important questions. At Songkick, we love checklists. I took a lot of inspiration from Atul Gawande’s research into how checklists helped hospitals reduce infection rates by giving them a simple list of things to do to provide the best care. !We’re not doing brain surgery or dealing with life or death issues, but at Songkick, our checklists make sure we go through the necessary steps to produce the best results. !Some checklist questions in our experiment kick-off: - What’s our hypothesis? What’s our goal in running this experiment? What’s our current conversion rate? What’s our expected improvement to the metric? How many trials do we need in order to get statistically significant results? What analytics events do we need to add to the product? !

Page 30: Songkick Product Discovery FOWA (Michelle You)

EXAMPLE: REMOTE USABILITY

UI What do they see?

Interaction How will people actually interact with it?

Feature/project How will this help your product solve the problem?

Product proposition How will your product solve the problem?

Customer discovery What problem are you solving and for whom?

We did also did lot of remote usability testing on Detour to test our explanations of the concept. We knew that Detour is quite a strange experience when compared to buying a normal ticket. We’re asking you to pledge to buy a ticket for a band who hasn’t announced a concert yet. Unlike crowdfunding sites, the creator has not yet been involved until we aggregate their demand and reach out to them. We knew there was a big uphill battle in educating fans about how the system works and to earn their trust.

Page 31: Songkick Product Discovery FOWA (Michelle You)

So we put prototypes and mock-ups in front of people on sites like Usertesting.com and Verify App and ask them questions to make sure they understood what was going on. Do they understand when they’ll be charged? Do they know why we’re asking for their credit card? How would they explain the concept to a friend? (That’s one of my favorite go-to questions to hear people talk about our product in their own words.) !A click test tool like Verify App can make sure your UI design is communicating effectively. We refined our design and copy until people really understood how the platform works. Those sites can give you really great, fast feedback on simple tasks -- the raw usability -- but won’t give you the deeper desirability insights that face to face interviews will provide. !But if your question is around “do people understand how to use this?”, these tools can give you the answers very quickly. Literally, in minutes, you’ll have videos of people struggling with your product, very confused, and unsure about what to do. !At the end of this discovery process, we learned enough to improve our conversion funnel by about 30%. !

Page 32: Songkick Product Discovery FOWA (Michelle You)

GOLDEN RULE OF PRODUCT DISCOVERY

01 SPEED & QUANTITY

OF LEARNING

This is just a glimpse into how we do product discovery at Songkick. We use methods as varied as face-to-face interviews, surveys, remote usability testing, and A/B tests to learn as fast as possible. !To recap my 3 golden rules of product discovery: your #1 goal is answering questions & learning, not shipping product. This learning then informs your product delivery phase when you’re spending the time & resources to build product based on what you’ve learned.

Page 33: Songkick Product Discovery FOWA (Michelle You)

GOLDEN RULE OF PRODUCT DISCOVERY

02 FRAME THE QUESTION

You need to ask questions before you learn. It’s important to know what you’re trying to understand, what your unknowns are.

Page 34: Songkick Product Discovery FOWA (Michelle You)

Asking the right questions

UI What do they see?

Interaction How will people actually interact with it?

Feature/project How will this help your product solve the problem?

Product proposition How will your product solve the problem?

Customer discovery What problem are you solving and for whom?

Know which level of the product you’re in when you’re learning. It makes no sense to test two different button colors at the UI level if you don’t even know whether your product is solving a real problem for people.

Page 35: Songkick Product Discovery FOWA (Michelle You)

GOLDEN RULE OF PRODUCT DISCOVERY

03 THE CHEAPEST,

FASTEST FEEDBACK

As a product team, you’re always fighting to find the cheapest, fastest way to answer your questions. You should have a variety of tools in your toolkit to get feedback and use all of them at different points in your product discovery journey. At our top speed, we’re running 3 experiments a week, but we’re always introspecting about how to go faster.

Page 36: Songkick Product Discovery FOWA (Michelle You)

Toolkit: quantitative and qualitative• Analytics • Real-time behavior • In-app feedback • Surveys

• Mock-up and click tests • Landing page tests • Remote usability testing • User research: in person

and over Skype

You should make use of a diverse range of tools to answer your questions. Both quantitative and qualitative feedback are necessary to building the right product. You’ll find that as a team, you might feel more comfortable with certain methods over others, quantitative rather than qualitative or vice versa, but both are necessary to product discovery, so force yourselves to use all of them.

Page 37: Songkick Product Discovery FOWA (Michelle You)

MICHELLE YOU CPO & CO-FOUNDER@WRECKINGBALL37

Thanks to @nfelger, @ClarkeTom, and @gideonb for feedback!

Happy discovering!