Getting Strategic with Digital: Intro to Analytics
-
Upload
nform-user-experience -
Category
Data & Analytics
-
view
234 -
download
0
Transcript of Getting Strategic with Digital: Intro to Analytics
![Page 1: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/1.jpg)
Getting Strategic with Digital Intro to Analytics
Research & Product Management: How I Learned to Stop Worrying and
Love the Customer
![Page 2: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/2.jpg)
Who am !?• ME = James Klassen• TALKING ABOUT = Research and putting the
customer 1st!• If you have a question, please just yell!
• Background in market research, online marketing, SEO, social media, analytics… blah blah blah
• Digital Product Manager @ AMA Insurance• In addition to product management, also lead a
small, elite team of marketing hackers
![Page 3: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/3.jpg)
Questions
• Who regularly uses analytics/research to learn about their customers? Online specifically?
• Who works in digital/ directly with people in digital?
![Page 4: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/4.jpg)
I have a case study!
![Page 5: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/5.jpg)
First, some results
![Page 6: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/6.jpg)
Which one converts better?
![Page 7: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/7.jpg)
Which one converts better?
50%increase
![Page 8: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/8.jpg)
Which one converts better?
![Page 9: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/9.jpg)
Which one converts better?
75%decrease
![Page 10: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/10.jpg)
Some context…
![Page 11: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/11.jpg)
What are we trying to learn?• Big question: who are the people using our
website?– What do they want, how do they want it, what do
they already know?– How can we organize our site to help them find
what they’re looking for?– Because it’s insurance, are they looking for the
right things?• We want to help educate about insurance
policies/coverage/etc.– What functionality should we add/delete?– How can we design our site better?
![Page 12: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/12.jpg)
Our path• Market research: understand broad strokes• Online survey 1: User intent
– We had no idea, clean slate• Traditional market research gave us direction, but we wanted to
validate online: were those people already talking to us?– Tried to keep in mind that the state of our current site was
going to influence responses• Online survey 2: Service preference
– Offline vs. online – many services not available online– How do people want to interact with us?– What type of insurance are they looking for?
• Online survey 3: Current customer?– Already insured? Already a member?
![Page 13: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/13.jpg)
Our path
• So, we decided to change our main-page templates to reflect what we learned
• For /insurance/ and central products:1. Make it easy to accomplish high-priority
tasks2. Make it more obvious that you’re on an
insurance-related page3. (/insurance/) make it easier to access
second-level self-service
![Page 14: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/14.jpg)
Thus…
![Page 15: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/15.jpg)
And…
![Page 16: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/16.jpg)
Successes are easy… so about that grey box...
• Initially:– Made the change without testing… OBVIOUSLY
THE NEW DESIGN IS BETTER– Saw big increase in bounce rate, decrease in
conversion rate– This was basically a before/after test
• Ok, we need to learn more:– Run it A/B style in a more controlled environment
(maybe external factors were impacting the results… this is what I was hoping for)
![Page 17: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/17.jpg)
2nd test: A/B
![Page 18: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/18.jpg)
2nd test: A/B
70%decreaseinquotes
![Page 19: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/19.jpg)
2nd test: A/B
194%increaseinbouncerate
![Page 20: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/20.jpg)
The horrible grey box
• Still really hate the grey box
• There must be something wrong inside of the 2-box setup– Let’s try different headings
![Page 21: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/21.jpg)
3rd test: A/B 3 treatments
![Page 22: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/22.jpg)
3rd test: A/B 3 treatments
58%increaseinquotes
133%increaseinquotes
![Page 23: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/23.jpg)
The horrible, terrible grey box• Simply removing 1 of the boxes, led to 133%
increase in quotes…• Unfortunately, it looks like 2 boxes are just a
losing combination
• SIGH
• So we switched back to the original grey box…
![Page 24: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/24.jpg)
The horrible, terrible, ugly grey box
• BUT I still think we can make it better!
• So let’s try making the single box better!– More visible text– More visible box colour– Remove overall page header
![Page 25: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/25.jpg)
4th test: A/B 4 treatments
![Page 26: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/26.jpg)
4th test: A/B 4 treatments
INCONCLUSIVE!!!
![Page 27: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/27.jpg)
So what’s the outcome ofJames vs. the grey box?
![Page 28: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/28.jpg)
So where are we now with the grey box?
• Slowly coming to terms with it
• We’re still learning and looking for ways to improve– “best practice” doesn’t work
• The fact is: it performs well!
![Page 29: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/29.jpg)
So where are we now with the grey box?
• What we should all learn:– This stuff doesn’t always go according to plan– Users don’t always want what you want them to
want– Testing isn’t magical, it’s actually more work– Check your bias – even THIS IS OBVIOUSLY
better doesn’t always work– The same approach in 2 different contexts might
not work… and you might not realize there are different contexts!
• Can’t set it and forget it... Once you start down the dark path...
![Page 30: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/30.jpg)
Some common mistakes• Not using the appropriate research tool for the
question• Making bias or stats-related errors in A/B tests
– Chasing 100% confidence– Not determining the parameters of your experiments– Not accounting for environmental factors like
seasonality• Putting too much weight into results• Testing too much, too big, too small• Testing things you can’t change
![Page 31: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/31.jpg)
Research tips• If you don’t have research/ stats experience, get help
– Ask your internal business intelligence and market research team(s)
– There are loads of great resources online• Test a lot, but know you can test too much
– Easy to test ”too big” or “too small”• Get to know your analytics platform• Research isn’t just a/b testing
– Don’t forget about surveys, focus groups, analytics, internal interviews (retail, product experts)
• Think about using Agile– Puts the customer at the centre and that’s really what
we’re talking about here
![Page 32: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/32.jpg)
Questions?
![Page 33: Getting Strategic with Digital: Intro to Analytics](https://reader031.fdocuments.us/reader031/viewer/2022030308/58f31feb1a28ab93398b4591/html5/thumbnails/33.jpg)
Thanks!James Klassen
Digital Product ManagerAMA Insurance
AMAInsurance.caca.linkedin.com/in/jrklassen
@forsoothed