How Spotify Tests World Class Apps
-
Upload
techwellpresentations -
Category
Technology
-
view
253 -
download
1
description
Transcript of How Spotify Tests World Class Apps
![Page 1: How Spotify Tests World Class Apps](https://reader036.fdocuments.us/reader036/viewer/2022062615/5483b70db4af9f8e428b4569/html5/thumbnails/1.jpg)
April 10, 2023
How Spotify Tests World Class AppsAlexander Andelkovic, Spotify
![Page 2: How Spotify Tests World Class Apps](https://reader036.fdocuments.us/reader036/viewer/2022062615/5483b70db4af9f8e428b4569/html5/thumbnails/2.jpg)
AgendaIntroduction
Background
Approval process
Future
Summary
Questions
![Page 3: How Spotify Tests World Class Apps](https://reader036.fdocuments.us/reader036/viewer/2022062615/5483b70db4af9f8e428b4569/html5/thumbnails/3.jpg)
Introduction – About me
- Lives in Stockholm, Sweden
- 15+ years testing experience
- Have a background in telecom and Med-Tech
- Test lead at Spotify
- Passionate about advanced test techniques
![Page 4: How Spotify Tests World Class Apps](https://reader036.fdocuments.us/reader036/viewer/2022062615/5483b70db4af9f8e428b4569/html5/thumbnails/4.jpg)
Background – Spotify Apps
- Apps(HTML5 & JS) integrated into Spotify music client(C++)
- Launched 30 Nov 2011 (at launch13 apps, today over 100 apps)
- Different categories of apps (discovery, concert, review, social, lyrics)
- Manual QA process
![Page 5: How Spotify Tests World Class Apps](https://reader036.fdocuments.us/reader036/viewer/2022062615/5483b70db4af9f8e428b4569/html5/thumbnails/5.jpg)
Background – QA Challenges- Spotify App = mini application
- Release high quality in no time
- Testing needs to scale over time(partners need to test)
- Stable test environment(new API, backend)
- Dependant on external development(partners)
![Page 6: How Spotify Tests World Class Apps](https://reader036.fdocuments.us/reader036/viewer/2022062615/5483b70db4af9f8e428b4569/html5/thumbnails/6.jpg)
Approval process - Introduction
- Steps that help partners ”release” high quality apps
- Approval steps: Concept > Development > Release > Update
- Average app development time 3 months
- Approval team around 10-15 persons
![Page 7: How Spotify Tests World Class Apps](https://reader036.fdocuments.us/reader036/viewer/2022062615/5483b70db4af9f8e428b4569/html5/thumbnails/7.jpg)
Approval process - Team
- Product owners(prioritize tasks, final calls)
- QA(manual testers)
- Test automators(regression test)
- Designers(concept, prototype)
- Release manager(silent/big bang release)
![Page 8: How Spotify Tests World Class Apps](https://reader036.fdocuments.us/reader036/viewer/2022062615/5483b70db4af9f8e428b4569/html5/thumbnails/8.jpg)
Approval process - Concept- An app shall bring ‘valuable content’ or something ‘unique’ to the Spotify experience.
- A good concept will save development time
- Summary, Mockups, Wireframes, Screenshots, Alpha version
- Designers gives valuable feedback
- No development should be started until concept is approved
![Page 9: How Spotify Tests World Class Apps](https://reader036.fdocuments.us/reader036/viewer/2022062615/5483b70db4af9f8e428b4569/html5/thumbnails/9.jpg)
Approval process - Development
- App development can start after concept approved and agreement signed
- App should be submited for approval when ”production ready”
- QA gives thorough feedback on app quality
- An app is approved when ”all” issues are resolved
![Page 10: How Spotify Tests World Class Apps](https://reader036.fdocuments.us/reader036/viewer/2022062615/5483b70db4af9f8e428b4569/html5/thumbnails/10.jpg)
Approval process - Release
- When an app is approved a release date and market(s) are decided
- PR involved(internal, external)
- Release manager inform partner when app is live
- Live app needs to be maintained
![Page 11: How Spotify Tests World Class Apps](https://reader036.fdocuments.us/reader036/viewer/2022062615/5483b70db4af9f8e428b4569/html5/thumbnails/11.jpg)
Approval process - Update
- Bugfixes, tweaks, small features
- Requires new App Submission
- App goes through the approval phase for Development again
- ”New” version treated as a new app concept(approval phase concept)
![Page 12: How Spotify Tests World Class Apps](https://reader036.fdocuments.us/reader036/viewer/2022062615/5483b70db4af9f8e428b4569/html5/thumbnails/12.jpg)
Approval process - Kanban- “Just-in-time” approval process
- App submission enter pipeline
- Approval process(Concept > Development > Release > Update) inside pipeline
- App submission approved/not approved emerges from the other end of the pipeline
- Effective development process for identifying bottlenecks
![Page 13: How Spotify Tests World Class Apps](https://reader036.fdocuments.us/reader036/viewer/2022062615/5483b70db4af9f8e428b4569/html5/thumbnails/13.jpg)
Approval process - Support
- Stack overflow
- Internal tech support
- Keeps guidelines updated
- Best practises posted to developer blog
![Page 14: How Spotify Tests World Class Apps](https://reader036.fdocuments.us/reader036/viewer/2022062615/5483b70db4af9f8e428b4569/html5/thumbnails/14.jpg)
Approval process - Partners
- One man shows
- Web developers
- App developers
- Inhouse development
![Page 15: How Spotify Tests World Class Apps](https://reader036.fdocuments.us/reader036/viewer/2022062615/5483b70db4af9f8e428b4569/html5/thumbnails/15.jpg)
Approval process - Checklist- Approximately 50 teststeps(manifest, navigation, performance, security etc..)- Should be run through by partner before app submission- Session based testing used(Charter, timeboxed, debrief)- Test session divided into smoke test and exploratory test(ET) phase- ET mindset crucial- Pair testing performed on first submission to find as many bugs as possible(reduces re-submissions)
![Page 16: How Spotify Tests World Class Apps](https://reader036.fdocuments.us/reader036/viewer/2022062615/5483b70db4af9f8e428b4569/html5/thumbnails/16.jpg)
Approval process – Test automation
- Regression vs bug finding test automation(TA)
- Robust TA a challenge(false positives)
- Empower partners to perform own TA
- App TA tests production environment
![Page 17: How Spotify Tests World Class Apps](https://reader036.fdocuments.us/reader036/viewer/2022062615/5483b70db4af9f8e428b4569/html5/thumbnails/17.jpg)
Approval process - Security
- App security tests performed by Security Penetration Testers
- All ways of input needs to be tested
- Are HTML5 and javascript apps a security risk?
- Partner homepages
![Page 18: How Spotify Tests World Class Apps](https://reader036.fdocuments.us/reader036/viewer/2022062615/5483b70db4af9f8e428b4569/html5/thumbnails/18.jpg)
Approval process - Metrics
- Performance metrics(daily, weekly, monthly goals) used to tweak approval process
- Enables greather understanding of user app behaviour
- Drives A/B testing of appfinder
- Continous app content tweaking
![Page 19: How Spotify Tests World Class Apps](https://reader036.fdocuments.us/reader036/viewer/2022062615/5483b70db4af9f8e428b4569/html5/thumbnails/19.jpg)
Future
- Quantity vs quality
- More test automation(bug finding, crawler)
- More advanced apps(WebGL)
- App SDK’s
![Page 20: How Spotify Tests World Class Apps](https://reader036.fdocuments.us/reader036/viewer/2022062615/5483b70db4af9f8e428b4569/html5/thumbnails/20.jpg)
Summary
- Developer performing tests early reduces app development time
- ”Tweak it” mindset crucial for an efficient approval process
- App develpment a new dicipline
- Apps are fun and easy try it out
![Page 22: How Spotify Tests World Class Apps](https://reader036.fdocuments.us/reader036/viewer/2022062615/5483b70db4af9f8e428b4569/html5/thumbnails/22.jpg)
April 10, 2023
Thank you!