Can WebRTC help musicians? - FOSDEM
Transcript of Can WebRTC help musicians? - FOSDEM
![Page 1: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/1.jpg)
Can WebRTC help musicians?Going beyond traditional and boring use cases to support the arts
Lorenzo Miniero@elminiero
FOSDEM 2021 Real Time devroom6th February 2021, Brussels My couch
![Page 2: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/2.jpg)
Who am I?
Lorenzo Miniero• Ph.D @ UniNA• Chairman @ Meetecho• Main author of Janus
Contacts and info• [email protected]• https://twitter.com/elminiero• https://www.slideshare.net/LorenzoMiniero• https://soundcloud.com/lminiero
![Page 4: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/4.jpg)
An amazing music ecosystem on Linux
![Page 5: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/5.jpg)
“Can WebRTC help musicians?”
https://linuxmusicians.com/viewtopic.php?t=21617
![Page 6: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/6.jpg)
Tsahi’s article back in 2017
https://bloggeek.me/sound-guru-webrtc/ (2017)
![Page 7: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/7.jpg)
Opus helps already!
• WebRTC mandates Opus, and it’s a good thing• High quality audio codec designed for the Internet• Very flexible in sampling rates, bitrates, etc.
• Different profiles for voice and music• Both encoding and decoding vary, in case• Can be mono and stereo (or more, as we’ll see in a minute!)
• With the right bitrate, it can sound quite good• ... and most importantly, with the help of WebRTC, real-time!
A “live” example: pre-recorded music via WebRTChttps://janus.conf.meetecho.com/streamingtest
![Page 8: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/8.jpg)
Opus helps already!
• WebRTC mandates Opus, and it’s a good thing• High quality audio codec designed for the Internet• Very flexible in sampling rates, bitrates, etc.
• Different profiles for voice and music• Both encoding and decoding vary, in case• Can be mono and stereo (or more, as we’ll see in a minute!)
• With the right bitrate, it can sound quite good• ... and most importantly, with the help of WebRTC, real-time!
A “live” example: pre-recorded music via WebRTChttps://janus.conf.meetecho.com/streamingtest
![Page 9: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/9.jpg)
Opus helps already!
• WebRTC mandates Opus, and it’s a good thing• High quality audio codec designed for the Internet• Very flexible in sampling rates, bitrates, etc.
• Different profiles for voice and music• Both encoding and decoding vary, in case• Can be mono and stereo (or more, as we’ll see in a minute!)
• With the right bitrate, it can sound quite good• ... and most importantly, with the help of WebRTC, real-time!
A “live” example: pre-recorded music via WebRTChttps://janus.conf.meetecho.com/streamingtest
![Page 10: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/10.jpg)
Opus helps already!
• WebRTC mandates Opus, and it’s a good thing• High quality audio codec designed for the Internet• Very flexible in sampling rates, bitrates, etc.
• Different profiles for voice and music• Both encoding and decoding vary, in case• Can be mono and stereo (or more, as we’ll see in a minute!)
• With the right bitrate, it can sound quite good• ... and most importantly, with the help of WebRTC, real-time!
A “live” example: pre-recorded music via WebRTChttps://janus.conf.meetecho.com/streamingtest
![Page 11: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/11.jpg)
Multiopus: 5.1 and 7.1 surround audio
• This is little known, but Chrome does support surround audio in WebRTC• Not really documented or standardized, though• Mostly just there because it’s used by Stadia, today
• Multiopus (5.1 and 7.1)• Each packet is basically OGG with multiple stereo Opus streams• Number of streams determines number of channels
• SDP munging needed on both offer and answer to specify the mapping
Some reading material if you’re curious• https://webrtcbydralex.com/index.php/2020/04/08/surround-sound-5-1-and-7-1-in-
libwebrtc-and-chrome/• https://github.com/meetecho/janus-gateway/pull/2059 (now supported in Janus)
![Page 12: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/12.jpg)
Multiopus: 5.1 and 7.1 surround audio
• This is little known, but Chrome does support surround audio in WebRTC• Not really documented or standardized, though• Mostly just there because it’s used by Stadia, today
• Multiopus (5.1 and 7.1)• Each packet is basically OGG with multiple stereo Opus streams• Number of streams determines number of channels
• SDP munging needed on both offer and answer to specify the mapping
Some reading material if you’re curious• https://webrtcbydralex.com/index.php/2020/04/08/surround-sound-5-1-and-7-1-in-
libwebrtc-and-chrome/• https://github.com/meetecho/janus-gateway/pull/2059 (now supported in Janus)
![Page 13: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/13.jpg)
Multiopus: 5.1 and 7.1 surround audio
• This is little known, but Chrome does support surround audio in WebRTC• Not really documented or standardized, though• Mostly just there because it’s used by Stadia, today
• Multiopus (5.1 and 7.1)• Each packet is basically OGG with multiple stereo Opus streams• Number of streams determines number of channels
• SDP munging needed on both offer and answer to specify the mapping
Some reading material if you’re curious• https://webrtcbydralex.com/index.php/2020/04/08/surround-sound-5-1-and-7-1-in-
libwebrtc-and-chrome/• https://github.com/meetecho/janus-gateway/pull/2059 (now supported in Janus)
![Page 14: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/14.jpg)
Tutoring and training
• A first simple use case: music lessons• Can be 1-to-1, or 1-to-many• In both cases, the closest to a “traditional” scenario
• It can basically be treated as a generic videocall or videoconference• Most of the interaction is conversational• No real need for plugging instruments directly in the call
• A regular mic is more than enough in this context
• My sister uses Skype to teach her students...• ... so why not WebRTC?
Cool add-on: pitch detection?• https://github.com/720kb/TeachMusicRTC (last updated 6 years ago, though)
![Page 15: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/15.jpg)
Tutoring and training
• A first simple use case: music lessons• Can be 1-to-1, or 1-to-many• In both cases, the closest to a “traditional” scenario
• It can basically be treated as a generic videocall or videoconference• Most of the interaction is conversational• No real need for plugging instruments directly in the call
• A regular mic is more than enough in this context
• My sister uses Skype to teach her students...• ... so why not WebRTC?
Cool add-on: pitch detection?• https://github.com/720kb/TeachMusicRTC (last updated 6 years ago, though)
![Page 16: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/16.jpg)
Tutoring and training
• A first simple use case: music lessons• Can be 1-to-1, or 1-to-many• In both cases, the closest to a “traditional” scenario
• It can basically be treated as a generic videocall or videoconference• Most of the interaction is conversational• No real need for plugging instruments directly in the call
• A regular mic is more than enough in this context
• My sister uses Skype to teach her students...• ... so why not WebRTC?
Cool add-on: pitch detection?• https://github.com/720kb/TeachMusicRTC (last updated 6 years ago, though)
![Page 17: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/17.jpg)
Tutoring and training
• A first simple use case: music lessons• Can be 1-to-1, or 1-to-many• In both cases, the closest to a “traditional” scenario
• It can basically be treated as a generic videocall or videoconference• Most of the interaction is conversational• No real need for plugging instruments directly in the call
• A regular mic is more than enough in this context
• My sister uses Skype to teach her students...• ... so why not WebRTC?
Cool add-on: pitch detection?• https://github.com/720kb/TeachMusicRTC (last updated 6 years ago, though)
![Page 18: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/18.jpg)
Broadcasting concerts
• A more interesting scenario: broadcasting concerts
• Basically a 1-to-many (maybe few-to-many?) streaming session
• Still traditional, if you will, but with a few caveats
• The audio source better not be the browser
• Browsers mess with the captured audio a lot (e.g., AEC, AGC, etc.)• You want the broadcasted audio to be as close as possible to what was captured
• OBS-WebRTC (via WHIP) or the Janus Streaming plugin (wink wink!) can help here
• This scenario is commonly done with HLS, today...
• ... but you may want less delay and/or a way to interact with the audience!
![Page 19: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/19.jpg)
Broadcasting concerts
• A more interesting scenario: broadcasting concerts
• Basically a 1-to-many (maybe few-to-many?) streaming session
• Still traditional, if you will, but with a few caveats
• The audio source better not be the browser
• Browsers mess with the captured audio a lot (e.g., AEC, AGC, etc.)• You want the broadcasted audio to be as close as possible to what was captured
• OBS-WebRTC (via WHIP) or the Janus Streaming plugin (wink wink!) can help here
• This scenario is commonly done with HLS, today...
• ... but you may want less delay and/or a way to interact with the audience!
![Page 20: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/20.jpg)
Broadcasting concerts
• A more interesting scenario: broadcasting concerts
• Basically a 1-to-many (maybe few-to-many?) streaming session
• Still traditional, if you will, but with a few caveats
• The audio source better not be the browser
• Browsers mess with the captured audio a lot (e.g., AEC, AGC, etc.)• You want the broadcasted audio to be as close as possible to what was captured
• OBS-WebRTC (via WHIP) or the Janus Streaming plugin (wink wink!) can help here
• This scenario is commonly done with HLS, today...
• ... but you may want less delay and/or a way to interact with the audience!
![Page 21: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/21.jpg)
About interacting with the audience...
![Page 22: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/22.jpg)
About interacting with the audience...
![Page 23: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/23.jpg)
Not really music, but close enough!
https://chrisuehlinger.com/blog/2020/06/16/unshattering-the-audience-building-theatre-on-the-web-in-2020
![Page 24: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/24.jpg)
Playing with music and WebRTC
• Many cool things that can be done withWebRTC
• e.g., browser or native app as an UI toa remote music setup
• Several more or less basic use casescome to mind
• Writing music in a browser• Interaction with (remote) instruments• Visual synchronization of music data• Integration in (remote) DAW• Distributed jam sessions• ...
![Page 25: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/25.jpg)
Playing with music and WebRTC
• Many cool things that can be done withWebRTC
• e.g., browser or native app as an UI toa remote music setup
• Several more or less basic use casescome to mind
• Writing music in a browser• Interaction with (remote) instruments• Visual synchronization of music data• Integration in (remote) DAW• Distributed jam sessions• ...
![Page 26: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/26.jpg)
A silly approach at online composition!
https://youtu.be/d1hOR27r4uY?t=1158
![Page 27: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/27.jpg)
Playing a keyboard with other people
https://youtu.be/8Hzg4hSJMsQ?t=790
![Page 28: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/28.jpg)
Playing a keyboard with other people
https://youtu.be/8Hzg4hSJMsQ?t=790
![Page 29: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/29.jpg)
Web MIDI API + Insertable Streams
https://twitter.com/komasshu/status/1329785808446836736
![Page 30: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/30.jpg)
Jam sessions
• What about really playing with other people, though?• Harder to do because of this ugly pandemic...
• Only apparently a traditional use case• Yes, we can see it as a “conference” of sorts...• ... but we’re not really talking, and latency is much more important
• Browsers are not a good option, here• Pipeline may be good for voice, but latency too high for live music
• Unfortunately, on Linux they don’t support Jack, only Pulseaudio
• Hard to capture anything else than a microphone
• Besides, as we said they’ll mess with the source audio anyway
![Page 31: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/31.jpg)
Jam sessions
• What about really playing with other people, though?• Harder to do because of this ugly pandemic...
• Only apparently a traditional use case• Yes, we can see it as a “conference” of sorts...• ... but we’re not really talking, and latency is much more important
• Browsers are not a good option, here• Pipeline may be good for voice, but latency too high for live music
• Unfortunately, on Linux they don’t support Jack, only Pulseaudio
• Hard to capture anything else than a microphone
• Besides, as we said they’ll mess with the source audio anyway
![Page 32: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/32.jpg)
Jam sessions
• What about really playing with other people, though?• Harder to do because of this ugly pandemic...
• Only apparently a traditional use case• Yes, we can see it as a “conference” of sorts...• ... but we’re not really talking, and latency is much more important
• Browsers are not a good option, here• Pipeline may be good for voice, but latency too high for live music
• Unfortunately, on Linux they don’t support Jack, only Pulseaudio
• Hard to capture anything else than a microphone
• Besides, as we said they’ll mess with the source audio anyway
![Page 33: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/33.jpg)
What about a native approach?
• A few, non-WebRTC, native solutions exist already• e.g., Jamulus and NINJAM (both open source)
• It might be interesting to experiment with WebRTC as well• e.g., Native client that uses Jack for audio input/output• WebRTC exchange of live streams (P2P or via a server)
Idea for a personal fun/pet project of mine• Native application based on GStreamer• Ability to add local instruments, captured via Jack and encoded with Opus• Janus as the reference WebRTC server for all the jam session “participants”
• Publishing local instruments, subscribing to remote ones
![Page 34: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/34.jpg)
What about a native approach?
• A few, non-WebRTC, native solutions exist already• e.g., Jamulus and NINJAM (both open source)
• It might be interesting to experiment with WebRTC as well• e.g., Native client that uses Jack for audio input/output• WebRTC exchange of live streams (P2P or via a server)
Idea for a personal fun/pet project of mine• Native application based on GStreamer• Ability to add local instruments, captured via Jack and encoded with Opus• Janus as the reference WebRTC server for all the jam session “participants”
• Publishing local instruments, subscribing to remote ones
![Page 35: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/35.jpg)
What about a native approach?
• A few, non-WebRTC, native solutions exist already• e.g., Jamulus and NINJAM (both open source)
• It might be interesting to experiment with WebRTC as well• e.g., Native client that uses Jack for audio input/output• WebRTC exchange of live streams (P2P or via a server)
Idea for a personal fun/pet project of mine• Native application based on GStreamer• Ability to add local instruments, captured via Jack and encoded with Opus• Janus as the reference WebRTC server for all the jam session “participants”
• Publishing local instruments, subscribing to remote ones
![Page 36: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/36.jpg)
The ugliest diagram you’ll see today
![Page 37: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/37.jpg)
A distributed/remote concert
• Whatever the approach, it might be useful to stream this remote session
• A truly distributed concert!
• If the session can be captured, it can be broadcast
• If an SFU is used, streams can be relayed as-is (they’re already there)
• Basically a few-to-many conferencing session
• Audio can also be mixed, though, either on the server or client side
• Many already use OBS for that, so OBS-WebRTC (WHIP) may be a simple option
• Server-side mixing may be more “balanced” in terms of delay, though?
• In general, same considerations made before on broadcasting apply here too
![Page 38: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/38.jpg)
A distributed/remote concert
• Whatever the approach, it might be useful to stream this remote session
• A truly distributed concert!
• If the session can be captured, it can be broadcast
• If an SFU is used, streams can be relayed as-is (they’re already there)
• Basically a few-to-many conferencing session
• Audio can also be mixed, though, either on the server or client side
• Many already use OBS for that, so OBS-WebRTC (WHIP) may be a simple option
• Server-side mixing may be more “balanced” in terms of delay, though?
• In general, same considerations made before on broadcasting apply here too
![Page 39: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/39.jpg)
A distributed/remote concert
• Whatever the approach, it might be useful to stream this remote session
• A truly distributed concert!
• If the session can be captured, it can be broadcast
• If an SFU is used, streams can be relayed as-is (they’re already there)
• Basically a few-to-many conferencing session
• Audio can also be mixed, though, either on the server or client side
• Many already use OBS for that, so OBS-WebRTC (WHIP) may be a simple option
• Server-side mixing may be more “balanced” in terms of delay, though?
• In general, same considerations made before on broadcasting apply here too
![Page 40: Can WebRTC help musicians? - FOSDEM](https://reader030.fdocuments.us/reader030/viewer/2022012701/61a3ad7fcc562821375da92c/html5/thumbnails/40.jpg)
Thanks! Questions? Comments?
Get in touch!• https://twitter.com/elminiero• https://twitter.com/meetecho• https://www.meetecho.com