Troubleshooting and monitoring Janus:a HEPIC journey!

Lorenzo Miniero@elminiero

OpenSIPS Summit2nd May 2018,

“Which Lorenzo are you, exactly?”

You may remember me from “Revenge ofthe Mutant Janus Instances” (not really...)

No, seriously, remember Janus?

• General purpose, open source WebRTC server• https://github.com/meetecho/janus-gateway• Demos and documentation: https://janus.conf.meetecho.com• Community: https://groups.google.com/forum/#!forum/meetecho-janus

Past OpenSIPS presentations: on Janus, SIP, and various salads• https://www.youtube.com/watch?v=SFeWYewoL7Q• https://www.youtube.com/watch?v=anmyMC6Ovl8&t=25112s

No, seriously, remember Janus?

• General purpose, open source WebRTC server• https://github.com/meetecho/janus-gateway• Demos and documentation: https://janus.conf.meetecho.com• Community: https://groups.google.com/forum/#!forum/meetecho-janus

Past OpenSIPS presentations: on Janus, SIP, and various salads• https://www.youtube.com/watch?v=SFeWYewoL7Q• https://www.youtube.com/watch?v=anmyMC6Ovl8&t=25112s

Troubleshooting Janus: a bit of history

• First approach (still widely used) was the Admin API• Request/response protocol available on different transports• Allows to inspect handles and WebRTC “internals” from the Janus perspective• Can tweak some settings too (e.g., enable/disable debugging)

• A different, asynchronous mechanism then followed: Event Handlers• Core and plugins generate events of different types• Custom modules can subscribe to and handle them

• e.g., save to DB, send to external service, CDR, etc.

Fun fact: Event Handlers were conceived during OpenSIPS 2016!• You can thank Lorenzo, Celeste, Alexandr (and a big burger!) for that ,

Troubleshooting Janus: a bit of history

• First approach (still widely used) was the Admin API• Request/response protocol available on different transports• Allows to inspect handles and WebRTC “internals” from the Janus perspective• Can tweak some settings too (e.g., enable/disable debugging)

• A different, asynchronous mechanism then followed: Event Handlers• Core and plugins generate events of different types• Custom modules can subscribe to and handle them

• e.g., save to DB, send to external service, CDR, etc.

Fun fact: Event Handlers were conceived during OpenSIPS 2016!• You can thank Lorenzo, Celeste, Alexandr (and a big burger!) for that ,

Troubleshooting Janus: a bit of history

• First approach (still widely used) was the Admin API• Request/response protocol available on different transports• Allows to inspect handles and WebRTC “internals” from the Janus perspective• Can tweak some settings too (e.g., enable/disable debugging)

• A different, asynchronous mechanism then followed: Event Handlers• Core and plugins generate events of different types• Custom modules can subscribe to and handle them

• e.g., save to DB, send to external service, CDR, etc.

Fun fact: Event Handlers were conceived during OpenSIPS 2016!• You can thank Lorenzo, Celeste, Alexandr (and a big burger!) for that ,

Routing and managing asynchronous events

“Sample Event Handler”: HTTP as a notifier

• Simply forwards all events as JSON to an HTTP backend• Supports basic authentication• Can group events (i.e., JSON array vs. multiple JSON objects)• Implements basic retransmissions (exponential back-off)

• Does nothing more than that: logic needs to be elsewhere• HTTP backend decides what to do with events, if anything• Behaviour can be tweaked via Admin API calls

Need something else? Check the alternatives or write your own!

A few other event handlers also available• RabbitMQ (in repo), MQTT (PR #1185), SQLite (by Mozilla)

“Sample Event Handler”: HTTP as a notifier

• Simply forwards all events as JSON to an HTTP backend• Supports basic authentication• Can group events (i.e., JSON array vs. multiple JSON objects)• Implements basic retransmissions (exponential back-off)

• Does nothing more than that: logic needs to be elsewhere• HTTP backend decides what to do with events, if anything• Behaviour can be tweaked via Admin API calls

Need something else? Check the alternatives or write your own!

A few other event handlers also available• RabbitMQ (in repo), MQTT (PR #1185), SQLite (by Mozilla)

“Sample Event Handler”: HTTP as a notifier

• Simply forwards all events as JSON to an HTTP backend• Supports basic authentication• Can group events (i.e., JSON array vs. multiple JSON objects)• Implements basic retransmissions (exponential back-off)

• Does nothing more than that: logic needs to be elsewhere• HTTP backend decides what to do with events, if anything• Behaviour can be tweaked via Admin API calls

Need something else? Check the alternatives or write your own!

A few other event handlers also available• RabbitMQ (in repo), MQTT (PR #1185), SQLite (by Mozilla)

How do we handle events, now?

• Generating events is easy, evaluating them is another matter...• Event Handlers typically just relay events, and don’t do processing themselves• Analyzing and correlating tons of events is complicated

• A few different approaches, from easiest to trickiest1 https://github.com/stirlab/janus-event-server (dumping events to file)2 https://github.com/mozilla/janus-eventhandler-sqlite (dumping events to SQLite)3 Write your own backend, e.g.,

http://www.meetecho.com/blog/event-handlers-a-practical-example/http://www.meetecho.com/blog/correlating-janus-event-handlers/

• A much better approach: trust the smart guys!• http://www.sipcapture.org/• http://hepic.tel/

How do we handle events, now?

• Generating events is easy, evaluating them is another matter...• Event Handlers typically just relay events, and don’t do processing themselves• Analyzing and correlating tons of events is complicated

• A few different approaches, from easiest to trickiest1 https://github.com/stirlab/janus-event-server (dumping events to file)2 https://github.com/mozilla/janus-eventhandler-sqlite (dumping events to SQLite)3 Write your own backend, e.g.,

http://www.meetecho.com/blog/event-handlers-a-practical-example/http://www.meetecho.com/blog/correlating-janus-event-handlers/

• A much better approach: trust the smart guys!• http://www.sipcapture.org/• http://hepic.tel/

How do we handle events, now?

• Generating events is easy, evaluating them is another matter...• Event Handlers typically just relay events, and don’t do processing themselves• Analyzing and correlating tons of events is complicated

• A few different approaches, from easiest to trickiest1 https://github.com/stirlab/janus-event-server (dumping events to file)2 https://github.com/mozilla/janus-eventhandler-sqlite (dumping events to SQLite)3 Write your own backend, e.g.,

http://www.meetecho.com/blog/event-handlers-a-practical-example/http://www.meetecho.com/blog/correlating-janus-event-handlers/

• A much better approach: trust the smart guys!• http://www.sipcapture.org/• http://hepic.tel/

Event Handlers in HOMER since “day one”(*)

(*A trip down Memory Lane: FOSDEM 2017!)

https://archive.fosdem.org/2017/schedule/event/janus/https://archive.fosdem.org/2017/schedule/event/homer/

From SIP to WebRTC

• As anticipated, Event Handlers were conceived in a chat with QXIP• “How can I dump SIP messages with Sofia SIP?”• “Which other events would be useful to have?”• “What about a new modular approach, where HEP could be a plugin?”

• The only “caveat”: it was SIP only /• Only events coming from the SIP plugin were tracked, to follow the signalling• Other events related to SIP handles only saved as “related info”• Still very used and widespread, but limited to WebRTC/SIP scenarios

• Lately, big step to start tracking generic WebRTC applications instead• How to handle users with more than one PeerConnection?• How to relate a PeerConnection to another, for issues and more?• How to reconstruct a session topology from the media perspective?

From SIP to WebRTC

• As anticipated, Event Handlers were conceived in a chat with QXIP• “How can I dump SIP messages with Sofia SIP?”• “Which other events would be useful to have?”• “What about a new modular approach, where HEP could be a plugin?”

• The only “caveat”: it was SIP only /• Only events coming from the SIP plugin were tracked, to follow the signalling• Other events related to SIP handles only saved as “related info”• Still very used and widespread, but limited to WebRTC/SIP scenarios

• Lately, big step to start tracking generic WebRTC applications instead• How to handle users with more than one PeerConnection?• How to relate a PeerConnection to another, for issues and more?• How to reconstruct a session topology from the media perspective?

From SIP to WebRTC

• As anticipated, Event Handlers were conceived in a chat with QXIP• “How can I dump SIP messages with Sofia SIP?”• “Which other events would be useful to have?”• “What about a new modular approach, where HEP could be a plugin?”

• The only “caveat”: it was SIP only /• Only events coming from the SIP plugin were tracked, to follow the signalling• Other events related to SIP handles only saved as “related info”• Still very used and widespread, but limited to WebRTC/SIP scenarios

• Lately, big step to start tracking generic WebRTC applications instead• How to handle users with more than one PeerConnection?• How to relate a PeerConnection to another, for issues and more?• How to reconstruct a session topology from the media perspective?

Challenge #1: getting events in HOMER/HEPIC

• Basically, this was there already in HOMER 5.x• HTTP Event Handler + hepipe.js = Magic!• hepipe.js as an Events server, statically injecting events in the platform

• Anyway, considerable refactoring done for HOMER 7.x• Modular protocols and backends (e.g., Prometheus vs. InfluxDB)

• The solution: HEPop as a way to index/tag incoming heterogeneous data• Support for several different data sources (e.g., Janus and mediasoup)• For Janus, it acts as an events server exactly as hepipe.js did

A potential Janus-specific improvement?As anticipated, a MQTT event handler is on the way (thanks Olle!)• Might be a much better alternative to HTTP as a way to channel events

Challenge #1: getting events in HOMER/HEPIC

• Basically, this was there already in HOMER 5.x• HTTP Event Handler + hepipe.js = Magic!• hepipe.js as an Events server, statically injecting events in the platform

• Anyway, considerable refactoring done for HOMER 7.x• Modular protocols and backends (e.g., Prometheus vs. InfluxDB)

• The solution: HEPop as a way to index/tag incoming heterogeneous data• Support for several different data sources (e.g., Janus and mediasoup)• For Janus, it acts as an events server exactly as hepipe.js did

A potential Janus-specific improvement?As anticipated, a MQTT event handler is on the way (thanks Olle!)• Might be a much better alternative to HTTP as a way to channel events

Challenge #1: getting events in HOMER/HEPIC

• Basically, this was there already in HOMER 5.x• HTTP Event Handler + hepipe.js = Magic!• hepipe.js as an Events server, statically injecting events in the platform

• Anyway, considerable refactoring done for HOMER 7.x• Modular protocols and backends (e.g., Prometheus vs. InfluxDB)

• The solution: HEPop as a way to index/tag incoming heterogeneous data• Support for several different data sources (e.g., Janus and mediasoup)• For Janus, it acts as an events server exactly as hepipe.js did

A potential Janus-specific improvement?As anticipated, a MQTT event handler is on the way (thanks Olle!)• Might be a much better alternative to HTTP as a way to channel events

Challenge #1: getting events in HOMER/HEPIC

• Basically, this was there already in HOMER 5.x• HTTP Event Handler + hepipe.js = Magic!• hepipe.js as an Events server, statically injecting events in the platform

• Anyway, considerable refactoring done for HOMER 7.x• Modular protocols and backends (e.g., Prometheus vs. InfluxDB)

• The solution: HEPop as a way to index/tag incoming heterogeneous data• Support for several different data sources (e.g., Janus and mediasoup)• For Janus, it acts as an events server exactly as hepipe.js did

A potential Janus-specific improvement?As anticipated, a MQTT event handler is on the way (thanks Olle!)• Might be a much better alternative to HTTP as a way to channel events

Challenge #2: storing and processing events

• Ex-post evaluations vs. live monitoring• Analyzing a dump after the session ended is different from live monitoring

• Postgres, MySQL, InfluxDB, Prometheus, etc.• Several options in the new HOMER/HEPIC, which is exciting!

• JSON databases and queries• Janus events are in JSON format, so why not use that effectively?• HOMER/HEPIC backends support JSON as a native format

Storing and tagging events

Smart choice is to store events AND tag them as they flow• Good way to create quick correlations and recap what happened• Media stats can then be turned in Time Series (monitoring/alerting)

Challenge #2: storing and processing events

• Ex-post evaluations vs. live monitoring• Analyzing a dump after the session ended is different from live monitoring

• Postgres, MySQL, InfluxDB, Prometheus, etc.• Several options in the new HOMER/HEPIC, which is exciting!

• JSON databases and queries• Janus events are in JSON format, so why not use that effectively?• HOMER/HEPIC backends support JSON as a native format

Storing and tagging events

Smart choice is to store events AND tag them as they flow• Good way to create quick correlations and recap what happened• Media stats can then be turned in Time Series (monitoring/alerting)

Challenge #2: storing and processing events

• Ex-post evaluations vs. live monitoring• Analyzing a dump after the session ended is different from live monitoring

• Postgres, MySQL, InfluxDB, Prometheus, etc.• Several options in the new HOMER/HEPIC, which is exciting!

• JSON databases and queries• Janus events are in JSON format, so why not use that effectively?• HOMER/HEPIC backends support JSON as a native format

Storing and tagging events

Smart choice is to store events AND tag them as they flow• Good way to create quick correlations and recap what happened• Media stats can then be turned in Time Series (monitoring/alerting)

Challenge #2: storing and processing events

• Ex-post evaluations vs. live monitoring• Analyzing a dump after the session ended is different from live monitoring

• Postgres, MySQL, InfluxDB, Prometheus, etc.• Several options in the new HOMER/HEPIC, which is exciting!

• JSON databases and queries• Janus events are in JSON format, so why not use that effectively?• HOMER/HEPIC backends support JSON as a native format

Storing and tagging events

Smart choice is to store events AND tag them as they flow• Good way to create quick correlations and recap what happened• Media stats can then be turned in Time Series (monitoring/alerting)

Challenge #3: correlating events

• Sessions might be used to identify users, but can’t always be “trusted”• Works fine when users talk to Janus directly (1↔1 relationship)• There are server-side applications with a single session for all users, though

• Enter opaque_id: a client-provided opaque identifier• Just set the same opaque ID for all handles belonging to the same user• The opaque ID can be whatever you want, Janus doesn’t care

• Useful for intra-plugin, but also inter-plugin, correlations• e.g., to know a user is publishing and subscribing in a VideoRoom• ... but also, for instance, in a SIP call in the same application

Want to learn more?http://www.meetecho.com/blog/correlating-janus-event-handlers/

Challenge #3: correlating events

• Sessions might be used to identify users, but can’t always be “trusted”• Works fine when users talk to Janus directly (1↔1 relationship)• There are server-side applications with a single session for all users, though

• Enter opaque_id: a client-provided opaque identifier• Just set the same opaque ID for all handles belonging to the same user• The opaque ID can be whatever you want, Janus doesn’t care

• Useful for intra-plugin, but also inter-plugin, correlations• e.g., to know a user is publishing and subscribing in a VideoRoom• ... but also, for instance, in a SIP call in the same application

Want to learn more?http://www.meetecho.com/blog/correlating-janus-event-handlers/

Challenge #3: correlating events

• Sessions might be used to identify users, but can’t always be “trusted”• Works fine when users talk to Janus directly (1↔1 relationship)• There are server-side applications with a single session for all users, though

• Enter opaque_id: a client-provided opaque identifier• Just set the same opaque ID for all handles belonging to the same user• The opaque ID can be whatever you want, Janus doesn’t care

• Useful for intra-plugin, but also inter-plugin, correlations• e.g., to know a user is publishing and subscribing in a VideoRoom• ... but also, for instance, in a SIP call in the same application

Want to learn more?http://www.meetecho.com/blog/correlating-janus-event-handlers/

Challenge #3: correlating events

• Sessions might be used to identify users, but can’t always be “trusted”• Works fine when users talk to Janus directly (1↔1 relationship)• There are server-side applications with a single session for all users, though

• Enter opaque_id: a client-provided opaque identifier• Just set the same opaque ID for all handles belonging to the same user• The opaque ID can be whatever you want, Janus doesn’t care

• Useful for intra-plugin, but also inter-plugin, correlations• e.g., to know a user is publishing and subscribing in a VideoRoom• ... but also, for instance, in a SIP call in the same application

Want to learn more?http://www.meetecho.com/blog/correlating-janus-event-handlers/

Challenge #3: storing and processing events

Challenge #3: storing and processing events

Challenge #3: storing and processing events

Challenge #4: client-side events

• All the discussion so far has been for server-side events• Janus originates events for what happens there• What about the client side perspective?

• In WebRTC, standard mechanism for getting statistics on a PeerConnection• https://www.w3.org/TR/webrtc-stats/• Calls to getStats() return useful info, but in JavaScript

• How to get it to HOMER/HEPIC, possibly via Janus/Event Handlers as well?

Solution: a new Janus plugin!• Client attaches and sends stats on a regular basis to the plugin via Janus API• Plugin relays stats via Event Handlers (same opaque ID as media channels)• Note: still WIP (working demo using rtcstats), but not open source yet

Challenge #4: client-side events

• All the discussion so far has been for server-side events• Janus originates events for what happens there• What about the client side perspective?

• In WebRTC, standard mechanism for getting statistics on a PeerConnection• https://www.w3.org/TR/webrtc-stats/• Calls to getStats() return useful info, but in JavaScript

• How to get it to HOMER/HEPIC, possibly via Janus/Event Handlers as well?

Solution: a new Janus plugin!• Client attaches and sends stats on a regular basis to the plugin via Janus API• Plugin relays stats via Event Handlers (same opaque ID as media channels)• Note: still WIP (working demo using rtcstats), but not open source yet

Challenge #4: client-side events

• All the discussion so far has been for server-side events• Janus originates events for what happens there• What about the client side perspective?

• In WebRTC, standard mechanism for getting statistics on a PeerConnection• https://www.w3.org/TR/webrtc-stats/• Calls to getStats() return useful info, but in JavaScript

• How to get it to HOMER/HEPIC, possibly via Janus/Event Handlers as well?

Solution: a new Janus plugin!• Client attaches and sends stats on a regular basis to the plugin via Janus API• Plugin relays stats via Event Handlers (same opaque ID as media channels)• Note: still WIP (working demo using rtcstats), but not open source yet

Challenge #4: client-side events

• All the discussion so far has been for server-side events• Janus originates events for what happens there• What about the client side perspective?

• In WebRTC, standard mechanism for getting statistics on a PeerConnection• https://www.w3.org/TR/webrtc-stats/• Calls to getStats() return useful info, but in JavaScript

• How to get it to HOMER/HEPIC, possibly via Janus/Event Handlers as well?

Solution: a new Janus plugin!• Client attaches and sends stats on a regular basis to the plugin via Janus API• Plugin relays stats via Event Handlers (same opaque ID as media channels)• Note: still WIP (working demo using rtcstats), but not open source yet

A real use case: IETF meetings!

Remote participation at the IETF

https://www.slideshare.net/LorenzoMiniero/ietf-remote-participation-via-meetecho-webrtc-meetup-stockholmhttps://www.vuc.me/2017/vuc640-ietf-remote-participation-with-lorenzo-miniero/

Studying data from IETF 101 in London

• IETF remote participation based on multiple Janus plugins• SIP plugin for mixed audio• Streaming plugin for static sources• VideoRoom plugin for remote speakers

• Collected events related to 10 different Working Group sessions• https://datatracker.ietf.org/meeting/agenda.html#2018-03-23-083000• MMUSIC, HOMENET, ICCRG, DETNET, ROLL, SECEVENT, DTN, SPRING,

IPSECME, LAMPS

• Good mix of heterogeneous sessions• Some shorter, some longer (DETNET lasted 5 hours!)• Some had many active remote speakers• All had attendees from all over the world (different network conditions)

Studying data from IETF 101 in London

• IETF remote participation based on multiple Janus plugins• SIP plugin for mixed audio• Streaming plugin for static sources• VideoRoom plugin for remote speakers

• Collected events related to 10 different Working Group sessions• https://datatracker.ietf.org/meeting/agenda.html#2018-03-23-083000• MMUSIC, HOMENET, ICCRG, DETNET, ROLL, SECEVENT, DTN, SPRING,

IPSECME, LAMPS

• Good mix of heterogeneous sessions• Some shorter, some longer (DETNET lasted 5 hours!)• Some had many active remote speakers• All had attendees from all over the world (different network conditions)

Studying data from IETF 101 in London

• IETF remote participation based on multiple Janus plugins• SIP plugin for mixed audio• Streaming plugin for static sources• VideoRoom plugin for remote speakers

• Collected events related to 10 different Working Group sessions• https://datatracker.ietf.org/meeting/agenda.html#2018-03-23-083000• MMUSIC, HOMENET, ICCRG, DETNET, ROLL, SECEVENT, DTN, SPRING,

IPSECME, LAMPS

• Good mix of heterogeneous sessions• Some shorter, some longer (DETNET lasted 5 hours!)• Some had many active remote speakers• All had attendees from all over the world (different network conditions)

First experiments: dumping events to Postgres

• Postgres supports JSON natively, and is used by HOMER/HEPIC too• Why not play with this locally, first?

• Created a single database, and a different table per each WG• Each table just has a unique index and the JSON value• All queries can be done on JSON fields

• Played with different queries to identify different events• How many concurrent participants? How many were active?• Who subscribed to what? (SIP, slides, camera, remotees)• Any network errors? If so, caused by what?• What can we extract from the statistics?

First experiments: dumping events to Postgres

• Postgres supports JSON natively, and is used by HOMER/HEPIC too• Why not play with this locally, first?

• Created a single database, and a different table per each WG• Each table just has a unique index and the JSON value• All queries can be done on JSON fields

• Played with different queries to identify different events• How many concurrent participants? How many were active?• Who subscribed to what? (SIP, slides, camera, remotees)• Any network errors? If so, caused by what?• What can we extract from the statistics?

First experiments: dumping events to Postgres

• Postgres supports JSON natively, and is used by HOMER/HEPIC too• Why not play with this locally, first?

• Created a single database, and a different table per each WG• Each table just has a unique index and the JSON value• All queries can be done on JSON fields

• Played with different queries to identify different events• How many concurrent participants? How many were active?• Who subscribed to what? (SIP, slides, camera, remotees)• Any network errors? If so, caused by what?• What can we extract from the statistics?

Summarizing relevant user/handle events

Summarizing relevant user/handle events

Summarizing relevant user/handle events

Summarizing relevant user/handle events

The ugliest charts you’ll ever see!

Some bugs to be fixed in Janus too, I guess...

• Analysing dumps, one thing popped to the eye: RTCP RTT > 4M?!

• RTT was either quite low, or super-high, never in the middle

• Turned out to be a bug in the RTCP parsing code, though... (that we fixed, now!)

How does it look like in HOMER/HEPIC?

How does it look like in HOMER/HEPIC?

Cool demo with Lorenzo tomorrow!

Next steps

• Collecting real data and analysing it was very useful• Helped fix a couple of bugs (I’m looking at you, RTT!)• Showed we needed better info in some events (e.g., in selected-pair)

• Very helpful in terms of HOMER/HEPIC integration as well• What were we looking for?• Was the available information enough?• Did we have all the instruments to monitor/navigate/search?

What’s next?Not the end of the journey, just the beginning!• Need to generalize IETF meeting considerations to other use cases• Live monitoring and troubleshooting is definitely of interest

• Most of the analysis was done on ex-post processing• Alerting is ready, though, so looking forward to that!

Next steps

• Collecting real data and analysing it was very useful• Helped fix a couple of bugs (I’m looking at you, RTT!)• Showed we needed better info in some events (e.g., in selected-pair)

• Very helpful in terms of HOMER/HEPIC integration as well• What were we looking for?• Was the available information enough?• Did we have all the instruments to monitor/navigate/search?

What’s next?Not the end of the journey, just the beginning!• Need to generalize IETF meeting considerations to other use cases• Live monitoring and troubleshooting is definitely of interest

• Most of the analysis was done on ex-post processing• Alerting is ready, though, so looking forward to that!

Next steps

• Collecting real data and analysing it was very useful• Helped fix a couple of bugs (I’m looking at you, RTT!)• Showed we needed better info in some events (e.g., in selected-pair)

• Very helpful in terms of HOMER/HEPIC integration as well• What were we looking for?• Was the available information enough?• Did we have all the instruments to monitor/navigate/search?

What’s next?Not the end of the journey, just the beginning!• Need to generalize IETF meeting considerations to other use cases• Live monitoring and troubleshooting is definitely of interest

• Most of the analysis was done on ex-post processing• Alerting is ready, though, so looking forward to that!

Next steps

• Collecting real data and analysing it was very useful• Helped fix a couple of bugs (I’m looking at you, RTT!)• Showed we needed better info in some events (e.g., in selected-pair)

• Very helpful in terms of HOMER/HEPIC integration as well• What were we looking for?• Was the available information enough?• Did we have all the instruments to monitor/navigate/search?

What’s next?Not the end of the journey, just the beginning!• Need to generalize IETF meeting considerations to other use cases• Live monitoring and troubleshooting is definitely of interest

• Most of the analysis was done on ex-post processing• Alerting is ready, though, so looking forward to that!

Next steps

• Collecting real data and analysing it was very useful• Helped fix a couple of bugs (I’m looking at you, RTT!)• Showed we needed better info in some events (e.g., in selected-pair)

• Very helpful in terms of HOMER/HEPIC integration as well• What were we looking for?• Was the available information enough?• Did we have all the instruments to monitor/navigate/search?

What’s next?Not the end of the journey, just the beginning!• Need to generalize IETF meeting considerations to other use cases• Live monitoring and troubleshooting is definitely of interest

• Most of the analysis was done on ex-post processing• Alerting is ready, though, so looking forward to that!

Thanks! Questions? Comments?

Get in touch!• https://twitter.com/elminiero• https://twitter.com/meetecho• http://www.meetecho.com