FOSDEM 2016: The State of XMPP and Instant Messaging, The Awakening
Real life XMPP Instant Messaging
-
Upload
mickael-remond -
Category
Technology
-
view
5.281 -
download
2
description
Transcript of Real life XMPP Instant Messaging
![Page 1: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/1.jpg)
XMPP in real life: attacks, bad behaviour and how to cope with them2009, 7th february - FOSDEM 2009Mickaël Rémond <[email protected]>
![Page 2: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/2.jpg)
Background: What we do
Company created in 1999
![Page 3: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/3.jpg)
Background: What we do
Company created in 199920 employees
![Page 4: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/4.jpg)
Background: What we do
Company created in 199920 employeesSpecialized in Instant Messaging since 2002
![Page 5: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/5.jpg)
Background: What we do
Company created in 199920 employeesSpecialized in Instant Messaging since 2002Involved in ejabberd since 2002. ProcessOne produced 98% of the code.
![Page 6: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/6.jpg)
Background: What we do
Company created in 199920 employeesSpecialized in Instant Messaging since 2002Involved in ejabberd since 2002. ProcessOne produced 98% of the code.Complete stack of Instant Messaging software
![Page 7: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/7.jpg)
Background: What we do
Company created in 199920 employeesSpecialized in Instant Messaging since 2002Involved in ejabberd since 2002. ProcessOne produced 98% of the code.Complete stack of Instant Messaging softwareTwo main activities
software: complete software solution for IMexpertise: renowed company for high-availability, scalability and custom solutions (consulting, development and hosting)
![Page 8: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/8.jpg)
Background: What we do
Company created in 199920 employeesSpecialized in Instant Messaging since 2002Involved in ejabberd since 2002. ProcessOne produced 98% of the code.Complete stack of Instant Messaging softwareTwo main activities
software: complete software solution for IMexpertise: renowed company for high-availability, scalability and custom solutions (consulting, development and hosting)
Several tens of large customers, spread across the worldLarge scale worldwide leaderSpecific needs renowed expertise
![Page 9: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/9.jpg)
XMPP deployments types
Visible public serversThe «Sandbox»Not necessarily large scale but very unusual behaviours, clients, usage pattern
![Page 10: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/10.jpg)
XMPP deployments types
Visible public serversThe «Sandbox»Not necessarily large scale but very unusual behaviours, clients, usage pattern
Large scale serversLarge scale in term of registered or simultaneous users
Large scale starts after a million of registered users and / or hundred of thousands simultaneous connections
![Page 11: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/11.jpg)
XMPP deployments types
Visible public serversThe «Sandbox»Not necessarily large scale but very unusual behaviours, clients, usage pattern
Large scale serversLarge scale in term of registered or simultaneous users
Large scale starts after a million of registered users and / or hundred of thousands simultaneous connections
Large scale in term of thoughputAt least tens of thousands of packets per seconds, millions of users of MUC / Pubsub, millions of nodes.
![Page 12: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/12.jpg)
XMPP deployments types
Visible public serversThe «Sandbox»Not necessarily large scale but very unusual behaviours, clients, usage pattern
Large scale serversLarge scale in term of registered or simultaneous users
Large scale starts after a million of registered users and / or hundred of thousands simultaneous connections
Large scale in term of thoughputAt least tens of thousands of packets per seconds or tens of thousands users in MUC room or subscribed to pubsub node, etc.
Experience of large clusters with several tens of millions registered users and more than 500 000 simultaneous users.
![Page 13: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/13.jpg)
Challenges of real life XMPP
Uptime
![Page 14: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/14.jpg)
Challenges of real life XMPP
UptimeUptime !
![Page 15: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/15.jpg)
Challenges of real life XMPP
UptimeUptime !Uptime !!
![Page 16: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/16.jpg)
Challenges of real life XMPP
UptimeUptime !Uptime !!
Everything else derives from thisChallenge (performance, scalability)
![Page 17: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/17.jpg)
Challenges of real life XMPP
UptimeUptime !Uptime !!When a server is restarted:
it faces a reconnect storm from client that login againit needs to resync the complete presence states with most of its known s2s serversit reconnects the users accounts on gateways ...
Everything else derives from thisChallenge (performance, scalability)
![Page 18: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/18.jpg)
Challenges of real life XMPP
UptimeUptime !Uptime !!When a server is restarted:
it faces a reconnect storm from client that login againit needs to resync the complete presence states with most of its known s2s serversit reconnects the users accounts on gateways ...
You need to:Be able to monitor lots of values to detect troubles and have tools to keep the server online during trouble phase (otherwise it crash:get worse)Be able to perform maintenance task and upgrade code live
Everything else derives from thisChallenge (performance, scalability)
![Page 19: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/19.jpg)
Case 1: XMPP as a proxy
Symptom: A «sandbox» XMPP server crashes regularly
![Page 20: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/20.jpg)
Case 1: XMPP as a proxy
Symptom: A «sandbox» XMPP server crashes regularlyFirst challenge: Detect possible abuser
![Page 21: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/21.jpg)
Case 1: XMPP as a proxy
Symptom: A «sandbox» XMPP server crashes regularlyFirst challenge: Detect possible abuserUse of our toolkit (TeamLeader console) to analyse traffic patterns.
![Page 22: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/22.jpg)
Case 1: XMPP as a proxy
Symptom: A «sandbox» XMPP server crashes regularlyFirst challenge: Detect possible abuserUse of our toolkit (TeamLeader console) to analyse traffic patterns.Correlate the crash to a given user
Large number of packets send when onlineLarge bandwidth consumption
![Page 23: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/23.jpg)
Case 1: XMPP as a proxy
Symptom: A «sandbox» XMPP server crashes regularlyFirst challenge: Detect possible abuserUse of our toolkit (TeamLeader console) to analyse traffic patterns.Correlate the crash to a given user
Large number of packets send when onlineLarge bandwidth consumption
Dump traffic of this user for analysis
![Page 24: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/24.jpg)
Case 1: XMPP as a proxy
Symptom: A «sandbox» XMPP server crashes regularlyFirst challenge: Detect possible abuserUse of our toolkit (TeamLeader console) to analyse traffic patterns.Correlate the crash to a given user
Large number of packets send when onlineLarge bandwidth consumption
Dump traffic of this user for analysisTraffic reveals that user:
has deployed XMPP bot at work on his serversis using the public server to get control of his serverbasically «Shell over XMPP»
![Page 25: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/25.jpg)
Case 1: XMPP as a proxy
Symptom: A «sandbox» XMPP server crashes regularlyFirst challenge: Detect possible abuserUse of our toolkit (TeamLeader console) to analyse traffic patterns.Correlate the crash to a given user
Large number of packets send when onlineLarge bandwidth consumption
Dump traffic of this user for analysisTraffic reveals that user:
has deployed XMPP bot at work on his serversis using the public server to get control of his serverbasically «Shell over XMPP»
Response: Need to detect abnormal usage pattern and trigger alerts
![Page 26: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/26.jpg)
Case 2: Client bad behaviours
Symptom: Abnormal memory consumption / sometime leading to crash
![Page 27: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/27.jpg)
Case 2: Client bad behaviours
Symptom: Abnormal memory consumption / sometime leading to crashSource problem had been client behaviour
![Page 28: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/28.jpg)
Case 2: Client bad behaviours
Symptom: Abnormal memory consumption / sometime leading to crashSource problem had been client behaviourGenerates an undue load on the server
Example: Client does not reply to some IQ stanzas (PEPS / CAPS)Server waits for reply until timeoutDepending on the type of processing it can be blocking
Example: Client that send too many presencesLarge presence broadcast, especially in MUC rooms
![Page 29: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/29.jpg)
Case 2: Client bad behaviours
Symptom: Abnormal memory consumption / sometime leading to crashSource problem had been client behaviourGenerates an undue load on the server
Example: Client does not reply to some IQ stanzas (PEPS / CAPS)Server waits for reply until timeoutDepending on the type of processing it can be blocking
Example: Client that send too many presencesLarge presence broadcast, especially in MUC rooms
Need to restrict the ability to perform those patterns:Limit the interval for sending presences in chat roomsLimit resourc consumption in general
![Page 30: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/30.jpg)
Case 2: Client bad behaviours
Symptom: Abnormal memory consumption / sometime leading to crashSource problem had been client behaviourGenerates an undue load on the server
Example: Client does not reply to some IQ stanzas (PEPS / CAPS)Server waits for reply until timeoutDepending on the type of processing it can be blocking
Example: Client that send too many presencesLarge presence broadcast, especially in MUC rooms
Need to restrict the ability to perform those patterns:Limit the interval for sending presences in chat roomsLimit resourc consumption in general
![Page 31: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/31.jpg)
Case 3: Multi User chat
MUC rooms attacksMost common case of abuse
![Page 32: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/32.jpg)
Case 3: Multi User chat
MUC rooms attacksMost common case of abuseCreate a lot of MUC persistant MUC rooms
![Page 33: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/33.jpg)
Case 3: Multi User chat
MUC rooms attacksMost common case of abuseCreate a lot of MUC persistant MUC roomsJoin a lot of MUC rooms
![Page 34: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/34.jpg)
Case 3: Multi User chat
MUC rooms attacksMost common case of abuseCreate a lot of MUC persistant MUC roomsJoin a lot of MUC roomsJoin / leave a MUC room fastly
![Page 35: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/35.jpg)
Case 3: Multi User chat
MUC rooms attacksMost common case of abuseCreate a lot of MUC persistant MUC roomsJoin a lot of MUC roomsJoin / leave a MUC room fastlyJoin lots of users in a single room
![Page 36: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/36.jpg)
Case 3: Multi User chat
MUC rooms attacksMost common case of abuseCreate a lot of MUC persistant MUC roomsJoin a lot of MUC roomsJoin / leave a MUC room fastlyJoin lots of users in a single roomChange presence to bypass voice
![Page 37: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/37.jpg)
Case 3: Multi User chat
MUC rooms attacksMost common case of abuseCreate a lot of MUC persistant MUC roomsJoin a lot of MUC roomsJoin / leave a MUC room fastlyJoin lots of users in a single roomChange presence to bypass voiceFlood with messages
![Page 38: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/38.jpg)
Case 3: Multi User chat
MUC rooms attacksMost common case of abuseCreate a lot of MUC persistant MUC roomsJoin a lot of MUC roomsJoin / leave a MUC room fastlyJoin lots of users in a single roomChange presence to bypass voiceFlood with messagesUse large values to «attack» the server or the client (large room names, large nick names, etc)
![Page 39: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/39.jpg)
Case 4: Bots
Symptom: Server crash
![Page 40: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/40.jpg)
Case 4: Bots
Symptom: Server crashReduce to a crash when some special user connects (every time)
![Page 41: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/41.jpg)
Case 4: Bots
Symptom: Server crashReduce to a crash when some special user connects (every time)Bots send messages to their users on a public server
![Page 42: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/42.jpg)
Case 4: Bots
Symptom: Server crashReduce to a crash when some special user connects (every time)Bots send messages to their users on a public serverThey do not often use headline message type (which means they are not intended to be stored offline).They sometimes rely on presence, but it can be inaccurate after a force server shutdown.We have seen users of public servers with more than 500 000 messages in the offline store.
![Page 43: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/43.jpg)
Case 4: Bots
Symptom: Server crashReduce to a crash when some special user connects (every time)Bots send messages to their users on a public serverThey do not often use headline message type (which means they are not intended to be stored offline).They sometimes rely on presence, but it can be inaccurate after a force server shutdown.We have seen users of public servers with more than 500 000 messages in the offline store.
Limit the size of the offline storeAbility to detect abusers and limit their ability to send massive amount of messages
![Page 44: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/44.jpg)
Case 5: Large flow / small pipes
An XMPP server is a pipeData flows from on connection to another.
![Page 45: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/45.jpg)
Case 5: Large flow / small pipes
An XMPP server is a pipeData flows from on connection to another.
Problem:What happens if you try to send data faster than the target client can receive (mobile) ?What happens if you try to send data faster than the target server can receive (limited bandwith, Karma limitation) ?
![Page 46: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/46.jpg)
Case 5: Large flow / small pipes
An XMPP server is a pipeData flows from on connection to another.
Problem:What happens if you try to send data faster than the target client can receive (mobile) ?What happens if you try to send data faster than the target server can receive (limited bandwith, Karma limitation) ?
Challenge:Detect congestions and decide what to do when this happensThis has to been done right otherwise the service might seems unreliable
Federation rules / pattern needed ?
![Page 47: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/47.jpg)
Interesting new challenges ahead
Massive numbers of XMPP servers deployedLots of s2s connections to maintain for large serversWill XMPP scale to millions of servers ?
![Page 48: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/48.jpg)
Interesting new challenges ahead
Massive numbers of XMPP servers deployedLots of s2s connections to maintain for large serversWill XMPP scale to millions of servers ?
Large servers connected through s2s:Several large servers need to keep users presence in syncImagine what happen when one of them goes down ...
Yes, massive presence resync is needed
![Page 49: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/49.jpg)
Interesting new challenges ahead
Massive numbers of XMPP servers deployedLots of s2s connections to maintain for large serversWill XMPP scale to millions of servers ?
Large servers connected through s2s:Several large servers need to keep users presence in syncImagine what happen when one of them goes down ...
Yes, massive presence resync is needed
New usage patternsUbiquitous XMPP: A single users can have many connections: Increase in size of XMPP platformsDevices / Machine to Machine communication: increase of volume of messages
![Page 50: Real life XMPP Instant Messaging](https://reader033.fdocuments.us/reader033/viewer/2022042613/54c3aa674a79591d428b45df/html5/thumbnails/50.jpg)
Questions and challenges to share ?