Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja...
Transcript of Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja...
![Page 2: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/2.jpg)
ArchitectureMatchMood architecture
![Page 3: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/3.jpg)
Data Node• X3 data nodes
• Cassandra Database
• Spark slaves
• hdfs YARN data node
• Storing market/twitter/aggregated data
• Spark jobs run close to data
Spark Slave
Cassandra
HDFS Datanode
![Page 4: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/4.jpg)
Name Node
• 1 name node
• Spark Master
• hdfs YARN namenode
• Hosts Matchmood Application
• Hosts Moodbuilder Application
Matchmood Application
Spark Master
HDFS Namenode
![Page 5: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/5.jpg)
Cassandra
• Masterless cluster, Gossip communication
• Good for time series
• Fast Read/Writes
• CAP - Manageable Consistency, Availability, Partition
![Page 6: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/6.jpg)
A
BC
Matchmood hashedKeyKey range
![Page 7: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/7.jpg)
A
BC
Matchmood (Client App) hashedKey
Key range
Replicate
Replicate
![Page 8: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/8.jpg)
Spark
• Batch/Real time large-scale data processing
• MapReduce in memory
• Make use of RDD(Resilient Distributed Datasets)
• Uniform API(s); Spark SQL
![Page 9: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/9.jpg)
A
BC
Spark Master
Spark jobs close to data
Every minute
![Page 10: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/10.jpg)
A
BC
Save to Cassandra
Spark Master
![Page 11: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/11.jpg)
ApplicationMatchMood
![Page 12: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/12.jpg)
Market Data
• Betfair API
• JSON-RPC
• Request every 250 ms for a given market
• Match Odds market; Odds for the selections1X2
![Page 13: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/13.jpg)
Market Data Example
![Page 14: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/14.jpg)
Twitter Data
• Twitter API
• JSON
• Push service through filtered stream
• Listen to keywords, hashtags, handles
![Page 15: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/15.jpg)
Args passed to Filter• ['@Swansea', '@MCFC', '#SWACITY', '#SWAvCITY', '#SWA',
'#swans', '#CITY', '#mcfc', '@WatfordFC', '@SunderlandAFC', '#watfordfcSUN', '#watfordfcvSUN', '#watfordfc', '#SUN', '#SAFC', '@SouthamptonFC', '@CPFC', '#SaintsFCCRY', '#SaintsFCvCRY', '#SaintsFC', '#CRY', '#CPFC', '@stokecity', '@whufc_official', '#STKWHU', '#STKvWHU', '#STK', '#SCFC', '#WHU', '#WestHam', '@ManUtd', '@afcbournemouth', '#MNUBOU', '#MNUvBOU', '#MNU', '#mufc', '#BOU', '#afcb', '@NUFC', '@SpursOfficial', '#NEWTOT', '#NEWvTOT', '#NEW', '#NUFC', '#TOT', '#COYS', '@ChelseaFC', '@LCFC', '#CHELEI', '#CHEvLEI', '#CHE', '#CFC', '#LEI', '#lcfc', '@Everton', '@NorwichCityFC ', '#EFCNOR', '#EFCvNOR', '#EFC', '#COYB', '#NOR', '#OTBC', '#NCFC', '@Arsenal', '@AVFCOfficial', '#ARSAVL', '#ARSvAVL', '#ARS', '#Arsenal', '#AVL', '#avfc', '@WBAFCofficial', '@LFC', '#WBALIV', '#WBAvLIV', '#WBA', '#LIV', '#LFC']
![Page 16: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/16.jpg)
Twitter Data Example
![Page 17: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/17.jpg)
Twitter Data Example (cont)
![Page 18: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/18.jpg)
MoodBuilder• Application written in Scala
• 1 Minute intervals - aggregate data
• 2 main jobs
• Run as near real time or rebuilding jobs(for older data)
![Page 19: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/19.jpg)
Aggregate Twitter Data• Select data from last_minute
from Cassandra twitter_data table
• Map for each tuple
• (tag,entity) -> (fear,positive,negative,….,joy)
• ReduceByKey (tag,entity) -> sum each tuple value ((fear,pos,neg,…,joy))
• Save result to Cassandra table
![Page 20: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/20.jpg)
Aggregate Twitter Data Example
![Page 21: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/21.jpg)
Aggregate Market Data • Select data from last minute
from Cassandra table market_data
• Map tuple for each operation (Home,Away,Draw,Volume)
• ReducebyKey for each tuple (max,min,open,close,average,volume difference)
• Save result to Cassandra table
![Page 22: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/22.jpg)
Aggregate Market Data Example
![Page 23: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/23.jpg)
RESTful API
• GET http://api.angrylogic.com/matches/
• GET http://api.angrylogic.com/matches/<TAG>/market/
• GET http://api.angrylogic.com/matches/<TAG>/entities/
• GET http://api.angrylogic.com/matches/<TAG>/entities/<ENTITY>
• GET http://api.angrylogic.com/matches/<TAG>/entities/<ENTITY>/minutes/
![Page 24: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/24.jpg)
Future Work
• Topic Extraction; Word clouds
• Prediction Dictionary
• Extend application to other Prediction Markets, such as Political campaigns, Eurovision
• Twitter + Financial Markets (Stock Exchange)
![Page 25: Matchmood - Matthew Zammitzammitmatthew.com/resources/matchmood-keynote.pdfMatchmood Sean Bugeja sean.bugeja.12@um.edu.mt Matthew Zammit matthew.zammit.09@um.edu.mt](https://reader034.fdocuments.us/reader034/viewer/2022042811/5fa2d9b187e13d672d66e7d5/html5/thumbnails/25.jpg)
Demonstration
• RESTful API
• https://youtu.be/AGyrniRent8
• GUI Application