Post on 27-Jan-2015
description
ADAPTIVE FILTERING OF TWEETS WITH MACHINE LEARNINGNeri Van Otten
Data Scientist, Conversocial
About Conversocial• Social customer service platform
Intelligent Prioritization
Continual Improvement• Incorporate feedback from customers• Constant monitoring performance
How to Process Text• Create features• Tf-idf for bag of words• N-grams• Other features e.g. length of text, contains url, …• Convert to matrix of 0/1• Singular value decomposition (SVD)• Machine learning models
Parameter Optimization• Separate server• Runs on backup copy of database
Training Models• Daily training• Queued and trained when servers are under utilized• Lower priority as the system still has working models• Store to S3
Models in Production• Servers specific for prioritization• Chef to configure new servers• Servers download models from S3• Cache as many models in memory as possible• Evict older models• Use the client specific model to classify message
Questions