Smart421 web sphere user group (wug) conference presentation
Smart421 star testing event big data, london 26 sept 2013 r meehan v1.0
-
Upload
smart421 -
Category
Technology
-
view
189 -
download
1
description
Transcript of Smart421 star testing event big data, london 26 sept 2013 r meehan v1.0
![Page 1: Smart421 star testing event big data, london 26 sept 2013 r meehan v1.0](https://reader035.fdocuments.us/reader035/viewer/2022062511/54b6e3254a7959aa3d8b461d/html5/thumbnails/1.jpg)
10 April 2023
Star Testing Event – Big DataRobin Meehan, CTO, Smart421
![Page 2: Smart421 star testing event big data, london 26 sept 2013 r meehan v1.0](https://reader035.fdocuments.us/reader035/viewer/2022062511/54b6e3254a7959aa3d8b461d/html5/thumbnails/2.jpg)
http://commons.wikimedia.org/wiki/File:Encyclopedia_Britannica_series.JPG
![Page 3: Smart421 star testing event big data, london 26 sept 2013 r meehan v1.0](https://reader035.fdocuments.us/reader035/viewer/2022062511/54b6e3254a7959aa3d8b461d/html5/thumbnails/3.jpg)
http://commons.wikimedia.org/wiki/File:Rail_disruption_at_Buckley.jpg
Cloud…
![Page 4: Smart421 star testing event big data, london 26 sept 2013 r meehan v1.0](https://reader035.fdocuments.us/reader035/viewer/2022062511/54b6e3254a7959aa3d8b461d/html5/thumbnails/4.jpg)
http://commons.wikimedia.org/wiki/File:Felicidade_A_very_happy_boy.jpg
![Page 5: Smart421 star testing event big data, london 26 sept 2013 r meehan v1.0](https://reader035.fdocuments.us/reader035/viewer/2022062511/54b6e3254a7959aa3d8b461d/html5/thumbnails/5.jpg)
http://commons.wikimedia.org/wiki/File:Eyjafjallaj%C3%B6kull_%284530326521%29.jpg
…Big Data!
![Page 6: Smart421 star testing event big data, london 26 sept 2013 r meehan v1.0](https://reader035.fdocuments.us/reader035/viewer/2022062511/54b6e3254a7959aa3d8b461d/html5/thumbnails/6.jpg)
http://commons.wikimedia.org/wiki/File:Woolworths_Beccles_-_geograph.org.uk_-_1077646.jpg
![Page 7: Smart421 star testing event big data, london 26 sept 2013 r meehan v1.0](https://reader035.fdocuments.us/reader035/viewer/2022062511/54b6e3254a7959aa3d8b461d/html5/thumbnails/7.jpg)
So what’s changed?
![Page 8: Smart421 star testing event big data, london 26 sept 2013 r meehan v1.0](https://reader035.fdocuments.us/reader035/viewer/2022062511/54b6e3254a7959aa3d8b461d/html5/thumbnails/8.jpg)
Big data exploitation – in practice
8
![Page 9: Smart421 star testing event big data, london 26 sept 2013 r meehan v1.0](https://reader035.fdocuments.us/reader035/viewer/2022062511/54b6e3254a7959aa3d8b461d/html5/thumbnails/9.jpg)
Case study
Aviva have a number of brands/channels to market including insurance aggregators (e.g. CompareThe Market, GoCompare…)
The raw aggregator quote data is of a scale to present a ‘Big Data’ problem – there is great potential for gaining additional insights from this data
9
Define some candidate business questions
Test them against
significant volumes of data
Measure cluster size/£/time
performance
![Page 10: Smart421 star testing event big data, london 26 sept 2013 r meehan v1.0](https://reader035.fdocuments.us/reader035/viewer/2022062511/54b6e3254a7959aa3d8b461d/html5/thumbnails/10.jpg)
Driving AWS EMR…
10
AWS Elastic Map Reduce…configuring a Hadoop Cluster...
![Page 11: Smart421 star testing event big data, london 26 sept 2013 r meehan v1.0](https://reader035.fdocuments.us/reader035/viewer/2022062511/54b6e3254a7959aa3d8b461d/html5/thumbnails/11.jpg)
Some pig…
register 's3n://ashaw-1/jars/myudfs.jar';register 's3n://ashaw-1/jars/dom4j-1.6.1.jar';A = load 's3n://ashaw-1/Intermediate/duplicated/lots' using PigStorage();Arac = load 's3n://ashaw-1/Intermediate/duplicated/lotsrac' using PigStorage();A1 = limit A 5000000;Arac1 = limit Arac 5000000;B = foreach A1 generate myudfs.Flatten((chararray)$5);Brac = foreach Arac1 generate myudfs.Flatten2((chararray)$5);C = join B by (chararray)($0.$21), Brac by (chararray)($0.$21);D = filter C by $1.$0 == 1 OR $0.$0 == 1;STORE D INTO ‘s3n://ashaw-1/myoutputfolder/’;
Query B: ~10 million quotes (5m each channel). Joining quote data across different channels.
![Page 12: Smart421 star testing event big data, london 26 sept 2013 r meehan v1.0](https://reader035.fdocuments.us/reader035/viewer/2022062511/54b6e3254a7959aa3d8b461d/html5/thumbnails/12.jpg)
Visualisation
![Page 13: Smart421 star testing event big data, london 26 sept 2013 r meehan v1.0](https://reader035.fdocuments.us/reader035/viewer/2022062511/54b6e3254a7959aa3d8b461d/html5/thumbnails/13.jpg)
Costs per run…
Cluster size: Time to execute: Approx. cost:10 x Small nodes 64 minutes 11 compute hours - $1.155 per hour (approx. £0.72)19 x Small nodes 31 minutes 20 compute hours - $2.10 per hour (approx. £1.30) 8 x Large nodes 19 minutes 8 compute hours - $3.78 per hour (approx. £2.34)
But we could have used spot instances…
![Page 14: Smart421 star testing event big data, london 26 sept 2013 r meehan v1.0](https://reader035.fdocuments.us/reader035/viewer/2022062511/54b6e3254a7959aa3d8b461d/html5/thumbnails/14.jpg)
http://commons.wikimedia.org/wiki/File:Binoculars_25x100.jpg
Challenges…
![Page 15: Smart421 star testing event big data, london 26 sept 2013 r meehan v1.0](https://reader035.fdocuments.us/reader035/viewer/2022062511/54b6e3254a7959aa3d8b461d/html5/thumbnails/15.jpg)
http://flickr.com/photos/42033648@N00/61053542
Storm
Spark
Dremel/Drill
Impala
AWS Redshift
Accumulo
etc etc
Trends…
![Page 16: Smart421 star testing event big data, london 26 sept 2013 r meehan v1.0](https://reader035.fdocuments.us/reader035/viewer/2022062511/54b6e3254a7959aa3d8b461d/html5/thumbnails/16.jpg)
Blog: http://smart421.wordpress.com/tag/big-data/
Thank you!
Robin MeehanCTO, Smart421
![Page 17: Smart421 star testing event big data, london 26 sept 2013 r meehan v1.0](https://reader035.fdocuments.us/reader035/viewer/2022062511/54b6e3254a7959aa3d8b461d/html5/thumbnails/17.jpg)
Spare slides…
![Page 18: Smart421 star testing event big data, london 26 sept 2013 r meehan v1.0](https://reader035.fdocuments.us/reader035/viewer/2022062511/54b6e3254a7959aa3d8b461d/html5/thumbnails/18.jpg)
10 April 2023
http://commons.wikimedia.org/wiki/File:Loud_environment_headphones.jpg
![Page 19: Smart421 star testing event big data, london 26 sept 2013 r meehan v1.0](https://reader035.fdocuments.us/reader035/viewer/2022062511/54b6e3254a7959aa3d8b461d/html5/thumbnails/19.jpg)
10 April 2023
http://commons.wikimedia.org/wiki/File:Ferrari_156_85_in_2011.jpg
![Page 20: Smart421 star testing event big data, london 26 sept 2013 r meehan v1.0](https://reader035.fdocuments.us/reader035/viewer/2022062511/54b6e3254a7959aa3d8b461d/html5/thumbnails/20.jpg)
10 April 2023
http://commons.wikimedia.org/wiki/File:Hundreds_and_thousands.jpg
![Page 21: Smart421 star testing event big data, london 26 sept 2013 r meehan v1.0](https://reader035.fdocuments.us/reader035/viewer/2022062511/54b6e3254a7959aa3d8b461d/html5/thumbnails/21.jpg)
http://www.flickr.com/photos/krishaamer/2836262962/