Extending Ruby by harnessing other languages
-
Upload
brendon-mclean -
Category
Software
-
view
34 -
download
1
Transcript of Extending Ruby by harnessing other languages
![Page 1: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/1.jpg)
Extending Ruby by harnessing other languages
Brendon McLean@brendon9x
![Page 2: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/2.jpg)
What if Ruby doesn’t do everything well?
But you want to use Ruby anyway
![Page 3: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/3.jpg)
A long time ago, in a startup far, far away…
![Page 4: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/4.jpg)
The Challenge a.k.a “The value proposition”
![Page 5: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/5.jpg)
To bring meaningful insight and intelligence to market research*
*In two weeks please
![Page 6: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/6.jpg)
![Page 7: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/7.jpg)
SolutionMVP on Ruby on Rails
![Page 8: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/8.jpg)
The Requirement a.k.a “The Problem”
a.k.a Numerical Ruby Please
![Page 9: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/9.jpg)
Dimensionality* *lots of it
![Page 10: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/10.jpg)
RESP AGE GENDER USE_IOS USE_ANDROID CAT_PERSON1 18-24 M 1 1 N2 25-35 M N3 35-50 F 1 1 Y4 35-50 F Y5 18-24 M 1 1 Y6 25-35 M 1 Y7 25-35 M 1 1 Y8 35-50 F 1 N9 18-24 F 1 Y10 18-24 F 1 N11 35-50 F 1 N12 35-50 F 1 N13 18-24 F 1 Y14 35-50 M 1 N15 18-24 F 1 N16 35-50 M 1 N17 35-50 M 1 1 NN 25-35 M Y
![Page 11: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/11.jpg)
up to 40GB* *per dataset
![Page 12: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/12.jpg)
Loosely structured
![Page 13: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/13.jpg)
Interrogate Anything* *goodbye clever caching strategy
![Page 14: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/14.jpg)
Long story short…
![Page 15: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/15.jpg)
Approx FLOPS
1,000
1,000,000
1,000,000,000
1,000,000,000,000
Plain Ruby (2011) Bitset (2012) GSL (2012) NArray (2013)
Complexity inflection point
![Page 16: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/16.jpg)
Performance Abstraction Power Complexity
![Page 17: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/17.jpg)
Time to consider other options
![Page 18: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/18.jpg)
If, hypothetically, we weren’t using Ruby, what else is out there?
![Page 19: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/19.jpg)
![Page 20: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/20.jpg)
HaskellComes with free beard*
*beard must compile
![Page 21: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/21.jpg)
NOBODY EXPECTED PYTHON!
![Page 22: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/22.jpg)
Why Python?
![Page 23: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/23.jpg)
Size of scientific community
![Page 24: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/24.jpg)
Depth of ecosystem
![Page 25: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/25.jpg)
Similar* attitude to usability and expressiveness first
*ish
![Page 26: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/26.jpg)
Performance
![Page 27: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/27.jpg)
Numpy
• Lineage goes back to 1995
• Array computing — vectorised operations for Python
• NArray is based on Numpy
• Is the bedrock upon which the rest of scientific Python is built
![Page 28: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/28.jpg)
Vectorisation
$> array.reduce(&:+)
$> a.zip(b).map do |l, r| l * r end
$> array.sum
$> a * b
![Page 29: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/29.jpg)
Pandas
• Built on Numpy
• Basically ports the best bits of R into Python
• Fast
• Cognitively simpler for general programmers
• Munging!
![Page 30: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/30.jpg)
Bonus extras
• Scipy: Linear Algebra, FFT, Clustering, Stats
• IPython Notebooks
• Sympy: Computer Algebra System
• nltk: Natural Language Toolkit
• scikit-learn: Machine Learning
![Page 31: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/31.jpg)
Strength of community
![Page 32: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/32.jpg)
Total commits
NArray GSL Pandas Numpy
12,588
10,865
193141
![Page 33: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/33.jpg)
Contributers
NArray GSL Numpy Pandas
310
249
44
![Page 34: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/34.jpg)
Issues: Open and Closed
GSL NArray Numpy Pandas
1051
651
181
4681
2691
185
![Page 35: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/35.jpg)
Using Python from Ruby ❤️
![Page 36: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/36.jpg)
Problem statement
![Page 37: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/37.jpg)
Flexibility of Pandas
![Page 38: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/38.jpg)
Speed of Numpy
![Page 39: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/39.jpg)
Scales horizontally
![Page 40: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/40.jpg)
Ruby ❤️ API
![Page 41: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/41.jpg)
API Inspiration
![Page 42: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/42.jpg)
ActiveRecord scopesDeferred, composable
![Page 43: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/43.jpg)
API implementation problemGetting to Ruby to run Python === Getting Python to run Ruby
![Page 44: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/44.jpg)
Ruby => Data => Python*
*or other
![Page 45: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/45.jpg)
“Code is data”— people with LISP personality disorder
![Page 46: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/46.jpg)
S-Expressions(function arg1 arg2 arg3 ...)
![Page 47: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/47.jpg)
Simple s-expression example
$> (+ 1 1) => 2
$> (find User 1 2)
$> 1 + 1 => 2
$> User.find(1, 2) $> User.send(:find, 1, 2)
![Page 48: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/48.jpg)
Example with nesting
$> => 5
2 2* 1+
![Page 49: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/49.jpg)
Example with nesting
$> => 5
22*1+( ( ) )
![Page 50: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/50.jpg)
ActiveLISP
User.select(:state). where(no_spam: false). group(:state). count
(count (group :state (where :no_spam false (select :state User) ) ) )
![Page 51: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/51.jpg)
Ruby => S-Expressions
S-Expressions => Python
![Page 52: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/52.jpg)
Does have limitations*
![Page 53: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/53.jpg)
Added benefits
![Page 54: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/54.jpg)
Optimisation (tree rewrites)
![Page 55: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/55.jpg)
Automatic query sharding
![Page 56: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/56.jpg)
Target multiple backends through a common API
![Page 57: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/57.jpg)
Enough talking…
![Page 58: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/58.jpg)
Live Demo
![Page 59: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/59.jpg)
![Page 60: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/60.jpg)
Thanks
• Min RK Initial iRuby Kernel
• Daniel Mendler for continued work on iRuby Kernel
• My Team @ Intellection
• And all the gems!
![Page 61: Extending Ruby by harnessing other languages](https://reader038.fdocuments.us/reader038/viewer/2022102903/55c2bd75bb61ebc52d8b45dc/html5/thumbnails/61.jpg)
London Cape Townhttp://www.public-domain-image.com/architecture/bridge/slides/bridge-london-england.html
intellection