Post on 13-Jan-2017
Agenda
★ Basics of Functional Programming★ Functional Programming and Python★ Everyday Functional Programming★ Scaling with Functional Programming★ Performance Implications★ Data Explorations
House rules
★ Show of hands please★ Ask anytime - Raise your hands★ Be hands on★ Free to use online resources★ One step at a time★ No-one left out policy
Basics of Functional Programming
★ Function as the atomic entity ○ First-class functions ○ Higher-order functions ○ Pure functions
Basics of Functional Programming
★ Recursion, Tail Recursion, Iterators, Currying, Sequences, pattern matching, monads....
Myths of Functional Programming
★ It requires immutability/pureness
★ It requires an advanced type system
★ It is significantly less efficient
★ It makes you learn advanced math
★ You must give up all your imperative
programming notions
★ Object orientation and functional paradigms
are incompatible
★ Functional programs are easier to debug
★ Dynamic types are better than Static types
Functional methodology in Python
★ itertools and functools★ Decorators and Generators★ What python CAN do
○ lazy eval, lambda, map, reduce, filter
★ What python CANNOT do (pragmatically)○ Eg: Tail Recursion, Pure immutability, Pure Functions
Functional Programming:Done everyday
★ map
★ reduce
★ filter
★ enumerate
★ iter
★ list comprehension
★ Generator Expressions
★ Generators
★ itertools
★ functools
★ operator
★ decorator
★ Lambda Functions
operator module
import operator
operator.add(40,2)
operator.sub(45,3)
data = {'company': 'Acme Corp.'}
getCompany = operator.itemgetter('company')
getCompany(data)
Map - Explained
from urllib import urlopen
urls = ['http://www.google.com',
'http://www.wikipedia.com',
'http://www.apple.com',
'http://www.python.org'
]
result = []
for item in urls:
result.append(urlopen(item))
return result
def fib(n):
a, b = 0, 1
for i in range(n):
a, b = b, a + b
....
integers = [1, 2, 3, 4, 5]
result = []
for item in integers:
result.append(fib(item))
return result ?
Map - Explained
from urllib import urlopen
urls = ['http://www.google.com',
'http://www.wikipedia.com',
'http://www.apple.com',
'http://www.python.org'
]
result = []
for item in urls:
result.append(urlopen(item))
return result
def fib(n):
a, b = 0, 1
for i in range(n):
a, b = b, a + b
...
integers = [1, 2, 3, 4, 5]
result = []
for item in integers:
result.append(fib(item))
return result
def map(function, sequence):
result = []
for item in sequence:
result.append(function(item))
return result
html_texts = map(urlopen, urls)
fib_integers = map(fib, integers)
Lambda
count_lambda =lambda w: len(w)
print map(count_lambda, 'It is raining cats and dogs'.split())
#conditions in lambda
lambda x: True if x % 2 == 0 else False
Reduce - Explained
# Sum of a list of numbers
def add(x, y):
return x + y
def sum(data):
result = 0
for x in data:
result = add(result, x)
return result sum([5, 2, 3])
# Smallest in a list
def lesser(x, y):
if x < y:
return x
else:
return y
def min(data):
result = 999999999999
for x in data:
result = lesser(result, x)
return result min([5, 2, 3])
?
Reduce - Explained
# Sum of a list of numbers
def add(x, y):
return x + y
def sum(data):
result = 0
for x in data:
result = add(result, x)
return result sum([5, 2, 3])
# Smallest in a list
def lesser(x, y):
if x < y:
return x
else:
return y
def min(data):
result = 999999999999
for x in data:
result = lesser(result, x)
return result min([5, 2, 3])
# Sum
result = sum(data)
result = reduce(add, data, 0)
# Min
result = min(data)
result = reduce(lesser, data, 9999999999)
filter
def is_even(x): return (x % 2) == 0
filter(is_even, range(10))
#even better
from itertools import ifilter
filter_iterator = ifilter(is_even,data)
filter_iterator.next()
iter function
numbers = [1,2,3]
it = iter(numbers)
# using while and StopIteration Exception
try:
while True:
print it.next()
except StopIteration:
print "Complete"
# as iterator in for loop
it = iter(numbers)
for value in it:
print value
import itertools
itertools.count(100)
itertools.cycle([1,2,3,4,5])
itertools.repeat("Hello", 5)
itertools - count, cycle, repeat
itertools - groupby
city_list = [('Cochin', 'KL'),
('Bengaluru', 'KA'),
('Chennai', 'TN'),
('Mumbai', 'MH'),
('Trivandrum', 'KL'),
('Salem', 'TN'),
('Pune', 'MH')]
for city, city_iterator in groupby(city_list, lambda x: x[0]):
for city_state in city_iterator:
print "%s is in %s." % (city, city_state[1])
itertools - combinations, permutations
import itertools
itertools.permutations([1, 2, 3])
itertools.combinations([1, 2, 3, 4, 5], 2)
functools - partial
import functools
def log (message, subsystem):
print '%s: %s' % (subsystem, message)
server_log = functools.partial(log,
subsystem='server')
server_log('Unable to open socket')
Decorators
def p_decorate(func):
def func_wrapper(name):
return "Super %s" func
(name)
return func_wrapper
@p_decorate
def get_text(name):
return "Hello %s" % name
print get_text("John")
Yield - creating custom generators
data_list = ["Orange", "Microsoft", "Apple"]
def one_at_a_time(data_list):
for item in data_list:
yield item
one_at_a_time.next()
Don't do unnecessary Classes
class Greeting(object):
def __init__(self, greeting="hello"):
self.greeting = greeting
def greet(self, name):
return "{greeting} {name}!".format
(greeting=greeting, name=name)
hola = Greeting("hola")
print hola.greet("bob")
or ?
Don't do unnecessary Classes
class Greeting(object):
def __init__(self, greeting="hello"):
self.greeting = greeting
def greet(self, name):
return "{greeting} {name}!".format
(greeting=greeting, name=name)
hola = Greeting("hola")
print hola.greet("bob")
def greet(greeting, name):
return "{greeting} {name}!".format(greeting=greeting,
name=name)
hello = functools.partial(greet, "Hello")
hello("Dude")
or
The Bad Parts
★ Memory Inefficiencies★ Purity★ No Tail Recursion★ Innately imperative (Guido)
★ Class based type system★ Only imperative Error Handling
(Exception)
★ Function Overloading★ Mutable variables
Python vs Functional
Thinking about Scalability with Functions
★ map-reduce-filter - recipe for distributed computing
★ shared states - to be or not to be
★ immutable 'variables'★ independent functions★ Execution Pipelines - chained map-reduce
Performance v/s Scalability
★ Functional Programs vs Object Oriented Programs★ CPU intensive processes vs I/O intensive processes★ The curse of GIL - workarounds
○ multiprocessing
★ Benchmarking○ %timeit
multiprocessing.Pool
import multiprocessing
def worker(): print 'Execute your function here' return
if __name__ == '__main__': jobs = [] for i in range(5): p = multiprocessing.Process(target=worker) jobs.append(p) p.start()
why!
★ no side effects - no state, no deadlocks, no semaphores
★ automatic parallelization - unlimited scalability
★ composability - break down into smaller functions
★ Testing - independent functions; well defined arguments and return values
★ partial evaluation - pass around half baked functions instead of objects
★ elegant code - forces to write logically correct programs
Hands On: Let’s do some Data Wrangling
Blockbuster Database (http://www.crowdflower.com/data-for-everyone)
➢ Which genre has most movies?➢ Which movie studio gross the most?➢ Sort by most grossed and Highly rated movies
demo code here
Questionscode samples in this presentation here
atm@profoundis.com
@atmb4u