The Data Structures of Python

8/7/2019 The Data Structures of Python

http://slidepdf.com/reader/full/the-data-structures-of-python 1/47

Data structures and

PythonAlex Gaynor

PyCon 2011

Monday, April 11, 2011



Who here remembers

their CS data structures class?




Who cares?




“We read Knuth soyou don't have to.“

- Tim Peters




If this isn’t CS102,what is it?




Why Python is

awesomeAs seen through data structures




I. What’s builtin, and how do we usethem?

II. Reaching for a little more

III.A little “do it yourself”




• list

• tuple

• dict

• set

• frozenset

The builtins




A brief interlude




Big O Notation

(in 30 seconds)

• How efficient an operation is, in termsof the number of items.

• O(1) - does the same number of

operations, regardless of how many

items

• O(n) - does a constant number

operation per item




Now where were we?




Awesome cars




lists, tuples, sets,

frozensets, and camels

• You might notice I didn’t include thedict, and it’s not just because camels

are awesome.

• dicts are a mapping, these are all

sequences

• That doesn’t make them

interchangeable




The pairings

• list vs tuple

• list vs set

• set vs frozenset




list vs tuple

• The obvious: lists are mutable andtuples aren’t

• So...




First the semantics

• I have a pretty simple rule about whento use tuples: only if using a namedtuple would be equally

approrpiate




class ArticleModelAdmin(admin.ModelAdmin):list_display = ["title", "author", "published", "get_absolute_url"]

class ArticleModelAdmin(admin.ModelAdmin):list_display = ("title", "author", "published", "get_absolute_url")

It’s a 2 character difference...

Which is right?




stocks = [("GOOG", "Google"),("T", "AT&T"),("AAPL", "Apple"),

]




And sometimes you

don’t have a choice

def memoize(func):cache = {}

def inner(*args): if args not in cache:

cache[args] = func(*args) return cache[args] return inner




The 2 Commandments

(so far)

1. Use types idiomatically.

2. Sometimes you don’t get a choice.




sets vs lists

• lists have an order, sets don’t

• list items can be anything, set itemsmust be hashable

• Computational complexity




def remove_dupes(seq):

seen = set()items = [] for item in seq: if item not in seen:

seen.add(item)items.append(item) return items

O(n)




list set

append/add

in/not in

remove

O(1) O(1)

O(n) O(1)

O(n) O(1)




3 Commandments

(and counting...)



3. Be efficient, when it doesn’t cost you

anything.




set vs frozenset

• Immutable vs immutable




class Lexer(object):keywords = frozenset([

"for", "if", "while", # etc...])




Why no frozendict?

• lists and tuples, sets and frozensets

• Not a ton of usecases

• Semantics might surprise people: do

both the key and the value need to beimmutable?




Its kind of amazing

how far you can gowith just those

Or: what to do if you need a little more




The stdlib




collections.OrderedDict

• New in 2.7/3.1

• Previous implementations existed in

tons of 3rd party libs

•For when you’ve got a dict that has

order!




class Food(Model):

name = Field()kind = Field()delicious = Field()




Solution!

OrderedDict([("name", Field()),

("kind", Field()),("delicious", Field()),])




4 Articles of the data

structure constitution



3. Be efficient, when it doesn’t cost youanything.

4. Sometimes you have more than oneconcern to deal with. The standard lib

can help!




collections.deque

• Ok I lied, I do use linked lists sometimes.

• Fact: list.pop(0) and list.insert(0) are

slow.


f



Also a really nifty ring

buffer

• Since 2.7 takes a maxlen parameter

• You keep on appending, it never getstoo big.

• Good for in memory logs and such.




See also

• The rest of collections

• array

• heapq (sort of)

• ... and more




DIY




collections.abc




Or: don’t subclass dict

ever>>> class SpecialDict(dict):... def __getitem__ (self, key):... return 42

... >>> SpecialDict()["a"]42>>> SpecialDict().get("a")>>> SpecialDict().get("a", 12)

12




• Subclassing Python’s builtincontainers tends not to produce the

results we want or expect.

• Subclassing the ABCs does.




>>> from collections import Mapping>>> >>> class SpecialDict(Mapping):... def __getitem__ (self, key):... return 42...

>>> SpecialDict()Traceback (most recent call last):

File "<stdin>", line 1, in <module>TypeError: Can't instantiate abstract class SpecialDictwith abstract methods __iter__, __len__

A tiny bit of checking for us




>>> from collections import Mapping>>> class SpecialDict(Mapping):... def __len__ (self):

... return 0

... def __iter__ (self):

... return iter([])

... def __getitem__ (self, key):

... return 72 # Best number evar!... >>> SpecialDict()["a"]72>>> SpecialDict().get("a")

72

Hallelujah!


I O d dDi t



I see your OrderedDict

and raise you:

OrderedSet

• Ordered collection, with that O(1) “in”

check we want

• But Python doesn’t include what,

whatever will I do!




import collections

KEY, PREV, NEXT = range(3)

class OrderedSet(collections.MutableSet):

def __init__ (self, iterable=None):self.end = end = []end += [None, end, end] # sentinel node for doubly linked listself.map = {} # key --> [key, prev, next]

if iterable is not None:self |= iterable

def __len__ (self): return len(self.map)

def __contains__ (self, key): return key in self.map

def add(self, key): if key not in self.map:

end = self.endcurr = end[PREV]curr[NEXT] = end[PREV] = self.map[key] = [key, curr, end]

def discard(self, key): if key in self.map:

key, prev, next = self.map.pop(key)prev[NEXT] = nextnext[PREV] = prev

def __iter__ (self):end = self.endcurr = end[NEXT]

while curr is not end: yield curr[KEY]

curr = curr[NEXT]

def __reversed__ (self):end = self.end

curr = end[PREV] while curr is not end: yield curr[KEY]

curr = curr[PREV]

def pop(self, last=True): if not self: raise KeyError('set is empty')

key = next(reversed(self)) if last else next(iter(self))self.discard(key)

return key

def __repr__ (self): if not self: return '%s()' % (self.__class__ .__name__,) return '%s(%r)' % (self.__class__ .__name__, list(self))

def __eq__ (self, other): if isinstance(other, OrderedSet): return len(self) == len(other) and list(self) == list(other) return set(self) == set(other)

def __del__ (self):self.clear() # remove circular references

http://code.activestate.com/recipes/576694/


5 t bl t f d t





















5 tablets of data

structures from on high1. Use types idiomatically.


3. Be efficient, when it doesn’t cost youanything.

4. Sometimes you have more than one concern

to deal with. The standard lib can help!

5. Don’t do more than you have to: ABCs are

there to help.




Questions?Comments? Thrown fruit?




http://alexgaynor.net

http://alexgaynor.net/

http://alexgaynor.net/

The Data Structures of Python

Documents

Transcript of The Data Structures of Python