Cython - Making Python as Fast as C

CYTHON — MAKING PYTHON AS FAST AS CMosky

MOSKY

2

MOSKY

Python Charmer at Pinkoi

2

http://mosky.tw

MOSKY


An author of the Python Packages

MoSQL, Clime, … and more

2

http://mosky.tw

MOSKY




A speaker of the conferences, PyCon TW/JP mostly.

2

http://mosky.tw

MOSKY





A Python instructor

2

http://mosky.tw

MOSKY





A Python instructor

mosky.tw2

http://mosky.tw

OUTLINE

3

OUTLINE

Introduction

3

OUTLINE

Introduction

Setup

3

OUTLINE

Introduction

Setup

Foundation

3

OUTLINE

Introduction

Setup

Foundation

Practicing

3

OUTLINE

Introduction

Setup

Foundation

Practicing

Tips

3

OUTLINE

Introduction

Setup

Foundation

Practicing

Tips

Uncovered Topics

3

OUTLINE

Introduction

Setup

Foundation

Practicing

Tips

Uncovered Topics

Is Cython the best solution?

3

INTRODUCTION

CYTHON

5

CYTHON

Cython is a source-to-source compiler (aka. transcompiler).

5

CYTHON


Cython is a superset of Python.

5

CYTHON



Provides optional static type declarations.

5

CYTHON




Makes writing C extensions for Python easier.

5

CYTHON




Makes writing C extensions for Python easier.

Makes Python program faster by pre-compiling and static type. (sometimes faster by orders of magnitude)

5

6

.pyx

6

.pyx .c

Cython

6

.pyx .c

Cython

C compiler

.so

6

.pyx .c

Cython

C compiler

.so .py

6

.pyx .c

Cython

C compiler

.so .py

Python

6

.pyx .c

Cython

C compiler

import.so .py

Python

INSTALL C COMPILER

8

INSTALL C COMPILER

Mac:

xcode-select --install

8

http://docs.cython.org/src/quickstart/install.html

INSTALL C COMPILER

Mac:


Ubuntu / Debian:

sudo apt-get install build-essential

8


INSTALL C COMPILER

Mac:


Ubuntu / Debian:

sudo apt-get install build-essential

Other:

http://docs.cython.org/src/quickstart/install.html8


INSTALL CYTHON

9

INSTALL CYTHON

Recommend to use PIP:

sudo pip install cython

9


INSTALL CYTHON

Recommend to use PIP:

sudo pip install cython

Other:


9


THE SETUP.PY

from distutils.core import setup

from Cython.Build import cythonize

!

setup(

name = 'cython-lab',

ext_modules = cythonize('*.pyx'),

)10

THE HELLO_CYTHON.PYX

print 'Hello Cython!'

11

BUILD

12

BUILD

Into package folder for development:

python setup.py build_ext --inplace

12

BUILD



Into system for production:

python setup.py install

12

BUILD



Into system for production:

python setup.py install

If clang: error: unknown argument: '-mno-fused-madd',

export CFLAGS=-Qunused-arguments12

FOUNDATION— the difference of Cython and Python

DEFINE STATIC TYPES

cdef int i, j, k

cdef float f, g[42], *h

14

cdef struct Grail:

int age

float volume

!

cdef union Food:

char* spam

float* eggs

!

cdef enum CheeseType:

cheddar, edam,

camembert

!

cdef enum CheeseState:

hard = 1

soft = 2

runny = 315

ctypedef unsigned long ULong

!

ctypedef int* IntPtr

16

cdef struct Point:

int x

int y

!

# either `struct` or `ctypedef` is not need

cdef Point p

17

cdef:

struct Point:

int x

int y

!

Point p

18

DEFINE FUNCTION

def say_hello(name='World'):

return 'Hello, %s!' % name

!

cdef say_hello(name='World'):

return 'Hello, %s!' % name19

cdef say_hello(object name='World'):


!

cdef say_hello(char* name='World'):


20

cdef int add(int a, int b):

return a+b

!

cpdef say_hello(char* name='World'):


21

.PXD EXPOSES CDEF FUNC.

# mylib.pxd

cdef say_hello(char* name=?)

!

# another.pyx

from mylib cimport say_hello22

USING C LIB.

from libc.math cimport sin

# or

cdef extern from "math.h":

double sin(double x)

23

FUNCTION VISIBILITY

24

SAME FILE

OTHER .PYX .PY

FUNC. IN .H/.C

Visible directly

Visible via cdef extern

InvisibleCDEF

Visible via .pxd & cimport

CPDEFVisible

via importDEF

BINDING DURING

25

SAME FILE

OTHER .PYX .PY

FUNC. IN .H/.C

compile-time

compile-time xCDEF

CPDEF

run-timeDEF

TYPE CONVERSIONS

26

C FROM PY TO PY

[UNSIGNED] CHAR/SHORT INT

int/long

intLONG

UNSIGNED INT/LONG

long[UNSIGNED] LONG LONG

27

C FROM PY TO PY

FLOAT/DOUBLE

int/long/float float

LONG DOUBLE

CHAR* str/bytes

STRUCT dict

PRACTICING

SUGGESTION

29

SUGGESTION

LIB_NAME.pyx

has an execute_self_tests function

29

SUGGESTION

LIB_NAME.pyx

has an execute_self_tests function

test_LIB_NAME.py

call the execute_self_tests function

29

OVERFLOW

30

OVERFLOW

Static types may also overflow in Cython silently.

30

http://j.mp/test_overflow_in_c_c

http://j.mp/overflow_in_pyx_pyx

OVERFLOW


Try to make an overflow!

30



OVERFLOW



Hint:


30



OVERFLOW



Hint:


Ans:

http://j.mp/overflow_in_pyx_pyx 30



FUNCTIONS

31

FUNCTIONS

Write three func. defined in def, cdef, and cpdef.

31

http://j.mp/lib_in_pyx_pyx

http://j.mp/use_lib_in_pyx_pyx

http://j.mp/test_lib_in_pyx_py

FUNCTIONS


Try to call them in

the same file,

another .pyx file,

and a .py file.

31




FUNCTIONS


Try to call them in

the same file,

another .pyx file,

and a .py file.

Hints:

Refer to the table, “Function Visibility”.


31




FUNCTIONS


Try to call them in

the same file,

another .pyx file,

and a .py file.

Hints:

Refer to the table, “Function Visibility”.


Ans:


http://j.mp/test_lib_in_pyx_py 31




USING C FUNCTION

32

USING C FUNCTION

Try to use the functions in C.

32

http://j.mp/test_fork_c

http://j.mp/fork_in_pyx_pyx

USING C FUNCTION


Playing with fork, the system call, may be fun.

32



USING C FUNCTION



Hint:


32



USING C FUNCTION



Hint:


Ans:

http://j.mp/fork_in_pyx_pyx 32



CYTHON -A

34

CYTHON -A

cython -a NAME.pyx

34

CYTHON -A

cython -a NAME.pyx

open NAME.html

34

CYTHON -A

cython -a NAME.pyx

open NAME.html

Lines are colored according to the level of “typedness” – white lines translates to pure C without any Python API calls.

34

PYXIMPORT

import pyximport; pyximport.install()

import my_pyx_lib # compile .pyx into .so

!

# or

pyximport.install(pyimport=True)

import my_py_lib # compile .py into .so35

UNCOVERED TOPICS

UNCOVERED TOPICS

37

UNCOVERED TOPICS

Differences between C and Cython expressions

http://docs.cython.org/src/userguide/language_basics.html#differences-between-c-and-cython-expressions

37


http://docs.cython.org/src/userguide/language_basics.html#error-return-values

UNCOVERED TOPICS

Differences between C and Cython expressions


Propagating Exceptions in cdef


37



Extension Type — cdef class

http://docs.cython.org/src/userguide/extension_types.html

38


http://docs.cython.org/src/userguide/fusedtypes.html

http://docs.cython.org/src/userguide/language_basics.html#conditional-compilation



Generic programming using Cython's Template


38






Generic programming using Cython's Template


Conditional Compilation


38




Profiling

http://docs.cython.org/src/tutorial/profiling_tutorial.html

39


http://docs.cython.org/src/userguide/parallelism.html

http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html

Profiling


Parallelism (No GIL + OpenMP)


39




Profiling


Parallelism (No GIL + OpenMP)


Using C++ in Cython


39




IS CYTHON THE BEST SOLUTION?

PYTHON COMMUNITY

IS DORAEMON!

Cython - Making Python as Fast as C

Technology

Transcript of Cython - Making Python as Fast as C