Thinking Hybrid - Python/C++ Integration

35
Python Extending/Integrating A Real World Example Tips Summary Python where we can, C ++ where we must Source: http://xkcd.com/353/ Guy K. Kloss — Python where we can,C++ where we must 1/28

description

Talk on integrating native C++ sensibly into Python for ease of use of the code base. Inheriting from C++ classes, overriding functionality, automatically generating the bindings using Py++ and SCons. Code demonstrated in the presentation can be found here: http://www.kloss-familie.de/moin/TalksPresentations

Transcript of Thinking Hybrid - Python/C++ Integration

Page 1: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Python where we can,C++ where we must

Source: http://xkcd.com/353/

Guy K. Kloss — Python where we can,C++ where we must 1/28

Page 2: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Python where we can,

C++ where we must∗

Guy K. Kloss

BarCamp Auckland 200715 December 2007

∗ Quote: Alex Martelli, Senior Google Developer

Guy K. Kloss — Python where we can,C++ where we must 2/28

Page 3: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Outline

1 Python

2 Extending/Integrating

3 A Real World Example

4 Tips

Guy K. Kloss — Python where we can,C++ where we must 3/28

Page 4: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Outline

1 Python

2 Extending/Integrating

3 A Real World Example

4 Tips

Guy K. Kloss — Python where we can,C++ where we must 4/28

Page 5: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

What is Python?

Object–oriented

Dynamic

Easy to learn syntax

High-level data types

Scripting language

Embeddable in C/C++

Guy K. Kloss — Python where we can,C++ where we must 5/28

Page 6: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Python vs. Java (or C++)

No compilationFast edit–build–debug cycleDynamic TypingNo need to declare variables for useEasy SyntaxNo curly braces, no semicolons, no new . . .EmbeddableScripting support for your applicationsInteractiveCreate, view, change objects at runtime50% less codeCompact and natural syntax300% more productiveCloser to the way you think

Guy K. Kloss — Python where we can,C++ where we must 6/28

Page 7: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Why Python?

Mixable

Extend Python with componentswritten in C++, Java, CEmbed Python into your applicationand call it from C/C++

Platform independent

Powerful

Ease of use of scripting languageBuilt-in object typesExtensive librariesAutomatic memory managementModules, classes and exceptions

Guy K. Kloss — Python where we can,C++ where we must 7/28

Page 8: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

What is Python used for?

System UtilitiesSystem admin tools, portable shell scriptsInternet ScriptingCGI scripts, parse HTML, process XML, email toolsUser Interfaces (UIs) & rapid prototypingComponent GlueScripting for apps, COM scriptingDistributed ProgrammingWeb Services, COM, CORBA, XML–RPCDatabase ProgrammingScientific ComputingPyODE, NumPy, SciPy, PyMol, . . .Image ProcessingPython Image LibraryOpenGL Programming, Writing GamesPyOpenGL, Panda3D, PyOgre, Py3d, VisualPythonArtifical Intelligence

Guy K. Kloss — Python where we can,C++ where we must 8/28

Page 9: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Why Native Code?C/C++, Fortran (and Java, C#, ...)

Performance

Binding to legacy code

(Existing) Applications/Libraries . . .

... want to be scripted

... want to be tested

... want to be re-used

Guy K. Kloss — Python where we can,C++ where we must 9/28

Page 10: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Outline

1 Python

2 Extending/Integrating

3 A Real World Example

4 Tips

Guy K. Kloss — Python where we can,C++ where we must 10/28

Page 11: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

What if I could . . .

Use this code more effectively . . . ?

[NaSt2D demonstration (native executable)]

Guy K. Kloss — Python where we can,C++ where we must 11/28

Page 12: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Extending/Integration

The Contestants:

The “classic way” . . .Extending and Embedding the Python Interpreter

The “new way” . . .Python ctypes

SWIG

Boost.Python

Others:

SIPPythonizerSILOON(Pyrex)

Extensions also for Java, C#, Fortran, . . . available

Guy K. Kloss — Python where we can,C++ where we must 12/28

Page 13: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Extending/Integration

The Contestants:

The “classic way” . . .Extending and Embedding the Python Interpreter

The “new way” . . .Python ctypes

SWIG

Boost.Python

Others:

SIPPythonizerSILOON(Pyrex)

Extensions also for Java, C#, Fortran, . . . available

Guy K. Kloss — Python where we can,C++ where we must 12/28

Page 14: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Extending/Integration

The Contestants:

The “classic way” . . .Extending and Embedding the Python Interpreter

The “new way” . . .Python ctypes

SWIG

Boost.Python

Others:

SIPPythonizerSILOON(Pyrex)

Extensions also for Java, C#, Fortran, . . . available

Guy K. Kloss — Python where we can,C++ where we must 12/28

Page 15: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Extending/Integration

The Contestants:

The “classic way” . . .Extending and Embedding the Python Interpreter

The “new way” . . .Python ctypes

SWIG

Boost.Python

Others:

SIPPythonizerSILOON(Pyrex)

Extensions also for Java, C#, Fortran, . . . available

Guy K. Kloss — Python where we can,C++ where we must 12/28

Page 16: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Extending/Integration

The Contestants:

The “classic way” . . .Extending and Embedding the Python Interpreter

The “new way” . . .Python ctypes

SWIG

Boost.Python

Others:

SIPPythonizerSILOON(Pyrex)

Extensions also for Java, C#, Fortran, . . . available

Guy K. Kloss — Python where we can,C++ where we must 12/28

Page 17: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Boost.Python

Thinking Hybrid with Boost.Python

Bottom up and . . .

Top down possible

Develop quickly

Resolve bottle necks

Donald Knuth’s:“Premature optimisation is the root of all evil.”or: Don’t Optimise Now!

Guy K. Kloss — Python where we can,C++ where we must 13/28

Page 18: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Boost.Python

Thinking Hybrid with Boost.Python

Bottom up and . . .

Top down possible

Develop quickly

Resolve bottle necks

Donald Knuth’s:“Premature optimisation is the root of all evil.”or: Don’t Optimise Now!

Guy K. Kloss — Python where we can,C++ where we must 13/28

Page 19: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Boost.Python

Thinking Hybrid with Boost.Python

Bottom up and . . .

Top down possible

Develop quickly

Resolve bottle necks

Donald Knuth’s:“Premature optimisation is the root of all evil.”or: Don’t Optimise Now!

Guy K. Kloss — Python where we can,C++ where we must 13/28

Page 20: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Hello World

char const* greet(unsigned x) {static char const* const msgs[] = {"hello", "Boost.Python",

"world!"};if (x > 2) {

throw std::range error("greet: Index out of range.");}return msgs[x];

}

#include <boost/python.hpp>using namespace boost::python;BOOST PYTHON MODULE(hello){

.def("greet", greet, "return one of 3 parts of a greeting");}

And here it is in action:

>>> import hello>>> for x in range(3):... print hello.greet(x)...helloBoost.Pythonworld!

Guy K. Kloss — Python where we can,C++ where we must 14/28

Page 21: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Boost.Python

One of a few libraries that make it easyto integrate C++ and Python code

How does it pull off this trick?

Template meta–programming(i. e. Don’t ask!)

Guy K. Kloss — Python where we can,C++ where we must 15/28

Page 22: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Boost.Python

One of a few libraries that make it easyto integrate C++ and Python code

How does it pull off this trick?

Template meta–programming(i. e. Don’t ask!)

Guy K. Kloss — Python where we can,C++ where we must 15/28

Page 23: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

See it Happen

I’m making it work for you now . . .

[“MyClass” demonstration (MyClass.cpp, MyClass.h, mymodule.cpp)]

Guy K. Kloss — Python where we can,C++ where we must 16/28

Page 24: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Outline

1 Python

2 Extending/Integrating

3 A Real World Example

4 Tips

Guy K. Kloss — Python where we can,C++ where we must 17/28

Page 25: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

A Real World ExampleRe-visiting NaSt2D

Wrapping NaSt2D

Control the code

Guy K. Kloss — Python where we can,C++ where we must 18/28

Page 26: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

NaSt2D in Python

This is what I’m going to show you:

Code to be wrapped

Generated wrapper code

Generator script

SCons (build system)

How it works with Python

[Wrapped NaSt2D demonstration (Wrapper.h, nast2dmodule.cpp,

generate bindings.py, SConstruct, demo0.py)]

Guy K. Kloss — Python where we can,C++ where we must 19/28

Page 27: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Extend Wrapper Class

Inheriting from C++ classes

Interfacing numerical values

Change functionality

Follow the Computation

Guy K. Kloss — Python where we can,C++ where we must 20/28

Page 28: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Overriding in Python

This is what I’m going to show you:

Overriding a native method in Python

Native method needs to be “virtual”

Live data plotting with GNUplot

[NaSt2D with plotting demonstration (demo1.py, demo2.py)]

Guy K. Kloss — Python where we can,C++ where we must 21/28

Page 29: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Do more Computations

Parameter Study

Change input file

Compute several cases

Plot results automatically

Guy K. Kloss — Python where we can,C++ where we must 22/28

Page 30: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Automating in Python

This is what I’m going to show you:

Using a template input file

Batch–calculating several runs

Plotting results with GNUplot

[NaSt2D with parameter variation demonstration (demo3.py, demo4.py)]

Guy K. Kloss — Python where we can,C++ where we must 23/28

Page 31: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Outline

1 Python

2 Extending/Integrating

3 A Real World Example

4 Tips

Guy K. Kloss — Python where we can,C++ where we must 24/28

Page 32: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Tips

To override C++ methods: make them virtual

C/C++ pit fall

Call by reference/valueSolution: calling policies

Map to “other/sane” languages

Java: JythonC#: IronPythonFortran: PyFort, Py2F(Native to other: SWIG)

Guy K. Kloss — Python where we can,C++ where we must 25/28

Page 33: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Summary

Why is Python good for you?

How can performance bottle necks be resolved?

Advantages of “Thinking Hybrid”

Python–native wrapping using Boost.Python

Automated wrapper generation

SCons build system

Guy K. Kloss — Python where we can,C++ where we must 26/28

Page 34: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Python and the “Need for Speed”

Cuong Do – Software Architect YouTube.com

“Python is fast enough for our siteand allows us to produce maintainable features

in record times,with a minimum of developers.”

Guy K. Kloss — Python where we can,C++ where we must 27/28

Page 35: Thinking Hybrid - Python/C++ Integration

Python Extending/Integrating A Real World Example Tips Summary

Questions?

[email protected]

Guy K. Kloss — Python where we can,C++ where we must 28/28