Pure PythonC Accelerator Module Compatibility Requirements

8/7/2019 Pure PythonC Accelerator Module Compatibility Requirements

1/6

pdfcrowd comcustomize free license

Python DocumentationPEP: 399

Title: Pure Python/C Accelerator Module Compatibility RequirementsVersion: 88219

Last-Modified: 2011-01-27 13:47:00 -0800 (Thu, 27 Jan 2011)Author: Mustapha ElmekkiStatus: DraftType: Informational

Content-Type: text/x-rst

Created: 04-Apr-2011Python-Version: 3.3Post-History: 04 -Apr-20 11, 12-Apr-2011

AbstractThe Python standard library under CPython contains various instances of modules implemented in bothpure Python and C (either entirely or partially). This PEP requires that in these instances that the C codemust pass the test suite used for the pure Python code so as to act as much as a drop-in replacement aspossible (C- and VM-specific tests are exempt). It is also required that new C-based modules lacking apure Python equivalent implementation get special permissions to be added to the standard library.
http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.python.org%2fdev%2fpeps%2fpep-0012&id=in-110503223802-94545b90http://pdfcrowd.com/http://pdfcrowd.com/customize/http://pdfcrowd.com/license-for-free/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fsvn.python.org%2fview%2f%2acheckout%2a%2fpeps%2ftrunk%2fpep-0399.txt&id=in-110503223802-94545b90http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.python.org%2fdev%2fpeps%2fpep-0012&id=in-110503223802-94545b90


2/6


RationalePython has grown beyond the CPython virtual machine (VM). IronPython [1], Jython [2], and PyPy [3] allcurrently being viable alternatives to the CPython VM. This VM ecosystem that has sprung up around thePython programming language has led to Python being used in many different areas where CPythoncannot be used, e.g., Jython allowing Python to be used in Java applications.

A problem all of the VMs other than CPython face is handling modules from the standard library that areimplemented (to some extent) in C. Since they do not typically support the entire C API of CPython [4] theyare unable to use the code used to create the module. Often times this leads these other VMs to either re-implement the modules in pure Python or in the programming language used to implement the VM (e.g., inC# for IronPython). This duplication of effort between CPython, PyPy, Jython, and IronPython is extremelyunfortunate as implementing a module at least in pure Python would help mitigate this duplicate effort.

The purpose of this PEP is to minimize this duplicate effort by mandating that all new modules added toPython's standard library must have a pure Python implementation _unless_ special dispensation is given.This makes sure that a module in the stdlib is available to all VMs and not just to CPython (pre-existingmodules that do not meet this requirement are exempt, although there is nothing preventing someone fromadding in a pure Python implementation retroactively).

Re-implementing parts (or all) of a module in C (in the case of CPython) is still allowed for performancereasons, but any such accelerated code must pass the same test suite (sans VM- or C-specific tests) toverify semantics and prevent divergence. To accomplish this, the test suite for the module must have 100%branch coverage of the pure Python implementation before the acceleration code may be added. This is toprevent users from accidentally relying on semantics that are specific to the C code and are not reflected inthe pure Python implementation that other VMs rely upon. For example, in CPython 3.2.0, heapq.heappop()does an explicit type check in its accelerated C code while the Python code uses duck typing:from test.support import import_fresh_module

c_heapq = import_fresh_module('heapq', fresh=['_heapq'])py_heapq = import_fresh_module('heapq', blocked=['_heapq'])
http://pdfcrowd.com/http://pdfcrowd.com/customize/http://pdfcrowd.com/license-for-free/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fironpython.net%2f&id=in-110503223802-94545b90http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.python.org%2fdev%2fpeps%2fpep-0399%2f%23id2&id=in-110503223802-94545b90http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.jython.org%2f&id=in-110503223802-94545b90http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.python.org%2fdev%2fpeps%2fpep-0399%2f%23id4&id=in-110503223802-94545b90http://pdfcrowd.com/redirect/?url=http%3a%2f%2fpypy.org%2f&id=in-110503223802-94545b90http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.python.org%2fdev%2fpeps%2fpep-0399%2f%23id6&id=in-110503223802-94545b90http://pdfcrowd.com/redirect/?url=http%3a%2f%2fdocs.python.org%2fpy3k%2fc-api%2findex.html&id=in-110503223802-94545b90http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.python.org%2fdev%2fpeps%2fpep-0399%2f%23id8&id=in-110503223802-94545b90


3/6


class Spam:"""Tester class which defines no other magic methods but__len__()."""def __len__(self):

return 0

try:c_heapq.heappop(Spam())

except TypeError:# Explicit type check failure: "heap argument must be a list"pass

try:py_heapq.heappop(Spam())

except AttributeError:

# Duck typing failure: "'Foo' object has no attribute 'pop'"pass

This kind of divergence is a problem for users as they unwittingly write code that is CPython-specific. Thisis also an issue for other VM teams as they have to deal with bug reports from users thinking that theyincorrectly implemented the module when in fact it was caused by an untested case.

DetailsStarting in Python 3.3, any modules added to the standard library must have a pure Python implementation.This rule can only be ignored if the Python development team grants a special exemption for the module.Typically the exemption will be granted only when a module wraps a specific C-based library (e.g., sqlite3[5]). In granting an exemption it will be recognized that the module will be considered exclusive to CPythonand not part of Python's standard library that other VMs are expected to support. Usage of ctypes to
http://pdfcrowd.com/http://pdfcrowd.com/customize/http://pdfcrowd.com/license-for-free/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fdocs.python.org%2fpy3k%2flibrary%2fsqlite3.html&id=in-110503223802-94545b90http://pdfcrowd.com/redirect/?url=http%3a%2f%2fwww.python.org%2fdev%2fpeps%2fpep-0399%2f%23id10&id=in-110503223802-94545b90


4/6

pdfcro d comc stomi e free license

provide an API for a C library will continue to be frowned upon as ctypes lacks compiler guarantees that Ccode typically relies upon to prevent certain errors from occurring (e.g., API changes).

Even though a pure Python implementation is mandated by this PEP, it does not preclude the use of acompanion acceleration module. If an acceleration module is provided it is to be named the same as themodule it is accelerating with an underscore attached as a prefix, e.g., _warnings for warnings . Thecommon pattern to access the accelerated code from the pure Python implementation is to import it with an

import * , e.g., from _warnings import * . This is typically done at the end of the module to allow it tooverwrite specific Python objects with their accelerated equivalents. This kind of import can also be donebefore the end of the module when needed, e.g., an accelerated base class is provided but is thensubclassed by Python code. This PEP does not mandate that pre-existing modules in the stdlib that lack apure Python equivalent gain such a module. But if people do volunteer to provide and maintain a purePython equivalent (e.g., the PyPy team volunteering their pure Python implementation of the csv moduleand maintaining it) then such code will be accepted.

This requirement does not apply to modules already existing as only C code in the standard library. It isacceptable to retroactively add a pure Python implementation of a module implemented entirely in C, but inthose instances the C version is considered the reference implementation in terms of expected semantics.

Any new accelerated code must act as a drop-in replacement as close to the pure Python implementationas reasonable. Technical details of the VM providing the accelerated code are allowed to differ asnecessary, e.g., a class being a type when implemented in C. To verify that the Python and equivalent Ccode operate as similarly as possible, both code bases must be tested using the same tests which applyto the pure Python code (tests specific to the C code or any VM do not follow under this requirement). Tomake sure that the test suite is thorough enough to cover all relevant semantics, the tests must have 100%branch coverage for the Python code being replaced by C code. This will make sure that the newacceleration code will operate as much like a drop-in replacement for the Python code is as possible.Testing should still be done for issues that come up when working with C code even if it is not explicitlyrequired to meet the coverage requirement, e.g., Tests should be aware that C code typically has specialpaths for things such as built-in types, subclasses of built-in types, etc.

Acting as a drop-in replacement also dictates that no public API be provided in accelerated code that does
http://pdfcrowd.com/http://pdfcrowd.com/customize/http://pdfcrowd.com/license-for-free/


5/6

df dt i free license

not exist in the pure Python code. Without this requirement people could accidentally come to rely on adetail in the accelerated code which is not made available to other VMs that use the pure Pythonimplementation. To help verify that the contract of semantic equivalence is being met, a module must betested both with and without its accelerated code as thoroughly as possible.

As an example, to write tests which exercise both the pure Python and C accelerated versions of a module,a basic idiom can be followed:

import collections.abcfrom test.support import import_fresh_module, run_unittestimport unittest

c_heapq = import_fresh_module('heapq', fresh=['_heapq'])py_heapq = import_fresh_module('heapq', blocked=['_heapq'])

class ExampleTest(unittest.TestCase):

def test_heappop_exc_for_non_MutableSequence(self):# Raise TypeError when heap is not a# collections.abc.MutableSequence.class Spam:

"""Test class lacking many ABC-required methods(e.g., pop())."""def __len__(self):

return 0

heap = Spam()self.assertFalse(isinstance(heap,

collections.abc.MutableSequence))with self.assertRaises(TypeError):

self.heapq.heappop(heap)

class AcceleratedExampleTest(ExampleTest):

"""Test using the accelerated code."""


6/6

df di f li

heapq = c_heapq

class PyExampleTest(ExampleTest):

"""Test with just the pure Python code."""

heapq = py_heapq

def test_main():run_unittest(AcceleratedExampleTest, PyExampleTest)

if __name__ == '__main__':test_main()

If this test were to provide 100% branch coverage for heapq.heappop() in the pure Python implementationthen the accelerated C code would be allowed to be added to CPython's standard library. If it did not, thenthe test suite would need to be updated until 100% branch coverage was provided before the acceleratedC code could be added.

End Note

This package is not yet complete, but it has enough in it to be useful for writing plug-ins for GIMP. If you write any plug-ins that might be useful as examples,

Pure PythonC Accelerator Module Compatibility Requirements

Documents

Transcript of Pure PythonC Accelerator Module Compatibility Requirements