«Отладка в Python 3.6: Быстрее, Выше, Сильнее» Елизавета...

Debugging in Python 3.6: Better, Faster, Stronger

Elizaveta Shashkova JetBrains

Bio

• Software developer of PyCharm IDE at JetBrains

• Debugger

• Saint Petersburg

2

Debugging

• Adding print statements

• Logging

3

Debugging

• Adding print statements

• Logging

• Special tools: debuggers

4

Debugger’s Performance

5

Run

Debug

Debuggers are 30 times slower

Contents

• Tracing debugger

• Python 3.6

• Frame evaluation debugger

• Results

6

Contents


• Python 3.6


• Results

7

8

def tracefunc(frame, event, arg): print(frame.f_lineno, event) return tracefunc

sys.settrace(tracefunc)

1 2 3 4 5 6

sys.settrace(tracefunc) - set the system tracing function

Tracing Function

def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends)

sys.settrace(tracefunc) foo()

1 2 3 4 5 6 7 8 9

Tracing Function

9

1 2 3 4 5 6 7 8 9

1 call def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends)


Tracing Function

10

1 2 3 4 5 6 7 8 9

1 call 2 line



Tracing Function

11

1 2 3 4 5 6 7 8 9

1 call 2 line 3 line 4 line Hi Bob!



Tracing Function

12

1 2 3 4 5 6 7 8 9

1 call 2 line 3 line 4 line Hi Bob! 3 line 4 line Hi Tom!



Tracing Function

13

1 2 3 4 5 6 7 8 9

1 call 2 line 3 line 4 line Hi Bob! 3 line 4 line Hi Tom! 5 line 5 return



Tracing Function

14

Build Python Debugger

• Breakpoints

• Stepping

15

Tracing Debugger

• Suspend program if breakpoint’s line equals frame.f_lineno

• Handle events for stepping

16

Performance



1 2 3 4 5 6 7 8 9

1 call 2 line 3 line 4 line Hi Bob! 3 line 4 line Hi Tom! 5 line 5 return

17

Example 1

def calculate(): sum = 0 for i in range(10 ** 7): sum += i return sum

1 2 3 4 5 6 7 8 9

18

Example 1


def tracefunc(frame, event, arg): return tracefunc

1 2 3 4 5 6 7 8 9

19

Performance

20

Run

Tracing

Breakpoints

0,80 sec

6,85 sec

19,81 sec

Performance

21

Run

Tracing

Breakpoints

0,80 sec

6,85 sec

19,81 sec

Performance

22

Run

Tracing

Breakpoints

0,80 sec

6,85 sec

19,81 sec

Performance

23

Run

Tracing

Breakpoints

0,80 sec

6,85 sec

19,81 sec

~ 25 times slower!

Problem

• Tracing call on every line

24

Optimizations

25

Optimization 1

• Skip scopes without breakpoints

• frame.f_code.co_name - name of the function

26

Optimization 2

• Rewrite tracing function in C

27

Optimization 2

• Rewrite tracing function in C

• Works only for CPython

• ~ 4 times faster

28

Problem


29

Contents


• Python 3.6


• Results

30

Contents


• Python 3.6


• Results

31

Python 3.6

32

Python 3.6

• New frame evaluation API

• PEP 523

33

PEP 523

• Handle evaluation of frames

• Add a new field to code objects

• C API

34

Frame Evaluation

cdef frame_eval(PyFrameObject *frame, int exc): func_name = <object> frame.f_code.co_name line_number = <object> frame.f_lineno print(line_number, func_name) return _PyEval_EvalFrameDefault(frame, exc)

1 2 3 4 5 6 7 8 9

35

cdef frame_eval(PyFrameObject *frame, int exc): func_name = <object> frame.f_code.co_name f line_number = <object> frame.f_lineno print(line_number, func_name) return _PyEval_EvalFrameDefault(frame, exc)

1 2 3 4 5 6 7 8 9

Frame Evaluation

36


1 2 3 4 5 6 7 8 9

Frame Evaluation

37


1 2 3 4 5 6 7 8 9

Frame Evaluation

38


1 2 3 4 5 6 7 8 9

Frame Evaluation

39


def set_frame_eval(): cdef PyThreadState *state = \ PyThreadState_Get() state.interp.eval_frame = frame_eval

1 2 3 4 5 6 7 8 9

Frame Evaluation

40

Example

1 2 3 4 5 6 7 8 9 10 11

def first(): second()

def second(): third()

def third(): pass

set_frame_eval() first()

41

Example

def first(): second()

def second(): third()

def third(): pass

set_frame_eval() first()

1 2 3 4 5 6 7 8 9 10 11

1 first 4 second 7 third

42

Custom Frame Evaluation

• It works

• Executed while entering a frame

• Access to frame and code object

43

Contents


• Python 3.6


• Results

44

Problem


45

Problem


• Remove the tracing function

46

Replace tracing function with frame evaluation function

47

Contents


• Python 3.6


• Results

48


• Breakpoints

• Stepping

49

Breakpoints

• Access to the whole code object

50

Breakpoints

• Access to the whole code object

• Insert breakpoint’s code into frame’s code

51

Breakpoints

def maximum(a, b): if a > b: return a else: return b

1 2 3 4 5 6 7 8

52

Breakpoints

def maximum(a, b): if a > b: return a # breakpoint else: return b

1 2 3 4 5 6 7 8

53

Breakpoints


1 2 3 4 5 6 7 8

breakpoint()

54

Breakpoints

def maximum(a, b): if a > b: breakpoint() return a # breakpoint else: return b

1 2 3 4 5 6 7 8

55

Python Bytecode


import dis dis.dis(maximum)

1 2 3 4 5 6 7 8

56

Python Bytecode



1 2 3 4 5 6 7 8

2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12

3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE

5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE

57

Python Bytecode



1 2 3 4 5 6 7 8




58

Python Bytecode



1 2 3 4 5 6 7 8




59

Python Bytecode



1 2 3 4 5 6 7 8




60

Python Bytecode



1 2 3 4 5 6 7 8




61

Python Bytecode2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12



62




Python Bytecode

63




Python Bytecode

64




Python Bytecode

65

Bytecode Modification2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12



breakpoint()

66

Bytecode Modification

• Insert breakpoint’s code

• Update arguments and offsets

67

Bytecode Modification

• Insert breakpoint’s code

• Update arguments and offsets

• 200 lines in Python

68

Bytecode Modification2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12



breakpoint()

?!

69

Breakpoint Bytecode

def _stop_at_break(): # a lot of code here

def breakpoint(): _stop_at_break()

1 2 3 4 5 6 7 8

0 LOAD_GLOBAL 0 2 CALL_FUNCTION 0 4 POP_TOP 6 LOAD_CONST 0 8 RETURN_VALUE

70


• Breakpoints

• Stepping

71

Stepping

• Inserting temporary breakpoint on every line

• Use old tracing function

72

Frame evaluation debugger is ready

73

Example 1


1 2 3 4 5

74

Example 1

75

Run

Tracing

Frame evaluation

0,80 sec

19,81 sec

0,81 sec

Example 1

76

Run

Tracing

Frame evaluation

0,80 sec

19,81 sec

0,81 sec

Example 2

def foo(): pass

def calculate(): sum = 0 for i in range(10 ** 7): foo() sum += i return sum

1 2 3 4 5 6 7 8 9

77

Example 2

78

Run

Tracing

Frame evaluation

1,73 sec

43,58 sec

37,41 sec

PEP 523



79

PEP 523

• Expand PyCodeObject struct

• co_extra - “scratch space” for the code object

• Mark frames without breakpoints

80

Mark Frames Cython

1 2 3 4 5 6 7 8 9

81

cdef frame_eval(PyFrameObject *frame, int exc): flag = _PyCode_GetExtra(frame.f_code, index) if flag == NO_BREAKS_IN_FRAME: return _PyEval_EvalFrameDefault(frame, exc)

# check for breakpoints ...

Example 2

82

Run

Tracing

Frame evaluation

1,73 sec

43,58 sec

1,91 sec

PEP 523



83

Contents


• Python 3.6


• Results

84

Contents


• Python 3.6


• Results

85

Real Life Example

86

Real Life Example

• Included into PyCharm 2017.1

• Works in production

87

PyCharm

88

Tracing w/o Cython

Tracing with Cython

Frame evaluation

11,59 sec

5,66 sec

0,28 sec

Disadvantages

• More complicated

• Only with CPython

• Only with Python 3.6

89

Frame Evaluation

• Let’s move to Python 3.6!

90

Frame Evaluation


• Let’s find another use cases

91

Use cases


1 2 3 4 5 6 7 8

breakpoint()

92

PEP 523

• Pyjion project

• JIT for Python

93

Frame Evaluation


• Let’s find another use cases

94

Links

• Prototype: https://github.com/Elizaveta239/frame-eval

• PyCharm Community Edition source code

• bytesinsert on PyPi

95

https://github.com/Elizaveta239/frame-eval

• Prototype: https://github.com/Elizaveta239/frame-eval

• PyCharm Community Edition source code

• bytesinsert on PyPi

Questions?

96

@lisa_shashkova

https://github.com/Elizaveta239/frame-eval

«Отладка в Python 3.6: Быстрее, Выше, Сильнее» Елизавета...

Internet

Transcript of «Отладка в Python 3.6: Быстрее, Выше, Сильнее» Елизавета...