PiterPy#3. DSL in Python. How and why?
-
Upload
ivan-tsyganov -
Category
Technology
-
view
385 -
download
2
Transcript of PiterPy#3. DSL in Python. How and why?
![Page 1: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/1.jpg)
DSL в Python.
Цыганов Иван Positive Technologies
Как и зачем?
![Page 2: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/2.jpg)
– Мартин Фаулер
Предметно ориентированный язык — это язык программирования с ограниченными выразительными возможностями, ориентированный на некую конкретную предметную область
![Page 3: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/3.jpg)
SQL
SELECT * FROM Users WHERE Age>20
![Page 4: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/4.jpg)
SQL
SELECT * FROM Users WHERE Are>20 REGEXP
[A-Z][A-Za-z0-9]*
![Page 5: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/5.jpg)
SQL REGEXP
[A-Z]\w+ LaTeX
E &=& mc^2\\
![Page 6: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/6.jpg)
SQL REGEXPLaTeX
E &=& mc^2\\HTML
<a href='http://piterpy.ru'>PiterPy</a>
![Page 7: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/7.jpg)
SQL REGEXPLaTeXHTML
<a href='http://piterpy.ru'>PiterPy</a>PonyORM
select(p for p in Person if p.age > 20)
![Page 8: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/8.jpg)
SQL REGEXPLaTeXHTML
PonyORMWTForms
class UsernameForm(Form): username = StringField('Username')
![Page 9: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/9.jpg)
✤ SQL✤ REGEXP✤ TeX/LaTeX✤ HTML
Виды DSL
DSL
ВнутренниеВнешние✤ PonyORM✤ WTForm✤ Django models
![Page 10: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/10.jpg)
Высокая скорость разработки
import re html_source = '''...'''links = re.findall( pattern=r'''<a href=("|')(?P<URL>.*?)\1>(?P<Text>.*?)</a>''', string=html_source
![Page 11: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/11.jpg)
Дополнительный уровень абстракции
persons = select(p for p in Person if p.age > 20)[:]
![Page 12: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/12.jpg)
Код могут писать "не-программисты"
server { location / { proxy_pass http://localhost:8080/; }
location ~ \.(gif|jpg|png)$ { root /data/images; }}
![Page 13: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/13.jpg)
Проблемы и решения
Высокая стоимость разработки DSL
![Page 14: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/14.jpg)
Проблемы и решения
Достаточно один раз разобраться
Высокая стоимость разработки DSL
![Page 15: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/15.jpg)
Проблемы и решения
Достаточно один раз разобраться
Высокая стоимость разработки DSL
Нет специалистов со знанием языка
![Page 16: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/16.jpg)
Проблемы и решения
Достаточно один раз разобраться
Высокая стоимость разработки DSL
Язык должен иметь ограниченные возможности
Нет специалистов со знанием языка
![Page 17: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/17.jpg)
Проблемы и решения
Достаточно один раз разобраться
Высокая стоимость разработки DSL
Язык должен иметь ограниченные возможности
Нет специалистов со знанием языка
Работает не для всех задач
![Page 18: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/18.jpg)
Проблемы и решения
Достаточно один раз разобраться
Высокая стоимость разработки DSL
Язык должен иметь ограниченные возможности
Нет специалистов со знанием языка
Стоит попробовать, что бы понять
Работает не для всех задач
![Page 19: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/19.jpg)
Перейдем к практике
![Page 20: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/20.jpg)
Зачем нужна модель
✤ Хранит в себе всю бизнес-логику
✤ Позволяет использовать различные DSL
✤ Обеспечивает возможность тестирования
![Page 21: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/21.jpg)
Семантическая модель
![Page 22: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/22.jpg)
Внутренние DSL
![Page 23: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/23.jpg)
Внутренние DSL
Все возможности базового языка
Привычный синтаксис
Легко работать
Ограничен базовым языком
![Page 24: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/24.jpg)
Цепочки вызовов
✤ Все методы заполняют модель и возвращают объект
✤ Методы именуются исходя из смыслового контекста
FileUpdater()\ .path('\music')\ .mask('.*metallica.*')\ .set(Genre='Rock')\ .set(Artist='Metallica'\ .do()
![Page 25: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/25.jpg)
Вложенные функции
✤ Для заполнения модели вызываются функции
✤ Функции именуются исходя из смыслового контекста
update( settings( path('./music'), mask('.*\.mp3') ), set(Artist='Metallica'), set(Genre='Rock') )
![Page 26: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/26.jpg)
Import hooks
Очень мощный инструмент
Вмешиваемся в работу интерпретатора
Очень сложная отладка
Непредсказуемые side-эффекты
![Page 27: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/27.jpg)
Import hooks
Не используйте это в реальном мире!
![Page 28: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/28.jpg)
PathFinder
FileLoader
import requests
def source_to_code(self, data, path, *, _optimize=-1): … source = self.get_source(path) return compile( source, path, … )
![Page 29: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/29.jpg)
WITH ".*\.mp3"IN "../tests/music/"SET Artist="Metallica"SET Genre="Rock"
my_script.py
import internal.import_tokenizerimport examples.internal_data.my_script as scriptscript.task.process_rules()
![Page 30: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/30.jpg)
import my_script as script
PathFinder
FileLoader
def source_to_code(self, data, path, *, _optimize=-1): … tokens = translate(path) return compile( tokenize.untokenize(tokens), path, … )
![Page 31: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/31.jpg)
def translate(path): tokens = tokenize.generate_tokens(…) while tokens: ... if token_value in ('IN', ‘WITH'): ... yield from create_task() ... elif ... else: yield (tok_type, value)
![Page 32: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/32.jpg)
def translate(path): tokens = tokenize.generate_tokens(…) while tokens: ... if token_value in ('IN', ‘WITH'): ... yield from create_task() ... elif ... else: yield (tok_type, value)
yield from create_task()
![Page 33: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/33.jpg)
def create_task(): yield from [ (tokenize.NAME, 'from'), (tokenize.NAME, 'model'), (tokenize.NAME, 'import'), (tokenize.NAME, 'Task'), (tokenize.OP, ','), (tokenize.NAME, 'Rule'), (tokenize.NEWLINE, '\n'), (tokenize.NAME, 'task'), (tokenize.OP, '='), (tokenize.NAME, 'Task'), (tokenize.OP, '('), (tokenize.OP, ')'), (tokenize.NEWLINE, '\n') ]
![Page 34: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/34.jpg)
def create_task(): yield from [ (tokenize.NAME, 'from'), (tokenize.NAME, 'model'), (tokenize.NAME, 'import'), (tokenize.NAME, 'Task'), (tokenize.OP, ','), (tokenize.NAME, 'Rule'), (tokenize.NEWLINE, '\n'), (tokenize.NAME, 'task'), (tokenize.OP, '='), (tokenize.NAME, 'Task'), (tokenize.OP, '('), (tokenize.OP, ')'), (tokenize.NEWLINE, '\n') ]
![Page 35: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/35.jpg)
DSL Ruby
task = Task.new do with '*.\.mp3' inside './music' rule do Artist 'Metallica' end rule do Genre 'Rock' endendtask.run()
![Page 36: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/36.jpg)
Внешние DSL
![Page 37: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/37.jpg)
Внешние DSL
Нет базового языка
Сами выбираем синтаксис
Нет базового языка
Необходимо разрабатывать анализаторы
![Page 38: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/38.jpg)
Свой язык
WITH ".*\.mp3" IN "./music" SET Artist="Metallica" SET Genre="Rock"
![Page 39: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/39.jpg)
Python Lex-Yacc (PLY)
![Page 40: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/40.jpg)
ply.lex
ply.yacc
ASTRunCode
source
![Page 41: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/41.jpg)
ply.lexWITH ".*\.mp3" IN "./music" SET Artist="Metallica" SET Genre="Rock"
Type ValueWITH WITHIN INSET SET
EQUALS =VALUE \".*?\"
ATTRIBUTE [A-Za-z][A-Za-z0-9]*
![Page 42: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/42.jpg)
ply.lexWITH ".*\.mp3" IN "./music" SET Artist="Metallica" SET Genre="Rock"
SET EQUALS
SET Artist="Metallica"
ATTRIBUTE
Artist
VALUE
"Metallica"
![Page 43: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/43.jpg)
ply.lex WITH Artist IN "Metallica"
LexToken(WITH,'WITH',1,0) LexToken(ATTRIBUTE,'Artist',1,5) LexToken(IN,'IN',1,12) LexToken(VALUE,'"Metallica"',1,15)
![Page 44: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/44.jpg)
ply.yaccWITH ".*\.mp3" IN "./music" SET Artist="Metallica" SET Genre="Rock"
rule : SET ATTRIBUTE EQUALS VALUE with : WITH VALUE in : IN VALUE
rule_list : rule_list rule | rule
task : with in rule_list | in rule_list
![Page 45: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/45.jpg)
PLYГенерируем AST
![Page 46: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/46.jpg)
ply.yaccWITH ".*\.mp3" IN "./music" SET Artist="Metallica" SET Genre="Rock"
simple_token = namedtuple( 'simple_token', ['Name', 'Value'] ) def p_rule(self, p): '''rule : SET ATTRIBUTE EQUALS VALUE''' p[0] = simple_token(Name='RULE', Value=(p[2], p[4]))
![Page 47: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/47.jpg)
ply.yaccWITH ".*\.mp3" IN "./music" SET Artist="Metallica" SET Genre="Rock"
Task
With In RuleList
Rule Rule.*\.mp3 "./music"
Artist Genre"Metallica" "Rock"
![Page 48: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/48.jpg)
PLYЗаполняем модель
![Page 49: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/49.jpg)
ply.yaccWITH ".*\.mp3" IN "./music" SET Artist="Metallica" SET Genre="Rock"
def p_rule(p): '''rule : SET ATTRIBUTE EQUALS VALUE''' p[0] = Rule(**{p[2]: p[4]})
![Page 50: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/50.jpg)
ply.yacc
Taskroot_dir = "./music"file_mask = ".*\\.mp3"rules = [
]
RuleArtist = "Metallica"
RuleGenre = "Rock"
WITH ".*\.mp3" IN "./music" SET Artist="Metallica" SET Genre="Rock"
![Page 51: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/51.jpg)
PLY
Гибкость
Отладка
Обработка ошибок
Понятный код библиотеки
Высокий порог входа
Многословный
![Page 52: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/52.jpg)
funcparserlib
![Page 53: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/53.jpg)
funcparserlibWITH ".*\.mp3" IN "./music" SET Artist="Metallica" SET Genre="Rock"
task = Task()root = keyword('In') + value_of('Value') >> set_rootmask = keyword('With') + value_of('Value') >> set_maskrule = keyword('Set') + \ value_of('Attribute') + \ keyword('Equals') + \ value_of('Value') \ >> make_ruleparser = maybe(mask) + root + many(rule)parser.parse(source)
![Page 54: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/54.jpg)
funcparserlibIN "./music" WITH ".*\.mp3" SET Artist="Metallica" SET Genre="Rock"
get_value = lambda x: x.valuevalue_of = lambda t: some(lambda x: x.type == t) >> get_valuekeyword = lambda s: skip(value_of(s))set_root = lambda value: task.set_root_dir(value[1:-1]) set_mask = lambda value: task.set_mask(value[1:-1]) make_rule = lambda x: task.add_rule(Rule(**{x[0]: x[1]}))
![Page 55: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/55.jpg)
funcparserlib
Компактный
Гибкий
Для любителей функционального программирования :)
Многое приходится делать руками
Для любителей функционального программирования :)
![Page 56: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/56.jpg)
pyparsing
![Page 57: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/57.jpg)
pyparsingWITH ".*\.mp3" IN "./music" SET Artist="Metallica" SET Genre="Rock"
rule = ( Keyword('SET') + Word(alphanums)('key') + '=' + QuotedString('"')('value') ).setParseAction(lambda r: {r.key: r.value})
>>> rule.parseString('SET Artist="Metallica"') {'Artist': 'Metallica'}
![Page 58: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/58.jpg)
pyparsingWITH ".*\.mp3" IN "./music" SET Artist="Metallica" SET Genre="Rock"
{ 'mask': '.*\.mp3', 'root_dir': './music', 'rules': [ {'Artist': 'Metallica'}, {'Genre': 'Rock'} ]}
![Page 59: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/59.jpg)
pyparsing
Понятная грамматика
Базовые компоненты
Свои компоненты
Постобработка каждого элемента
Документация
Отладка
![Page 60: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/60.jpg)
Если сомневаетесь в необходимости DSL - попробуйте
Начните с семантической модели, это ведь просто библиотека
![Page 61: PiterPy#3. DSL in Python. How and why?](https://reader038.fdocuments.us/reader038/viewer/2022103010/58ee857e1a28aba0218b4645/html5/thumbnails/61.jpg)
mi.0-0.im