Clayton Parker, Senior Web Developer

Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010

Thursday, October 28, 2010

PLONE CONFERENCE 2010Who Am I?

• claytron

• Python dev since 2003

• Plone Core Committer

• Foundation Member

Thursday, October 28, 2010

PLONE CONFERENCE 2010What Will We Learn?• What is Transmogrifier

• How to use it in Plone

• How to package a migration

Thursday, October 28, 2010

PLONE CONFERENCE 2010Migrations

• One off scripts

• In multiple places

• No re-use

Thursday, October 28, 2010

PLONE CONFERENCE 2010Transmogrifier

• A framework for migrations

• Re-usable parts

Thursday, October 28, 2010

PLONE CONFERENCE 2010Basics

• Pipeline

• Blueprints

• Sources

Thursday, October 28, 2010

PLONE CONFERENCE 2010Pipeline[transmogrifier]pipeline = blog title text text-mimetype date type transitions urlnormalizer path constructor schemaupdate comment-insert comments workflow reindexobject

[blog]blueprint = transmogrify.bloggerfilename = acost.policy.import:export.xml

[title]blueprint = collective.transmogrifier.sections.manipulatorkeys = _transmogrify.blogger.titledestination = string:titledelete = ${title:keys}

[text]blueprint = collective.transmogrifier.sections.manipulatorkeys = _transmogrify.blogger.contentdestination = string:textdelete = ${text:keys}

[text-mimetype]blueprint = collective.transmogrifier.sections.inserterkey = string:text.mimetypevalue = string:text/html

[date]blueprint = collective.transmogrifier.sections.manipulatorkeys = _transmogrify.blogger.published.rfc822destination = string:effectiveDatedelete = ${date:keys}

[type]blueprint = collective.transmogrifier.sections.inserterkey = string:_typevalue = string:BlogEntry

[transitions]blueprint = collective.transmogrifier.sections.inserterkey = string:_transitionsvalue = string:publishcondtion = python:item["_transmogrify.blogger.state"] == "published"

[urlnormalizer]blueprint = plone.app.transmogrifier.urlnormalizersource-key = titledestination-key = string:_id

[path]blueprint = collective.transmogrifier.sections.inserterkey = string:_pathvalue = string:${settings:blog-root}/${item/_id}

[constructor]blueprint = collective.transmogrifier.sections.constructor

[schemaupdate]blueprint = plone.app.transmogrifier.atschemaupdater

[comment-insert]blueprint = collective.transmogrifier.sections.manipulatorkeys = _transmogrify.blogger.commentsdestination = string:_commentsdelete = ${comment-insert:keys}

[comments]blueprint = transmogrify.commentscomment-type = plone.app.discussion

[workflow]blueprint = plone.app.transmogrifier.workflowupdater

[reindexobject]blueprint = plone.app.transmogrifier.reindexobject

[debug]blueprint = collective.transmogrifier.sections.inserterkey = string:debugvalue = python:modules['pdb'].set_trace()

Thursday, October 28, 2010

PLONE CONFERENCE 2010Blueprints

• Python code

• Packages

Thursday, October 28, 2010

PLONE CONFERENCE 2010Sources

• A blueprint

• First item in your pipeline

Thursday, October 28, 2010

PLONE CONFERENCE 2010configure.zcml

<configure xmlns="http://namespaces.zope.org/zope">

<include package="collective.transmogrifier" />

<utility component=".blueprint.MySection" name="my.section" />

</configure>

Thursday, October 28, 2010

PLONE CONFERENCE 2010blueprint.pyfrom zope.interface import implementsfrom zope.interface import classProvidesfrom collective.transmogrifier.interfaces import ISectionfrom collective.transmogrifier.interfaces import ISectionBlueprint

class MySection(object): """A blueprint for importing something into plone """ classProvides(ISectionBlueprint) implements(ISection)

def __iter___(self): # add any other sources into the stream for item in self.previous: # Do something with each item # Pass it on to the next blueprint yield item

Thursday, October 28, 2010

PLONE CONFERENCE 2010Items

• Each item is a mapping

• Keys are fields

• Keys with a leading underscore are controllers

Thursday, October 28, 2010

PLONE CONFERENCE 2010Pipeline Settings

• Use annotations

• Globally imported settings

Thursday, October 28, 2010

PLONE CONFERENCE 2010GenericSetup

• Make migration part of your release

• Ability to package migrations

Thursday, October 28, 2010

PLONE CONFERENCE 2010Package Layout

my.migration!"" __init__.py!"" configure.zcml!"" import#   !"" __init__.py#   !"" configure.zcml#   !"" my_items.csv#   %"" my_migration.cfg%"" profiles %"" default %"" transmogrifier.txt

Thursday, October 28, 2010

PLONE CONFERENCE 2010import/configure.zcml

<configure xmlns="http://namespaces.zope.org/zope" xmlns:transmogrifier="http://namespaces.plone.org/transmogrifier">

<include package="collective.transmogrifier"/> <include package="collective.transmogrifier" file="meta.zcml"/>

<transmogrifier:registerConfig name="my_migration" title="Import items based on CSV file" description="" configuration="my_migration.cfg" />

</configure>

Thursday, October 28, 2010

PLONE CONFERENCE 2010transmogrifier.txt

my_migration

Thursday, October 28, 2010

PLONE CONFERENCE 2010Example Pipeline[transmogrifier]pipeline = csv_file constructor schemaupdater

[csv_file]blueprint = collective.transmogrifier.sections.csvsourcefilename = my.migration.import:my_items.csv

[constructor]blueprint = collective.transmogrifier.sections.constructor

[schemaupdater]blueprint = plone.app.transmogrifier.atschemaupdater

Thursday, October 28, 2010

PLONE CONFERENCE 2010my_items.csv

_path,_type,title,description/folder1,Folder,First Folder,This is folder One/folder2,Folder,Second Folder,This is folder Two/folder1/foo,Document,One Foo,A document named foo/folder2/foo,Document,Two Foo,Another doc named foo

Thursday, October 28, 2010

PLONE CONFERENCE 2010The Result

Thursday, October 28, 2010

PLONE CONFERENCE 2010Stock Blueprints• Constructor section

• Folders section

• Codec section

• Inserter section

• Condition section

• Manipulator section

• Splitter section

• Savepoint section

• CSV source sectionThursday, October 28, 2010

PLONE CONFERENCE 2010Useful Sources and Blueprints• plone.app.transmogrifier

• transmogrify.filesystem

• transmogrify.sqlalchemy

• transmogrify.webcrawler

• wordpress / zine / blogger

Thursday, October 28, 2010

PLONE CONFERENCE 2010plone.app.transmogrifier

• ATSchema updater section

• UID updater section

• Workflow updater section

• Browser default section

• Criterion adder section

• Portal Transforms section

• URL Normalizer section

• Mime encapsulator section

• Indexing sectionThursday, October 28, 2010

PLONE CONFERENCE 2010Links

• collective.transmogrifier

http://pypi.python.org/pypi/collective.transmogrifier/

• plone.app.transmogrifier

http://pypi.python.org/pypi/plone.app.transmogrifier/

Thursday, October 28, 2010

Check out

sixfeetup.com/demos

Thursday, October 28, 2010