Transmogrifier

Lennart Regebro

Plone Conference 2009, Budapest

collective.transmogrifier

Created by Martijn Pieters, Jarn

in 2008

Released inversion 1.0

August 2009

collective.transmogrifier

Import/migration framework

Both from Plone 2 and from other websites or data

Super-simple architecture

You set up a pipeline of sections

Each section send items to the next

section.

Sections do something with the

items (or not)

What are you talking about?

A simple example

1. A section that reads a CSV file with a list of events and creates items based on that data

2. A section that will generate some plone-specific data for each item, like path and type.

3. A section that creates the Plone event4. A section that updates the Archetypes fields5. A section that publishes the event

The CSV file

title,startDate,endDatePlone Conference,2009-10-28 09:00,2009-10-30 18:00Christmas,2009-12-24 00:00,2009-12-26:23:59

Setting it up

1. Define a pipeline in a .cfg file

2. Register it as a name in ZCML

3. Call Transmogrifier from Python

The configuration file[transmogrifier]pipeline = section1 section2

[section1]blueprint = name.of.the.blueprintsize = 5

[section2]blueprint = another.blueprint

ZCML

<configure xmlns:transmogrifier= "http://namespaces.plone.org/transmogrifier"> <transmogrifier:registerConfig name="Zap event import" title="Import of events" description="" configuration="transmogrifier.cfg" />

</configure>

setuphandlers.py

from collective.transmogrifier.transmogrifier import \ Transmogrifier

def setupVarious(context):

if context.readDataFile('zap_various.txt') is None: return

transmogrifier = Transmogrifier(context.getSite()) transmogrifier('Zap event import')

What is a generator

def my_generator(): i = 1 while i < 1000: yield i i = i * 2

for x in my_generator(): print x

Never ending story

def my_generator(): i = 1 while True: yield i i = i * 2

for x in my_generator(): print x

EventSource

class EventSource(object): def __iter__(self): for event is csv.DictReader(open(self.options['csvfile'])) yield event

for item in self.previous: yield item

EventSource

class EventSource(object): def __iter__(self): for event is csv.DictReader(open(self.options['csvfile'])) yield event

for item in self.previous: yield item

EventSource

class EventSource(object): def __iter__(self): for event is csv.DictReader(open(self.options['csvfile'])) yield event

for item in self.previous: yield item

The item

{ 'title': 'Plone Conference', 'startDate': '2009-10-27 09:00', 'endDate': '2009-10-30 18:00',}

EventUpdater

class EventUpdater(object): def __iter__(self): for item in self.previous: id = item['title'].lower().replace(' ', '-') item['_path'] = '/events/'+id item['_type'] = 'Event' item['_transitions'] = ('publish',) yield item

The item

{ '_type': 'Event', '_path': '/events/plone-conference', '_transitions': ('publish,), 'title': 'Plone Conference', 'startDate': '2009-10-27 09:00', 'endDate': '2009-10-30 18:00',}

Registering the section

<utility component=".eventsource.EventSource" name="zap.eventsource" />

<utility component=".eventsource.EventUpdater" name="zap.eventupdater" />

The configuration file[transmogrifier]pipeline = eventsource eventupdater

[eventsource]blueprint = zap.eventsourcecsvfile = /path/to/events.csv

[eventupdater]blueprint = zap.eventupdater

The configuration file cont.

[constructor]blueprint = collective.transmogrifier.sections.constructor

[schemaupdater]blueprint = plone.app.transmogrifier.atschemaupdater

[workflow]blueprint = plone.app.transmogrifier.workflowupdater

The configuration file cont.

[transmogrifier]pipeline = eventsource eventupdater constructor schemaupdater workflow

SectionsCreate items

Modify items

Drop items

Split the pipeline in two

Construct content

Modify content

Migrating from another CMS

Similar to the simple examplebut using SQL instead of CSV

Migration from Plone 2

1. Write an export script2. Write an import section

Migrate from Plone 2

importsourceconstructor

criterionadderschemaupdaterbrowserdefault

workflowsavepoint

Migrating from HTML

1. Get HTML from web

2. Extract fields like title, publish date, etc.

3. Make all urls either relative or into UID's

From small to huge

Largest migration:28 sections

527 lines of configuration16 custom sections

Caveat Emptor

Don't migrate from HTML on disk

Caveat Emptor

Newly created objects are not indexed.

Caveat Emptor

Debugging is backwards

Lennart Regebro

regebro@gmail.com

IRC nick: regebro