EC2 Plone Presentation - Plone Conference Washington DC, October 8,2008
Journeys with Transmogrifier and friends or How not to get stuck in the Plone dark ages
-
Upload
daniel-jowett -
Category
Software
-
view
308 -
download
2
Transcript of Journeys with Transmogrifier and friends or How not to get stuck in the Plone dark ages
![Page 1: Journeys with Transmogrifier and friends or How not to get stuck in the Plone dark ages](https://reader033.fdocuments.us/reader033/viewer/2022042817/55a13d8b1a28ab79188b4593/html5/thumbnails/1.jpg)
Journeys with Transmogrifier & friends
or How not to get stuck in
the Plone dark ages
Daniel JowettJowett Enterprises Ltd
![Page 2: Journeys with Transmogrifier and friends or How not to get stuck in the Plone dark ages](https://reader033.fdocuments.us/reader033/viewer/2022042817/55a13d8b1a28ab79188b4593/html5/thumbnails/2.jpg)
About me
● Software Professional since 1997
● Background in C, Java, SQL, etc …. even P*P
● Using Plone since 2008 on starting self-employment
● Once upon a time...
● … I came to a Plone Conference in Bristol...
● … in 2010
● And everyone seemed to be cleverer than me
● And now 4 years later? ...
● I know most of them are cleverer than me
● But I have done 4 or 5 projects with Transmogrifer / Jsonmigrator
![Page 3: Journeys with Transmogrifier and friends or How not to get stuck in the Plone dark ages](https://reader033.fdocuments.us/reader033/viewer/2022042817/55a13d8b1a28ab79188b4593/html5/thumbnails/3.jpg)
Foundational Principles
● Calvin & Hobbes
● Calvin is an over-imaginative 6 year old
● Hobbes is his toy tiger
● A cardboard box can be anything
● But in this case it's a “Transmogrifier”
![Page 4: Journeys with Transmogrifier and friends or How not to get stuck in the Plone dark ages](https://reader033.fdocuments.us/reader033/viewer/2022042817/55a13d8b1a28ab79188b4593/html5/thumbnails/4.jpg)
Transmogrifier: ...what is it?
● Transmogrifier – turns something into anything elselike Calvin into a toad, or tiger
● Plone's Transmogrifier: a way to migrate content
– typically between Plone versions
– also from plain html sites & other CMS databases
– Aims for and achieves a decent amount of code reuse● JsonMigrator: Builds on top of Transmogrifier so you don't have to
install it in the source Plone site
![Page 5: Journeys with Transmogrifier and friends or How not to get stuck in the Plone dark ages](https://reader033.fdocuments.us/reader033/viewer/2022042817/55a13d8b1a28ab79188b4593/html5/thumbnails/5.jpg)
Transmogrifier Variants
● Plain Transmogrifier – 2 stage process to export/import from Plone site via xml files (or csv files)
● Funnelweb - Crawls and parses static sites for import, which are then 'pushed' into Plone
● Jsonmigrator – 1 stage process that crawls Json views of an old site
● Transmogrifier blueprints also exist that can read from SQL Dbs, so for example Drupal data
We will be focussing on Jsonmigrator
![Page 6: Journeys with Transmogrifier and friends or How not to get stuck in the Plone dark ages](https://reader033.fdocuments.us/reader033/viewer/2022042817/55a13d8b1a28ab79188b4593/html5/thumbnails/6.jpg)
When to use Jsonmigrator
● From old Plone versions, like 2.5, but even as far back as 1.0 in theory (but requires Python >= 2.2)
● Even from old Zope/CMF sites
● particularly when you have no buildout
● When changing content type framework from archetypes to dexterity
● To clean cruft out of an error prone old site
● When scared to upgrade because you don't know what might bite you later
Note: This isn't the whole story - you still have to Upgrade Your Plone products & custom code
![Page 7: Journeys with Transmogrifier and friends or How not to get stuck in the Plone dark ages](https://reader033.fdocuments.us/reader033/viewer/2022042817/55a13d8b1a28ab79188b4593/html5/thumbnails/7.jpg)
When NOT to use Transmogrifier
● From Plone 4.x
● Probably not from 3.3.6 with buildout
● Without considering the alternatives:
– running Plone upgrade steps (plone.app.upgrade)– plone.app.contenttypes migration for AT to DX based types
![Page 8: Journeys with Transmogrifier and friends or How not to get stuck in the Plone dark ages](https://reader033.fdocuments.us/reader033/viewer/2022042817/55a13d8b1a28ab79188b4593/html5/thumbnails/8.jpg)
Technology Stack
● Transmogrifier:
– plone.app.transmogrifier
– collective.transmogrifier● collective.jsonify
● collective.jsonmigrator
● Products containing extra pipeline sections, such as:
– transmogrify.dexterity
– quintagroup.transmogrifier
– transmogrify.print
![Page 9: Journeys with Transmogrifier and friends or How not to get stuck in the Plone dark ages](https://reader033.fdocuments.us/reader033/viewer/2022042817/55a13d8b1a28ab79188b4593/html5/thumbnails/9.jpg)
Setting Up Jsonmigrator
* These steps are outside the scope of this presentation – hopefully you've got an idea of what to do here (though step 5 may well not be trivial)
1) * Duplicate old Plone in your staging environment
2) install collective.jsonify to this duplicate
3) Add “external methods”
4) * Create a brand new Plone 4.x instance (try 4.3.3 for starters)
5) * Install the updated or replacement products you need to the new instance
6) install collective.jsonmigrator & any special pipeline sections to new instance
![Page 10: Journeys with Transmogrifier and friends or How not to get stuck in the Plone dark ages](https://reader033.fdocuments.us/reader033/viewer/2022042817/55a13d8b1a28ab79188b4593/html5/thumbnails/10.jpg)
Install collective.jsonify
This is the really old way to install stuff & generally bad practise, but if you are using a virtualenv (or dedicated python) it's passable. It's much easier if you're using buildout, but I'll assume that you're not!
1. Download the egg (my version in this case) to your Plone directory:
2. Unzip it
3. Add it to your python (it will also pull in simplejson)
cd collective.jsonify-0.1.8b/../bin/zopepy setup.py install…Adding collective.jsonify 0.1.8b to easy-install.pth file…… simplejson...
unzip 0.1.8b.zip
wget https://github.com/djowett/collective.jsonify/archive/0.1.8b.zip
![Page 11: Journeys with Transmogrifier and friends or How not to get stuck in the Plone dark ages](https://reader033.fdocuments.us/reader033/viewer/2022042817/55a13d8b1a28ab79188b4593/html5/thumbnails/11.jpg)
Add “External Methods”
● Add an Extensions folder
● Add json_methods.py
● Then run your Plone instance, go to the Zope root and create the three equivalently named External Methods, ie: “get_item”, “get_children” & “get_catalog_results”, eg:
from collective.jsonify import get_itemfrom collective.jsonify import get_childrenfrom collective.jsonify import get_catalog_results
![Page 12: Journeys with Transmogrifier and friends or How not to get stuck in the Plone dark ages](https://reader033.fdocuments.us/reader033/viewer/2022042817/55a13d8b1a28ab79188b4593/html5/thumbnails/12.jpg)
Install collective.jsonmigrator
● Add the highlighted lines to your buildout's: develop.cfg
● You might also want to add transmogrify pipelines like transmogrify.dexterity or quintagroup.transmogrifier to your eggs section
● Then run buildout & run up your instance
Note: We check out the collective.jsonmigrator egg for development so that we can tweak the pipeline configuration when we need to
[sources]...collective.jsonmigrator = git https://github.com/djowett/collective.jsonmigrator.git
[buildout]eggs +=... collective.jsonmigrator
![Page 13: Journeys with Transmogrifier and friends or How not to get stuck in the Plone dark ages](https://reader033.fdocuments.us/reader033/viewer/2022042817/55a13d8b1a28ab79188b4593/html5/thumbnails/13.jpg)
Looking closer
● Installing Collective.jsonify & the external methods gives a (fairly) complete json view of every piece of content on your old site
● For example...
● Collective.jsonmigrator can pick up these json data dictionarys, and using transmogrifier 'massage' them and create the corresponding objects on your new site
![Page 14: Journeys with Transmogrifier and friends or How not to get stuck in the Plone dark ages](https://reader033.fdocuments.us/reader033/viewer/2022042817/55a13d8b1a28ab79188b4593/html5/thumbnails/14.jpg)
Have a go!
● Go to: http://<your-plone>:<port>/<instance>/@@jsonmigrator
● Select one of the pre-loaded sample configurations
● Select username, password & a small folder for your first run
![Page 15: Journeys with Transmogrifier and friends or How not to get stuck in the Plone dark ages](https://reader033.fdocuments.us/reader033/viewer/2022042817/55a13d8b1a28ab79188b4593/html5/thumbnails/15.jpg)
Demo
![Page 16: Journeys with Transmogrifier and friends or How not to get stuck in the Plone dark ages](https://reader033.fdocuments.us/reader033/viewer/2022042817/55a13d8b1a28ab79188b4593/html5/thumbnails/16.jpg)
So what happened there? (I)
● This was our “pipeline”
[transmogrifier]pipeline = remotesource removeid strip-leading-path correct-zope-file-types fix_index_html_in_id fix_index_html_in_path print_seq_in_desc description_remove_seq tag_general_if_ccsnews_in_desc tag_service_if_ccsnews_not_in_desc description_remove_ccsnews find_path_depth repl_relative_urls_in_text get_hardcoded_urls_in_text print_hardcoded_url_post_replace get_siteImage_urls_in_text print_siteImage_urls fix_index_html_refs_in_text find_h1_title_in_text
remove_h1_title_from_text constructor schema uid datafields fix_defaultPage# print_defaultpage browser-default# workflow-history get-last-workflow-state calc-workflow-transition publish_all_folders workflowupdater properties# permissions owner local_roles mimetype commit IMPORTING
![Page 17: Journeys with Transmogrifier and friends or How not to get stuck in the Plone dark ages](https://reader033.fdocuments.us/reader033/viewer/2022042817/55a13d8b1a28ab79188b4593/html5/thumbnails/17.jpg)
So what happened there? (II)
● Here are some pipeline section definitions:
[constructor]blueprint = collective.transmogrifier.sections.constructor
# Zope Images & Files get incorrectly reported as Folders [correct-zope-file-types]blueprint = collective.transmogrifier.sections.manipulatorkeys = _classnamedestination = python:"_type"condition = python:(item['_type'] == 'Folder' and item.has_key('_classname')
and item['_classname'] in ['Image', 'File'])[strip-leading-path]blueprint = collective.transmogrifier.sections.inserterkey = string:_pathvalue = python: item['_path'].replace('ccs/', '', 1)
[tag_general_if_ccsnews_in_desc]blueprint = collective.transmogrifier.sections.inserterkey = string:subjectvalue = python: item['subject'] + ['general']condition = python: item['_type'] == "News Item" and 'ccsnews' in item['description']
![Page 18: Journeys with Transmogrifier and friends or How not to get stuck in the Plone dark ages](https://reader033.fdocuments.us/reader033/viewer/2022042817/55a13d8b1a28ab79188b4593/html5/thumbnails/18.jpg)
Real life isn't Blue Peter*
So, even if it did work just now, it will probably break for you first time, so make some adjustments in the pipeline (<instance>/src/collective.jsonmigrator/collective/jsonmigrator/config_plone3.cfg)
● Exclude objects from the import (particularly plone tools)
● Use print a lot (from transmogrify.print)
● Delete the import results (if you had any) & run again
* Blue Peter was the staple TV diet for 1980's Kids in the UK, featuring the oft used phrase “here's one I made earlier” when things didn't work
![Page 19: Journeys with Transmogrifier and friends or How not to get stuck in the Plone dark ages](https://reader033.fdocuments.us/reader033/viewer/2022042817/55a13d8b1a28ab79188b4593/html5/thumbnails/19.jpg)
Benefits compared to Legacy Plone
● A fresh start
● A chance to use new Plone technologies like Diazo, Dexterity, new Collections, plone.app.contenttypes. etc. etc.
● Moving from retired products to cutting edge equivalents
● The speed of Plone 4 & blob storage
● Using buildout to control your site
![Page 20: Journeys with Transmogrifier and friends or How not to get stuck in the Plone dark ages](https://reader033.fdocuments.us/reader033/viewer/2022042817/55a13d8b1a28ab79188b4593/html5/thumbnails/20.jpg)
Caveats
● This doesn't export/import users (though see collective.blueprint.usersandgroups)
● And it doesn't do portlets (portlets can be exported by generic setup, but that doesn't include “deep” content portlets – you might be stuck with a manual portlet setup)
● Collective.jsonify ignores Plone's security model and is a security loophole – add this to a publicly accessible site at your own risk
● The docs can be a little misleading at times
● Many folk using this are probably working off checked out code, so egg releases are few & far between – you are going to have to get your hands dirty
● “Debugging is backwards” in transmogrifier
![Page 21: Journeys with Transmogrifier and friends or How not to get stuck in the Plone dark ages](https://reader033.fdocuments.us/reader033/viewer/2022042817/55a13d8b1a28ab79188b4593/html5/thumbnails/21.jpg)
Credits
● Transmogrifier – Martjin Pieters, Jarn
● Jsonmigrator - Rok Garbas
● RCS – letting me loose on this
● Calvin & Hobbes – Bill Watterson – creating the original Transmogrifier
Links
● http://www.jowettenterprises.com/blog/jsonmigrator
● http://collectivejsonify.readthedocs.org/
● http://collectivejsonmigrator.readthedocs.org/