Post on 16-Apr-2017
PowerPoint Presentation
Content Migration:Quantum Leap
Plone Conference 2008Washington, DCOctober 10, 2008Vitaliy Podobahttp://quintagroup.com
Good day!Now I'm going to present our approach to migration in Plone.
But before that, I'd like to introduce myself.
Vitaliy PodobaPlone Developer at Quintagroup.compiv (plone.org)vipod (#plone irc channel)
Who am I?
My name is Vitaliy Podoba.
I'm Zope/Plone developer at Quintagroup
'piv' this is my user name on plone.org
And 'vipod' this is my nickname at #plone irc channel.
As we all know Plone is celebrating his 7th birthday these days. So Happy Birthday Plone:-)
Plone Evolution
And all this time (7 years) Plone people were working hard to
implement more and more Plone sites.
As time was passing not all sites were regularly upgraded to fresh
Plone versions. And this is natural process.
But with the recent brand new superb Plone 3 release, customers
are willing to move their old 2.0, 2.1, 2.5 websites to new Plone
3.
And this is also natural. Because, as you see, Plone 3 is really
good;-)
So, what to do?
Plone Portal Migrationhttp://plone.org/documentation/manual/upgrade-guide
For the time being the main approach for Plone migration is its
native built-in migration procedure. This approach is vital for us
and it's difficult to overvalue it. We have been using it since the
first Plone releases and we are sure that we will stillbe using it
for quite a while. Plone portal migration is very well described in
Plone migration Guide at plone.org.
But there are several rough edges in built-in plone migration
mechanism in terms of content migration.
Plone Portal Migrationcan failloose controliterative (step-by-step) processno way back
1. The first and the worst thing about plone portal migration is
that it can fail. It especially concerns very customized and really
old Plone versions like 2.0 or 2.1.Plone is a very flexible system,
but when migrating this will affect the outcome based on what
changes you have made to your system.
For example:
* If you have a standard Plone site with simple customizations, it
will likely work very well.
* But if you have installed and depend on a lot of third-party
products produced by different developers, it's hard to say
something definite
The migration tool handles most cases, but your mileage may vary.
Heavily customized sites should take into account some extra time
to do the transition.
2. Site Upgrades currently, utilize a function registry to enable
upgrades, this process is labour-intensive and not introspectable
with regards to the changes it makes to a site. So this is another
rough edge in portal migration, called here as loose control.
3. You can not migrate, for example, from plone 2.1 to plone 3.1
omitting plone 2.5 version during the migration. You can do the
migration only step-by-step.
4. Moreover, built-in migration procedure doesn't know how to
migrate from newer Plone to the older one. So you can't take a step
back.
Use Case: CMS.Info
http://www.contentmanagementsoftware.info
And now a few words about our use case which actually involved us in a Quantum Leap session.
We decided to test the content migration strategy on one of our well known websites: contentmanagementsoftware.info, which contains listings of Plone and Zope products.
From now on I will call it just CMS.Info..
Historically the website was migrated from Plone 2.0.5 to Plone 2.1 long time ago. Content was converted from regular CMF to Archetypes, leaving a lot of cruft in a database.
So we had to migrate from Plone 2.1 to Plone 3.1. And we took care of the content only. We didn't care about existing site's functionality, settings or preferences, because we wanted to replace existing functionality and features with new ones.
And what is more: a) CMS.Info portal had it's own blog, built using SimpleBlog product which isn't supported in Plone 3; b) and CMS.Info site had a few forms created with a help of PloneFormMailer which is also not compatible with Plone 3.
The last two tasks could definitely not be accomplished using standard plone migration.
That's why CMS.Info was ideal use case for Quantum Leap content migration.
Quantum Leap
This is a photo from my last mountains trip. Here I'm doing my
quantum leap ;-) Taking off from Plone 2.1 version, flying over
Plone 2.5 and landing on Plone 3.1 version. This all is true about
our use case. We migrated content from Plone 2.1 to Plone 3.1
without intermediate, version by version steps.
That is why this session is called Quantum Leap. It's about a large
and significant jump forward from Plone 2.1 to Plone 3.1 without
intermediate major 2.5 version.
This approach can be well covered with a help of GenericSetup
product, introduced by Tres Seaver.
Aside:
In physics, a quantum leap or quantum jump is a change of an
electron from one energy state to another within an atom.. In real
physical systems a quantum leap is not necessarily a large change,
and can in fact be very insignificant.. In the popular sense, the
term is usually applied to mean a large or significant change,
which is thus not strictly correct.
GenericSetupTres Seaver
And now a short introduction to GenericSetup. Just in a few
words for those who don't know what it is.
From product's README.txt: This product provides a mini-framework
for expressing the configured state of a Zope Site as a set of file
system artefacts. These artefacts consist of declarative XML files,
which spell out the configuration settings for each "tool" in the
site.GenericSetup introduces the idea of "setup profiles", which
are a collection of XML files that describe different aspects of
the site configuration.
But actually we need only one step from GenericSetup tool called
'Content' which is responsible for exporting and importing site
content.
Now let's consider why GenericSetup approach is so good in our
case.
Content Migrationmigrate only contentclean databasetransform portal type on-the-flymanipulate/transform an exported XMLtransfer content back and forwardand between different Plone versions
1. First of all, as we are not willing to migrate any portal
settings or other functionality, we care only about content. Thus,
we won't be doing unnecessary work.
2. We will acquire a clean, created from scratch database, without
any trash which was possibly collected by previous
migrations.
3. With this migration we will be able to transform one portal type
into another one, if it's necessary. And I know, it really is,
because we have some content based on no longer supported Plone
products.
4. After export we will have a bunch of xml files which we will be
able to manipulate/transform and import them back in as a new site.
We will also be able to even exchange site content with other
content consumers.
5. This approach is reciprocal. Saying it in other words it means
that anything that can be exported can be re-imported.
6. And it could be done between different Plone versions.
I think now we have sufficient amount of arguments to start our
investigation of what we have in Plone 'out-of-the-box'.
But at this point I should say that unfortunately nothing is
perfect. Neither is Plone's export/import story.
Nothing is perfect
PSPS 7826: content import/exporthttp://thread.gmane.org/
gmane.comp.web.zope.plone.devel/
18968
As a proof of the fact that not everything is good with
export/import in Plone, here is a link to plone's devel thread as
for PSPS 7826 called 'content import/export'. And it is still in
progress.
This means that at the moment Plone doesn't provide us with a solid
content migration solution.
But anyway, let's consider what does it provides
.
CMFPloneGenericSetupCMFCoreCMFTopicArchetypes & ATContentTypesMarshall
What do we have?
Well, Plone together with GenericSetup, CMFCore, CMFTopic,
Archetypes and Marshall product furnish us with the next
things:
AT content types:FolderATDocumentATNewsItem
AT schema fields:+ UID+ schema fields-- reference fields--
image/file fields
CMF:typeworkflow historylocal roles
We can migrate 'out-of-the-box' Folder, ATDocument and
ATNewsItem. And that is all about content types.
These content types will be handled with almost all their
archetypes schema fields and unique id. You would probably ask why
almost all - but not all? Well, reference fields, due to some
unknown for me reasons, are simply skipped by Marshall during
content export.
And file fields are not fetched properly either. It's because
archetypes file field's accessor does not return file data, but
only url to that file.
Apart from archetypes fields Marshall will process content object's
type, workflow history and local roles for us.
That's all what we have by default.As you can see , this list is
far from being complete. A lot of important issues are still
missing.
The main task
or
what is missingtransfer all standard types
including topics with criteriatransfer AT Referencestransfer AT
File fieldstransfer Properties (PropertyManager)transfer Comments
(Disscussion items)
Here is a list of different issues that we needed to carry out
to migrate CMS.Info
As you see, the main task here is to make it possible to migrate
all the standard plone content types, like events, links, images,
files, etc.
We don't have any mechanism to export/import AT references.
Also it is vital to fix a problem with AT file fields accessor
which doesn't want to return us file data.
We have qSEOptimizer installed in our CMS.Info. This product
intensively uses object properties (PropertyManager).
And we have a lot of objects with custom qSEOptimizer properties.
So it is important for us to transfer such properties too.
Last but not the least, we don't have any migration handlers for
comments, also known as discussion items.
After we figure out what do we need we can look at the possible
existing solutions.
Possible Solutionscollective.plone.gsxmlRamon BartlStefan
Eletzhofer
collective.transmogrifier
plone.app.transmogrifierMartijn Pieters
It seems like these two packages are the best we have for
content import/export.
The first one collective.plone.gsxml was written by Ramon Bartl and
Stefan Eletzhofer. The most important thing about this package is
that it allows to export/import at least all standard plone AT
based content. This package relies pretty much on the Marshall. So
the problems with Marshall will persist with gsxml too. And in
addition this package can transfer AT references.
Another one is collective.transmogrifier, written by Martijin
Pieters, is really well designed and it's easy to extend it. But at
the moment transmogrifier does not have any
sections/handlers/sources which can help us to export content.
Instead it contains a lot of different useful import
hanlders.
That's why finally we choose transmogrifier as a base for our
Quantum Leap content migration approach.
So thanks a lot to Martijn Pieters for it's great
collective.transmogrifier package.
And of course thanks to Ramon Bartl and Stefan Eletzhofer for their
work from what we could benefit.
And finally the result looks like this:
The Solutionquintagroup.transmogrifierhttp://svn.quintagroup.com/products
This package provides a lot of extension handlers for
collection.transmogrifier like:
. PropertyManager property handler
handler for discussion items
references file fields topics with criteria etc.
But this is not all we managed to accomplish...
Some extra tasksSimpleBlog to QuillsPloneFormMailer to PloneFormGenmigrate Users and Membersmigrate portletstransfer RedirectionTool settings
We had a few more extra tasks during the migration.
The first task was to migrate SimpleBlog content to Quills.
The second one was similar: migrate PloneFormMailer forms to
PloneFormGen forms.
Of course we had to move Users and Members.
We also had to migrate old style portlets to plone 3 ones, and
transfer RedirectionTool settings.
For the first two items we created two more packages:
Extra Solutionsquintagroup.transmogrifier.simpleblog2quillsquintagroup.transmogrifier.pfm2pfgother.packages.follow
So if you need to migrate blogs based on SimpleBlog product to
Quills content then it will be useful to use
quintagroup.transmogrifier.simpleblog2quills package.
To move PloneFormMailer forms to PloneFormGen ones just use
quintagroup.transmogrifier.pfm2pfg.
All these packages are available at
http://svn.quintagroup.com/products svn repository.
We hope these packages will be useful for you. And we are looking
forward to get some feedback from you to know what to improve, what
to fix and what if missing.
Roadmapcontent versionslocal permission settingsblobssomething
else?
To conclude, I would mention what is missed right now in our
solution.
These are: content versions, local permission settings,
blobs.
And I think there are a lot of other different things which I don't
remember or don't even know about ;-)
LinksPlone Mailing List
http://thread.gmane.org/gmane.comp.web.zope.plone.devel/18968 Plone
Feature Request
http://dev.plone.org/plone/ticket/7826 CMS.Info
http://www.contentmanagementsoftware.infocollective.transmogrifier
http://svn.plone.org/svn/collective/collective.transmogrifiercollective.plone.gsxml
http://pypi.python.org/pypi/collective.plone.gsxml/0.4.5Quintagroup
Repository
http://svn.quintagroup.com/products
Here I listed a few links mentioned previously during this
session.
Questions
Time for Questions.
and Thank you!
Good day!Now I'm going to present our approach to migration in Plone.
But before that, I'd like to introduce myself.My name is Vitaliy
Podoba.
I'm Zope/Plone developer at Quintagroup
'piv' this is my user name on plone.org
And 'vipod' this is my nickname at #plone irc channel.
As we all know Plone is celebrating his 7th birthday these days.
So Happy Birthday Plone:-)And all this time (7 years) Plone people
were working hard to implement more and more Plone sites.
As time was passing not all sites were regularly upgraded to fresh
Plone versions. And this is natural process.
But with the recent brand new superb Plone 3 release, customers
are willing to move their old 2.0, 2.1, 2.5 websites to new Plone
3.
And this is also natural. Because, as you see, Plone 3 is really
good;-)
So, what to do?
For the time being the main approach for Plone migration is its
native built-in migration procedure. This approach is vital for us
and it's difficult to overvalue it. We have been using it since the
first Plone releases and we are sure that we will stillbe using it
for quite a while. Plone portal migration is very well described in
Plone migration Guide at plone.org.
But there are several rough edges in built-in plone migration
mechanism in terms of content migration.
1. The first and the worst thing about plone portal migration is
that it can fail. It especially concerns very customized and really
old Plone versions like 2.0 or 2.1.Plone is a very flexible system,
but when migrating this will affect the outcome based on what
changes you have made to your system.
For example:
* If you have a standard Plone site with simple customizations, it
will likely work very well.
* But if you have installed and depend on a lot of third-party
products produced by different developers, it's hard to say
something definite
The migration tool handles most cases, but your mileage may vary.
Heavily customized sites should take into account some extra time
to do the transition.
2. Site Upgrades currently, utilize a function registry to enable
upgrades, this process is labour-intensive and not introspectable
with regards to the changes it makes to a site. So this is another
rough edge in portal migration, called here as loose control.
3. You can not migrate, for example, from plone 2.1 to plone 3.1
omitting plone 2.5 version during the migration. You can do the
migration only step-by-step.
4. Moreover, built-in migration procedure doesn't know how to
migrate from newer Plone to the older one. So you can't take a step
back.
And now a few words about our use case which actually involved us in a Quantum Leap session.
We decided to test the content migration strategy on one of our well known websites: contentmanagementsoftware.info, which contains listings of Plone and Zope products.
From now on I will call it just CMS.Info..
Historically the website was migrated from Plone 2.0.5 to Plone 2.1 long time ago. Content was converted from regular CMF to Archetypes, leaving a lot of cruft in a database.
So we had to migrate from Plone 2.1 to Plone 3.1. And we took care of the content only. We didn't care about existing site's functionality, settings or preferences, because we wanted to replace existing functionality and features with new ones.
And what is more: a) CMS.Info portal had it's own blog, built using SimpleBlog product which isn't supported in Plone 3; b) and CMS.Info site had a few forms created with a help of PloneFormMailer which is also not compatible with Plone 3.
The last two tasks could definitely not be accomplished using standard plone migration.
That's why CMS.Info was ideal use case for Quantum Leap content
migration.This is a photo from my last mountains trip. Here I'm
doing my quantum leap ;-) Taking off from Plone 2.1 version, flying
over Plone 2.5 and landing on Plone 3.1 version. This all is true
about our use case. We migrated content from Plone 2.1 to Plone 3.1
without intermediate, version by version steps.
That is why this session is called Quantum Leap. It's about a large
and significant jump forward from Plone 2.1 to Plone 3.1 without
intermediate major 2.5 version.
This approach can be well covered with a help of GenericSetup
product, introduced by Tres Seaver.
Aside:
In physics, a quantum leap or quantum jump is a change of an
electron from one energy state to another within an atom.. In real
physical systems a quantum leap is not necessarily a large change,
and can in fact be very insignificant.. In the popular sense, the
term is usually applied to mean a large or significant change,
which is thus not strictly correct.
And now a short introduction to GenericSetup. Just in a few
words for those who don't know what it is.
From product's README.txt: This product provides a mini-framework
for expressing the configured state of a Zope Site as a set of file
system artefacts. These artefacts consist of declarative XML files,
which spell out the configuration settings for each "tool" in the
site.GenericSetup introduces the idea of "setup profiles", which
are a collection of XML files that describe different aspects of
the site configuration.
But actually we need only one step from GenericSetup tool called
'Content' which is responsible for exporting and importing site
content.
Now let's consider why GenericSetup approach is so good in our
case.
1. First of all, as we are not willing to migrate any portal
settings or other functionality, we care only about content. Thus,
we won't be doing unnecessary work.
2. We will acquire a clean, created from scratch database, without
any trash which was possibly collected by previous
migrations.
3. With this migration we will be able to transform one portal type
into another one, if it's necessary. And I know, it really is,
because we have some content based on no longer supported Plone
products.
4. After export we will have a bunch of xml files which we will be
able to manipulate/transform and import them back in as a new site.
We will also be able to even exchange site content with other
content consumers.
5. This approach is reciprocal. Saying it in other words it means
that anything that can be exported can be re-imported.
6. And it could be done between different Plone versions.
I think now we have sufficient amount of arguments to start our
investigation of what we have in Plone 'out-of-the-box'.
But at this point I should say that unfortunately nothing is
perfect. Neither is Plone's export/import story.
As a proof of the fact that not everything is good with
export/import in Plone, here is a link to plone's devel thread as
for PSPS 7826 called 'content import/export'. And it is still in
progress.
This means that at the moment Plone doesn't provide us with a solid
content migration solution.
But anyway, let's consider what does it provides
.Well, Plone together with GenericSetup, CMFCore, CMFTopic,
Archetypes and Marshall product furnish us with the next
things:
We can migrate 'out-of-the-box' Folder, ATDocument and ATNewsItem.
And that is all about content types.
These content types will be handled with almost all their
archetypes schema fields and unique id. You would probably ask why
almost all - but not all? Well, reference fields, due to some
unknown for me reasons, are simply skipped by Marshall during
content export.
And file fields are not fetched properly either. It's because
archetypes file field's accessor does not return file data, but
only url to that file.
Apart from archetypes fields Marshall will process content object's
type, workflow history and local roles for us.
That's all what we have by default.As you can see , this list is
far from being complete. A lot of important issues are still
missing. Here is a list of different issues that we needed to carry
out to migrate CMS.Info
As you see, the main task here is to make it possible to migrate
all the standard plone content types, like events, links, images,
files, etc.
We don't have any mechanism to export/import AT references.
Also it is vital to fix a problem with AT file fields accessor
which doesn't want to return us file data.
We have qSEOptimizer installed in our CMS.Info. This product
intensively uses object properties (PropertyManager).
And we have a lot of objects with custom qSEOptimizer properties.
So it is important for us to transfer such properties too.
Last but not the least, we don't have any migration handlers for
comments, also known as discussion items.
After we figure out what do we need we can look at the possible
existing solutions.
It seems like these two packages are the best we have for
content import/export.
The first one collective.plone.gsxml was written by Ramon Bartl and
Stefan Eletzhofer. The most important thing about this package is
that it allows to export/import at least all standard plone AT
based content. This package relies pretty much on the Marshall. So
the problems with Marshall will persist with gsxml too. And in
addition this package can transfer AT references.
Another one is collective.transmogrifier, written by Martijin
Pieters, is really well designed and it's easy to extend it. But at
the moment transmogrifier does not have any
sections/handlers/sources which can help us to export content.
Instead it contains a lot of different useful import
hanlders.
That's why finally we choose transmogrifier as a base for our
Quantum Leap content migration approach.
So thanks a lot to Martijn Pieters for it's great
collective.transmogrifier package.
And of course thanks to Ramon Bartl and Stefan Eletzhofer for their
work from what we could benefit.
And finally the result looks like this:
This package provides a lot of extension handlers for
collection.transmogrifier like:
. PropertyManager property handler
handler for discussion items
references file fields topics with criteria etc.
But this is not all we managed to accomplish...
We had a few more extra tasks during the migration.
The first task was to migrate SimpleBlog content to Quills.
The second one was similar: migrate PloneFormMailer forms to
PloneFormGen forms.
Of course we had to move Users and Members.
We also had to migrate old style portlets to plone 3 ones, and
transfer RedirectionTool settings.
For the first two items we created two more packages:
So if you need to migrate blogs based on SimpleBlog product to
Quills content then it will be useful to use
quintagroup.transmogrifier.simpleblog2quills package.
To move PloneFormMailer forms to PloneFormGen ones just use
quintagroup.transmogrifier.pfm2pfg.
All these packages are available at
http://svn.quintagroup.com/products svn repository.
We hope these packages will be useful for you. And we are looking
forward to get some feedback from you to know what to improve, what
to fix and what if missing.To conclude, I would mention what is
missed right now in our solution.
These are: content versions, local permission settings,
blobs.
And I think there are a lot of other different things which I don't
remember or don't even know about ;-)
Here I listed a few links mentioned previously during this
session.
Time for Questions.
and Thank you!