Drupal upgrades and migrations. BAD Camp 2013 version

105
Upgrades and Migrations, how? David Lanier @nadavoid ThinkShout PREFACE

description

Originally presented at PNW Drupal Summit 2013. Revised for BADCamp 2013. You have an aging Drupal 6 or even a Drupal 5 site. You know it's time to move up to Drupal 7. Now, how? There are two main ways to get there. You can perform a traditional upgrade, or you can migrate the data from the old site to a brand new site. In this session I will show how you can use these methods and discuss their benefits and drawbacks, including a thought process to go through when evaluating these options, drawing from some recent projects.

Transcript of Drupal upgrades and migrations. BAD Camp 2013 version

Page 1: Drupal upgrades and migrations. BAD Camp 2013 version

Upgrades and Migrations, how?

David Lanier@nadavoidThinkShout

PREFACE

Page 2: Drupal upgrades and migrations. BAD Camp 2013 version

Overview

• Getting your Drupal 6 site onto Drupal 7

• Use traditional upgrade or migrate data to new site

• Look at the benefits and drawbacks of both approaches

• Primarily for people who have not gone through either process

Your site is unique, so take what you find today and let it help you in your decision making. I won't be saying always do it one way.Both approaches do require work.

Page 3: Drupal upgrades and migrations. BAD Camp 2013 version

An Architecture Metaphor

• Changing the foundation (codebase) of your house

Page 4: Drupal upgrades and migrations. BAD Camp 2013 version

Basic pieces

• Codebase (core, modules, themes, libraries)

• Database

• Files (images, pdfs, mp3s, etc.)

Files = content.Database & Files together = "data"

Page 5: Drupal upgrades and migrations. BAD Camp 2013 version

Deployment

• Same issues as replacing any site, unless the D6 site is incredibly simple.

I won't be dwelling on this today because it is such a common task.

Page 6: Drupal upgrades and migrations. BAD Camp 2013 version

Upgrade overview

• Swap out the foundation, house in place.

• Replace codebase, run upgrade scripts on existing data & files.

• Built into core and contrib.

• UI based

Lift the house, swap in a new foundation, set the house down on the new foundation.The upgrade scripts essentially upgrade the plumbing and wiring to be compatible with the new foundation.UI centered, but drush can be used for most upgrades as well. 'drush updb'

Page 7: Drupal upgrades and migrations. BAD Camp 2013 version

Upgrade drawbacks

• Hard to get rid of legacy cruft

• Field names

• Content type structures

• Anything else that seems old or odd

Page 8: Drupal upgrades and migrations. BAD Camp 2013 version

Migrate overview

• Build new house on a new foundation.

• Build new site. Import data from old site.

• Write custom code to define the specifics of the migration.

The details of a migration must be handled by writing custom code. There is also helpful support in the UI.

Page 9: Drupal upgrades and migrations. BAD Camp 2013 version

Migrate drawbacks

• Very code-centric

• Very granular

Averse to custom coding? Maybe prefer upgrade. But Migrate isn't so hard, I promise!Granular: you generally have to specify everything (image alt, title, language, etc.). Of course, that could also be a good thing. You're in full control.

Page 10: Drupal upgrades and migrations. BAD Camp 2013 version

Feeds

• Best fit is to regularly import new data.

• Not usually a good fit for full migrations.

• Easy to learn, but it usually falls short when getting into the details, such as handling relationships and very large datasets.

I really like feeds, and it can be super for importing data into Drupal. But it's just not a module I generally recommend for a full site migration.

Page 11: Drupal upgrades and migrations. BAD Camp 2013 version

Initial preparation

• Regular D7 system requirements (drupal.org/requirements)

• Local development environment that can support several copies of the site

• Reference site

• Data source

• Development

CHAPTER 1D7: PHP 5.3 recommended, MySQL 5.Local Dev: for testing, experimenting, practicing

Page 12: Drupal upgrades and migrations. BAD Camp 2013 version

Architecture of current site

• Inventory of modules, libraries, themes

•drush pml | grep Enabled

• Content Types, fields

• Views, other configuration

• Important pages, capabilities, workflows

• Reference copy of the site

Save lists of directories (list of modules)Save drush outputSave certain pagesHave a clear idea of exactly what it is you're upgrading from

Page 13: Drupal upgrades and migrations. BAD Camp 2013 version

Handy "archeology" modules

• drupal.org/project/sitedoc

• drupal.org/project/hacked

Page 14: Drupal upgrades and migrations. BAD Camp 2013 version

Determine upgrade path

• Module to D7 options

• same module (pathauto)

• different module

• moved into core (CCK to fields)

• nothing available, so custom solution

This is where most of the upgrade work will be.Be sure to check if D7 version of modules have their own dependencies. Views adds a ctools dependency. Others will likely require the libraries module.

Page 15: Drupal upgrades and migrations. BAD Camp 2013 version

Determine upgrade path

• Custom modules

• Converting 6.x modules to 7.x:http://drupal.org/node/224333

• Coder module

Any custom code will need to be updated to match the Drupal 7 API and coding standards.

Page 16: Drupal upgrades and migrations. BAD Camp 2013 version

Determine upgrade path

• Theme

• Update or replace base theme

• Update or rebuild subtheme

• Upgrading 6.x themes to 7.xhttps://drupal.org/node/254940

A custom theme is custom code, so needs same attention.

Page 17: Drupal upgrades and migrations. BAD Camp 2013 version

Prepare the source

• Install a copy of the site to use as the source.

• Remove all disabled or unused modules.

• Update all remaining modules to their latest D6 versions.

Page 18: Drupal upgrades and migrations. BAD Camp 2013 version

How to Upgrade

• Practice on a copy of the site.

• 3 dev sites:

• Pristine copy of site for reference.

• Copy of site with everything D6 updated.

• Copy that is actually getting upgraded.

CHAPTER 2

Page 19: Drupal upgrades and migrations. BAD Camp 2013 version

Today's Example

• RidePDW.com Portland Design Works

• Portland company that develops bicycle accessories

I experimented with an upgrade, but ended up doing a migration instead, primarily because of keeping data in sync.

Page 20: Drupal upgrades and migrations. BAD Camp 2013 version
Page 21: Drupal upgrades and migrations. BAD Camp 2013 version

Process

• Commit after each step.

• Definitely after updating each module.

• Include db exports, as outside files.

DB Exports: if not in the repo, in their own folder.

Page 22: Drupal upgrades and migrations. BAD Camp 2013 version

Upgrade core

• Refer back to your list of all enabled modules

• Disable all enabled contrib modules

• Switch theme to Garland, including the admin theme

Page 23: Drupal upgrades and migrations. BAD Camp 2013 version

Upgrade core

• Use the copy of the site that you have for actually upgrading

• Replace core files manually

• Delete core directories & files

• Add the new ones

• Download Drupal, move the files into place

I generally prefer to swap in the new files/folders, rather than building up a new codebase. (List of "incompatible" modules helps me know where I am)

Page 24: Drupal upgrades and migrations. BAD Camp 2013 version

Update the database

• Do 'drush updb' (Or use /update.php)

• Watch output, observe warnings

• Save output to refer back to

Page 25: Drupal upgrades and migrations. BAD Camp 2013 version

Update, drush example

Page 26: Drupal upgrades and migrations. BAD Camp 2013 version

Update, drush example

Page 27: Drupal upgrades and migrations. BAD Camp 2013 version

Update, drush example

Page 28: Drupal upgrades and migrations. BAD Camp 2013 version

Upgraded! (core only)

Page 29: Drupal upgrades and migrations. BAD Camp 2013 version

Core upgrade complete!

• Yay!

• You're running Drupal 7 now

• Commit the current state to a repo

• Next step, contrib modules

Page 30: Drupal upgrades and migrations. BAD Camp 2013 version

Upgrading contrib modules

• Upgrade one at a time

• Module's d.o page

• Module's README.txt or INSTALL.txt or UPGRADE.txt

• Check for special steps, instructions

• Check for new dependencies

Do a little research on each module before trying. Then go ahead and try.Dependencies: views needs ctools. ubercart needs rules. rules needs entity.

Page 31: Drupal upgrades and migrations. BAD Camp 2013 version

Add the new code

• This works: drush dl modulename

• Or, delete existing, then add the new

Page 32: Drupal upgrades and migrations. BAD Camp 2013 version

Where to start?

• Start with the most foundational

• ctools

• views

• Or most standalone

• pathauto

• devel

Page 33: Drupal upgrades and migrations. BAD Camp 2013 version

Views

• Replace views code

• Update database

• Enable the upgraded module

This is the usual process. But it's still best to see the module's upgrade documentation first.

Page 34: Drupal upgrades and migrations. BAD Camp 2013 version

Views database updated

Page 35: Drupal upgrades and migrations. BAD Camp 2013 version

Enable views

Views requires ctools now.

Page 36: Drupal upgrades and migrations. BAD Camp 2013 version

Enable views

After downloading ctools

Page 37: Drupal upgrades and migrations. BAD Camp 2013 version

Upgrade CCK

• Download D7 version of CCK

• Enable the content_migrate module

• Use the UI to migrate CCK fields to drupal core fields

Kind of a special case, because it's a major architectural change, bringing fields into core.Also common pattern: contrib module providing upgrade path into core.

Page 38: Drupal upgrades and migrations. BAD Camp 2013 version

Available fields

Provides suggestions of things to do, and notes about what it will do.Caveat: it lets you migrate fields, even if the relevant content type does not yet exist properly in the upgraded site. Should content types come before fields?

Page 39: Drupal upgrades and migrations. BAD Camp 2013 version

Converted Fields

Page 40: Drupal upgrades and migrations. BAD Camp 2013 version

Unavailable Fields

Page 41: Drupal upgrades and migrations. BAD Camp 2013 version

Migrate CCK fields

• Install required modules

• Enable modules that define content types

• Sometimes several layers of dependencies emerge

• Can end up being messy and difficult to fully successfully migrate all CCK fields.

dependency chain: Nodereference field is used to refer to products. Products are defined by ubercart. Ubercart requires Rules. Rules requires entity api.

Page 42: Drupal upgrades and migrations. BAD Camp 2013 version

That was Upgrade

• Mainly a taste of what's in store when doing an upgrade

Page 43: Drupal upgrades and migrations. BAD Camp 2013 version

Next up: Migrate

CHAPTER 3

Page 44: Drupal upgrades and migrations. BAD Camp 2013 version

How to Migrate

• Building a whole new house, with its own new foundation.

• Then we will move all of our data into it.

Page 45: Drupal upgrades and migrations. BAD Camp 2013 version

Plan and get organized

• Collect data and files into a safe place, to use as a pristine, read-only source.

• Make a spreadsheet or list to plan out your mapping

• Build out the architecture of the new site

Identify questionable or troublesome pieces, e.g. whether to keep certain fields.Build: at least content types & fields.Anything else that is needed by your migrated data.

Page 46: Drupal upgrades and migrations. BAD Camp 2013 version

Set up environments

• Install or get access to a reference site

• Install source database

• Install a built destination site

• An IDE for exploring the migrate API

reference site = this is how it currently is. must be inspectable.

Page 47: Drupal upgrades and migrations. BAD Camp 2013 version

Migrate using IDE

Page 48: Drupal upgrades and migrations. BAD Camp 2013 version

Start a new module

• Install the migrate module

• Create a new module that extends classes that are in the migrate module

Page 49: Drupal upgrades and migrations. BAD Camp 2013 version

Important functions

• addFieldMapping()

• addSimpleMappings()

• defaultValue()

• sourceMigration()

• addUnmigratedDestinations()

• addUnmigratedSources()

Just to get a flavor of what you will be using.

Page 50: Drupal upgrades and migrations. BAD Camp 2013 version

Module structure

• .info file: file list

• .module file: specify migrations via hook_migrate_api()

• Migration class files: handle the details and mappings of a migration

• name: sitename_migration

.info file: Mainly lists files that are used.

.module file: register migrations and their classes by using hook_migrate_api().Migration class files: I generally prefer one class per file, but there's nothing wrong with having multiple classes in one file too.

Page 51: Drupal upgrades and migrations. BAD Camp 2013 version

Directory list

Most basic structure that provides a functional migration. But there are many more complicated examples.

Page 52: Drupal upgrades and migrations. BAD Camp 2013 version

.info file

Page 53: Drupal upgrades and migrations. BAD Camp 2013 version

.module file

Page 54: Drupal upgrades and migrations. BAD Camp 2013 version

class file

Page 55: Drupal upgrades and migrations. BAD Camp 2013 version

How did you find out?

• Comments in the migrate module

• migrate_example module (inside migrate)

• articles on btmash.com

Page 56: Drupal upgrades and migrations. BAD Camp 2013 version

Additional Docs

• Source code

• Suggestions in IDE

• Migrate handbook at drupal.org/migrate

Page 57: Drupal upgrades and migrations. BAD Camp 2013 version

Before enabling custom migrate module

Page 58: Drupal upgrades and migrations. BAD Camp 2013 version

Migration class

• Set description

• Define database source

• Create query to get data

• Tell migrate that this query is the source of this migration

• Do the mappings

This is essential before you can enable your custom module without errors.

Page 59: Drupal upgrades and migrations. BAD Camp 2013 version

Set description

Page 60: Drupal upgrades and migrations. BAD Camp 2013 version

Set database

Page 61: Drupal upgrades and migrations. BAD Camp 2013 version

Define the query

Page 62: Drupal upgrades and migrations. BAD Camp 2013 version

Set the source

Page 63: Drupal upgrades and migrations. BAD Camp 2013 version

Set the destination

Page 64: Drupal upgrades and migrations. BAD Camp 2013 version

Set up table mapping

Enables migrate to map source rows to destination rows

Page 65: Drupal upgrades and migrations. BAD Camp 2013 version

Map first field

Page 66: Drupal upgrades and migrations. BAD Camp 2013 version

Enable the module

List of migrations & their status.Status of mappings.There are equivalent tools in drush.

Page 67: Drupal upgrades and migrations. BAD Camp 2013 version

Overview

Page 68: Drupal upgrades and migrations. BAD Camp 2013 version

Destination

Page 69: Drupal upgrades and migrations. BAD Camp 2013 version

Source

Page 70: Drupal upgrades and migrations. BAD Camp 2013 version

Done

Page 71: Drupal upgrades and migrations. BAD Camp 2013 version

Mapping process

• Add fields as needed to the query

• Add mappings until the mapping screen clear.

Adding fields will usually mean adding joins.

Page 72: Drupal upgrades and migrations. BAD Camp 2013 version

Mapping a subfield

Page 73: Drupal upgrades and migrations. BAD Camp 2013 version

Mapping a reference

From a ProductHelpMigration

Page 74: Drupal upgrades and migrations. BAD Camp 2013 version

Self references

• Handles creating references to same import

• Uses a createStub function

Page 75: Drupal upgrades and migrations. BAD Camp 2013 version

Mapping a file field

Page 76: Drupal upgrades and migrations. BAD Camp 2013 version

Term References

• Use a dependent migration

• Or create as needed

Page 77: Drupal upgrades and migrations. BAD Camp 2013 version

Repeats

• "Update" completely overwrites

• Highwater mark

In addition to rollback and reimport.highwater mark, important for bringing in added content.brings in any new items with a greater value of highwater than last recorded.No other comparisons. (no import "changed")

Page 78: Drupal upgrades and migrations. BAD Camp 2013 version

Manipulating source data

• prepareRow() function

Looking up a value for 'weight_units' from another table.

Page 79: Drupal upgrades and migrations. BAD Camp 2013 version

migrate_extras

• Provides support for many contrib modules

• Addressfield, entity api, more

• Excellent examples of custom handlers

Page 80: Drupal upgrades and migrations. BAD Camp 2013 version

migrate_d2d

• Current stable version (2.0)

• Ends up being somewhat cumbersome.

• Dev version (2.1-beta1)

• Looks very promising

Not migrate_d2d 2.0:- complicated things for me as a newcomer to the migrate framework.- Added too many layers and did not streamline things for me.- Had to unmap many things that it automatically mapped for me.

Page 81: Drupal upgrades and migrations. BAD Camp 2013 version

migrate_d2d 2.1

• Provides migration setup wizard

• Provides UI for field mapping

• Requires 2.6 version of Migrate module, which is also still in development

Page 82: Drupal upgrades and migrations. BAD Camp 2013 version

Step 1

Page 83: Drupal upgrades and migrations. BAD Camp 2013 version

Step 2

Page 84: Drupal upgrades and migrations. BAD Camp 2013 version

Step 3

Page 85: Drupal upgrades and migrations. BAD Camp 2013 version

Step 4

Page 86: Drupal upgrades and migrations. BAD Camp 2013 version

Step 5

Page 87: Drupal upgrades and migrations. BAD Camp 2013 version

Dashboard (Groups)

Page 88: Drupal upgrades and migrations. BAD Camp 2013 version

Migrations

Page 89: Drupal upgrades and migrations. BAD Camp 2013 version

Product migration

Page 90: Drupal upgrades and migrations. BAD Camp 2013 version

Migration mappings

Page 91: Drupal upgrades and migrations. BAD Camp 2013 version

Migration mappings 2

Page 92: Drupal upgrades and migrations. BAD Camp 2013 version

Migration mappings 3

Page 93: Drupal upgrades and migrations. BAD Camp 2013 version

Pretty slick!

Page 94: Drupal upgrades and migrations. BAD Camp 2013 version

migrate_d2d drawbacks

• Still really new (alpha/beta/dev status)

• Making changes to initial settings overwrites existing migrate configuration

• Cannot (yet) export configuration

• drupal.org/node/1983404

• Uncertain how to add regular migrate code to what is done in the UI

• drupal.org/node/2118243

Making changes, such as database credentials or adding a content type.Regular programmatic code = prepareRow, anything that the UI doesn't provide but is available in the API.

Page 95: Drupal upgrades and migrations. BAD Camp 2013 version

Migrate

• Showed important parts

• There's a LOT more there

Page 96: Drupal upgrades and migrations. BAD Camp 2013 version

QA. Did it work?

• Regardless whether upgrade or migrate

• Have someone else check your work

• Have the client check your work

• Pass any existing acceptance tests

SIDEBARMake certain. This will be the content of the next version of this site.

Page 97: Drupal upgrades and migrations. BAD Camp 2013 version

QA Questions

• Correct content?

• Correct versions of content?

• Complete number of fields?

• Complete metadata? (urls, relations, dates, authors, tags)

• Complete number of nodes?

• Images and media? (block live site)

Page 98: Drupal upgrades and migrations. BAD Camp 2013 version

Which way do I go?

• WHY do you want to go?

CHAPTER 4First thing to think about is why specifically do you want to move to Drupal 7?Are there architectural things that need to be changed in your site, in order to take advantage of what you want from D7?

Page 99: Drupal upgrades and migrations. BAD Camp 2013 version

Traditional upgrade if:

• Drupal site, only one version back

• Simple site

• Few contrib modules

• End result should mirror current site

• Content is infrequently updated

• A little downtime is OK

Downtime can still be avoided, using good deployment methods.Also, if limited time is available (and the other conditions are met).

Page 100: Drupal upgrades and migrations. BAD Camp 2013 version

Migrate if:

• Drupal 5 site, or even a different platform

• Complex site

• Many contrib modules

• New site needs structural changes or enhancements

• Content is frequently updated

• Minimal downtime is very important

Page 101: Drupal upgrades and migrations. BAD Camp 2013 version

Conclusion (?)

• traditional upgrade

• migrate

If you want something that is quick and your site's upgrade path supports it, go with a traditional upgrade.If you want to make a lot of structural changes and need fine control over the upgrade path, go with migrate.

Page 102: Drupal upgrades and migrations. BAD Camp 2013 version

Future of upgrades and migrations

• Migrate is being added to Drupal 8 core, to provide a smooth upgrade path from D6 and D7 core.

• https://groups.drupal.org/imp

• http://www.drupal4hu.com/node/381

Page 103: Drupal upgrades and migrations. BAD Camp 2013 version

Bookmarks• http://drupalcode.org/project/drupal.git/

blob/refs/heads/7.x:/UPGRADE.txt

• https://drupal.org/project/drush_sup

• http://drupal.org/project/migrate

• http://drupal.org/project/migrate_d2d

• http://drupal.org/migrate

• http://drupal.org/project/coder (upgrade)

• http://drupal.org/project/hacked

• http://drupal.org/project/upgrade_status

Page 105: Drupal upgrades and migrations. BAD Camp 2013 version

Thanks!