Drupal Continuous Integration (European Drupal Days 2015)

129
© Ibuildings 2014/2015 - All rights reserved #DrupalDaysEU DRUPAL CONTINUOUS INTEGRATION by Claudio Beatrice

Transcript of Drupal Continuous Integration (European Drupal Days 2015)

Page 1: Drupal Continuous Integration (European Drupal Days 2015)

© Ibuildings 2014/2015 - All rights reserved

#DrupalDaysEU

DRUPAL CONTINUOUS INTEGRATION

by Claudio Beatrice

Page 2: Drupal Continuous Integration (European Drupal Days 2015)

#DrupalDaysEU

© Ibuildings 2014/2015 - All rights reserved

Gold Sponsors

Page 3: Drupal Continuous Integration (European Drupal Days 2015)

#DrupalDaysEU

© Ibuildings 2014/2015 - All rights reserved

Media Sponsors

Silver Sponsors

Page 4: Drupal Continuous Integration (European Drupal Days 2015)

DRUPAL CONTINUOUS INTEGRATION

Introduction

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 5: Drupal Continuous Integration (European Drupal Days 2015)

Continuous Integration“Continuous Integration” is a software development practice where members of a team integrate work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including tests) to detect integration errors as quickly as possible.

— Martin Fowler

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 6: Drupal Continuous Integration (European Drupal Days 2015)

Continuous Integration

• automating repetitive tasks such as build, QA and deploy

• fast and frequent integrations

• fast feedback loop, catch errors asap

• ability to deploy quickly and keep track of every released set of code

ADVANTAGES

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014-2015http://tenwarp.com/

Page 7: Drupal Continuous Integration (European Drupal Days 2015)

Continuous Integration

developercommits triggers

config

static analysis

unit / functional

tests

runs

runs

runs

report

FAIL

deploy

smoke / functional

testsreport

SUCCESS

SUCCESS

FAIL

CI server

revision control

log

logs

logs

logs

BUILD PROCESS EXAMPLE

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 8: Drupal Continuous Integration (European Drupal Days 2015)

Continuous IntegrationTOOLS

• SCM: git, svn, cvs, …

• CI SERVER: jenkins

• BUILD TOOL: phing

• DRUPAL AUTOMATION: drush

• DEPENDENCIES MANAGEMENT: drush make & composer

• DRUPAL CDD TOOLS: features & migrate

• TESTING: PHPUnit & Behat

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 9: Drupal Continuous Integration (European Drupal Days 2015)

Continuous IntegrationDRUPAL - I

Drupal, as opposed to “classic” frameworks such as Symfony or Zend, does not make the setup and maintenance a Continuous Integration cycle easy due to the high coupling between data and configuration within its database.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 10: Drupal Continuous Integration (European Drupal Days 2015)

Continuous IntegrationDRUPAL - II

To overcome this problem it’s necessary to adopt a strictly controlled, disciplined work methodology and to make use of a set of tools that allow us to transform configuration into code, to easily deal with content and to incrementally update database’s schema.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 11: Drupal Continuous Integration (European Drupal Days 2015)

Continuous IntegrationWORKFLOW - I

A developer’s workflow consist not only in the plain old features implementation, but also in the correct, precise and complete export of all the element of configuration that are necessary to rebuild the functionalities on several environments. !

Key modules: Features and Strongarm.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 12: Drupal Continuous Integration (European Drupal Days 2015)

Continuous IntegrationWORKFLOW - II

It’s often necessary to setup some default content that may vary during development and that might be expanded or updated once the site has been published. It’s therefore necessary to formalise in code all the data import procedures in order to better administrate them and to be able to replicate them at any given time. !

Key module: Migrate.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 13: Drupal Continuous Integration (European Drupal Days 2015)

Continuous IntegrationWORKFLOW - III

As soon as a self-contained task such a bug-fix or a new feature implementation has been completed, it’s necessary to commit and push that unit of work to the central repository in order to integrate, test and deploy it automatically on the development environment. !

Key tools: git/svn.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 14: Drupal Continuous Integration (European Drupal Days 2015)

Continuous IntegrationWORKFLOW - IV

!

Writing automated tests is necessary in order to guarantee the behaviour of a given implementation and to make sure that future changes won’t break the work already done, creating regressions and dealing potential damage to users and clients. !

Key tools: PHPUnit, Behat.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 15: Drupal Continuous Integration (European Drupal Days 2015)

Q & A

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 16: Drupal Continuous Integration (European Drupal Days 2015)

Continuous IntegrationEXERCISES

1. What are the main advantages of a Continuous Integration system?

2. When do Unit Tests are executed during the build? How about functional? Why are they executed at different times?

3. Why is it important to write tests?

4. Which benefits come from using a Source Control Management system?

5. Why is Drupal so hard -compared to other tools such as “classic” frameworks- to use in a Continuous Integration environment?

6. What tools allow Drupal to be successfully employed in a Continuous Integration development cycle and for what reasons?

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 17: Drupal Continuous Integration (European Drupal Days 2015)

DRUPAL CONTINUOUS INTEGRATION

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 18: Drupal Continuous Integration (European Drupal Days 2015)

Phing is a build system written in PHP that allows developers to describe the automated operations used during development, deployment and maintenance using plain XML files.

Phing

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 19: Drupal Continuous Integration (European Drupal Days 2015)

• Simplicity: it’s based on well-known, declarative XML syntax and features a very rich documentation

• Power: thanks to the big amount of available tasks, it is able to perform almost anything you could do using a command line interpreter

• Extensibility: as it is written in PHP, it’s easy to extend and debug

• Automation: the whole application configuration and deployment process can be handled by build files

PhingBENEFITS

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 20: Drupal Continuous Integration (European Drupal Days 2015)

• execute tests and log their results

• perform filesystem operations

• interact with source configuration management systems

• generate documentation

• transfer files from/to remote systems

• fire drush commands

• invoke shell commands

PhingFEATURES

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 21: Drupal Continuous Integration (European Drupal Days 2015)

There are two available file types in Phing to configure the build: .properties and .xml files. The former contains a list of key/value variables, while the latter contains the “instructions” to execute.

In a usual setup there are up to five different environments: loc, dev, stage, debug and prod. Each of these ones will be configured using a dedicated .properties file in order to make the build file environment-agnostic so to reuse most of its code.

PhingMAIN FILES AND ENVIRONMENTS

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 22: Drupal Continuous Integration (European Drupal Days 2015)

PhingPROPERTY FILE

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 23: Drupal Continuous Integration (European Drupal Days 2015)

PhingBUILD FILE - PROJECT TAG

For every build file, there must be a single, unique, root “project" tag. Within that "target", "task" and “type" tags can be used.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 24: Drupal Continuous Integration (European Drupal Days 2015)

PhingBUILD FILE - TYPE TAG

Besides default types such as numbers or strings, it is possible to define more complex, nested types. An interesting feature of Phing’s types is the ability to be referenced by id later.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 25: Drupal Continuous Integration (European Drupal Days 2015)

PhingBUILD FILE - TARGET TAG

“Targets” are collections of components that are assigned a unique name within the project. It is possible to invoke other targets within a target and it’s also possible to set dependencies between different targets.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 26: Drupal Continuous Integration (European Drupal Days 2015)

PhingBUILD FILE - TASK TAG

“Task”s are responsible for the execution of specific units of work and are usually configurable through attributes and nested tags. A few examples of what tasks do could be: files copying, archive compression or drush commands execution.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 27: Drupal Continuous Integration (European Drupal Days 2015)

Q & A

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 28: Drupal Continuous Integration (European Drupal Days 2015)

PhingEXERCISES

1. What are the main benefits of using a build system?

2. Why is Phing more convenient than a bunch of shell scripts?

3. What files does Phing use to work? What’s their use?

4. What are the fundamental elements of a build file? What’s their use?

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 29: Drupal Continuous Integration (European Drupal Days 2015)

DRUPAL CONTINUOUS INTEGRATION

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 30: Drupal Continuous Integration (European Drupal Days 2015)

Drush

Drush is a command-line shell and scripting interface for Drupal, a veritable Swiss Army knife designed to make life easier for those who spend their working hours hacking away at the command prompt.

— Drush Team

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 31: Drupal Continuous Integration (European Drupal Days 2015)

Drush

• Drupal Power: ability to simply, swiftly and programmatically execute Drupal-specific tasks

• Integration: working with other command-line tools is very easy and effective

BENEFITS

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 32: Drupal Continuous Integration (European Drupal Days 2015)

Drush• code and database backup, restore e sync

• cron tasks execution

• automatic installations

• variables, cache and logs CRUD operations

• modules and themes administration

• SQL commands execution

• users administration

• dependency management

• remote tasks execution

FEATURES

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 33: Drupal Continuous Integration (European Drupal Days 2015)

Drush

Drush make is a Drush extension that deals with modules, themes, libraries and patches management. It makes use of a manifest file to index and download the required versions of the packages used to build a Drupal website.

MAKE - I

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 34: Drupal Continuous Integration (European Drupal Days 2015)

DrushMAKE - II

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 35: Drupal Continuous Integration (European Drupal Days 2015)

Drush

Drush Make allows developers to specify a destination sub-folder using the “subdir” option. We’ll leverage that to establish the convention of having four sub-folders for modules: contrib, custom, devel and features. This division will help in keeping a cleaner division between the modules and will save us the burden of versioning “contrib” modules.

MAKE - III

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 36: Drupal Continuous Integration (European Drupal Days 2015)

Drush

Drush allows the developers to create a kind of labels called Site Aliases: these are nothing but configuration groups that fall under an arbitrary name and make the execution of some Drush commands easier. In our case those aliases match the environments defined for our project, and will look something like myproject.loc and myproject.dev.

SITE ALIAS - I

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 37: Drupal Continuous Integration (European Drupal Days 2015)

DrushSITE ALIAS - II

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 38: Drupal Continuous Integration (European Drupal Days 2015)

DrushDrush looks for Aliases definitions in several locations:

1. in the path set in the “alias path” variable of the options array found in the drushrc.php file

2. in the path specified in the —alias-path command-line option

3. in one of the following folders: “/etc/drush”, “$HOME/.drush” or the current site’s “sites/all/drush”

In order to keep Drush Aliases clean and separated, we’ll use our projects’ “sites/all/drush” as the go-to location to store their configuration so not to pollute the global configuration.

SITE ALIAS - III

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 39: Drupal Continuous Integration (European Drupal Days 2015)

Drush

Drush implements two extremely useful commands to keep files and databases in sync between hosts: rsync e sql-sync.

• drush rsync @mysite.loc @mysite.dev

• drush sql-sync @mysite.loc @mysite.dev

SYNCHRONISATION

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 40: Drupal Continuous Integration (European Drupal Days 2015)

Q & A

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 41: Drupal Continuous Integration (European Drupal Days 2015)

DrushEXERCISES

1. What’s Drush?

2. Why is it convenient to use it?

3. What are its most important features?

4. What’s the use for a make file?

5. What’s a Drush Site Alias? What’s its use?

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 42: Drupal Continuous Integration (European Drupal Days 2015)

DRUPAL CONTINUOUS INTEGRATION

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 43: Drupal Continuous Integration (European Drupal Days 2015)

JenkinsJenkins is a Java application useful to manage the execution of those repetitive tasks that are needed to build a software project.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 44: Drupal Continuous Integration (European Drupal Days 2015)

Jenkins

• automation of deployments procedures

• speed-up of repetitive tasks such as test runs or the creation of build artifacts to deploy on our projects’ target environments

• logging and monitoring for a handful of code-related stats such as test results or code quality analysis

• convenient web ui to configure all the needed jobs

BENEFITS

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 45: Drupal Continuous Integration (European Drupal Days 2015)

JenkinsFEATURES

• integration with several source configuration management systems

• tests results and code quality reports generation

• integration with several build systems

• jobs status notifications

• configurable web ui

• hundreds of plugins

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 46: Drupal Continuous Integration (European Drupal Days 2015)

JenkinsINSTALLATION

Page 47: Drupal Continuous Integration (European Drupal Days 2015)

JenkinsPLUGINS

Page 48: Drupal Continuous Integration (European Drupal Days 2015)

JenkinsJOB CREATION - I

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 49: Drupal Continuous Integration (European Drupal Days 2015)

JenkinsJOB CREATION - II

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 50: Drupal Continuous Integration (European Drupal Days 2015)

JenkinsDASHBOARD

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 51: Drupal Continuous Integration (European Drupal Days 2015)

Q & A

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 52: Drupal Continuous Integration (European Drupal Days 2015)

JenkinsESERCIZI

1. What’s Jenkins?

2. What are its tasks?

3. What’s a Job?

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 53: Drupal Continuous Integration (European Drupal Days 2015)

DRUPAL CONTINUOUS INTEGRATION

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 54: Drupal Continuous Integration (European Drupal Days 2015)

Features

The Feature module is able to export parts of the configuration found in the database into “ordinary” Drupal modules. The ability to produce a set of files containing a specific functionality starting from the database makes them much easier to manage and deploy across several systems and installations.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 55: Drupal Continuous Integration (European Drupal Days 2015)

• Removes the need for manual, repetitive work that’s necessary to reproduce all of a website’s features and traits by encapsulating them in portable, standard Drupal modules;

• Exports directly to Drupal modules or, in other words, text files: that means it’s easy to version them using a source control management system, which in turn makes easy to keep the history of changes and allows multiple people to work on the project concurrently;

• Allows to transport functionalities across websites in an easy, reproducible and automatic fashion.

FeaturesBENEFITS

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 56: Drupal Continuous Integration (European Drupal Days 2015)

Features

Local StageDev

Prod

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 57: Drupal Continuous Integration (European Drupal Days 2015)

FeaturesCREATION - I

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 58: Drupal Continuous Integration (European Drupal Days 2015)

FeaturesCREATION - II

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 59: Drupal Continuous Integration (European Drupal Days 2015)

FeaturesADMINISTRATION

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 60: Drupal Continuous Integration (European Drupal Days 2015)

FeaturesOPERATIONS - ENABLING A FEATURE

Enabling a Features-exported module executes the same operations a “normal” module executes, such as:

• Schema installation;

• System Register and Cache update;

• Execution of hook_install();

• Execution of hook_enable().

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 61: Drupal Continuous Integration (European Drupal Days 2015)

FeaturesOPERATIONS - UPDATE & REVERT

Features Update: updating a feature consists in the act of saving some of the information found in the database into a specific Feature Module.

Features Revert: reverting a feature consists in the act of writing the information found in a Feature Module into the database.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 62: Drupal Continuous Integration (European Drupal Days 2015)

Features

features-components (fc) List features components.

features-diff (fd) Show the difference between the default and overridden state of a feature.

features-export (fe) Export a feature from your site into a module.

features-list (fl) List all the available features for your site.

features-revert (fr) Revert a feature module on your site.

features-revert-all (fra) Revert all enabled feature module on your site.

features-update (fu) Update a feature module on your site.

features-update-all (fua) Update all feature modules on your site.

DRUSH COMMANDS

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 63: Drupal Continuous Integration (European Drupal Days 2015)

Features

• Thanks to the power and flexibility granted by the Entity API it’s possible to easily create exportable Entities that are automatically picked up and handled by Features;

• A lot of contrib modules are already using such possibility in order to provide exportable configuration. Some examples could be the Rules, Views or Context modules.

BEYOND THE DEFAULT OPTIONS

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 64: Drupal Continuous Integration (European Drupal Days 2015)

Features• Create and maintain “vertical” features such as blog, forum and front page. Avoid

creating Feature Modules that group “exportables” by type, such as “all site’s views” or “all site’s rules”;

• Do always use English and specific namespaces when naming components (field_blog_topics instead of field_argomenti);

• Do not use abbreviations unless it’s strictly ncssry.

• Keep a standard format when naming fields: try not to mix casing, dashing, underscores and so on;

• Define conventions. The Kit Project, although abandoned, is a good source of inspiration;

• Strive to keep things clean and in order: try to always use Features for importing and exporting all the developed functionalities.

BEST PRACTICES

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 65: Drupal Continuous Integration (European Drupal Days 2015)

Q & A

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 66: Drupal Continuous Integration (European Drupal Days 2015)

FeaturesEXERCISES

1. What are the main goals of the Features module?

2. What components can it act upon?

3. What does feature-revert do?

4. What benefits does Features bring?

5. What’s Strongarm used for?

6. Can you name a few modules that allow their components to be exported by Features?

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 67: Drupal Continuous Integration (European Drupal Days 2015)

DRUPAL CONTINUOUS INTEGRATION

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 68: Drupal Continuous Integration (European Drupal Days 2015)

Migrate

The Migrate module makes a powerful and flexible object oriented framework for importing contents into Drupal available to developers: it offers out-of-the-box support for a handful of sources, such as databases and file formats, and destinations, such as nodes and taxonomy terms.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 69: Drupal Continuous Integration (European Drupal Days 2015)

Migrate

At macro-level, there’s four interesting areas to consider when working with Migrate:

• An API for defining and managing the migration process;

• A set of Drush commands for managing the migrations, such as import, rollback and status;

• An Admin UI featuring some of the features exposed by Drush;

• Some example modules that demonstrate a handful of techniques used to define custom migrations.

FEATURES

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 70: Drupal Continuous Integration (European Drupal Days 2015)

Migrate

MigrateSourceid

userNameemail

address

MigrateMapsourceId

destinationIdMigrateDestination

uid

name

mail

UnmigratedDestinationaddress

ARCHITECTURE

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 71: Drupal Continuous Integration (European Drupal Days 2015)

MigrateSOURCE

• Inherits from the abstract class MigrateSource

• Provides an interface to the data set

• Iterates on the source’s rows

• Provides a list of fields

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 72: Drupal Continuous Integration (European Drupal Days 2015)

MigrateDESTINATION

• Inherits from the abstract class MigrateDestination (and usually from the more specific MigrateDestinationEntity)

• Responsible for persisting data in a specific Drupal’s content type such as user, node, comment, file.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 73: Drupal Continuous Integration (European Drupal Days 2015)

MigrateMAPPING TABLE

• Inherits from MigrateSQLMap, which in turn inherits from the abstract class MigrateMap

• Describes the key formats for the source and the destination

• Stores all the information that’s used to import and rollback each and every single record

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 74: Drupal Continuous Integration (European Drupal Days 2015)

MigrateFIELDS MAPPING

• Allows to link each destination field with a source field

• Allows to transform data during the migration to fix or adapt to the destination requirements.

• Allows to exclude some fields from the migration

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 75: Drupal Continuous Integration (European Drupal Days 2015)

Migrate

Handlers are classes that allow to execute additional operations during the import process.

They are usually used to support migrations where some specific, non-standard data is connected to Core Entities: most of the time this will be related to custom field types.

HANDLER

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 76: Drupal Continuous Integration (European Drupal Days 2015)

Migrate

There’s two kinds of Handler: Destination and Field.

The former is used to manipulate the Destination Entity around saving (pre- and post- save) and to declare additional fields exposed by the Destination Entity.

The latter works much like the Destination Handler, except its methods are invoked for each Field (as opposed to each Entity).

HANDLER

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 77: Drupal Continuous Integration (European Drupal Days 2015)

Migrate

Sometimes there are pieces of content that hold circular references to each other: for instance a tutorial post “part 1” could contain a reference to the “part 2” and vice-versa.

STUB

NODEnid 1title tutorial part I

body …series_nids 2,3,4

NODEnid 2title tutorial part II

body …series_nids 1,3,4

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 78: Drupal Continuous Integration (European Drupal Days 2015)

Migrate

During the import process the first node won’t be able to correctly link to the second as the (new) id is still unknown: Stubs solve this problem by creating and linking a temporary, empty node that only holds an id. As soon as the import process gets to the stubbed node, it will populate it with all the due information.

STUB

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 79: Drupal Continuous Integration (European Drupal Days 2015)

Migrate

Most of the time migrations are defined as static as that source and the destination match one-to-one: one row, one migration.

However, sometimes it’s necessary to “join” multiple sources having the same structure into a single destination: this kind of migration is defined as dynamic.

An example could be a migration where you need to import users from several Drupal sites into a new one.

DYNAMIC MIGRATIONS

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 80: Drupal Continuous Integration (European Drupal Days 2015)

Migrate

When using this functionality, Drupal will take care of instantiating each migration multiple times. Besides that, a dynamic migration differs from a static migration in the following ways:

• its constructors accepts an array of arguments, where each of its keys identifies an instance of the migration class;

• it overrides the generateMachineName() method in order to generate a distinct machine name for each instance, so to be able to identify them.

DYNAMIC MIGRATIONS

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 81: Drupal Continuous Integration (European Drupal Days 2015)

Migrate migrate-analyze (maz) Analyze the source fields for a migration.

migrate-audit (ma) View information on problems in a migration.

migrate-auto-register (mar) Register any newly-defined migration classes.

migrate-deregister Remove all tracking of a migration.

migrate-fields-destination (mfd) List the fields available for mapping in a destination.

migrate-fields-source (mfs) List the fields available for mapping from a source.

migrate-import (mi) Perform one or more migration processes.

migrate-mappings View information on all field mappings in a migration.

migrate-messages (mmsg) View any messages associated with a migration.

migrate-reset-status (mrs) Reset a active migration's status to idle.

migrate-rollback (mr) Roll back the destination objects from a given migration.

migrate-status (ms) List all migrations with current status.

migrate-stop (mst) Stop an active migration operation.

migrate-wipe (mw) Delete all nodes from specified content types.

DRUSH COMMANDS

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 82: Drupal Continuous Integration (European Drupal Days 2015)

MigrateADMINISTRATION

http://tenwarp.com/

Page 83: Drupal Continuous Integration (European Drupal Days 2015)

Migrate• Migrate D2D - set of utilities for migrating content from

other versions of Drupal (5, 6, 7) to Drupal 7;

• Migrate Extras - integrates several contrib modules such as Rules or Pathauto;

• Commerce Migrate - Integrates Drupal Commerce’s Entities;

• Wordpress/Typo3/PHPBB Migrate - set of utilities to aid the import of content from other well-known platforms.

BEYOND THE DEFAULT OPTIONS

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 84: Drupal Continuous Integration (European Drupal Days 2015)

Q & A

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 85: Drupal Continuous Integration (European Drupal Days 2015)

MigrateEXERCISES

1. What’s the main goal of the Migrate module?

2. What could be the sources for a migration?

3. What’s the use for the mapping table?

4. What does migrate-rollback do?

5. When is appropriate to use Migrate?

6. What’s a dynamic migration? How is it used?

7. What’s a stub?

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 86: Drupal Continuous Integration (European Drupal Days 2015)

DRUPAL CONTINUOUS INTEGRATION

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 87: Drupal Continuous Integration (European Drupal Days 2015)

Behat

Behat is a testing framework written in PHP with the purpose of aiding the writing of Acceptance Tests using Behaviour Driven Development practices.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 88: Drupal Continuous Integration (European Drupal Days 2015)

Behat

IT’S NOT A FUNCTIONAL

TESTING FRAMEWORK

It’s a tool to verify that a given system satisfies the business

needs of the client. Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 89: Drupal Continuous Integration (European Drupal Days 2015)

Behat

Tests are written using a semi-formal language called Gherkin.

In a nutshell, Behat maps Gherkin sentences -usually written in English, but it could be any language- to a class method using to regular expressions.

GHERKIN

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 90: Drupal Continuous Integration (European Drupal Days 2015)

BehatGHERKIN FEATURE

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 91: Drupal Continuous Integration (European Drupal Days 2015)

BehatStoryBDD

Behaviour Driven Development based on User Stories -aka StoryBDD- is a practice that helps into bringing a development team’s understanding of business requirements to the same level of the client’s.

It does so by setting a formal process for asking questions about requirements and then expressing the outcome in a specific story-format.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 92: Drupal Continuous Integration (European Drupal Days 2015)

Behat

For each new feature, a developer should answer the following business questions:

• Who will use it?

• What’s its purpose?

• How is that achieved?

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

StoryBDD

Page 93: Drupal Continuous Integration (European Drupal Days 2015)

Behat

“I want users to access the site through a dedicated form”

• Who will use it? a user an anonymous visitor

• What’s the purpose? it’s not stated

• How is that achieved? by using the login form

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

StoryBDD

Page 94: Drupal Continuous Integration (European Drupal Days 2015)

“I want an anonymous visitor to be able to access the site using the login form in order to access her dashboard containing her aggregated data”

!

In order to see aggregated data on my dashboard as a site visitor I need to be able to login on the site

Behat

benefitrolecondition

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

StoryBDD

Page 95: Drupal Continuous Integration (European Drupal Days 2015)

Once a feature is defined, you should start writing all the scenarios that describe it, using the Context-Action-Outcome pattern:

!

Given I am in on the homepage When I login Then I should see “Welcome Back!” And I should be on my dashboard

BehatSCENARIOS

contextaction

outcome

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 96: Drupal Continuous Integration (European Drupal Days 2015)

Behat

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 97: Drupal Continuous Integration (European Drupal Days 2015)

BehatBEST PRACTICES

And I should see “message” inside “.notifications > p:first-child”

!

It’s recommended to avoid technical details in your scenarios as they increase the coupling between tests and code and make the stories harder for non-technical readers -such as the client- to understand.

!

And I should see “message” inside notification area

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 98: Drupal Continuous Integration (European Drupal Days 2015)

BehatMINK

Each test gets executed within a context where the methods invoked by the expressions used in the scenarios can access to a browser-abstraction API called Mink.

There are several drivers available for Mink such as Goutte, a headless browser written in PHP that’s able to parse HTML and XML and Selenium, a full-fledged framework able to programmatically control real browsers.

Oh and there’s a Drush Driver, too! :)

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 99: Drupal Continuous Integration (European Drupal Days 2015)

BehatMINK CONTEXT

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 100: Drupal Continuous Integration (European Drupal Days 2015)

BehatMINK BEST PRACTICES

The Page object pattern is a way of keeping context files clean by separating UI knowledge from the actions and assertions.

In other words, page objects hide the UI and expose clean services that can be used in the context classes: such

abstraction makes context classes much more clear and helps avoiding duplication and improving maintainability.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 101: Drupal Continuous Integration (European Drupal Days 2015)

BehatMINK DRUSH DRIVER

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 102: Drupal Continuous Integration (European Drupal Days 2015)

BehatEXECUTION EXAMPLE

Gherkin Feature

Behat Mink

Goutte

Selenium Firefox

Test Results

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 103: Drupal Continuous Integration (European Drupal Days 2015)

Q & A

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 104: Drupal Continuous Integration (European Drupal Days 2015)

BehatEXERCISES

1. What’s Behat and what’s its main goal?

2. What’s StoryBDD?

3. What’s a scenario?

4. What languages can be used to write the tests?

5. What questions should be answered by a feature description?

6. What are Mink’s main tasks?

7. What browsers can be used run the tests? What are the main differences between them?

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 105: Drupal Continuous Integration (European Drupal Days 2015)

DRUPAL CONTINUOUS INTEGRATION

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 106: Drupal Continuous Integration (European Drupal Days 2015)

PHPUnit

PHPUnit is a unit testing framework, it’s part of xUnit family and it has become the de-facto PHP standard over the years.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 107: Drupal Continuous Integration (European Drupal Days 2015)

PHPUnit

In software engineering, unit testing is a method by which individual units of code are tested to determine if they are fit for use.

A unit is defined as the smallest part of an application that can be tested in isolation: most of the time that unit is a class or a method.

UNIT TESTING

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 108: Drupal Continuous Integration (European Drupal Days 2015)

PHPUnit

A SUT -an acronym for System Under Test- is defined as that subset of the software being developed that’s exercised by a test. It can be a specific method, and object or even a set of objects.

DEFINITIONS

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 109: Drupal Continuous Integration (European Drupal Days 2015)

PHPUnit

Writing automated tests (not necessarily unit tests only) sets some goals:

• Improving the quality of the product and its code;

• Increasing developers’s understanding of the SUT;

• Decreasing the damage caused by unexpected bugs;

• Simplifying the creation, maintenance and execution of the tests;

• Documenting the behaviours of the applications’ components.

GOALS

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 110: Drupal Continuous Integration (European Drupal Days 2015)

PHPUnitTYPES OF TESTS

Unit

Integration

Functional

Functional: they operate on the system from the outside, often by simulating/employing a browser.

Integration: they check how multiple components of the system cooperate when exercised together.

Unit: they make sure that the single units of code behave as expects when exercised in isolation.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 111: Drupal Continuous Integration (European Drupal Days 2015)

PHPUnit

Should I test before or after development?

Traditionally tests -being them unitary or functional- have always been executed once the implementation was completed.

Such approach has been upturned by the rise of agile methodologies, where the emphasis is put on the ability to respond to change quickly rather than sticking to a plan.

Writing tests first makes code inherently testable and therefore safer to modify and usually produces less verbose and over-engineered implementations.

METHODOLOGIES

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 112: Drupal Continuous Integration (European Drupal Days 2015)

PHPUnit

Should I test outside-in or inside-out?

As soon as development begins, a very important decision has to be made: should it start from the big picture and “zoom in” or from one or some of the details and “zoom out” in order to compose the macro-functionality at the end?

METHODOLOGIES

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 113: Drupal Continuous Integration (European Drupal Days 2015)

PHPUnit

Should I test outside-in or inside-out?

By starting development(and testing) from the outside, the developer forces himself to think as a client first, a method that helps in making requirements and (temporarily missing) dependencies more explicit.

Developing “from the inside” on the other hand frees the developer from the problem of not existing dependencies at the cost of guessing what’s going to be the right set of things to implement before reaching the requested level of functionality.

METHODOLOGIES

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 114: Drupal Continuous Integration (European Drupal Days 2015)

PHPUnit

Testing status or behaviour?

There are two main ways to verify the correctness of a SUT:

• Testing the state: it’s often sufficient to verify that given an initial state, once the system is exercised the ending state then matches the expected one.

• Testing the behaviour: it’s sometimes necessary to look inside the black box of the SUT in order to check how the system works during the execution of the tests to make sure all the behavioural expectations were satisfied.

METHODOLOGIES

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 115: Drupal Continuous Integration (European Drupal Days 2015)

PHPUnitTOOLS

PHPUnit provides a number of tools and helpers to aid testing, such as:

• Assertions

• Fixtures

• Database testing helpers

• Stub and mock objects

• Code coverage analysis tools

• Several logging formats

• Integration with several functional testing tools such as Selenium

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 116: Drupal Continuous Integration (European Drupal Days 2015)

PHPUnitEXAMPLE

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 117: Drupal Continuous Integration (European Drupal Days 2015)

PHPUnit

PHPUnit features about 80 assertions (of which 40 are “straight” and 40 are “negated”). It’s up to the developer to pick the best one for the test in order to have the most expressive tests and the best error messages possible.

ASSERTIONS

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 118: Drupal Continuous Integration (European Drupal Days 2015)

PHPUnitFIXTURES

When writing tests, one of the most tedious and time consuming parts is the preparation of the environment status before the actual execution happens: such status is known as the fixture of the test .

PHPUnit features quite a big set of methods used to aid in the creation and sharing of these states, such as: setUpBeforeClass(), setUp(), assertPreConditions(), assertPostConditions(), tearDown(), tearDownAfterClass() and onNotSuccessfulTest(Exception $e).

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 119: Drupal Continuous Integration (European Drupal Days 2015)

PHPUnitDATA PROVIDERS

It can be useful to be able to run multiple iterations of the same test on a larger-than-one data set. Data Providers allows for a convenient way to do just that.

Such providers are nothing but a method that returns an array containing all the combinations of arguments a tests will be called with.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 120: Drupal Continuous Integration (European Drupal Days 2015)

PHPUnitTEST DOUBLES

It could be helpful -or necessary- to replace a SUT’s dependencies with so-called “test doubles”, a kind of objects that looks like the dependency but it’s created with the sole purpose of satisfying testing needs.

Such a replacement allows to isolate the SUT and manipulate its inputs as needed, making the test more effective and focused on the unit alone.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 121: Drupal Continuous Integration (European Drupal Days 2015)

PHPUnitTEST DOUBLES

There are various kinds of Test Doubles, each serving a specific purpose. Meszaros and Fowler suggested the following five definitions:

• Dummy objects are passed around but never actually used. Usually they are just used to fill parameter lists;

• Fake objects actually have working implementations, but usually take some shortcut which makes them not suitable for production (an InMemoryTestDatabase is a good example);

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 122: Drupal Continuous Integration (European Drupal Days 2015)

PHPUnitTEST DOUBLES

• Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what's programmed in for the test;

• Spies are stubs that also record some information based on how they were called. One form of this might be an email service that records how many messages it was sent;

• Mocks are pre-programmed with expectations which form a specification of the calls they are expected to receive. They can throw an exception if they receive a call they don't expect and are checked during verification to ensure they got all the calls they were expecting.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 123: Drupal Continuous Integration (European Drupal Days 2015)

PHPUnitTEST DOUBLES

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 124: Drupal Continuous Integration (European Drupal Days 2015)

PHPUnitOPTIONS

PHPUnit features a long list of options to configure its behaviour that’s accessible both from command line and from a configuration file.

The former is a convenient way to quickly get started and try thing out, but on the long run the phpunit.xml file is probably a more convenient choice to configure the tool.

It’s a good practice to commit a phpunit.xml.dist file and (git) ignore the phpunit.xml one: by doing such thing the other developers will be able to create a phpunit.xml file tailored to their own needs.

PHPUnit will automatically use phpunit.xml if present or phpunit.xml.dist if not.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 125: Drupal Continuous Integration (European Drupal Days 2015)

PHPUnitOPTIONS BEST PRACTICES

It’s a good practice to commit a phpunit.xml.dist file and (git) ignore the phpunit.xml one: by doing such thing the other developers will be able to create a phpunit.xml file tailored to their own needs, leaving the phpunit.xml.dist targeted at automatic builds, fore example.

PHPUnit is smart enough to automatically use phpunit.xml if present or phpunit.xml.dist if not.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 126: Drupal Continuous Integration (European Drupal Days 2015)

PHPUnitOPTIONS

Page 127: Drupal Continuous Integration (European Drupal Days 2015)

PHPUnitBEYOND THE DEFAULT OPTIONS

There are more tools provided by the community than what’s implemented by PHPUnit alone. Here’s a short list of some relevant ones:

• Faker - a library that generates fake-yet-plausible data for a handful of typical use cases such as names, addresses, numbers, dates, images, etc…

• Mockery - a powerful mocks creation library;

• Prophecy - a new, strongly-opinionated mocks creation library;

• PHP-VCR - a library to record and replay HTTP calls in tests;

• TuTu - a simple and flexible HTTP mocking tool.

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 128: Drupal Continuous Integration (European Drupal Days 2015)

Q & A

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/

Page 129: Drupal Continuous Integration (European Drupal Days 2015)

PHPUnitEXERCISES

1. What’s Unit Testing?

2. What are the main types of tests?

3. What are the benefits of writing the tests before the code?

4. What’s an assertion?

5. What’s a fixture?

6. What’s a data provider?

7. What’s the difference between a Stub and a Mock?

8. What’s the purpose of the phpunit.xml file?

Drupal Continuous Integration - By Claudio Beatrice - Copyright 2014http://tenwarp.com/