Lipstick On a Pig (+Audio)

25
30 th June 2009 Europython 2009, Birmingham, UK 1 Lipstick on a Pig Dynamically re-skinning a legacy .NET portal with python Matt Hamilton [email protected]

description

Dynamically skinning a legacy portal using Python, WSGI (the Python Web Server Gateway Interface), and Deliverance. So you have a big legacy portal application which you want to change the look of, but are contractually not allowed to touch? Here is a case study on how we used the power and flexibility of Python and WSGI and the wonder lxml to dynamically re-skin a proprietary .NET portal without even touching it. We take a giant lump of messy invalid HTML markup and dynamically strip it back, add semantic markup and CSS and present the user with a nice svelte valid site. I will cover the history of the legacy portal, the problems encountered, our cunning plan to dynamically re-skin the site, a technical overview of the parts of the system (lxml, WSGI, etc), and what we learned along the way.

Transcript of Lipstick On a Pig (+Audio)

Page 1: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 1

Lipstick on a Pig

Dynamically re-skinning a legacy .NET portal with python

Matt [email protected]

Page 2: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 2

Introduction

Dynamically re-skinning a .NET portal site Can't name the client Portal for teachers in the UK Aggregating content across legacy portal, Plone

and Moodle

Page 3: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 3

Who Am I

Technical Director of Netsight→ Web development firm in Bristol, UK

10 years experience with Zope/Plone More of an integrator than core developer

→ I get involved in all those sticky projects of merging Plone in with other systems in an enterprise

Page 4: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 4

Where Were We?

Page 5: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 5

Existing Portal (1.0)

Five years old by November 2009 User registrations: 46,681 Course enrolments: 33,664 Resource Bank views: 247,911

Page 6: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 6

Existing Portal (1.0)

Page 7: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 7

Problems with Current Portal Look-and-feel

→ Not very compelling Usability

→ Challenging in places A poor content management system

→ Can't really edit general content, so use a separate FTP server and Dreamweaver

Vendor lock-in→ Even small changes, very expensive

Page 8: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 8

The Future - Portal 2.0

Usability, Design and Content Review April 2008

Strategic Review August 2008 Feasibility Studies Jan 2009 Pilot Demonstrator (“Portal V1.5”) March 2009

Page 9: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 9

Design Review

Page 10: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 10

Architecture Review

Portal 1.0 - Monolithic, tightly coupled, poor separation of skin

Page 11: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 11

Architecture Review

Portal 2.0 - Extensible, loosely coupled, good separation of skin

Page 12: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 12

How Do We Get There?

Remember: We Can't Touch the Existing System!

Page 13: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 13

The Cunning Plan

Skin 1.0

Portal 1.0

ExistingPortal Module of

functionality e.g. portfolio

BrowserBrowser

Page 14: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 14

Total Skin Graft!

Portal 1.5New skin via

xpath and xslttransformation.

Web server needs to handle

SSL.

Together they give us nice

URLs.

Skin 1.0

web server

transformation proxySkin 2.0

BrowserBrowser

Page 15: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 15

Deliverance

Several Different Projects→ xdv→ Deliverance 0.3

To learn more on specifics of Deliverance go to http://deliverance.openplans.org

Page 16: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 16

WSGI

WSGI allows you to write small modules chained together in a 'pipeline'

Many small filters combined together as you need

Lots of existing components out there Very easy to write new ones

Page 17: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 17

WSGI Power - The Pipeline

[pipeline:portal]pipeline = theme.portal ploneinterceptor xslt linkrewrite htmlcleaner source.portal

[filter:theme.portal]use = egg:dv.xdvserver#xdvtheme_uri = file://%(here)s/theme/theme.htmlrules = %(here)s/rules/content.xml

[filter:ploneinterceptor]use = egg:ns.ploneinterceptor#ploneinterceptor

[filter:xslt]use = egg:dv.xdvserver#xsltxslt_file = %(here)s/rules/transform.xsl

[filter:linkrewrite]use = egg:ns.linkrewrite#linkrewrite

[filter:htmlcleaner]use = egg:ns.htmlcleaner#htmlcleaner

[app:source.portal]use = egg:Paste#proxyaddress = http://www.theexistingsite.org.uk/

Page 18: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 18

Link Rewriting

Old URL:→ http://www.theclient.org.uk/WebPortal.aspx?page=1&module=DB920A53-01EA-4886-8878-F2CDF5FA8CFD&mode=101&IsNonNewsDB920A53_01EA_4886_8878_F2CDF5FA8CFD=True&newsIdDB920A53_01EA_4886_8878_F2CDF5FA8CFD=11208#10

→ 205 characters! New URL:

→ http://www.theclient.org.uk/news/11208#10→ 41 characters!

Page 19: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 19

HTML Cleanup

LXML rules!→ from lxml.html.clean import Cleanercleaner = Cleaner(...)# Pretty print the HTMLdom = document_fromstring(body)body = etree.tostring(dom, pretty_print=True)# Clean the HTMLbody = cleaner.clean_html(body)

Page 20: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 20

Result

Old→ 70kb of HTML→ 120 Validation errors, 61 warnings

New→ 40Kb of HTML→ 27 Errors, 1 warning (mainly xhtml/html conflicts)→ No significant performance impact

Page 21: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 21

Putting all togetherComposite:main

pipeline:portal pipeline:plone pipeline:moodle

theme.content

navmerger

plonecontent

source.plone

theme.content

xslt

linkrewrite

htmlcleaner

theme.content

navmerger

moodlecontent

source.moodle

source.portal

Page 22: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 22

End ResultNew Style

Portal content

Page 23: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 23

Complications

Navigation→ One page, two content sources, how is the

navigation built? Search

→ Search needs to go across multiple systems→ Will soon be looking at Solr, Xapian, Google Mini

.NET viewstate postback→ Massive hidden state variable, form wraps entire

site!

Page 24: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 24

Questions?

[email protected]

Page 25: Lipstick On a Pig (+Audio)

30th June 2009 Europython 2009, Birmingham, UK 25

We are looking for Developers!

Come chat to me or

drop an email to

[email protected]