http://popcorn.cx/

Square pegs and round holes

A reflection on Class::DBI

http://popcorn.cx/

Who am I?

• Developer at Monash University

• Applications in and related to the staff and student portal

• Built upon open source– Perl, HTML::Mason, Apache

http://popcorn.cx/

Class::DBI

• Database abstraction layer for Perl

• Allows mapping of classes to database tables

• Each row of the table is an instance of the class

http://popcorn.cx/

The square peg

User directory

Student records

LearningManagement

System

Teaching staff

http://popcorn.cx/

Initial development

• Bulk written in Perl, remainder PL/SQL and Java

• Chose Class::DBI as it gave us a leg up due to time constraint– But not as much as we expected

http://popcorn.cx/

Clash of standards

Music::CD->has_a(

artist => 'Music::Artist’

);

print $cd->artist->name;

http://popcorn.cx/

Clash of standards

Music::CD->has_a(

artist_id => 'Music::Artist’

);

print $cd->artist_id->name;

http://popcorn.cx/

Clash of standards

Music::CD->has_a(

artist_id => 'Music::Artist’

);

*artist = \&artist_id;

print $cd->artist->name;

http://popcorn.cx/

Clash of standards

Music::CD->has_a(

artist_id => 'Music::Artist’

);

*get_artist = \&get_artist_id;

*set_artist = \&set_artist_id;

print $cd->get_artist->name;

http://popcorn.cx/

Playing nice with friends

• Our expectation was for error at runtime

• If remote service is not available:– Module loads– Only when used is error indicated

• By return value or an exception

http://popcorn.cx/

Playing nice with friends

• Class::DBI connects to database, and errors, at module load

• Which mean an error at load paradigm• If database is unavailable

– Module will not load– Thus stopping remaining code from loading

• We needed to be more defensive than usual and wrap the use in an eval

http://popcorn.cx/

I canna do it captain!

• Takes XML extract of all user accounts - upwards of 90,000

• Compares against data in database

• Updates accordingly

http://popcorn.cx/

First attempt

• Run through extract and call retrieve() for each record

• 90,000 database queries

• Database bottleneck

• Runtime of over an hour

http://popcorn.cx/

Second attempt

• Load all objects into memory via retrieve_all()

• Single database query– no longer the bottleneck

• Now encounter memory allocation issues

• Still a runtime of over an hour

http://popcorn.cx/

Third attempt

• Avoid Class::DBI and use DBI directly

• Single database query gave all records in an array of hash references

• Then we only call retrieve() if a change is required

• Runtime of less than ten minutes

http://popcorn.cx/

Fourth attempt

• Did not try as third attempt gave satisfactory results

• Would have probably loaded entire extract and then used iterator from retrieve_all()

http://popcorn.cx/

Overview of initial development

• A success

• Built working system in a few weeks

• Class::DBI gave significant head start. Although we needed to:– learn how to use it– bend to our standards– avoid in some cases for performance

http://popcorn.cx/

Maintenance and enhancement

• Now been running for over two years

• Huge increase in usage– Second largest installation in the world

• Also expansion of functionality

• Can at times be frustrating

http://popcorn.cx/

Where is the object?

print $section->get_course();

• Many hours wasted• A number is printed • Not the expected blessed referent• Due to stringification feature of Class::DBI

– That we had not mentioned in our documentation

http://popcorn.cx/

Unforeseen implications

• Extended update(), create() and delete() to require a username for an audit trail– $object->delete($username)

• Worked well

• Until we tried to use features such as cascade delete

http://popcorn.cx/

Unforeseen implications

• Class::DBI follows ‘has many’ relationships and calls delete()

• Does not know to pass our username• Which makes it fail• Our solution to a business need limited the

features we could use• Better knowledge of Class::DBI internals

should provide a better solution

http://popcorn.cx/

Building a round peg

Course database Public website

Administrators

http://popcorn.cx/

Development

• All Perl solution– Except for Oracle database

• Perl CGI for public website

• Perl under HTML::Mason for administration interface

• Perl for batch load process

http://popcorn.cx/

Class::DBI or not Class::DBI

• Our data model led to a matching set of classes and database schema

• But– Class::DBI was not available on the target

environment– Recent issues (at the time) soured our

opinion of Class::DBI

• Decided to build our classes from scratch

http://popcorn.cx/

It lives

• Quickly became apparent that some form of framework was needed.

• So we built our own:

http://popcorn.cx/

Basic setup

use base 'Base';

__PACKAGE__->_set_table_name( ’table' );

__PACKAGE__->_setup_attribtes(

qw{ code name ... },

);

http://popcorn.cx/

‘has_a’ relationship

__PACKAGE__->_setup_has_a_relationship(

'class' => 'Faculty',

'get_method' => 'get_faculty',

'set_method' => 'set_faculty',

);

http://popcorn.cx/

‘has_many’ relationship

__PACKAGE__->_setup_has_many_relationship(

'class' => 'OtherClass',

'table' => 'other_table',

'get_method' => 'get_others',

'add_method' => 'add_other',

'remove_method' => 'remove_other',

);

http://popcorn.cx/

‘has_many’ relationship

__PACKAGE__->_setup_has_many_relationship(

'class' => 'OtherClass',

'table' => 'other_table',

'get_method' => 'get_others',

);

http://popcorn.cx/

Compensating for ‘issues’

• Built to expect our standards• Follows error at runtime paradigm• Still need to bypass in some cases• Only built what we needed/wanted

– So no stringification to primary key

• Natively requires username for audit trail

http://popcorn.cx/

Was it worth it?

• Course Finder project was a success

• Framework allowed later changes to be implemented quickly

• We knew exactly how our framework worked

http://popcorn.cx/

Would we do it again?

• No.

• Because we can reuse this framework

• Which we have done in one other project and fed enhancements back

http://popcorn.cx/

Another peg

• There are other database abstraction frameworks:– DBIx::Class is biggest alternative

• Either didn’t exist or appear on our radar at the time

• They are not excluded for future projects

http://popcorn.cx/

Conclusion

• I don’t really have one…

http://popcorn.cx/

Conclusion

• Any framework is beneficial

• But you need one that does what you need

• And you will need to bypass it

http://popcorn.cx/

Questions?

http://popcorn.cx/

• Blackboard Learning System Vista Enterprise License Release 3 Patch something or other. Potato!

http://popcorn.cx/

This slide left intentionally blank