Bringing Solr to DrupalA General and a Library-Specific Use Case

Király PétereXtensible Catalog

1

2

Two ways of using Solr in DrupalGeneral solution: Apache Solr Search Integration and related

modules: Stats, Autocomplete, Multisite, Ajax, Biblio, Attachments, Übercart (e-commerce integration), Views, Multilingual, Geospatial and many othersFits for the overall needs, uses predefined fields

Library specific solution: eXtensible Catalog modulesFits for library needs, uses dynamic fields

3

Part 1 – the general solutionGeneral solution: Apache Solr Search Integration and related

modules: Stats, Autocomplete, Multisite, Ajax, Biblio, Attachments, Übercart (e-commerce integration), Views, Multilingual, Geospatial and many othersFits for the overall needs, uses predefined fields

Library specific solution: eXtensible Catalog modulesFits for library needs, uses dynamic fields

This part of the presentation is based on the works and previous presentations ofRobert Douglass.

4

Drupal architecture

Relational databasefor storage

Solr index for search

5

Purposes

Creating a general insfrastructure which is usable in every Drupal installation

Core module, and additional module for covering specific Solr functionalities (statistics, autocomplete etc.)

Replace the original (and still default) MySQL-based search feature

6

sort

Facet 1

Facet 2

7

categories

API version

module search

sort

List of modules page

8

Advanced search: issues

9

Whitehouse.gov on Drupal & Solr

sort

facets

10

Boosting by Drupal specific properties

11

Boosting and ignoring by document type

12

Boosting by fields / HTML tags

13

More like this implementation

14

Solr in Views integration

Views is a very popular module, helps creating interactive DB queries andresult pages. Now it can handle Apache Solr as data source.

15

Part of the Views admin page

You can specify fields,

sorting, filters, layout,arguments, behaviours and more

16

Using Tika: file search

17

Indexing/searching multiple sites

18

Search in comments

19

CCK date searching

Content Construction Kit: popular module to create document and field types. CCK date is a special field type handling dates.

20

statistics

impressive numbers – that’s why welove Solr…

21

Statistics of facet usage

22

autocomplete

23

Future plansCrawling with Nutch

Geospatial search

eDismax (Solr 1.5)

Drupal 7

API changes

Improving documentation

24

People behind these modulesRobert Douglass (DE) http://drupal.org/user/5449

Alejandro Garza (MX) http://drupal.org/user/153120

Peter Wolanin (US) http://drupal.org/user/49851

James McKinney (CA) http://drupal.org/user/472460

Scott Reynolds (US) http://drupal.org/user/60009

Mike O'Connor (US) http://drupal.org/user/104525

Markus Kalkbrenner (DE) http://drupal.org/user/124705

and others…

25

Links apachesolr http://drupal.org/project/apachesolr (this is the best starting point)

content recommendation patch http://drupal.org/node/372767

views integration http://drupal.org/project/apachesolr_views, http://acquia.com/node/911667

file search http://drupal.org/project/apachesolr_attachments, http://acquia.com/node/1129446

date facet for CCK field http://drupal.org/node/558160

statistics http://drupal.org/project/apachesolr_stats

multisite http://drupal.org/project/apachesolr_multisitesearch

autocomplete http://drupal.org/project/apachesolr_autocomplete

26

Part 2 – Library specific solutionGeneral solution: Apache Solr Search Integration and related

modules: Stats, Autocomplete, Multisite, Ajax, Biblio, Attachments, Übercart (e-commerce integration), Views, Multilingual, Geospatial and many othersFits for the overall needs, uses predefined fields

Library specific solution: eXtensible Catalog modulesFits for library needs, uses dynamic fields

27

About eXtensible Catalog

a project creating an open source next generation library ‘discovery interface’ and an FRBR-based metadata platform

started in 2007

driven by new theories of library science, cultural anthropology and the practice of web 2.0, library 2.0

Universities of Rochester, Notre Dame, Cornell, North Carolina at Charlotte, Rochester Institute of Technology and CARLI consortium

28

Architecture Drupal CMS

Integrated Library System Repository

XC Metadata Services Toolkit

XC OAI ToolkitXC NCIP Toolkit

MARC NormalizationDC Normalization

Aggregation Transformation

XC Drupal Toolkit

circulation data

29

Purposes of XC Drupal Toolkit integrate library data into a popular content management system

customizable functionalities

customizable interface(s)

internationalization, localization

5000+ custom modules, 20+ library specific modules

wide range on mashup options

all features are available through user interfaces

30

Search results

highlighted terms

availability information

bibliographical data

cover images

facets

31

Customized interface (Kyushu University)

32

Similar documents

33

XML attribute handling<subject type="OCLC">History</subject>

subject=„History” and subject_OCLC=„History”subject_OCLC=„History”subject=„History” and subject_type=„OCLC”nonecould be indexed as…

34

Mapping schema fields to Solr types

35

Setup a facet

Aggregating values ofdifferent fields into onefacet

custom PHP code to modify field values(conditions)

specifySolr type

36

Custom PHP code for displaying title

37

Getting records into Drupal: OAI harvesting

Harvest is running

List of scheduled harvests

38

Data flow between components

OAI-PMHprovider Drupal

MySQL Solr

batch delete/insert documentscreating nodes

39

Creating a ‘more like this’ parameter set

Saving parameters for ‘More like this’ functionality

40

Creating highlighter

wrapper around the highlighter’s parameters

41

Setting up field and date facet properties

Date facet properties

Field facet properties

42

Putting facets together: ‘facet group’

List of facets, and their type

General properties

43

Reordering facets

Just drag and drop

You haven’t saved changes!

44

Using facet term list in search form

dropdown fullfilled withlanguagefacet terms dropdown definition

45

Adding widgets to UI: navigation bar

definition of navigation bar

navigation bar in action

46

Links Project page http://eXtensibleCatalog.org

XC Drupal Toolkit http://drupal.org/project/xc

Metadata Services Toolkit http://code.google.com/p/xcmetadataservicestoolkit

OAI Toolkit http://code.google.com/p/xcoaitoolkit

NCIP Toolkit http://code.google.com/p/xcnciptoolkit

Developers:

Mlen-Too Wesley (GH) http://drupal.org/user/318924

Király Péter (H) http://drupal.org/user/352587, http://twitter.com/kiru