Perl, CGI and DBI Ashley George [email protected] March 7th, 2005.
Perl DBI Scripting with the ILS
-
Upload
roy-zimmer -
Category
Technology
-
view
1.062 -
download
2
description
Transcript of Perl DBI Scripting with the ILS
![Page 1: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/1.jpg)
Perl DBI Scripting with the ILS
Roy Zimmer
Western Michigan University
![Page 2: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/2.jpg)
What do I need for database interactions with Perl?
![Page 3: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/3.jpg)
What do I need for database interactions with Perl?
DBI – DataBase Interface (always)
![Page 4: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/4.jpg)
What do I need for database interactions with Perl?
DBI – DataBase Interface (always)
DBD – DataBase Driver (one for each type of database you must access)
![Page 5: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/5.jpg)
What do I need for database interactions with Perl?
DBI – DataBase Interface (always)
DBD – DataBase Driver (one for each type of database you must access)
a database? (some DBDs let you access non-database data, such as CSV files)
![Page 6: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/6.jpg)
What do I need for database interactions with Perl?
DBI – DataBase Interface (always)
DBD – DataBase Driver (one for each type of database you must access)
a database? (some DBDs let you access non-database data, such as CSV files)
some Perl proficiency
![Page 7: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/7.jpg)
What do I need for database interactions with Perl?
DBI – DataBase Interface (always)
DBD – DataBase Driver (one for each type of database you must access)
a database? (some DBDs let you access non-database data, such as CSV files)
some Perl proficiency
But we only care about using our Voyager database, so we’ll stick with that.
![Page 8: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/8.jpg)
What does this mean to you?
![Page 9: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/9.jpg)
What does this mean to you?
DBI – DataBase Interface (already on your Voyager box)
![Page 10: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/10.jpg)
What does this mean to you?
DBI – DataBase Interface (already on your Voyager box)
DBD – DataBase Driver (for , already on your Voyager box)
![Page 11: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/11.jpg)
What does this mean to you?
DBI – DataBase Interface (already on your Voyager box)
DBD – DataBase Driver (for , already on your Voyager box)
a database? (already on your Voyager box)
![Page 12: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/12.jpg)
What does this mean to you?
DBI – DataBase Interface (already on your Voyager box)
DBD – DataBase Driver (for , already on your Voyager box)
a database? (already on your Voyager box)
some Perl proficiency (the only thing you supply!)
![Page 13: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/13.jpg)
A simple program
Always needed for database access
nameletters.pl
![Page 14: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/14.jpg)
A simple program
Get access information from a setup file and connect to a database.
File format (the only record):
library.box.university.edu/username/password/VGER
nameletters.pl
![Page 15: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/15.jpg)
A simple program
Connect to the database and get a handle.
nameletters.pl
![Page 16: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/16.jpg)
A simple program…for each letter, how many patrons’ last names start with that letter
Create the query
sprintf is your friend
A sample query here is:
select count (*) from wmichdb.patron where last_name like ‘S%’
nameletters.pl
![Page 17: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/17.jpg)
A simple program…for each letter, how many patrons’ last names start with that letter
Prepare the query, associating it with a database, giving it a handle.
nameletters.pl
![Page 18: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/18.jpg)
A simple program…for each letter, how many patrons’ last names start with that letter
Run the query, get a return code.
nameletters.pl
![Page 19: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/19.jpg)
A simple program…for each letter, how many patrons’ last names start with that letter
Get the query result…
nameletters.pl
![Page 20: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/20.jpg)
A simple program…for each letter, how many patrons’ last names start with that letter
…and print it.
nameletters.pl
![Page 21: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/21.jpg)
A simple program…for each letter, how many patrons’ last names start with that letter
A: 003016B: 007113C: 005041D: 003792E: 001322F: 002605G: 003603H: 005388I: 000368J: 001970K: 004371L: 003763M: 007039N: 001622O: 001299P: 003792Q: 000121R: 003770S: 008217T: 002528U: 000248V: 001791W: 004487X: 000018Y: 000562Z: 000709
Output
nameletters.pl
![Page 22: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/22.jpg)
Nested query example…getting barcodes of retirees who’ve borrowed something within the past year
retirees.pl
![Page 23: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/23.jpg)
Nested query example…getting barcodes of retirees who’ve borrowed something within the past year
Outer query
Get patron information that meets the criteria, and connecting information for barcodes.
(sprintf actually not needed here)
retirees.pl
![Page 24: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/24.jpg)
Nested query example…getting barcodes of retirees who’ve borrowed something within the past year
Inner query
Remember the outer query?
Note the positional correspondence of the query parameters with the while statement receiving a query row.
retirees.pl
![Page 25: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/25.jpg)
Nested query example…getting barcodes of retirees who’ve borrowed something within the past year
Inner query
The inner query has a different query string name…
retirees.pl
![Page 26: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/26.jpg)
Nested query example…getting barcodes of retirees who’ve borrowed something within the past year
Inner query
…as do the other variables for this query.
retirees.pl
![Page 27: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/27.jpg)
Nested query example…getting barcodes of retirees who’ve borrowed something within the past year
Inner query
Doe, John R doe1 pgroup: EMERIT/RET
21141002502810 08/16/1999 Other
99952803601000 11/25/2001 Expired
Doe, Josephine doe2 pgroup: RETIREDSTF
21141001297289 08/16/1999 Other
21141002300660 08/16/1999 Other
11130755000000 09/04/2002 Expired
retirees.pl
Sample output:
![Page 28: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/28.jpg)
Termination Notes
$sth->finish finish a query
$dbh->disconnect disconnect from a database
finish seems like a good practice, but it’s not really needed. Yet…
disconnect isn’t really needed (for our purposes, read-only). However, without using the finish statement, you could run into error situations.
leading us to…
![Page 29: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/29.jpg)
CPAN
the Comprehensive Perl Archive Network
Great resource for *all* kinds of Perl modules
Documentation for modules
- comprehensively so for DBI and DBD
- about 100 pages for DBI and 50 pages for DBD
![Page 30: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/30.jpg)
A bit more documentation…
$sth->fetchrow_array
returns a single value, array, or list
There are a number of other ways to get query data, but this seems to work best.
You can also get lots of details about your database environment via DBI calls. See the DBI documentation for more information.
![Page 31: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/31.jpg)
A quick little DBI utility
dbddrivers.pl
![Page 32: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/32.jpg)
A quick little DBI utility
Available Perl DBD drivers on this systemDBMExamplePFileGoferOracleProxySponge
Output
dbddrivers.pl
![Page 33: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/33.jpg)
What if you need to look at every bib record you’ve got?
![Page 34: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/34.jpg)
What if you need to look at every bib record you’ve got?
And you need to access the marc record (synonymous with )?
Here’s one approach…
THE BLOB
![Page 35: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/35.jpg)
connect to the database
query bib_data table to get the maximum bib ID
set increment to 50,000
set ending bib ID to increment
set beginning bib ID to 0
Traversing every record and dealing with THE BLOB along the way
![Page 36: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/36.jpg)
connect to the database
query bib_data table to get the maximum bib ID
set increment to 50,000
set ending bib ID to increment
set beginning bib ID to 0
while beginning bid ID < maximum bib ID
chunkthroughdb()
provide feedback of progress
beginning bib ID = ending bib ID + 1
add increment to ending bib ID
end
provide final feedback
Traversing every record and dealing with THE BLOB along the way
![Page 37: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/37.jpg)
sub chunkthrudb
query bib_data for blob data based on bib IDs >= beginning bib ID and
bib IDs < ending bib ID
for each record from the query
assemble each bib ID's data into a marc record
do the required processing for this record
end
end
Traversing every record and dealing with THE BLOB along the way
![Page 38: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/38.jpg)
Traversing every record and dealing with THE BLOB along the way
Discussion of this approach
Alternative method: read the whole database at once!
- might be impossible
- might not be feasible
- probably not efficient
The program presented here…seems to be the better method.
![Page 39: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/39.jpg)
Traversing every record and dealing with THE BLOB along the way
Discussion of this approach
If nothing else, “chunking” your way through your database is more efficient
We have close to 1.6 million bib records. Based on the program we’re about to see, traversing our database…
using 50,000 record chunks takes about 50 minutes
without “chunking” it takes about 76 minutes
![Page 40: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/40.jpg)
Traversing every record and dealing with THE BLOB along the way
findbadleader.pl
![Page 41: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/41.jpg)
Traversing every record and dealing with THE BLOB along the way
Get our boundary condition
findbadleader.pl
![Page 42: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/42.jpg)
Traversing every record and dealing with THE BLOB along the way
0-50,000 – our first chunk of records
findbadleader.pl
![Page 43: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/43.jpg)
Traversing every record and dealing with THE BLOB along the way
the loop
findbadleader.pl
![Page 44: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/44.jpg)
Traversing every record and dealing with THE BLOB along the way
set up the next chunk
the loop
findbadleader.pl
![Page 45: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/45.jpg)
Traversing every record and dealing with THE BLOB along the way
- chunking subroutine (chunkthrudb)
“seqnum desc” is key to getting the blob data for each record
This gets one chunk’s worth of MARC records (blob data)
findbadleader.pl
![Page 46: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/46.jpg)
Traversing every record and dealing with THE BLOB along the way
- chunking subroutine (chunkthrudb)
Creates the MARC record for each bib retrieved.
For larger records that don’t fit in one table row, assemble the record in reverse order (from the query).
findbadleader.pl
![Page 47: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/47.jpg)
Traversing every record and dealing with THE BLOB along the way
- chunking subroutine (chunkthrudb)
Process each record.
In this case, we’re looking for records with bad leaders (not ending in “4500”).
Sample output:
findbadleader.pl
![Page 48: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/48.jpg)
Being in a is a good thing…bind
sprintf is a friend, but bind is a better friend
![Page 49: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/49.jpg)
Being in a is a good thing…bind
sprintf is a friend, but bind is a better friend
usual query method
![Page 50: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/50.jpg)
Being in a is a good thing…bind
sprintf is a friend, but bind is a better friend
usual query method
query with bind values method
![Page 51: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/51.jpg)
testbind.pl
Illustrating and contrasting the query with bind method
create, prepare, and execute a query
![Page 52: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/52.jpg)
testbind.pl
Illustrating and contrasting the query with bind method
prepare and execute a query
![Page 53: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/53.jpg)
testbind.pl
Illustrating and contrasting the query with bind method
prepare and execute a query, improved
prepare moved outside of the loop
![Page 54: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/54.jpg)
testbind.pl
Illustrating and contrasting the query with bind method
Using testbind.pl for about 80,000 patron records, these are the results:
Create, prepare and execute a query
20
Prepare and execute a query (with bind)
11
Prepare and execute a query, improved (with bind)
10
Method Runtime (seconds)
![Page 55: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/55.jpg)
Running reports, queries via Perl on your PC
ActiveState sells products and also has free versions. I recommend their free Perl.
Get the MSI version.
It comes with DBI and DBD for Oracle, among many other modules (and lots of documentation).
Still (!) requires external Oracle client software (at least as of version 5.10.0, build 1003)
![Page 56: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/56.jpg)
The files listed below are available at
http://homepages.wmich.edu/~zimmer/files/eugm2008
dbi.ppt this presentation
nameletters.pl for every letter, find number of patrons whose lastnames begin with that letter
retirees.pl get barcodes of retirees who’ve borrowed somethingin the last year
dbddrivers.pl get list of DBD drivers on your system
findbadleader.pl find bib records with bad leaders (not ending in 4500)
testbind.pl demonstrates varying query method efficiencies
Resources
![Page 57: Perl DBI Scripting with the ILS](https://reader033.fdocuments.us/reader033/viewer/2022061210/5490b0eab47959ed448b458b/html5/thumbnails/57.jpg)
CPAN http://cpan.org
DBI http://search.cpan.org/~timb/DBI-1.605/DBI.pm
DBD Oracle http://search.cpan.org/~pythian/DBD-Oracle-1.21/Oracle.pm
Active State Perl
http://activestate.com/downloads/index.mhtml
There is also a book on Perl and DBI. I’d recommend
using it along with the most current documentation
from the CPAN sites above.
Resources