Load Profile Training
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Agenda – Day One
Introductions Review of training goals Review of Innovative Record Structure Anatomy of a Data Load Load Tables (m2btab) Review of standard load profiles Global Variable Functions (@)
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Agenda – Day Two
Questions from day one Special Processing Functions (%) Command Functions (#) Translation Tables (m2bmap) Data analysis The m.marcload.local file Test loading records Review and support from Innovative
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Training Goals
Understand the database structure Understand existing load profiles Be familiar with issues to consider for
new load profiles Update and create load tables (m2btab) Create translation tables (m2bmap) Add menu options for loading records Test load profiles before production
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Topics Not Covered in This Training
Text to MARC conversion programs Loading non-MARC records Loading Contact, Course, Checkin
Cards, Invoice, License, Program, Resource, Session or Vendor records
Complex data manipulation Export tables
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Record Structure
Types of records CSDirect @ csdirect.iii.com (see chart) Innovative Guide and Reference no.
101113 The Getting Started Manual @
gsm.iii.com
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Each Record Type Contains:
Variable-Length Fields M>I>C>T and M>I>C>M
(Page 7 and IGR page 105783) Fixed-Length Fields
(Page 14 and IGR page 105775)
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Anatomy of a Data Load
Load Table(m2btab)
Loading prog.(marc2inn)
Trans. File(bibtemp)
Holding symboltable
m.marcload.local
Data Exchange Load Interface
Translation Table(m2bmap)
Fixed-length fields
Fixed-lengthvalues/codes
Variable-lengthfields
RecordTemplates
Indexing prog.(control)
Indexes
Database
Index rules
Hi/Lo watermark
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Load Tables (m2btab)
Control how records are loaded Fields in the incoming record that do not
match load table criteria will not be loaded Create separate load tables for different
loading requirements Data must be in MARC format
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Standard Load Tables
Authority records m2btab.a (p. 140) m2btab.anam (p. 142) m2btab.asub (p. 144)
Bibliographic and Item records m2btab.batch (p. 130) m2btab.lti (p. 137)
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Standard Load Tables
Bibliographic and Order records m2btab.order (p. 133)
Patron records m2btab.p (p. 146)
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Overview of Load Tables
Commenting out lines with “#” Twelve data elements Global Variable Functions (@) Special Processing Functions (%) Command Functions (#) “#” has 2 meanings in load tables
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Comment Lines in Load Tables
Any line that begins with “#” is a Comment
Include a brief summary of load table, identify the author and creation date
It is strongly recommended that you add a Comment to every load table you create
Commented lines are not read by the loading program
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Element 1 – MARC Tag 690|699|+|0|0|b|d|0|y|N|0|%first=“y”
MARC tags and indicators to be loaded Can be set as a regular expression Can load from any MARC field and the
Leader
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Element 2 – New MARC Tag 690|699|+|0|0|b|d|0|y|N|0|%first=“y”
Re-map the data from the tag in element one to the new MARC tag in element two
Cannot be a regular expression If indicators are specified, all indicators
must be present
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Element 3 – Subfields 690|699|+|0|0|b|d|0|y|N|0|%first=“y”
Indicates which subfields to load from MARC field in Element 1, MARC
tag Unless specified, subfields are loaded in
the order that they appear in the incoming data
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Element 4 - Offset 008|| |35|3|b| |24|n|N|0|lang(b) 690|699|+|0|0|b|d|0|y|N|0|%first=“y”
Used for loading data from positional fields, e.g. Leader, 006, 007, 008
Counting begins at 0 Loading data from single subfield, refers
to beginning of subfield Must be set to “0” when loading the
entire field (usual setting)
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Element 5 – Byte Count 008|| |35|3|b| |24|n|N|0|lang(b) 690|699|+|0|0|b|d|0|y|N|0|%first=“y”
Number of bytes to load starting from offset in Element 4
Counting begins at 1 Must be set to “0” when loading the
entire field (usual setting)
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Example of MARC 008 field
820305s1991####nyu###########001#0#eng#
Language code at byte 35 of 008 field:
Offset = 35
Byte count = 3
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Element 6 – Record Type 690|699|+|0|0|b|d|0|y|N|0|%first=“y”
Determines the record type to load the MARC data into for this line This can be set to:
“b” (Bibliographic) or “i” (Item) or “o” (Order) or “c” (Holdings/Checkin) or “a” (Authority) or “p” (Patron)
Will always correspond with Element 11, Pass Indicator
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Element 7 – Variable Field Group Tag 690|699|+|0|0|b|d|0|y|N|0|%first=“y”
Indicates the variable field group tag to load data into
Each line can map to a variable field or a fixed field, but not both
This should be set to “space” if using the line to load a fixed-length field
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Element 8 – Fixed Field No. 690|699|+|0|0|b|d|0|y|N|0|%first=“y”
Indicates the fixed-length field to load data into
Fixed-length fields must be referenced by line numbers. See page 14
If the line is used to load a variable-length field then this element should be “0”
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Element 9 – MARC/Non-MARC Indicator 690|699|+|0|0|b|d|0|y|N|0|%first=“y”
Determines if the incoming field is loaded in MARC format or not
Works in conjunction with the @marc trigger
Load lines for fixed-length fields always have a non-MARC indicator, “n”
Variable fields usually have a MARC indicator, “y”
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Element 10 - Permanence 690|699|+|0|0|b|d|0|y|N|0|%first=“y”
Used for loading during Interactive hardwired Serial Port Interface
Determines whether that setting is retained or not for subsequent loads
Usually set to “N” for most lines in a load table since Serial Port Interface is not used much
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Element 11 – Pass Indicator 690|699|+|0|0|b|d|0|y|N|0|%first=“y”
Determines at which pass the line is read during the load
Will always correspond with Element 6, Record Type
Each new record type loaded requires a new pass
Primary record type is always pass 0
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Element 12 – Function or Note 690|699|+|0|0|b|d|0|y|N|0|%first=“y”
Global Variable Function (@) Command Function (#) Special Processing Function (%) Text to be inserted Any other beginning character is a
comment and is not read by the loading program
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Training Exercise 1
“Creating and Evaluating Load Table Entries”
Turn to page 120 and complete training exercise 1
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Global Variable Functionsaka Triggers (@)
Specified in 998 or 999 lines that have “@” in Element 12, e.g. @main
Controls how the incoming record is loaded
Options are controlled on a “global” level and affect all records loaded
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Global Variable Functions
Used to control the following overlay options default record templates used holdings symbol processing field protection automatic dates in records and many more load options...
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Commonly Used Triggers
These triggers are found in most load tables:
- @main - @marc - @msg - @recs (not found in Authority load tables) - @atab (not found in Patron load tables)
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
@main
This trigger identifies the primary record to be created during the load
Will always correspond with Element 11, Pass Indicator
This can only be set to: “b” for Bibliographic records “a” for Authority records “p” for Patron records
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
@marc
This trigger is used to determine which records can have MARC fields
The settings of this trigger corresponds with Element 9, MARC/Non-MARC Indicator
Typically this is set for Bibliographic, Item, Holdings/Checkin and Authority records (and not Patron records)
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
@msg
This trigger adds a message to the top of the Output Messages display when a record load starts in Data Exchange
Usually used to specify types of records loaded
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
@recs
This trigger is used to specify the load table (m2btab file) during interactive loading
For all load tables (except Authority load tables) this trigger equals the m2btab’s file extension
Authority load tables do not have this -- @atab does the same for those tables
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
@atab
This trigger is used to set the Authority load table to use
In Authority load tables, this should equal the m2btab’s file extension
In Bibliographic load tables, this should equal “a”
This trigger is not found in Patron load tables
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Updating/Creating Load Profiles
X > Load Profile Maintenance Naming conventions Options for editing load tables (m2btab) Verifying load tables and checking for
errors Saving changes to load tables Restoring load tables if mistake is made
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Practice Exercise 1
“Copying and Editing an Existing Load Table (m2btab)”
Turn to page 111 and complete practice exercise 1
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Most Commonly Used Overlay Triggers
@ov_tag @ov_action @ov_protect @ov_rec_number @ldx
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
@ov_tag
Sets the overlay match point Match point must be an indexed
variable-length field Provisions can be made for specific
matches on field group tag, MARC tag and index tag
Alternate match points can be set
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
@ov_action
Sets the action to be taken if a match is found or not found
Possible actions are: Insert a new record Overlay an existing record Link records to a Bibliographic record Reject the record
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
@ov_protect
Specifies fields to protect from overlay Can be set for variable-length (V) and/or
fixed-length fields (F) Variable-length fields can be protected
based on field group tag and MARC tag Variable-length fields have a variety of
protect options (:d and :k)
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Overlay Based on Innovative Record Number
@ldx is used to overlay using the Innovative record number as the match point
Options for @ldx include the @ov_rec_number trigger
@ldx overrides the @ov_tag and @ov_action triggers
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Overview of Overlay Triggers
MOST USED
@ov_tag @ov_action @ov_protect @ov_rec_number @ldx
LESS USED
@ov_attach_delete @ov_attach_insert @ov_priority @ov_priority_action @ov_title
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Overlay using Innovative record number as Match Point
@ov_rec_number
@ldxmatch?
Add record tothe database
Overlay existingrecord
Do not add recordto the database
Use @ov_tag and@ov_action tooverlay record
YesBEGIN ofrecord load
END ofrecord load
1
1
“t”
“r”
No
“i”
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Overlay using Indexed Control Number as Match Point
@ov_action1 or 2 matches
Additional matchpoint?
@ov_action0 matches
Add record tothe database
Obey @ov_protect rules and overlayexisting record
Do not add recordto the database
2
2BEGIN of
Record load
Reject
Insert
END ofrecord load
1
1
Yes
@ov_tagmatch?
No No
Reject
Overlay
Yes
Insert
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Training Exercise 2
“Practice Applying Overlay Triggers”
Turn to page 122 and complete training exercise 2
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Busy Record Handling Triggers
Loading program will not overlay “records in use by system”
Two triggers control the handling of “busy” records: @busy – insert busy records? @busy_file – save busy records to file that
can be loaded later
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Automatic Date Selection Triggers
Load system date into fixed-length fields @cdate=“y” - automatic Cat. Date @odate=“y” - automatic Order Date @rdate=“y” - automatic Recv Date
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
@holdsymb
Innovative staff can construct a Holding Symbol table to issue loading instructions for: Bibliographic Call Number Bibliographic Location code 6XX fields based on 2nd indicator values
(e.g., value of 0 to load LCSH only) Holding symbol processing is activated
using @holdsymb trigger
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
@dflt
Fixed-length field values are assigned from the default Record Template when no value for the fixed field is present in the incoming record
The format is:@dflt=“biblio,item”(“pass 0, pass 1”)
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
@link
Linked (aka attached) records are loaded based on settings of @link trigger
The format of this trigger is:@link=rectype:pass#:MARC/count:tag
@link=“i:1:949 1”@link=“i:1:945:b”
@link=“i:1:#1”
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Additional Triggers
@comline @diac_sub_table @locmerge @m2b_normalize_020 @password (applicable if your library
has multiple accounting or serials units) @poprint
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Practice Exercise 2
“Specifying Record Templates, the Cat. Date Field and Creating Linked
Records”
Turn to page 116 and complete practice exercise 2
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Special Processing Functions (%)
Specified in 001-997 load lines that have “%” in Element 12, e.g. %008=“y”
MARC tag is processed in a special way when loaded
Examples of usage: 001 and 008 field processing Encrypt patron PINS Foreign currency conversion Activate a translation table (m2bmap)
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
%001
This function has five parts:%001(bcode,char,skip,start,valid)
bcode - RLIN code to Bib Code 1; set to “n” since RLIN db no longer used
char - valid characters for 001 field skip - number of characters to skip start - valid chars. for start of field valid - 001 have an OCLC prefix?
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Commonly Used Special Processing Functions
%008 - places Leader after 008 %bracket - place [ ] around GMD %encryptpin – encrypt patron PIN %foreign – foreign currency processing %replace – single character replacement %map - activates a translation table
(m2bmap) %first and %last – select first or last MARC
tag occurrence
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Training Exercise 3
“Using Special Functions”
Turn to page 123 and complete training exercise 3
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Command Functions (#)
Specified in 998 or 999 lines that have “#com” in Element 12
For example: #com=“ins”@password=“” Activated using @comline trigger during
OCLC interactive downloading (not batch)
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
More on Command Functions
Functions to: Specify fixed-length field information Activate triggers from command line Create linked records
Can new command functions be created locally?
See the Innovative Guide and Reference for more information about these commands
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Practice Exercise 3
“Troubleshooting Load Tables 101”
Turn to page 117 and complete practice exercise 3
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Translation Tables (m2bmap)
Used to change the data in incoming record during the load, e.g. change ABC to 123
Translations are stored in an m2bmap file
Activated in m2btab with the special processing function %map
More than one m2bmap file can be used in a load table
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Data Elements in Translation Tables<comparison> | <replacement>
Comparison expression is a string or regular expression to search for in incoming data file
If match is found, comparison expression will be replaced by the replacement expression
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Data Elements in Translation Tables@special_instruction=
Global instructions appear at top of file and start with @
They affect all lines in the m2bmap file @stop_on_map=true @delimiter=| @case=true
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Example of Simple Translation Table
@stop_on_map=true@case=trueJuv Ref|mjrefSouth|sSJUV|sjuvant|manthJUV|mjuv
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Using Regular Expressions
Regular Expressions can be used in comparison expressions
These make use of “meta-characters” listed on page 24
More information on regular expressions can be found in any basic UNIX manual
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Syntax for Regular Expressions
Comparison expressions(.*)$0|d(.*)$1
Replacement expressions \0|d19\1
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Tips on Using Translation Tables Order of lines in a map file is important
longest matching field should appear first Avoid unwanted case changes
@case=true Loading data into fixed-length fields
byte count in Element 5 must be > largest comparison
Must activate in load table Specify m2bmap in Element 12 using
%map trigger
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Creating and Updating Translation Tables
X > Load Profile Maintenance E > Edit m2btab and m2bmaps T > TRANSLATION Tables (m2bmap) N > Create a NEW m2bmap file Use %map trigger to activate translation
table in m2btab Must load records to test m2bmap file
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Training Exercise 4
“Translation Tables (m2bmap) ”
Turn to page 124 and complete training exercise 4
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Data Analysis before Profiling
Issues to consider before Profiling - MARC tags to load - Creating new records or overlay existing records - Remapping of MARC data - Loading linked records
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Use Data Exchange to Transferand Prep Files
Launch Millennium Cataloging Go | Data Exchange Select Process: Load records via
locally-created load profiles (.local) Data file must be transferred with “.lfts”
file extension (use “Get FTS” or “Get PC” buttons to transfer files to server)
“Prep” file to create “.lmarc” file for loading
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Use Data Exchange toView Records
Click on “.lfts” or “.lmarc” file Click “View” button Viewing features include:
“Count” button (counts number of records and MARC tags)
“Search” button (to find records that match specific criteria)
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
The m.marcload.local file
What does this file do? Data elements in this file Edit this file to create “Load” buttons for
your new load tables X > Load Profile Maintenance L > Edit m.marcload.local change –fzzz to –f<ext of m2btab.> Test new “Load” buttons in Data
Exchange
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Practice Exercise 4
“Applying Required Changes to an Existing Load Table (m2btab)”
Turn to page 118 and complete practice exercise 4
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Test Load Files Using Data Exchange
Admin | Clear Initials Admin | Reset Initials Go | Data Exchange Select Process: Load records via
locally-created load profiles (.local) Click once on “.lmarc” file Click correct “Load” button Click “Test” button at the bottom of
screen
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Practice Exercise 5
“Overlaying of Bibliographic Records”
Turn to page 119 and complete practice exercise 5
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Review: Creating a New Load Table (m2btab)
Plan load table settings View MARC records to be loaded Copy an existing load table Verify and check load table for errors Restore load table if errors are made
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Review: Using Translation Tables (m2bmap)
m2bmaps are optional But if you need one, remember to
activate the m2bmap in the m2btab file Specify m2bmap in Element 12 using
%map trigger
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Review: Editing m.marcload.local
Add m2btab file extension or brief description to Element 1 to distinguish between “Load” buttons
For example:C > LOAD a MARC file (.pbursar)
Use Clear Initials and Reset Initials to activate new “Load” buttons in Data Exchange
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Support from Innovative
Consultation [email protected] csdirect.iii.com
Technical Assistance Profiling Services
Product Code 830 [email protected]
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Review of Training Goals
Understand the database structure Understand existing load tables (m2btab) Be familiar with issues to be considered for
new load profiles Update and create load tables Create simple translation tables (m2bmap) Add “Load” buttons to Data Exchange Test load profiles before production
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Trainer Evaluation
Workshop Evaluation Form is available on csdirect.iii.com
Select “Contact Us” in upper right corner "Feedback" from left column "Workshop Feedback" from right column
Please take a few minutes to fill out the evaluation. Thank you!
Copyright Innovative Interfaces. Not to be duplicated or distributed without permission of Innovative Interfaces.
Top Related