ARC Basic Training

Idit Shachar-AbutbulMarch 2011

Part one

2

Copyright Statement All of the information and material inclusive of text, images, logos, product names is either the property of, or used with permission by Ex Libris Ltd. The information may not be distributed, modified, displayed, reproduced – in whole or in part – without the prior written permission of Ex Libris Ltd.

TRADEMARKS Ex Libris, the Ex Libris logo, Aleph, SFX, SFXIT, MetaLib, DigiTool, Verde, Primo, Voyager, MetaSearch, MetaIndex and other Ex Libris products and services referenced herein are trademarks of Ex Libris, and may be registered in certain jurisdictions. All other product names, company names, marks and logos referenced may be trademarks of their respective owners.

DISCLAIMER The information contained in this document is compiled from various sources and provided on an "AS IS" basis for general information purposes only without any representations, conditions or warranties whether express or implied, including any implied warranties of satisfactory quality, completeness, accuracy or fitness for a particular purpose.

Ex Libris, its subsidiaries and related corporations ("Ex Libris Group") disclaim any and all liability for all use of this information, including losses, damages, claims or expenses any person may incur as a result of the use of this information, even if advised of the possibility of such loss or damage.

© Ex Libris Ltd., 2009

3

Session Chapters

1. Introduction to ARC2. Getting Started3. ETL

1. Architecture 2. License Mechanism3. ARC in Multi-ADM environment4. Maintenance

Part one

Part two

4

Chapter one:Introduction to ARC

5

Chapter Objectives• What is ARC• ARC relationship with the Aleph 500 ILS• ARC Packages and Reports• ARC Studios

6

What is ARC• ARC = ALEPH Reporting Center• ARC is a web-based reporting, querying and

statistical environment.• ARC integrates the Cognos ReportNet™

software

7

The ARC Relation To The ALEPH 500™ ILS

ARCARCDatabaseDatabase

ALEPH ALEPH DatabaseDatabase

ARC Tables

ALEPH Data

ALEPH Tables

The ARC takes periodic “Snap Shots”

of the ALEPH 500™ALEPH 500™ set of tables and arranges

them into a completely separate database.

8

Cognos ReportNet Components

• Cognos Connection Web presentation layout/portal

• Query Studio Ad hoc reporting tool

• Report Studio Professional reporting tool

9

ARC Predefined Reports

• ARC comes complete with a selection of standard pre-defined reports and statistics

• These reports and statistics draw on information from data sources within the ALEPH system modules.

10

ARC Predefined Reports

11

ARC Packages

• Predefined Reports are organized into packages

• The packages are based on subjects related to various areas of library management.

12

To create a report about…

Choose Package Main Subjects for Analysis

Loans Loans Information about Loans and Loans History

Cash Cash Summary of Fines information and Patrons Cash Payments, based on Fines data

Requests Requests Information about Hold Requests and Booking Requests

Circulation Activities Events Circulation information and statistics about circulation activities, such as: Loans, Returns, Renewals, Lost and Claimed Returned Items - Based on circulation transactions and circulation events

Orders Acquisition Orders Information about Order Budgets, Expenditures, Vendors, and Claim and Acquisition statistics in general, based on Order data.

Patron information Patrons Local and global patron information

13

Items Inventory Information about Items and Items history and Circulation activities based on Item data.

Course Reading Course Reading Information from course reading data and its related Items.

Budgets Acquisition Budgets Information about Budget transactions, Orders, and Invoice Header based on Budget Transaction data.

Web OPAC OPAC OPAC Statistics

Cataloging Information Bibliographic Data Bibliographic and Catalogers Information

ETL Process ETL ETL process

ARC Usage Audit User Activities within ARC

14

ARC Studios – Query studio

• The Query Studio allows you to:

• Create ad hoc, simplified reports and statistics using simple “drag and drop” into the report authoring window

15

ARC Studios – Report studio

• The Report Studio allows you to:

• Create new sophisticated reports that analyze library data according to specific information needs.

• Report Studio provides many methods of formatting, presenting, and distributing your findings.

16

Chapter two:Getting Started

17

Chapter Objectives• Logging into ARC• ARC portal structure• My Folders vs. Customized Reports• Personalizing ARC• Actions on reports• Searching in ARC• Query Studio quick overview

18

Logging into ARC• The ARC URL syntax is

http://<server_name>:<port*>• Login for support is

administrator/administrator999

* Usually 8081

• This is the default login and is not used by customers.

• More about this later in part two.

19

ARC Portal

Tabs for Public folders and My folders

The connected user

and his scheme

Main tool bar

Main Menus

20

My Folders vs. Customized Reports

My Folders Customized ReportsCan only be accessed by you when you log-in

All users can access

Reports saved here can be modified

Reports saved here can be modified

Present in a separated tab- My Folders tab

Present under Public Folders tab

21

Personalizing ARC• Click on “My Preferences”

to Personalize the ARC portal for your user

• You can determine the following:• the amount of entries in a

list• the product and content

language • other defaults

22

Personalizing ARC

23

Personalizing ARC• For Content Language we support the

following languages:English, German, French, Italian, Czech, Danish, Swedish, Simplified & Traditional Chinese

24

Actions on reports• Main actions are:

Run, Open, Schedule and Create a Shortcut or a Report View

25

Actions- Create a report view

• A report view shares the same report specification as the source report, but other properties such as prompt values, schedules, languages, and output formats are independent of the source report.

26

Actions- Create a shortcut

• A shortcut is a pointer to another entry such as a report, report view, folder, job, or URL.

• Updating the source automatically updates all shortcuts to the entry.

• You can change access permissions for a shortcut entry, but it does not change the access permissions for the source entry.

27

Actions – Create a Schedule

• Reports can be scheduled individually or by using a job.

• Reports can be scheduled to run • by minute, hourly, daily, weekly, monthly or

yearly • for different output formats • saved or distributed to mailing lists

• A schedule can be associated with each report or job only once.

28

Actions – Create a Schedule

29

Searching in ARC• You can search for entries whose name,

description, or both match the string entered in the search criteria.

30

Searching in ARC

• You can find types of entries by leaving the search string empty and selecting a specific entry type in the advanced search criteria.

• The search is not case sensitive.

31

Query Studio quick overview• Use Query Studio to:

1. Create ad hoc reports that can be viewed by others.

2. Customize predefined reports created in Query Studio.

32

Query Studio quick overview

33

Query Studio quick overview

34

Report Example

35

Chapter three:ETL

36

Chapter objectives• Basic outlines of ETL process and architecture• ETL and ARC Database Layers• Extract in detail• Transform and Load in detail• ETL configuration and running• Related Documents

37

Basic outline of ETL process & architecture• The ETL process Extracts data from ALEPH,

Transforms it to a different physical structure, and Loads it into the data warehouse database.

Source system - ALEPH

Extraction, Transformation and Load tool

Data Warehouse

Client / server

38

ETL and ARC Database Layers

DWH

ARC schema

Extract Transform

Load

MRR STG

ALEPHDatabase

andConfiguration

39

• STG - The Staging area layer transforms data from the MRR layer, and builds the fact and dimension tables in a de-normalized structure (star schema). In every loading process this layer is truncated and filled again. At the end of this step the data from STG tables is loaded into the DWH layer.

ARC Database Layers (cont.)• MRR – A mirroring area. Includes tables which are

extracted from ALEPH but not necessary in the ALEPH table structure. This layer is static, i.e. no processing on MRR tables is done.

MRR STG DWH

ALEPH ARC schema

Database and

Configuration

MRR STG DWH

ALEPH ARC schemaDatabase

andConfiguration

40

ARC Database Layers (cont.)• DWH – Data Warehouse- The end-user layer

which includes tables with the same structure as the STG tables. This layer is fully indexed to support efficient retrieval.

MRR STG DWH

ALEPH ARC schemaDatabase

andConfiguration

41

ETL and ARC Database Layers (cont.)

MRR STG

ALEPH ARC schema

Extract Transform

STG layer is synonymous with the DWH layer

Database and

Configuration

Load

DWH

42

ARC ETL Architecture• The three ARC components – MRR, STG and DWH

– must reside on the same machine.

• It is recommended that ARC and ALEPH not be on the same machine.

• The Load phase is much heavier on IO, hence it will significantly affect the ALEPH online performance

43

Extract

• The base of the extraction is the administrative library.

• Extraction of all additional information, such as bibliographic data, is confined to records directly related to administrative records.

44

Extract

• The parallel processing is managed by a cycle table which is very similar to the cycle tables used for index building in ALEPH.

• Each cycle relates to a specific MRR table; as such it may gather information from different Oracle and configuration tables in ALEPH.

45

Extract

• p_arc_01.cycles example:

0001 + + BIB-INFO 000000101-000048621 $0002 + + CATALOG $0003 + + ITEMS 000000001-000048621 $0004 + + COURSE $0005 + + EVENT 000000001-000050000 $0006 + + EVENT 000050001-000100000 $0007 + + EVENT 000100001-000140816 $

46

Extract

• Each cycle creates a sequential file to be loaded into the respective MRR table.

• The extraction is done by a standard ALEPH dump procedure.

47

• It is possible to limit the amount of data that is extracted: • How many history records (items, loans and requests)

will be extracted• How many OPAC and Events transactions will be

extracted• If patron ID’s will be extracted

• This is done by configuring the file arc_conf_table.conf dlib <USM50> dt vi arc_conf_table.conf [ARC-HISTORY] from year = 2004 [OPAC] from date = 20050101

[EVENTS]From date = 20070101[PATRON-DATA]eliminate_data=N

Extract

48

• Following the extraction from ALEPH to the MRR tables, the following steps are performed:

• Truncate staging area tables. • Inserting data from the MRR layer into the STG

layer (relevant for STG tables whose structure is equal to the MRR tables).

• Building STG layer (mainly the fact tables and calculated dimensions like the periods and time dimensions).

Transform and Load

49

• The transformation phase is composed of stored procedures that run PL/SQL statements.

• These stored procedures reside in an Oracle table.

Transform and Load

50

• Changes related to the ETL (extract part) are done on the ALEPH side and distributed as part of the ALEPH Service Pack.

• In addition to the regular ALEPH Implementation notes, a special document is distributed that lists mandatory implementation notes that should be done on the ALEPH side by ARC users.

ALEPH SP

51

• Changes related to the reports and ETL (transform part) are done on the ARC side and distributed as an ARC Service Pack.

• In addition to the regular ARC Implementation notes, a special document called “ARC Service Pack” describes how the ARC service pack should be implemented.

• The latest ARC service pack should be installed as part of each ARC installation.

ARC SP

52

• Before running ETL, you must verify that you have the latest service packs installed on both the ALEPH and the ARC servers.

• Do all required configuration, according to the “ALEPH configuration for ARC” document.

Running ETL for the first time

53

Configure ETL

This configuration allows the connection between the ARC and the ALEPH DBs.

Thumb rules:Use ARC_TWO_TASK when ARC and ALEPH are installed on different servers.Use ARC_ORACLE_SID when they are installed on the same server.Exceptions:When ARC and Aleph are installed on the same sever, but have different Oracle version then also use ARC_TWO_TASK

54

exlarc.arc=)description= (address= (port

On ARC server type:cd $ORACLE_HOME/network/admincat tnsnames.ora

# Reporting Center Definitionssetenv ARC_TWO_TASK exlarc.arc#setenv ARC_ORACLE_SID aleph5

setenv ARC_ADMIN ARC_ADMIN [ Will usually be ARC_ADMINset arc_user = “USMR0” [ ARC schema name]

Configure ETLARC_TWO_TASK Example: dlib

<usm50>drvi prof_libraryOnly one should stay

unmarked

55

# Reporting Center Definitions#setenv ARC_TWO_TASK exlarc.arcsetenv ARC_ORACLE_SID aleph5

Configure ETLARC_ORACLE_SID Example:

For For ARC_ORACLE_SID:ARC_ORACLE_SID:From crn user type:echo $ORACLE_SID   aleph5

dlib <usm50>drvi prof_libraryOnly one should stay

unmarked

56

Additional configuration• You can define the size of the cycles in the extract

phase.• This is done by configuring $data_root/prof_libraryidits@il-aleph07(a20_3) USM50> grep p_arc_01 prof_library setenv p_arc_01_loop_length 50000 setenv p_arc_01_time_loop_length 002 setenv p_arc_01_bor_loop_length 3000

57

• Test Oracle connection with ARC DB by running csh -f p_arc_01_test as defined in prof_library.

Test the connection

58

csh –f p_arc_01 <usm50>,<ALL>,<Y>,<4> >& $data_root/report/p_arc_01.log.20110320_1113 &

parameters: <usm50> = ALEPH ADM library name <ALL> = all tables or one (i.e.- bib-info). If using 1 table - can run on debug mode. <Y> = Y/N lock library <4> = number of processes = number of processors-1. To know number of processors: on Sun – uname –X On Linux – top.data_root/report… = for log file. Make sure directory exists.

Run ETL

59

• ETL process creates a few log files:• $alephe_scratch/p_arc_01.log.<date> -

Main ETL log file (as any Aleph batch job)• $data_scratch/p_arc_01.log – log of records

loaded.• $data_scratch/p_arc_01_transform.log - log

of transform process.• $data_scratch/p_arc_01.cycles – log of the

cycles progress

ETL Log Files

60

ETL Log Files (cont.)• Example of info given by p_arc_01.log:Starting Process Mon Jan 31 13:58:26 IST 2011Table mrr_acquisition_method Successfully Loaded 3 of 3Table mrr_approval_department Successfully Loaded 2 of 2Table mrr_bib_info Successfully Loaded 31989 of 31989Table mrr_cash Loaded  126520 of 739484 - Not all records were loadedTable mrr_unfied_cataloging Successfully Loaded 0 of 0 End Extract Process Mon Jan 31 14:02:45 IST 2011TWO_TASK : exlarc.arcORACLE_SID : aleph9ARC_ADMIN: ARC_ADMINStarting Transform Process Mon Jan 31 14:02:45 IST 2011End Transform Process Mon Jan 31 14:16:57 IST 2011

61

ETL Log Files• Another way to view errors of the transform process is via

the ARC portal under Public Folders ETL ETL status:

62

Running ETL in Debug Mode• Sequential files created by the EXTRACT phase are not

deleted. They remain in the ADM $data_files directory.

• TRANSFORM phase is not applied.

• When executing ETL on a single table. The parameters noted above remain the same, except for the table name, replacing the ALL parameter.

• setenv ARC_DEBUG Yapcsh –f p_arc_01 USM50,BIB-INFO,N,3

63

ETL in debug mode- p_arc_01.log

Starting Process Thu Jun 22 18:50:41 EST 2006Table mrr_fact_items Successfully Loaded 8750976 of 8750976End Extract Process Thu Jun 22 20:29:16 EST 2006Debug Mode - No Transform

64

• ETL can be set to run on a regular basis from the job_list.

• To set it, you need to add corresponding records to two tables: job_list and job_list.conf.

• Both tables reside under $alephe_tab

ETL Configuration in the job_list

65

• Suppose you want your ETL to run: Once a week, on a Sunday morning at 07:00 apcsh -f p_arc_01 ABC50,ALL,N,3 and

• 2) job_list! Running ETL once a week (Sunday) for ALL tables!W8 07:00:00 Y                      ABC50 p_arc_01                       ABC50,ALL,N,03

• 1) Job_list.confW8    W YNNNNNN - run on Sunday

ETL Configuration in the job_list

66

Related Documents• 1. “ALEPH Configuration for ARC”

• Explains configuration steps that should be done on the ALEPH side before ETL activation.

• 2. “ALEPH to MRR Mapping”• Provides mapping between ALEPH Z Oracle tables and ARC

MRR layer tables. • 3. “MRR to DWH Mapping”

• Provides mapping between ARC MRR layer tables and ARC DWH layer tables.

• 4. “ARC Logical Model”• Provides mapping between ARC presentation layer packages

and ARC DWH layer tables.

Thank You!Idit.Shachar-Abutbul@exlibrisgroup.com