Metadata for Interchange of Files on Sequential Storage ... · ANSI/AIIM MS66-1999 M ETADATA FOR...

ANSI/AIIM MS66-1999

American National Standard for Information

and Image Management —

Metadata for Interchange of Files onSequential Storage Media Between FileStorage Management Systems (FSMSs)

An American National Standard prepared by the Association for Information and Image Management International

Approved as an American National Standard

April 6, 1999

Abstract:This document describes a standard for specifying metadata that describes how a File StorageManagement System (FSMS) has stored files on sequential media. This metadata descriptionis independent of any particular proprietary metadata format, but it may be used to describethe proprietary format that an FSMS uses to write files to sequential media.

ANSI/AIIM MS66-1999 METADATA FOR INTERCHANGE OF FILES ON SEQUENTIAL STORAGE MEDIA BETWEEN FILE

STORAGE MANAGEMENT SYSTEMS (FSMSS)

ASSOCIATION FOR INFORMATION AND IMAGE MANAGEMENT INTERNATIONALii

Contents

1 Scope and purpose.............................................................................................................................1

2 Conformance......................................................................................................................................3

3 References .........................................................................................................................................5

3.1 Referenced international standard .................................................................................................5

3.2 Related international standards .....................................................................................................5

4 Definitions ..........................................................................................................................................5

5 Abbreviations and acronyms...............................................................................................................8

6 The format of the metadata ................................................................................................................9

6.1 Record description ........................................................................................................................9

6.2 Reserved words .......................................................................................................................... 10

6.3 Extending the metadata ............................................................................................................. 10

6.4 The sections of the metadata export............................................................................................ 10

6.5 Order........................................................................................................................................... 10

6.6 Counting and counts.................................................................................................................... 10

6.7 Conventions in this document...................................................................................................... 11

7 Metadata describing the originating FSMS and its environment ....................................................... 11

7.1 Export version ............................................................................................................................. 11

7.2 Export hardware system.............................................................................................................. 11

7.3 Export operating system ............................................................................................................. 12

7.4 Export time.................................................................................................................................. 12

7.5 Exporting FSMS.......................................................................................................................... 12

7.6 Export root .................................................................................................................................. 13

8 Metadata describing the removable media being exchanged ............................................................ 13

8.1 Cartridge identifier ...................................................................................................................... 13

8.2 Cartridge info ............................................................................................................................. 14

8.3 Cartridge drive type..................................................................................................................... 14

8.4 Cartridge layout........................................................................................................................... 15

8.5 Cartridge number of blocks between tape marks ......................................................................... 16



ASSOCIATION FOR INFORMATION AND IMAGE MANAGEMENT INTERNATIONAL iii

8.6 Cartridge number of bytes after tape marks................................................................................. 16

8.7 Cartridge block size..................................................................................................................... 17

8.8 Cartridge family .......................................................................................................................... 17

8.9 Cartridge billing ID....................................................................................................................... 18

8.10 Cartridge volume group............................................................................................................. 18

8.11 Cartridge location ...................................................................................................................... 19

8.12 Cartridge lot .............................................................................................................................. 19

8.13 Cartridge statistics..................................................................................................................... 20

8.14 Cartridge byte order................................................................................................................... 20

8.15 Cartridge block structure............................................................................................................ 21

8.16 Cartridge recording FSMS......................................................................................................... 21

9 Metadata describing the file segments on the removable media being exchanged............................ 22

9.1 File UID....................................................................................................................................... 22

9.2 File segment file name................................................................................................................ 22

9.3 File segment number................................................................................................................... 23

9.4 File segment copy number .......................................................................................................... 23

9.5 File segment information............................................................................................................. 23

9.6 File segment header.................................................................................................................... 25

9.7 File segment trailer...................................................................................................................... 26

10 Metadata describing the directory and file structure being exchanged............................................. 26

10.1 Object attribute list .................................................................................................................... 26

10.2 Contents list .............................................................................................................................. 28

Annex A (informative) Examples of the record types ........................................................................ 31

A.1 Metadata describing the originating FSMS and its environment (7) ............................................. 30

A.2 Metadata describing the removable media being exchanged (8) ................................................. 31

A.3 Metadata describing the file segments on the removable media being exchanged (9)................. 35

A.4 Metadata describing the directory and file structure being exchanged (10).................................. 37

Figure

Figure 1 FSMS conceptual representation of relationship between file segment and physical media ...2



ASSOCIATION FOR INFORMATION AND IMAGE MANAGEMENT INTERNATIONALiv

Table

Table 1 Mandatory record types and their categories...........................................................................4



ASSOCIATION FOR INFORMATION AND IMAGE MANAGEMENT INTERNATIONAL v

Foreword

(This Foreword is not part of ANSI/AIIM MS66-1999 — Metadata for Interchange of Files on SequentialStorage Media Between File Storage Management Systems (FSMSs).)

This document describes a standard for specifying metadata that describes how a File Storage Manage-ment System (FSMS) has stored files on sequential media. This metadata description is independent ofany particular proprietary metadata format, but it may be used to describe the proprietary format that anFSMS uses to write files to sequential media.

Various proprietary schemes for writing files to sequential media have been discussed in the AIIM FileStorage Management Systems (FSMSs) Subcommittee, C21.1, and much of this standard has been de-signed to accommodate the practices of various vendors. For this reason, certain records described inthe standard do not apply to certain vendors, while other records do apply to those vendors. For exam-ple, one vendor places tape marks throughout a file segment for use in positioning, while another vendordoes not use tape marks at all, even to indicate an end-of-file. For each of these practices, certain rec-ords described by the standard accommodate that practice, and each vendor is expected to use the rec-ords that apply to its own scheme.

The record format describing the metadata has been designed so that vendor-specific or site-specificrecords or fields are easy to add. It is believed that for virtually any practice, the general record formatdefined in the standard is flexible enough to allow vendors to define records that accommodate thatpractice.

When it approved this standard, the AIIM Standards Board had the following members:

Name ofrepresentative Organization represented

Marilyn Wright, Chair AIIM International

Robert Breslawski Eastman Kodak

Jewel Drass Bell & Howell

Basil Manns U.S. TAG TC171

Fernando L. Podio Information Technology Laboratory,National Institute of Standards andTechnology

Suresh Shenoy Information Management Consultants

Herman Silbiger APPLICOM

Herbert J. White, II Genealogical Society of Utah

When it approved this standard, the AIIM Storage Devices and Applications Committee, C21, had thefollowing members:

Name of representative Organization represented

Fernando L. Podio, Chair Information Technology Laboratory,National Institute of Standards andTechnology

Soloman Appavu Cook County Hospital

Robert Blatt EID



ASSOCIATION FOR INFORMATION AND IMAGE MANAGEMENT INTERNATIONALvi

Betty S. Burton Department of Army

Benjamin Carter Carter & Associates

Jim Devoy Devoy Associates

Rawl Gelinas The Keyops Group

Becky L. Gingras Schema Systems

Kenneth J. Hallam ENDL Associates

Joseph G. Hardy Department of the Army

John E. Kulakowshi IBM Corporation

Francois Le Carvennec Consultant

R. Lenel James Foster Group

Basil Manns Library of Congress

Richard Murphy TOSOH USA, Inc.

Charles R. Obermeyer, II Woodside Summit Group Incorporated

When it prepared this standard, the AIIM File Storage Management Systems (FSMSs) Subcommittee,C21.1, had the following members:

Name of representative Organization represented

Fernando L. Podio, Chair Information Technology Laboratory,National Institute of Standards andTechnology

Brian Berg Berg Software Design

Bill Bullers Datatape Inc.

Don Crouse LSC Inc.

David Fisher High Performance Storage System

Marty Frary Storage Technology Corp.

P.C. Hariharan Systems Engineering & Security, Inc.

R. Lenel James Foster Group

James Johnson Performance Group

Theodore Johnson AT&T Laboratories

Stephen Jones National Media Laboratory

Gerry Joyce Terabank Systems

Ben Kobler NASA Goddard Space Flight Center

John Kodis Hughes STX Corp.

Thomas W. Lanzatella SGI

Harold Lloyd NOAA/NWS

Gary Lyng Veritas Software

Basil Manns Library of Congress



ASSOCIATION FOR INFORMATION AND IMAGE MANAGEMENT INTERNATIONAL vii

Ken Nicholson EMASS

Juliet Z. Pao NASA Langley Research Center

David L. Patton IBM Corporation

Frank Puleo BDM

Sarah Roy NOAA/NWS

Mark Saake UniTree Software, Inc

Ellen Salmon NASA Goddard Space Flight Center

Dave Skinner Red Cape Software

Joe Straub LOTS Technology

William Vollrath LOTS Technology

Joel Williams Systems Engineering & Security, Inc.

Steve Wilson IBM Corp. – Tucson



ASSOCIATION FOR INFORMATION AND IMAGE MANAGEMENT INTERNATIONALviii

AMERICAN NATIONAL STANDARD ANSI/AIIM MS66-1999

ASSOCIATION FOR INFORMATION AND IMAGE MANAGEMENT INTERNATIONAL 1

American National Standard for Information and Image Management —

Metadata for Interchange of Files on Sequential StorageMedia Between File Storage ManagementSystems (FSMSs)

1 Scope and purpose

This standard specifies the format and content of metadata used for interchange of files on removablesequential storage media. As defined in this standard, sequential storage media includes magnetic andoptical tapes and other media [e.g., a Digital Versatile Disk (DVD) written in sequential storage formats].This standard is not intended to be used to describe a file system.

A File Storage Management System (FSMS) is a software system that manages a collection of files.These files may be on tape, disk, or other media, and one of the responsibilities of the FSMS might be tomove files between different kinds of media to optimize performance and cost. Examples of FSMSs areHierarchical Storage Management systems (HSMs) and systems that manage large storage archives.FSMS are evolving rapidly, and each usually uses proprietary file formats. Files, particularly those thathave been updated frequently, may be scattered over several media (e.g., tapes), and the informationrequired to reconstruct the files is likely to be stored on disks that are separate from the sequential mediaon which the files reside. Some information may be embedded in headers or trailers on each block or fileon the sequential media, so that any program reading the file would have to identify this information andunderstand that it is not part of the file. Migrating from one FSMS to another would therefore most likelyrequire rewriting all of the files stored on sequential media written by the original system. For a large ar-chive, this would be extremely expensive.

This standard accommodates various scenarios, such as

the interchange of sequential storage media between two different (and proprietary) FSMS;

the interchange of sequential storage media between different physical volume repositories.

The metadata described in this standard will allow one FSMS to properly read files written on sequentialmedia by another FSMS. This will enable users to avoid rewriting an extensive archive when transferringsequential storage media in either of the two scenarios listed above. This metadata describes informationwritten by the FSMS, such as any data and control blocks. It does not describe the physical data-recording format on the media written by hardware, software, or firmware drivers. This distinction is illus-trated in Figure 1.

Striping to sequential media that is performed by a controller managing several tape drives is not ad-dressed by this standard, whereas striping performed by the FSMS is. Striping that involves writing outparity information is not addressed by this standard.



ASSOCIATION FOR INFORMATION AND IMAGE MANAGEMENT INTERNATIONAL2

Figure 1 FSMS conceptual representation of relationship

between file segment and physical media

An application program sees a file or a logically contiguous part of a file (file segment) as an orderedsequence of bytes. When the FSMS is given this file segment to process for storage on sequential me-dia, it may add control blocks, such as the header and trailer shown in Figure 1, and it may break up thefile segment into smaller parts. When the device driver writes the file or file segment to the cartridge,additional bytes may be added for error correction and detection. These additional bytes may be distrib-uted throughout the segment which is written to the cartridge, and they are not necessarily located at thebeginning or the end of the file segment.

It is understood that metadata not directly necessary for identifying and reading files may be handled byeach FSMS uniquely. Therefore the requirements for the receiving FSMS to handle such metadata areoutside the scope of this standard, and they can be part of the negotiations between users and FSMSvendors.

The purpose of this standard is to specify a way of describing the information added by one FSMS withother information sufficient to allow another FSMS to read what the first FSMS wrote and reconstruct theoriginal file as seen by the original application program.

For convenience, this standard accommodates the transfer of any cartridge known to the exporting sys-tem, whether application data has been written to the cartridge or not.

This standard assumes that appropriate device drivers exist to allow the cartridge to be read in a mannerconsistent with the way in which it was written. Describing the information added by the device driver isoutside the scope of this standard.

One critical assumption of this standard is that sequential media may be physically addressed to the file,block, record, or byte level (depending on the technology) so as to allow an application program (such asan FSMS) to begin reading at the specified address, such as the beginning of a file segment.

Certain records defined in this standard identify cartridge types, disk drive models, etc. Various membersof the committee felt that it would be desirable to have lists of "approved names" for these specifications.However, no practical method of creating and maintaining such lists could be devised. Therefore thespecification of such items will have to be left to negotiations between the parties involved.

Similarly, the method by which the metadata is to be transmitted shall be determined by negotiationsbetween the parties involved; it is not specified by this standard. Possible methods are the file transferprotocol (ftp) and the physical transfer of removable media that can be accessed by both parties.




The target audience for this standard is any organization with a large data system and vendors who sup-port them. Other data storage system users can also benefit from products conforming to this standard.Users that need to interchange sequential storage media will benefit with large data migration cost sav-ings and benefit from less need for additional computing capacity required to support the copy opera-tions.

2 Conformance

Since (according to 6.1) the records in the metadata export shall be either ASCII or UNICODE, confor-mance may be achieved for the ASCII version, the UNICODE version, or both versions. The two char-acter sets shall not be mixed in any one metadata export.

In the following section, the term user refers to the person who causes the metadata export to be writtenand thereby makes the choices of record types mentioned below.

A metadata export is in conformance with this standard if it meets all of the mandatory requirements ofthis standard.

A generating system is in conformance with this standard if it produces a metadata export that con-forms to all of the mandatory requirements of this standard.

A receiving system is in conformance with this standard if it can read the metadata export, identify anyfile from its fully qualified file name, and output it from the cartridges described in the export metadata sothat the file is bytewise identical to the file in the generating system.

The metadata export described by this standard consists of a collection of named records whose typesare defined in clauses 6 through 10. Each record consists of a sequence of fields. The name of the rec-ord is in the first field, and other fields in the record are either name fields or information fields. Informa-tion fields immediately follow their name fields. Each record is in one of two categories:

required;

selectable.

It is mandatory that any record type chosen by the user be included in the metadata.

It is mandatory that

if a record type is in the required category, all records of that type shall be included in the metadata,providing that the information in the record exists. If none of the information for that record type ex-ists, then the record shall be omitted. If the information for any field in a required record type doesnot exist, but the information for some other field in the record type does exist, then the record shallbe included with all fields for which information exists.

if a record type is in the selectable category, then all records of that type shall be included in themetadata if the user chooses that record type. If the user does not choose that record type, then norecords of that type shall be included in the metadata. The method by which the user chooses a rec-ord type is not specified in this standard. As with records in the required category, if a record type isselected, and none of the information for that record type exists, the record shall be omitted. If a rec-ord type is selected, and if the information for any field in the record type does not exist, but the in-formation for some other field in the record type does exist, then the record shall be included with allfields for which information exists.




It is also mandatory that all records of the metadata shall conform to the format and semantics describedin clauses 6 through 10.

Table 1 lists the mandatory record types and their categories. There are no optional record types, al-though 6.3 describes a method of extending the metadata to include additional information.

Table 1 Mandatory record types and their categories

Subclause Name of record typeCategory of record

type

7.1 Export version Required

7.2 Export hardware system Required

7.3 Export operating system Required

7.4 Export time Required

7.5 Exporting FSMS Required

7.6 Export root Required

8.1 Cartridge identifier Required

8.2 Cartridge info Required

8.3 Cartridge drive type Required

8.4 Cartridge layout Required

8.5 Cartridge number of blocks between tape marks Required

8.6 Cartridge number of bytes after tape marks Required

8.7 Cartridge block size Required

8.8 Cartridge family Selectable

8.9 Cartridge billing ID Selectable

8.10 Cartridge volume group Selectable

8.11 Cartridge location Selectable

8.12 Cartridge lot Selectable

8.13 Cartridge statistics Selectable

8.14 Cartridge byte order Required

8.15 Cartridge block structure Required

8.16 Cartridge recording FSMS Required

9.1 File UID Required

9.2 File segment file name Selectable

9.3 File segment number Required

9.4 File segment copy number Required

9.5 File segment information Required

9.6 File segment header Required




Subclause Name of record typeCategory of record

type

9.7 File segment trailer Required

10.1 Object attribute list Required

10.2 Contents list Required

3 References

3.1 Referenced international standard

ISO/IEC 10646-1:1993, Information technology Universal Multiple-Octet Coded Character Set (UCS) Part 1: Architecture and Basic Multilingual Plane.

3.2 Related international standards

ISO/IEC 9945-1:1996 (IEEE 1003.1), Information technology Portable Operating System Interface(POSIX) Part 1: System Application Program Interface (API) [C Language].

ISO/IEC 9945-2:1993, Information technology Portable Operating System Interface (POSIX) Part 2:Shell and Utilities.

Amendment 2 to ISO/IEC 9945-2:1993, Shell and Utilitity Addendum.

4 Definitions

For the purpose of this standard, the following definitions apply.

4.1applicationsynonymous with application program.

4.2application programa program that processes the contents of records belonging to a file or files and that may also processselected attribute data relating to the file or files or to the physical media on which they are recorded.

4.3billing IDa charge number or other identification that allows an organization to extract payment for a service or tomonitor resource usage.

4.4bita single digit in the binary number system. Binary digits are, in order, 0, 1.




4.5bytea string of eight bits that is operated on as a unit.

4.6cartridgea package that contains one or more physical volumes and that is made accessible by a mount opera-tion. This definition is meant to include single cartridges only; cartridges combined by a stacker are con-sidered separate cartridges, since their metadata is specified separately without any reference to astacker. In this context a cartridge is defined to include all removable media, including reel-to-reel mediaand optical disks.

4.7devicea piece of electronic equipment that writes or reads storage media. A device contains a media accesspoint and a mount point. A mount point is the mechanism used for associating a cartridge with a device.A media access point provides electronic access for reading or writing a cartridge. A mount operationassociates a cartridge with a mount point. A load operation associates a cartridge with a media accesspoint and makes it ready for reading or writing in exactly one physical volume.

4.8device drivera program that manages a device and causes it to read or write data storage media.

4.9directorya file that contains pointers to other files or directories.

4.10directory nameSee file name.

4.11drivesynonymous with device.

4.12filean ordered sequence of one or more records.

4.13file namedirectory nameIf the fully qualified name of a file or directory has components, as in a path name, this is the last com-ponent in the name. Otherwise, it is the fully qualified file or directory name.

NOTE A directory is just a type of file that logically contains other files (or pointer information to the files).Accordingly, directory and file name semantics are the same.

4.14file segmenta sequential part of a file written to sequential storage media. File segments may consist of a number ofblocks or records and may contain tape marks. Certain implementations of striping distribute a file seg-ment across more than one cartridge. A file consists of an ordered set of file segments.




4.15fully qualified directory nameSee fully qualified file name.

4.16fully qualified file namefully qualified directory namethe unique name of the file according to the system in which it resides.

4.17hexadecimalHEXa representation of a number in numeric base 16. Hexadecimal digits are, in order, 0, 1, 2, 3, 4, 5, 6, 7,8, 9, A, B, C, D, E, F.

4.18kilobytekb1024 bytes.

4.19labela record that identifies and characterizes a physical volume.

4.20logical blockan ordered sequence of logical records processed as a unit by an application. The processing may in-volve adding additional data for management purposes.

4.21logical recordan ordered sequence of bytes processed by an application. When one application transfers a logical rec-ord to another application (such as a File Storage Management System), it may modify that record byadding control information. The modified record is then a logical record for the second application.

4.22megabyteMb(1024 x 1024) bytes.

4.23metadatainformation about data or other information. The metadata described in this standard enables one to re-construct a file that has been stored by an FSMS and also to place that file in its proper position in a di-rectory.

4.24path nameIn a Unix or POSIX environment, this is the fully qualified name of an object, which identifies in order thehierarchy of directories in which a file or directory resides. For such a system, this is the fully qualifiedfile name.

4.25physical blockan ordered sequence of physical records written to electronic media as a unit.




4.26physical recordan ordered sequence of bytes written on electronic media. A physical record contains a logical recordand any additional data that may be added by a device driver.

4.27physical volumethe portion of recording media accessible for reading or writing without an intervening load operation.

4.28recordan ordered sequence of bytes treated as a unit of data.

4.29sequential mediarecording media, such as tape, that is managed and modified in a particular order. Accessing sequentialmedia requires the movement of the media in some direction (forwards or backwards, for instance), themovement of the reading or writing heads, or both.

4.30stripeSee striping.

4.31stripe block lengthSee striping.

4.32stripingStriping a file consists of mounting n cartridges (n > 1) simultaneously and writing the file sequentially tothe cartridges in order in a round-robin fashion. The stripe block length is the number of bytes written toone cartridge before moving to the next cartridge. A stripe is the portion of the file written to one car-tridge. A file segment is not necessarily identical to the part of the file written to one cartridge beforemoving to the next cartridge. Striping may have the additional feature that parity bits are placed on oneor more of the cartridges. This feature is not addressed in this standard.

4.33UIDunique identifier; an identifier that is unique among all objects in the universe of discourse. In this con-text, a UID is unique among all of the objects in the metadata export. It need not be reproducible, sinceone acceptable method of generating such an ID involves using a time stamp.

4.34userwith respect to the metadata export, the person who causes the metadata export to be written and in sodoing makes the choices of record types. More generally, a person or program that commands an appli-cation.

5 Abbreviations and acronyms

5.1DCEDistributed Computing Environment.




5.2FSMSFile Storage Management System.

5.3HEXhexadecimal.

5.4HSMHierarchical Storage Management.

5.5POSIXPortable Operating Systems Interface.

5.6SCSISmall Computer System Interface.

5.7UIDUnique Identifier.

5.8UTCUniversal Time Co-ordinate.

6 The format of the metadata

6.1 Record description

This subclause specifies the format of the records of the metadata. Each record consists of a sequenceof fields. The record name shall be the first field. Other fields shall be either name fields or informationfields. Name fields shall be used to identify the contents of information fields. Information fields shallimmediately follow their corresponding name fields. As specified in clauses 7 through 10, some informa-tion fields shall be named and others shall not be named.

All records shall consist either of ISO/IEC 10646 standard 16-bit characters (commonly referred to asunicode) or of ASCII 8-bit characters. In the case of unicode, each field of the record shall end with a2400 (HEX) character, which is defined by the standard as the SYMBOL FOR NULL, and each recordshall end with a 214E (HEX) character after the last SYMBOL FOR NULL. The 214E (HEX) character isdescribed by the standard as the SYMBOL FOR RECORD SEPARATOR. In the case of ASCII, eachfield of the record shall end with a binary zero character (commonly represented by \0), and each recordshall end with a 0A (HEX) character (commonly represented by \n). The exporting system shall specifyseparately which character set is used.

Record names and field names are case-insensitive; however, in the representations of clauses 7through 10, they are all upper case.

This standard does not impose a limit to the length of a record or to the length of any of the informationfields. The only length restrictions are imposed by the operating system that writes the metadata. Simi-larly, the size of the numbers that are likely to be encountered in the metadata is not limited by the stan-




dard, and in particular, any program that reads the metadata should expect to encounter numbers thatwill have to be stored as 64-bit integers.

6.2 Reserved words

Record and field names are reserved for their uses and shall not be duplicated or used for purposes dif-ferent from those defined in clauses 7 through 10. Information fields shall contain information as de-scribed in clauses 7 through 10. Because of the way in which the records are defined, there is no conflictin the case where the content of an information field is identical to a record or field name.

6.3 Extending the metadata

The exported metadata may be extended beyond the metadata specified in this standard only by the ad-dition of records of the same format (i.e., using the same record and field separators) as the records ofthis standard but with different names. Those wishing to extend the metadata shall prefix their recordnames with the characters "EXT_" and may include some site- or vendor-specific character string, suchas "XYZ-Someplace". Record names, identifiers, and record content in such an extension may be in anylanguage. Additional named fields may not be added to existing record types. These extensions shall bedocumented separately.

6.4 The sections of the metadata export

The metadata for media exchange is divided into four types:

1. metadata describing the originating FSMS and its environment;

2. metadata describing the removable media being exchanged;

3. metadata describing the file segments on the removable media being exchanged;

4. metadata describing the directory and file structure being exchanged.

The metadata export shall consist of four sections, one corresponding to each of the four types. If themetadata export is presented as one file, then the first section shall contain all of the metadata of the firsttype, and each of the other sections shall contain all of the metadata of one of the other types and mayoccur in any order. For metadata export, it is also acceptable that each of these sections be in a separatefile if the files are appropriately identified.

6.5 Order

If the order of a particular record is not specified, it may appear in any order, if that does not conflict withrecords whose order is specified. Similarly, if the order of fields is not specified, then they may be in anyorder. For example, in the second section of the metadata, all records following a Cartridge Identifierrecord refer to the cartridge identified in it, until another Cartridge Identifier record is encountered. Therecords between the Cartridge Identifier records may be in any order.

6.6 Counting and counts

All counting in the metadata shall be zero-based. That is, the first element shall be element 0; the secondelement shall be element 1, etc. For example, a stripe set consisting of five cartridges shall have the




cartridges numbered 0, 1, 2, 3, and 4. The number of cartridges in this set shall be specified as 5. Allnumbers shall be decimal.

6.7 Conventions in this document

The examples that follow are intended to represent either unicode characters or ASCII characters. Inthese examples, the field separator, which is either the SYMBOL FOR NULL (in unicode) or \0 (inASCII), will be represented by the character ‘Ø’, and the record separator, which is either the SYMBOLFOR RECORD SEPARATOR (in unicode) or \n (in ASCII), will be represented by the character ‘§’. Rec-ord and field names are all upper case.

In some of the following examples of records, it has been necessary to use more than one line. However,there are no carriage returns or line feeds in the records represented by these examples.

7 Metadata describing the originating FSMS and its environment

This clause specifies the record types in the section of the metadata that contains information about theexport itself. The names of all records specified in this clause begin with the characters "EXPORT_".

7.1 Export version

The export version record shall be the first record in this section, which is the first section in the export.

Record name: EXPORT_VERSION

Description: The name and version of this standard export format.

Named fields: None.

Record category: Required.

Example: EXPORT_VERSIONØANSI/AIIM MS55-1997xØ§

Rationale: This record identifies the version, allowing different versions of the standard to be distin-guished in the future.

7.2 Export hardware system

Record name: EXPORT_HARDWARE_SYSTEM

Description: Computer system central processing unit make and model that wrote the export.

Named fields: VENDOR, NAME, and VERSION


Example:

EXPORT_HARDWARE_SYSTEMØVENDORØVendorNameØNAMEØX1764Ø

VERSIONØAB1.5Ø§




Rationale: This record identifies the hardware system that wrote the export, to aid in reading the export.

7.3 Export operating system

Record name: EXPORT_OS

Description: Operating system and version running on the computer system on which the export waswritten. The version specification may include a patch level specification if desired.

Named fields: VENDOR, NAME, and VERSION


Example:

EXPORT_OSØVENDORØVendorNameØNAMEØ

OperatingSystemØVERSIONØ6.2Ø§

Rationale: This record identifies the software system that wrote the export, to aid in reading the export.

7.4 Export time

Record name: EXPORT_TIME

Description: Starting date and time of the export: yyyy/mm/dd/hh:mm:ss. The time zone shall be UTC.

Named fields: None.


Example:

EXPORT_TIMEØ2000/02/29/12:23:15Ø§

Rationale: This record identifies the time at which the export was written, in case multiple exports havebeen made.

7.5 Exporting FSMS

Record name: EXPORT_FSMS

Named fields:

VENDOR: Specifies the vendor of the FSMS.

FSMS: Specifies the FSMS name.

VERSION: Specifies the version. The version specification may include a patch level specification if de-sired.

Description: Vendor name, FSMS name, and FSMS version.





Example:

EXPORT_FSMSØVENDORØVendorNameØFSMSØHGZYØ

VERSIONØ9.12O§

Rationale: For additional identification, this record identifies the vendor of the FSMS. This informationmay be helpful in interpreting some details of the export.

7.6 Export root

Record name: EXPORT_ROOT

Description: A list of the highest level directory or directories in this export. All of the directories andfiles in the export are below one of the roots listed in this record.

Named fields: None.


Example:

To specify that the directories /chemistry/accounting and /architecture/design and all of the directoriesbelow are included, the following could be used:

EXPORT_ROOTØ/chemistry/accountingØ/architecture/designØ§

Rationale: This record may be used in two ways by some implementations. By specifying one or moreexport roots, one identifies the logical mount points under which all of the files in this export are defined.This allows one to partition the export if necessary. On the import side, if there is a name space conflict,one can change the names of the export roots so that the files underneath these roots will not cause aname space conflict.

8 Metadata describing the removable media being exchanged

This clause specifies the record types in the section of the metadata that contains information about thecartridges being exported. The names of all records specified in this clause begin with the characters"CART_". The Cartridge Identifier record shall come first, and all records between this record and thenext record of this type shall refer to the cartridge identified.

If cartridges having no application data are described with records defined in this clause, only those rec-ords that apply to such cartridges shall be included. Additional information for such cartridges may beadded as extensions.

8.1 Cartridge identifier

Record name: CART_ID

Description: External label the Visual Serial Number on the outside of the cartridge. Duplicate Car-tridge Identifiers in a set of cartridges being exported are not allowed.




Named fields: None.


Example: CART_IDØA60032Ø§

Rationale: This is to provide unique identification of each of the cartridges being exported. The commit-tee decided that the name (or identifier) of a cartridge is sufficient to uniquely identify it and that a sepa-rate unique identifier is not needed. If the name of a cartridge to be imported into a system conflicts withan existing cartridge, then the conflict will have to be resolved to complete the import and another uniquecartridge identifier (such as a UID) will not solve that problem.

8.2 Cartridge info

Record name: CART_INFO

Description: This record, with the record specified in 8.3, gives sufficient information to remove anyambiguity involved in drive selection for this cartridge.

Named fields:

MEDIA_TYPE: The vendor-supplied name for the type of media of the cartridge. This is not to be con-fused with the form factor of the cartridge. Different media may be included within the same form factor,although the configuration of identification marks sensed by the drive on the cartridge is generally differ-ent.

PHYSICAL_RECORDING_FORMAT: The vendor-supplied name for the way in which the informationwas put on the cartridge by the drive. This is not intended to refer to a higher-level recording format,such as the format of a vendor-specific FSMS.


Example:

CART_INFOØMEDIA_TYPEØXYZ2000ØPHYSICAL_RECORDING_FORMAT

Ø XYZ1000§

Rationale: Sometimes a cartridge may not be written in its "native" format, but in an earlier version ofthat format. MEDIA_TYPE is provided because a number of different makes and models of cartridgesmay be exported. PHYSICAL_RECORDING_FORMAT is provided because some cartridges can be re-corded in a variety of ways.

8.3 Cartridge drive type

Record name: CART_DRIVE_TYPE

Description: This record describes the type of drive that wrote the cartridge. For a SCSI device, much ofthis information may be obtained from the device by the Drive Inquiry operation.

Named fields:

MFR: The Vendor ID.




MODEL: The product revision level.

TYPE: The product ID.

DRIVE_COMPRESSION: YES if the drive can do compression; NO if not.


Example: CART_DRIVE_TYPEØMFRØABCØMODELØXYZ4000Ø

TYPEØMNZØDRIVE_COMPRESSIONØYESØ§

Rationale: This record, with the record described in 8.2, is designed to remove any ambiguity in speci-fying a drive for reading or writing the cartridge.

8.4 Cartridge layout

Record name: CART_LAYOUT

Description: This record describes the high level layout of the data on the cartridge. For the purposes ofthis standard, all cartridges are assumed to be partitioned into at least one partition, even if partitioning isnot supported by that cartridge technology.

Named fields:

NO_PARTITIONS: The number of partitions on the cartridge. This number shall always be at least 1.

NO_SIDES: The number of sides on the cartridge. This number shall always be at least 1.

SIDE_NO, PARTITION_NO, PARTITION _NAME, LABEL_TYPE, INTERNAL_LABEL, SIZE_MB,FREE_SPACE_MB: These fields shall appear with the SIDE_NO and PARTITION_NO fields first in thatorder. The remaining fields (LABEL_TYPE, INTERNAL_LABEL, SIZE_MB, FREE_SPACE_MB) mayappear in any order, and all refer to the partition on the side specified by the two previous fields. Thisentire group of fields shall be repeated for as many partitions as there are on the cartridge.

SIDE_NO: The side number.

PARTITION_NO: The number of one of the partitions on the tape. Partitions are to be numbered con-secutively, in physical order, 0, 1, 2, 3, ...n-1, where there are n partitions on a side of the cartridge.

PARTITION_NAME: The name, if any, of the partition. If the partition is not named, then this field maybe omitted.

SIZE_MB: The estimated capacity of the partition, in MB, represented as an integer, rounded to thenearest MB (1024 x 1024).

FREE_SPACE_MB: The estimated free space of the partition, in MB, represented as an integer, roundedto the nearest MB (1024 x 1024).

LABEL_TYPE: The name of the label format, assuming that there is an internal label on the cartridge orpartition. If this label conforms to a standard, this is the name and number of the standard. Otherwise thelabel format shall be named and described in a separate document, and the name specified in thatdocument shall be used for this field.

INTERNAL_LABEL: The contents of the internal label, sufficient to do label checking.





Example:

CART_LAYOUTØNO_PARTITIONSØ3ØNO_SIDESØ1ØSIDE_NOØ0Ø

PARTITION_NOØ0ØLABEL_FORMATØXYZ1.2ØINTERNAL_LABELØ

My_First_PartitionØSIZE_MBØ1506000000ØFREE_SPACE_MBØ494000000Ø

SIDE_NOØ0ØPARTITION_NOØ1ØLABEL_TYPEØXYZ2.1ØINTERNAL_LABELØ

My_Second_PartitionØSIZE_MBØ1206000000ØFREE_SPACE_MBØ794000000Ø

SIDE_NOØ0ØPARTITION_NOØ2ØLABEL_FORMATØXYZ2.1Ø

INTERNAL_LABELØMy_Third_PartitionØSIZE_MBØ1001000000Ø

FREE_SPACE_MBØ2398Ø§

Rationale: This record specifies the top level structure of the cartridge so that the cartridge can be prop-erly identified and positioned at the proper partition.

8.5 Cartridge number of blocks between tape marks

Record name: CART_NO_BLOCKS_BETWEEN_TM

Description: This record specifies the number of blocks between tape marks, if constant throughout thecartridge (except possibly for the last collection of blocks).

Named fields: None.


Example:

For a cartridge where the number of blocks between tape marks is constant,

CART_NO_BLOCKS_BETWEEN_TMØ48Ø§

Rationale: At least one FSMS implementation sets a maximum for the number of fixed length datablocks that may appear contiguously on a tape without being interrupted by a tape mark. This is a char-acteristic of the cartridge, imposed by the FSMS, and the presence of a tape mark does not necessarilyindicate the end of a file or file segment.

8.6 Cartridge number of bytes after tape marks

Record name: CART_NO_BYTES_AFTER_TM

Description: This record specifies the number of bytes after a tape mark that are not part of the originalfile but instead consist of control information written by the FSMS.

Named fields:




NO_BLOCKS: The number of blocks containing the specified number of bytes.

NO_BYTES: The total number of bytes in the blocks.


Example:

For a cartridge where a 64 byte block after the tape mark is used for control information,

CART_NO_BYTES_AFTER_TMØNO_BLOCKSØ1ØNO_BYTESØ64Ø§

For a cartridge where 64 bytes of control information is after the tape mark but in two blocks (not neces-sarily of equal length),


Rationale: At least one FSMS implementation follows each tape mark with a fixed amount of controlinformation that is not part of a file segment.

8.7 Cartridge block size

Record name: CART_BLOCK_SIZE

Description: This record specifies the block structure on the cartridge as it is seen and written by theFSMS, giving the block size in bytes if it is fixed (constant) throughout the cartridge.

Named fields:

MODE: Either FIXED or VARIABLE, depending on whether the block size is fixed or variable.

SIZE: The size in bytes of the blocks, if MODE is FIXED. Otherwise this field shall be omitted.


Example:

CART_BLOCK_SIZEØMODEØFIXEDØSIZEØ1024Ø§

Rationale: MODE refers to the data blocks on a cartridge, not to any control information that may bewritten in a different block size. If the data blocks are all of the same size, then MODE should be FIXEDand SIZE should be provided, otherwise mode should be VARIABLE.

8.8 Cartridge family

Record name: CART_FAMILY

Description: If families are supported, this record names the family to which this cartridge belongs. Ifmultiple families are supported for a single cartridge, then this should list the families for this cartridge inany order.

Named fields: None.

Record category: Selectable.




Example:

For the case where there is only one family,

CART_FAMILYØDAAC_TRMMØ§

For the case where there are two families,

CART_FAMILYØAccountingØChemistyØ§

Rationale: At least one FSMS implementation allows the arbitrary grouping of cartridges into sets whichare called families. No additional information is implied in this grouping.

8.9 Cartridge billing ID

Record Name: CART_BILLING_ID

Description: If a billing ID is associated with this cartridge, this field specifies the billing ID.

Named fields: None.


Example:

CART_BILLING_IDØJSC5872Ø§

Rationale: Some implementations allow for billing on the cartridge level.

8.10 Cartridge volume group

Record name: CART_VOL_GROUP

Description: If volume groups are supported, this record specifies the volume group to which this car-tridge belongs.

Named fields:

NAME: This specifies the name of the volume group.

BLOCKSIZE: This specifies the block size in uncompressed bytes for all of the cartridges in this volumegroup.

COMPRESSION_ALLOWED: This specifies whether compression is allowed or not. The value of thisfield is to be either YES or NO.


Example:

CART_VOL_GROUPØNAMEØAccountingØBLOCKSIZEØ4096Ø

COMPRESSION_ALLOWEDØNOØ§




Rationale: At least one implementation allows the association of specific cartridges with a directorystructure and its contents. This allows the user to specify that the contents of a directory structure is al-ways placed in a defined set of cartridges. In addition, the user can specify the block size and whethercompression is allowed for these cartridges or not. In most cases, users want the data compressed bythe drive when it is written onto the cartridges. However in some cases, the users forbid any compressionbecause they fear loss of data or they have already compressed their data and do not want the drive tocompress it again. Examples are medical data and video data.

8.11 Cartridge location

Record name: CART_LOC

Description: This record specifies the current location of the cartridge.

Named fields:

CART_LIB: The name of the Library or Physical Volume Repository in which the cartridge resides. Thereis no universal naming scheme for libraries. The name could be a string, a SCSI-ID, or an IP address.There will be cases where a library will hold thousands of cartridges. During an FSMS migration, it maybe unreasonable to assume that each cartridge will be ejected and then reloaded into the same libraryunder the new FSMS. This allows for the translation of the name of the library from the exporting FSMSto the importing FSMS. The meaning of the library name shall be documented by the exporting FSMS.

CART_PLACE: A string describing the current location of the cartridge within the Library or Physical Vol-ume Repository. There is no universal location naming scheme within libraries. There will be caseswhere a library will hold thousands of cartridges. During an FSMS migration, it may be unreasonable toassume that each cartridge will be ejected and then reloaded into the same library under the new FSMS.This field allows for the translation of the location from the exporting FSMS to the importing FSMS. Themeaning of the location string shall be documented by the exporting FSMS.


Example:

CART_LOCØCART_LIBØVault1Ø CART_PLACEØRack6,Shelf3Ø§

Rationale: This record is intended to aid in the situation where one FSMS is being replaced by another inthe same facility.

8.12 Cartridge lot

Record name: CART_LOT

Description: The lot number or lot identifier of the cartridge. This may include a cartridge vendor name.

Named fields: None.


Example:

CART_LOTØ1764903Ø§

Rationale: This information is useful for management purposes, since cartridge quality may vary by lot.




8.13 Cartridge statistics

Record name: CART_STATS

Description: Statistics on the use of the cartridge.

Named fields:

NO_MOUNTS: The number of times the cartridge has been mounted.

CREATION: The time the cartridge was created, in the form yyyy/mm/dd/hh:mm:ss. The time zone shallbe UTC. This is the time at which the cartridge becomes known to the FSMS.

MODIFIED: The time at which the cartridge was last modified, in the form yyyy/mm/dd/hh:mm:ss. Thetime zone shall be UTC.

LAST_MOUNT: The time of last access, in the form yyyy/mm/dd/hh:mm:ss. The time zone shall be UTC.

NO_ERRORS: The number of errors that have occurred on this cartridge. Note that there is no standardway of reporting errors (hard, soft, partial, etc.). Hence this field can only be given on a "best effort" ba-sis.


Example:

CART_STATSØNO_MOUNTSØ12ØCREATIONØ1984/03/15:08:15Ø

MODIFIEDØ1985/06/25:15:20:32ØLAST_MOUNTØ1985/06/25:15:20:32Ø

NO_ERRORSØ10Ø§

Rationale: This information may be useful for management purposes.

8.14 Cartridge byte order

Record name: CART_BYTE_ORDER

Description: This record exists to handle cases where the natural order of the bytes has been changedon the cartridge. It gives the physical representation on the cartridge of the characters "ABCD" where"ABCD" is the natural order.

Named fields: None.


Example:

CART_BYTE_ORDERØBADCØ§

Rationale: This information is necessary to read the cartridge in some instances.




8.15 Cartridge block structure

Record name: CART_BLOCK_STRUCT

Description: Certain implementations embed headers within a physical block on the media. There is aconstant-length logical data block containing part of the original file with a header or trailer containingcontrol information. This is repeated throughout the physical block, which is of fixed length.

Named fields:

BLOCK_LENGTH: The length in bytes of the physical block.

DATA_BLOCK_LENGTH: The length in bytes of the logical data block.

HDR_LENGTH: The length in bytes of the header.

TRL_LENGTH: The length in bytes of the trailer.

Record category: Required, if this practice is followed.

Example:

For an instance with a header but no trailer,

CART_BLOCK_STRUCTØBLOCK_LENGTHØ40960ØDATA_BLOCK_LENGTHØ

40950ØHDR_LENGTHØ10Ø§

NOTE If the physical block is not completely filled, resulting in some unused space at the end, this will be ac-counted for in the specification of the length of the file segment written in the block. See 9.5.

Rationale: At least one FSMS implementation uses this kind of format.

8.16 Cartridge recording FSMS

Record name: CART_RECORDING_FSMS

Description: This record names the FSMS that recorded the cartridge. If system version information isavailable, it shall be included.

Named fields: None.


Example:

CART_RECORDING_FSMSØAcme storage system 1.3 Ø§

Rationale: This information, with the file segment addressing information, facilitates reading files fromthe cartridge, particularly if the cartridge was not originally recorded by the exporting system.




9 Metadata describing the file segments on the removable media beingexchanged

This clause specifies the record types in the section of the metadata that contains information about thefile segments on the cartridges being exported. The names of all records specified in this clause beginwith the characters "FS_".

The File UID and File Segment File Name in order shall be the first two records in this section of themetadata. Until there is a new record with a different File Segment UID, all subsequent records refer tothe file segment identified by the UID specified in the first record.

9.1 File UID

Record name: FS_FILE_UID

Description: An identifier that is unique within the objects described by this metadata. All file segmentsof the same file have the same file UID and are distinguished by their segment and copy numbers. Ifthere are hard or soft links to a file, then the file shall only be represented once in this clause. Softlinkinformation is contained in clause 10, and hard links to the same file all have the same file UID. If the filehas different copies, each copy shall have the same UID. If the file has different versions, each versionshall have a different UID.

Named fields: None.

Record category: Required. If the exporting system does not have such a UID, then the generatingsystem must provide it. For the purposes of this export, this UID does not have to be generated by analgorithm that would give the same UID at different times.

Example:

FS_FILE_UIDØ2fac5432-31f8-11b4-a222-08002b34c003Ø§

Rationale: This provides a unique identifier for the file. The original path name of the file may not beunique in the receiving system and so may have to be modified when imported. UIDs are probably easierto match than path names, particularly if the path names are mapped during import. The DCE UUID maybe used for this purpose, but other solutions are possible as well.

9.2 File segment file name

Record name: FS_FILE_NAME

Description: The fully qualified name of the file, unique within the exporting system name space. This isthe unique identifier of the file in the operating system where the file resides. If the file has different, non-identical versions (as opposed to different identical copies), then the version identifier shall be includedas part of the file name.

Named fields: None.


Examples:

FS_FILE_NAMEØ/usr/joelw/Cartridge_Stuff/PSD_R12.htmlØ§




FS_FILE_NAMEØC:windows\Personal\joelw\Cartridge_Stuff\PSD_R12.htmØ§

Rationale: This provides the file name. Note that this name also appears in 10.1.

9.3 File segment number

Record name: FS_NO

Description: For a given file, file segments are an ordered set, numbered 0, 1, 2, 3, ... . This is thenumber of the file segment that specifies its logical order.

Named fields: None.


Examples: FS_NOØ12Ø§

Rationale: A file is an ordered set of file segments. There is no requirement that the logical order is thesame as the physical order. By specifying the logical order, it is possible for the receiving system toidentify different file segments and order them in their logical order.

9.4 File segment copy number

Record name: FS_COPY

Description: If having multiple copies of a given file is supported, and if there is more than one copy,then this is the copy number: 0, 1, 2, ... .

Record category: Required, if there is more than one copy of the file. If there are no copies of the file,then this record shall be omitted.

Example: For the third copy,

FS_COPYØ2Ø§

Rationale: This allows the identification of different copies of the same file.

9.5 File segment information

Record name: FS_INFO

Description: This record describes the way in which this file segment is written to one or more car-tridges. If the file is striped, then the information on the striping of this file segment is contained in thisrecord. Fields having to do with striping shall be omitted if the file segment is not striped. Note that allcounts start at zero.

Named fields:

STRIPE_WIDTH: The number of cartridges used to write this file segment.

STRIPE_BLOCK_LENGTH: The number of bytes written to one cartridge before beginning to write to thenext cartridge.




OFFSET: The offset in the file that is, the byte number in the file of the first byte in the file segment.This field shall be present.

LENGTH: The length, in bytes, of the file segment, not including any control blocks, such as headers andtrailers, added by the FSMS.

The following set of fields is repeated in order, one set for each stripe if the file segment is striped; oth-erwise, of course, there is only one set; that is, the CART_ID and the remaining fields of this record de-scribe the starting location of the file segment on the specified cartridge. For a striped file segment, thecartridges and the starting addresses shall be specified in the order in which data is written to the car-tridges. All of the following fields refer to the cartridge identified in this field. Note also that if the filesegment has a header part and/or a trailer part, the header and trailer shall be specified in the headerand trailer records; that the starting address specified in this record is the first byte of the header; but thatthe length specified above does not include the length in bytes of any header or trailer blocks.

CART_ID: The ID of the cartridge containing this file segment. The value of this field shall match someentry in the Cartridge identifier record specified in 8.1.

PARTITION: This specifies the partition number on the cartridge in which the file segment starts. If thisfield is not specified, it is assumed that the file segment begins in partition zero.

SIDE: This specifies the side number on the cartridge in which the file segment starts. If this field is notspecified, it is assumed that the file segment begins in side zero.

TAPE_MARK: This specifies the number of tape marks within the partition in which the file segmentstarts to skip in positioning to the file. If this field is not specified, it is assumed that the file segment be-gins in file zero.

BLOCK: This specifies the block after the tape mark, within the specified partition, in which the file seg-ment begins. If this field is not specified, it is assumed that the file segment begins in block zero. In gen-eral, a block is created by physical I/O to the sequential media by the device driver.

RECORD: This specifies the record within the block in which the file segment begins. If this field is notspecified, it is assumed that the file segment begins in record zero. For the purposes of this standard, arecord is a part of a block. Note that not all sequential media technologies preserve record boundaries.

BYTE: This specifies the byte within the block or record in which the file segment begins. If this field isnot specified, it is assumed that the file segment begins in byte zero.

COOKIE: Application-dependent value for fast positioning on a cartridge. The exact nature of this entryshall be documented separately by the exporting FSMS. Not all cartridges support this.


Examples:

For non-striped file segments on a cartridge having cartridge identifier AB0003.

The file segment begins after the first tape mark in the third partition, begins immediately at the begin-ning of the partition, is of length 6083 bytes, and represents bytes 2365 through 8447 of the original file:

FS_INFOØOFFSETØ2364ØLENGTHØ6083ØCART_IDØAB0003ØPARTITIONØ2Ø§

The file segment is the first thing on the cartridge, is of length 2089, and represents bytes 10365 through12453 of the original file:




FS_INFOØOFFSETØ10364ØLENGTHØ2089ØCART_IDØAB0003Ø§

The file segment is on a non-partitioned cartridge, is after the 13th tape mark, is of length 1024, and rep-resents bytes 89455 through 90478 of the original file:

FS_INFOØOFFSETØ89454ØLENGTHØ1024ØCART_IDØAB0003ØTAPE_MARKØ 13Ø§

The file segment is on the fifth partition, begins at the sixth block, is of length 5349, and represents bytes2170 through 7519:

FS_INFOØOFFSETØ2169ØLENGTHØ5349ØCART_IDØAB0003ØPARTITIONØ4Ø

BLOCKØ5Ø§

For a striped file segment on cartridges having cartridge identifiers AB0010, AB0011, AB0012,and where the file segment is striped across those cartridges in that order.

The file segment is after the third tape mark on each of the cartridges, it is the beginning of the file, thefile segment length is 2397960, and stripe block length is 159864 bytes:

FS_INFOØSTRIPE_WIDTHØ3ØSTRIPE_BLOCK_LENGTHØ159864ØOFFSETØ

0ØLENGTHØ2397960ØCART_IDØAB0010ØTAPE_MARKØ3ØCART_IDØAB0011

ØTAPE_MARKØ3ØCART_IDØAB0012ØTAPE_MARKØ3Ø§

Rationale: This record allows for the specification of a file segment address on a cartridge in a numberof different ways. Different technologies allow for different addressing or for a variety of addressingschemes. At least one helical scan implementation, for instance, maintains the original record bounda-ries as written by the host system.

9.6 File segment header

Record name: FS_HEADER

Description: This record specifies the length in bytes of the header, if any, which is added by the FSMS.The number of bytes in the header is not included in the specification of the file segment length.

Named fields: None.

Record category: Required, if there is a header.

Examples:

If there is a header for the file segment of length 12 bytes,

FS_HEADERØ12Ø§

Rationale: Some implementations add a header containing control information to each file segment.This is not to be confused with implementations that place headers on data blocks, which may not be filesegments.




9.7 File segment trailer

Record name: FS_TRAILER

Description: This record specifies the length in bytes of the trailer, if any, which is added by the FSMS.The number of bytes in the trailer is not included in the specification of the file segment length.

Named fields: None.

Record category: Required, if there is a trailer.

Examples:

If there is a trailer of length 12 bytes,

FS_TRAILERØ12Ø§

Rationale: Some implementations add a trailer containing control information to each file segment. Thisis not to be confused with implementations that place trailers on data blocks, which may not be file seg-ments.

10 Metadata describing the directory and file structure being exchanged

There are only two record types defined in this clause. The first describes an object in the file system thatmay be a directory, an ordinary file, or some kind of special file. The second record type applies only toobjects such as directories or tar files that are known by the FSMS to contain files. It gives a list of filescontained in the object described in the previous record. Other than this, there is no restriction on theorder of the records in this clause.

The names of all records specified in this clause begin with the characters "D_".

10.1 Object attribute list

Record name: D_OBJ_ATTR_LIST

Description: This record identifies an object and gives information about it. Hard links are identified bytwo or more files having the same UID but different fully qualified path names.

The absence of clauses describing Access Control Lists, Discretionary Access Control, Mandatory Ac-cess Control, and other security information is deliberate because there is no common practice evidentamong different FSMS or operating system vendors. This type of information should be transferred ex-plicitly and under the direction of the appropriate security administrators. It would not be prudent to ex-pect an importing FSMS to update or change security information in any way.

Named fields:

UID: An identifier of the object (file, directory, etc.) which is unique within the objects described by thismetadata.

NAME: The fully qualified name of this object.

OBJ_TYPE: One of the following: DIR for a directory, FILE for an ordinary file, SYM for a symbolic link,and CONTAINER for a file that the FSMS uses to hold other files.




PERMISSIONS: The octal representation (in ASCII or Unicode characters) of the absolute mode of thePOSIX access permissions on this object. Deviations from what is below shall be documented sepa-rately.

An absolute mode is specified using octal numbers nnnn where

n is a number from 0 to 7.

An absolute mode is constructed from the OR of any of the following modes:

4000: Set user ID on execution.

2000: Set group ID on execution.

0400: Allow read by owner.

0200: Allow write by owner.

0100: Allow execute (search in directory) by owner.

0040: Allow read by group.

0020: Allow write by group.

0010: Allow execute (search in directory) by group.

0004: Allow read by others.

0002: Allow write by others.

0001: Allow execute (search in directory) by others.

OWNER_ID: The ID of the owner of the object.

GROUP_ID: The group ID associated with this object.

LENGTH: The length, in bytes, if the OBJ_TYPE is FILE. This field is necessary because it enableschecking to see if there is unused space (a "hole") at the end of a file. For a directory, this field shall beomitted.

CTIME: The time that the object's metadata was last changed, expressed in the formyyyy/mm/dd/hh:mm:ss. The time zone shall be UTC.

ATIME: The time that the object was last accessed, expressed in the form yyyy/mm/dd/hh:mm:ss. Thetime zone shall be UTC.

MTIME: The time that the object was last modified, expressed in the form yyyy/mm/dd/hh:mm:ss. Thetime zone shall be UTC.

BILLING_ID: If accounting is associated directly with this object, this is the billing ID.

FAMILY_ID: If object families are supported, this identifies the family.

CONTAINER_TYPE: If the FILE_TYPE is CONTAINER, this is the name of the program to be used (withany command line options that are required) to extract the files contained in this file.

SYM_CONTENTS: The contents of the symbolic link, if the FILE_TYPE is SYM.





Example:

D_FILE_ATTR_LISTØUIDØ2fac1234-31f8-11b4-a219-08002b34c003ØNAMEØ

/usr/joelwØPERMISSIONSØ0755ØOWNER_IDØjoelwØGROUP_IDØsoftwareØ

CTIMEØ1997/03/15/09:10:26ØATIMEØ1997/05/20/11:30:14ØMTIMEØ

1997/04/10/08:26:56ØBILLING_IDØA3972ØFAMILY_IDØsoftwareØOBJ_TYPEØ

DIRØ§

Rationale: This record allows the transmission of information that varies by object, in particular a file, butnot by cartridge or file segment.

10.2 Contents list

Record name: D_CONTENTS_LIST

Description: If the object type in the record described in 10.1 is DIR or CONTAINER, then this recordmust follow immediately. It shall contain the UID of the directory or container file (named in the previousrecord) followed by a list of the locally qualified names and the corresponding UIDs of all of the objectscontained in the DIR or CONTAINER specified in the previous record. In the list, the UID shall immedi-ately follow the name. If the directory or container file is empty, then this record shall contain only theUID of the directory or container file.

Named fields:

DIR or CONTAINER, depending on the OBJ_TYPE of the entry in 10.1.

LIST, to indicate the start of the list of contained files and/or directories.


Examples:

For a non-empty directory,

D_CONTENTS_LIST ØDIRØ2fac1234-31f8-11b4-a2108002b34c103Ø

LISTØsoftwareØ2fac1234-31f8-11b4-a2108002b34c103Ø

hardwareØ2fac1234-31f8-11b4-a2108322b34c103Ø§

For an empty directory,

D_CONTENTS_LIST ØDIRØ2fac1234-31f8-11b4-a2108002b34c103Ø§

Rationale: This record provides a list of objects contained in a directory or some other kind of containerfile. Note that in the case of directories, this information could be deduced from the full path names in therecords specified in 10.1.




Annex A(informative)

Examples of the record types

Introduction

This annex is not part of ANSI/AIIM MS66-1999 Metadata for Interchange of Files on SequentialStorage Media Between File Storage Management Systems (FSMSs). It is included for informationalpurposes only. It presents at least one example of each record type, just as it would appear in a meta-data export. Real vendor names are used for the purpose of being complete and concrete and not for thepurpose of endorsing any vendor. Each example stands on its own and is not intended to relate to anyother example.

The original clause or subclause numbers are listed in parentheses after the name of the clause or sub-clause.

A.1 Metadata describing the originating FSMS and its environment (7)

The export version record (7.1)

Example: EXPORT_VERSIONØANSI/AIIM MS55-1997xØ§

Rationale: This record identifies the version, allowing for different versions of the standard to be distin-guished in the future.

Export hardware system (7.2)

Example:

EXPORT_HARDWARE_SYSTEMØVENDORØSGIØNAMEØOriginØ

VERSIONØ200Ø§

Rationale: This record identifies the hardware system that wrote the export, to aid in reading the export.In this instance, the export was written on an Origin200, manufactured by Silicon Graphics Incorporated(SGI).

Export operating system (7.3)

Example:

EXPORT_OSØVENDORØSGIØNAMEØIRIXØVERSIONØ6.2Ø§

Rationale: This record identifies the software system that wrote the export, to aid in reading the export.In this instance, the operating system was IRIX 6.2, supplied by Silicon Graphics Inc. (SGI).




Export time (7.4)

Example:

EXPORT_TIMEØ2000/02/29/12:23:15Ø§

Rationale: This record identifies the time at which the export was written, in case multiple exports havebeen made.

Exporting FSMS (7.5)

Example:

EXPORT_FSMSØVENDORØEMASSØFSMSØAMASSØVERSIONØ1.0Ø§

Rationale: For additional identification, this record identifies the vendor of the FSMS. This informationmay be helpful in interpreting some details of the export. This example specifies that the FSMS usedwas AMASS, supplied by EMASS, and that the version was 1.0.

Export root (7.6)

Example:

To specify that the directories /chemistry/accounting and /architecture/design are the top-level directo-ries,

EXPORT_ROOTØ/chemistry/accountingØ/architecture/designØ§

Rationale: This record may be used in two ways by some implementations. By specifying one or moreexport roots, one identifies the logical mount points under which all of the files in this export are defined.This allows one to partition the export if necessary. On the import side, if there is a name space conflict,one can change the names of the export roots so that the files underneath these roots will not cause aname space conflict.

A.2 Metadata describing the removable media being exchanged (8)

Cartridge identifier (8.1)

Example: CART_IDØA60032Ø§

Rationale: This is to provide unique identification of each of the cartridges being exported. The commit-tee decided that the name (or identifier) of a cartridge is sufficient to uniquely identify it and that a sepa-rate unique identifier is not needed. If the name of a cartridge to be imported into a system conflicts withan existing cartridge, then the conflict will have to be resolved to complete the import, and anotherunique cartridge identifier (such as a UID) will not solve that problem.

Cartridge info (8.2)

Example:

CART_INFOØMEDIA_TYPEØDLT2000ØPHYSICAL_RECORDING_FORMAT

Ø DLT1000§




Rationale: Sometimes a cartridge might not be written in its "native" format but might instead be writtenin an earlier version of that format. MEDIA_TYPE is provided because a number of different makes andmodels of cartridges may be exported. PHYSICAL_RECORDING_FORMAT is provided because somecartridges can be recorded in a variety of ways.

Cartridge drive type (8.3)

Example: CART_DRIVE_TYPEØMFRØQuantumØMODELØDLT4000Ø

DRIVE_COMPRESSIONØYESØ§

Rationale: This record, with the record described in 8.2, is designed to remove any ambiguity in speci-fying a drive for reading or writing the cartridge.

Cartridge layout (8.4)

Example:

CART_LAYOUTØNO_PARTITIONSØ3ØNO_SIDESØ1ØSIDE_NOØ0Ø

PARTITION_NOØ0ØLABEL_FORMATØXYZ1.2ØINTERNAL_LABELØ

My_First_PartitionØSIZE_MBØ1506000000ØFREE_SPACE_MBØ494000000Ø

SIDE_NOØ0ØPARTITION_NOØ1ØLABEL_TYPEØXYZ2.1ØINTERNAL_LABELØ

My_Second_PartitionØSIZE_MBØ1206000000ØFREE_SPACE_MBØ794000000Ø

SIDE_NOØ0ØPARTITION_NOØ2ØLABEL_FORMATØXYZ2.1Ø

INTERNAL_LABELØMy_Third_PartitionØSIZE_MBØ1001000000Ø

FREE_SPACE_MBØ2398Ø§

Rationale: This record specifies the top level structure of the cartridge so that the cartridge can be prop-erly identified and positioned at the proper partition.

Cartridge number of blocks between tape marks (8.5)

Example:

For a cartridge where the number of blocks between tape marks is constant,

CART_NO_BLOCKS_BETWEEN_TMØ48Ø§

Rationale: At least one FSMS implementation sets a maximum for the number of fixed length datablocks that may appear contiguously on a tape without being interrupted by a tape mark. This is a char-acteristic of the cartridge, imposed by the FSMS, and the presence of a tape mark does not necessarilyindicate the end of a file or file segment.

Cartridge number of bytes after tape marks (8.6)

Example:

For a cartridge where a 64 byte block after the tape mark is used for control information,





For a cartridge where 64 bytes of control information is after the tape mark but in two blocks (not neces-sarily of equal length),


Rationale: At least one FSMS implementation follows each tape mark with a fixed amount of controlinformation that is not part of a file segment.

Cartridge block size (8.7)

Example:

CART_BLOCK_SIZEØMODEØFIXEDØSIZEØ1024Ø§

Rationale: MODE refers to the data blocks on a cartridge, not to any control information that may bewritten in a different block size. If the data blocks are all of the same size, then MODE should be FIXED,and SIZE should be provided, otherwise mode should be VARIABLE.

Cartridge family (8.8)

Example:

For the case where there is only one family,

CART_FAMILYØDAAC_TRMMØ§

For the case where there are two families,

CART_FAMILYØAccountingØChemistyØ§

Rationale: At least one FSMS implementation allows the arbitrary grouping of cartridges into sets thatare called families. No additional information is implied in this grouping.

Cartridge billing ID (8.9)

Example:

CART_BILLING_IDØJSC5872Ø§

Rationale: Some implementations allow for billing on the cartridge level.

Cartridge volume group (8.10)

Example:

CART_VOL_GROUPØNAMEØAccountingØBLOCKSIZEØ4096Ø

COMPRESSION_ALLOWEDØNOØ§

Rationale: At least one implementation allows the association of specific cartridges with a directorystructure and its contents. This allows the user to specify that the contents of a directory structure is al-ways placed in a defined set of cartridges. In addition, the user can specify the block size and whethercompression is allowed for these cartridges or not. In most cases, users want the data compressed bythe drive when it is written onto the cartridges. However in some cases, the users forbid any compression




because they fear loss of data or they have already compressed their data and do not want the drive tocompress it again. Examples are medical data and video data.

Cartridge location (8.11)

Example:

CART_LOCØCART_LIBØVault1Ø CART_PLACEØRack6,Shelf3Ø§

Rationale: This record is intended to aid in the situation where one FSMS in a facility is being replacedby another in the same facility.

Cartridge lot (8.12)

Example:

CART_LOTØ1764903Ø§

Rationale: This information is useful for management purposes, since cartridge quality may vary by lot.

Cartridge statistics (8.13)

Example:

CART_STATSØNO_MOUNTSØ12ØCREATIONØ1984/03/15:08:15Ø

MODIFIEDØ1985/06/25:15:20:32ØLAST_MOUNTØ1985/06/25:15:20:32Ø

NO_ERRORSØ10Ø§

Rationale: This information may be useful for management purposes.

Cartridge byte order (8.14)

Example:

CART_BYTE_ORDERØBADCØ§

Rationale: This information is necessary to read the cartridge in some instances.

Cartridge block structure (8.15)

Example:

For an instance with a header but no trailer,

CART_BLOCK_STRUCTØBLOCK_LENGTHØ40960ØDATA_BLOCK_LENGTHØ

40950ØHDR_LENGTHØ10Ø§

NOTE If the physical block is not completely filled, resulting in some unused space at the end, this will be ac-counted for in the specification of the length of the file segment written in the block. See 9.5.

Rationale: At least one FSMS implementation uses this kind of format.




Cartridge recording FSMS (8.16)

Examples:

CART_RECORDING_FSMSØHPSS 1.2Ø§

CART_RECORDING_FSMSØAMASS 5.4Ø§

Rationale: This information, with the file segment addressing information, facilitates reading files fromthe cartridge, particularly if the cartridge was not originally recorded by the exporting system.

A.3 Metadata describing the file segments on the removable media beingexchanged (9)

File UID (9.1)

Example:

FS_FILE_UIDØ2fac5432-31f8-11b4-a222-08002b34c003Ø§

Rationale: This provides a unique identifier for the file. The original path name of the file may not beunique in the receiving system and so may have to be modified when imported. UIDs are probably easierto match than path names, particularly if the path names are mapped during import. The DCE UUID maybe used for this purpose, but other solutions are possible as well.

File segment file name (9.2)

Examples:

For a Unix implementation,

FS_FILE_NAMEØ/usr/joelw/Cartridge_Stuff/PSD_R12.htmlØ§

For a Windows implementation,

FS_FILE_NAMEØC:windows\Personal\joelw\Cartridge_Stuff\PSD_R12.htmØ§

Rationale: This provides the file name. Note that this name also appears in 10.1.

File segment number (9.3)

Examples: FS_NOØ12Ø§

Rationale: A file is an ordered set of file segments. There is no requirement that the logical order is thesame as the physical order. By specifying the logical order, it is possible for the receiving system toidentify different file segments and order them in their logical order.

File segment copy number (9.4)

Example: For the third copy,

FS_COPYØ2Ø§

Rationale: This allows the identification of different copies of the same file.




File segment information (9.5)

(1) For non-striped file segments on a cartridge having cartridge identifier AB0003.

The file segment begins after the first tape mark in the third partition, begins immediately at the begin-ning of the partition, is of length 6083 bytes, and represents bytes 2365 through 8447 of the original file:

FS_INFOØOFFSETØ2364ØLENGTHØ6083ØCART_IDØAB0003ØPARTITIONØ2Ø§

The file segment is the first thing on the cartridge, is of length 2089, and represents bytes 10365 through12453 of the original file:

FS_INFOØOFFSETØ10364ØLENGTHØ2089ØCART_IDØAB0003Ø§

The file segment is on a non-partitioned cartridge, is after the 13th tape mark, is of length 1024, and rep-resents bytes 89455 through 90478 of the original file:

FS_INFOØOFFSETØ89454ØLENGTHØ1024ØCART_IDØAB0003ØTAPE_MARKØ 13Ø§

The file segment is on the fifth partition, begins at the sixth block, is of length 5349, and represents bytes2170 through 7519:

FS_INFOØOFFSETØ2169ØLENGTHØ5349ØCART_IDØAB0003ØPARTITIONØ4Ø

BLOCKØ5Ø§

(2) For a striped file segment on cartridges having cartridge identifiers AB0010, AB0011, AB0012,and where the file segment is striped across those cartridges in that order.

The file segment is after the third tape mark on each of the cartridges, it is the beginning of the file, thefile segment length is 2397960, and stripe block length is 159864 bytes:

FS_INFOØSTRIPE_WIDTHØ3ØSTRIPE_BLOCK_LENGTHØ159864ØOFFSETØ

0ØLENGTHØ2397960ØCART_IDØAB0010ØTAPE_MARKØ3ØCART_IDØAB0011

ØTAPE_MARKØ3ØCART_IDØAB0012ØTAPE_MARKØ3Ø§

Rationale: This record allows for the specification of a file segment address on a cartridge in a numberof different ways. Different technologies allow for different addressing or for a variety of addressingschemes. At least one helical scan implementation, for instance, maintains the original record bounda-ries as written by the host system.

File segment header (9.6)

Examples:

If there is a header for the file segment of length 12 bytes,

FS_HEADERØ12Ø§

Rationale: Some implementations add a header containing control information to each file segment.This is not to be confused with implementations that place headers on data blocks, which may not be filesegments.




File segment trailer (9.7)

Examples:

If there is a trailer of length 12 bytes,

FS_TRAILERØ12Ø§

Rationale: Some implementations add a trailer containing control information to each file segment. Thisis not to be confused with implementations that place trailers on data blocks, which may not be file seg-ments.

A.4 Metadata describing the directory and file structure being exchanged (10)

Object attribute list (10.1)

Example:

D_FILE_ATTR_LISTØUIDØ2fac1234-31f8-11b4-a219-08002b34c003ØNAMEØ

/usr/joelwØPERMISSIONSØ0755ØOWNER_IDØjoelwØGROUP_IDØsoftwareØ

CTIMEØ1997/03/15/09:10:26ØATIMEØ1997/05/20/11:30:14ØMTIMEØ

1997/04/10/08:26:56ØBILLING_IDØA3972ØFAMILY_IDØsoftwareØOBJ_TYPEØ

DIRØ§

Rationale: This record allows the transmission of information that varies by object, in particular a file, butnot by cartridge or file segment.

Contents list (10.2)

Examples:

For a non-empty directory,

D_CONTENTS_LIST ØDIRØ2fac1234-31f8-11b4-a2108002b34c103Ø

LISTØsoftwareØ2fac1234-31f8-11b4-a2108002b34c103Ø

hardwareØ2fac1234-31f8-11b4-a2108322b34c103Ø§

For an empty directory,

D_CONTENTS_LIST ØDIRØ2fac1234-31f8-11b4-a2108002b34c103Ø§

Rationale: This record provides a list of objects contained in a directory or some other kind of containerfile. Note that in the case of directories, this information could be deduced from the full path names in therecords specified in 10.1.




ANSI/AIIM MS66-1999© by the Association for Information and Image Management International

1100 Wayne Avenue, Suite 1100Silver Spring, MD 20910-5603

Tel: 301/587-8202FAX: 301/587-8202

e-mail: [email protected]: http://www.aiim.org

ISBN 0-89258-365-7Printed in the United States of America

Metadata for Interchange of Files on Sequential Storage ... · ANSI/AIIM MS66-1999 M ETADATA FOR...

Documents

Transcript of Metadata for Interchange of Files on Sequential Storage ... · ANSI/AIIM MS66-1999 M ETADATA FOR...