Metadata & Interoperability: Free Tools

the future in hand | createasphere.com

Metadata & Interoperability: Free Tools


The Instructors

Roger Howard ([email protected])• The J. Paul Getty Trust, eVox, Playboy, Blizzard, North Plains Systems

• Digital media technologist, photographer, cataloger

• Software developer, workflow automator

• Always looking for better tools to make life easier for my users (and to make me

look better to my boss and, now, clients).


The Instructors

Mike Jennings ([email protected])• Technicolor/PRN, Adobe Systems

• Quality Program Manager, Workflow Analyst, Metadata Maven

• Asset pack-rat


The Problem

Your DAM needs to evolve• Business needs change

• Users demand refinements

• Measurable gains come from workflow improvements

• Out-of-the-box features/services sometimes disappointing

…but your budget can’t keep up


Your DAM is only as good as your metadata

Realizing the value of a DAM always comes down to

the metadata.• Asset retrieval

• Business processes

• Reporting and analysis

• Integration with other business systems


…But it’s difficult to ensure metadata quality

• People don’t want to do it

• People aren’t very good at it

• People aren’t very consistent about it

• Integrated systems don’t always interoperate well


The Strategies

Add services to improve metadata quality and

workflow efficiency• Validation – Make sure the metadata is correct, or at least in the correct format

• Automation – The less people have to do, the fewer mistakes they make

• Deduplication – efficient use of resources, metadata version control

• Simplification – minimize user interaction = happier users = buy-in


The Answer

DIY integration of free tools• Lots of free/cheap/open-source tools to fill where the DAM leaves off

• Most DAMs support integrating external tools and services

• It doesn’t always require an integrator or professional coder

• No procurement = agility and simplicity

• We’re not lawyers, but using open source does not mean giving away your farm


The Tools

Community built is community supported• Field tested, IT approved

• Typically cross-platform

• Often found “rolled into” trusted commercial products

• Robust communities and accessible developers for support

• Professional training or integrators usually available if you’re stuck


The Methods

Scripting• Our favorite content creation tools support it with native scripting APIs

• Power tools with a command line interface are inherently scriptable

• Many good free tools support it

• Nearly all DAMs support scripting external tools in one way or another


The Scripts

Shell Scripts• Usually little more than the name of the tool followed by the particulars

(“arguments”)

• Can get as elaborate as you need


The Scripts

Embedded JavaScript• JavaScript and variants (e.g. Adobe’s ExtendScript and Flash ActionScript) are

similar – you learn one, you learn them all

• Can get as elaborate as you need

• Loads of great learning resources out there, and you likely have people on staff

already with the baseline skills


Intra-Application Automation

Content creation tools often include workflow

automation features• Can be trivially simple to create and use

• Can extend to significant levels of sophistication

• Distributable among workgroups

• Can address all four strategies


Intra-Application Automation

Example: Adobe Creative Suite• Photoshop actions: Easy and fun!

• Photoshop Event Scripting: JavaScript automation for processes and metadata

• Bridge scripting for automated workflows between Adobe applications

• Custom File Info panels, metadata schemas

• Flash-based custom palettes highly tuned to business needs, appear as bespoke

design


DAM Tooling

DAMs provide many paths to integrate external tools• Direct calling of external tools, often used in ingest and delivery processes

• Native, Web Services and other APIs

• Product-specific integration, such as InDesign plugins

• Direct access to the underlying database


DAM Tooling

Example: Final Cut Server• Metadata mapping pane for EXIF, IPTC, QuickTime or XMP metadata on ingest

• Executes scripts and executables post-archive, pre-restore, from poll watchers,

or subscription events (asset, job or production) – with parameters

• It is itself scriptable from the command-line


DAM Tooling

Example: Alfresco ECM and Adobe Drive• Adobe Drive treats a version-controlled content repository as a mounted volume

usable within content creation apps

• Alfresco provides a “CMIS Connector” to interact with it in a standardized way

• Content creators can therefore use version control/asset management as easily

as not using it


Metadata Toolkit

Exiftool• Cross-platform, free, extremely mature and frequently updated

• Supports 100+ file formats, from images to video to PDF

• Focused primarily on explicit metadata

• Extract, read, write, and sanitize embedded metadata

• Obscures many of the ugly details of embedded metadata


Metadata Toolkit

ffmpeg• Cross-platform, free, extremely mature and frequently updated

• Primarily focused on being the universal video and audio format converter

• Can be used purely to analyze and report on file format characteristics


Metadata Toolkit

ImageMagick• Cross-platform, free, extremely mature and frequently updated

• Primarily focused on being the universal image file format converter and

programmatic editor

• Also has extensive reporting capabilities for analyzing supported file

formats via the “describe” command


Metadata Toolkit

MediaInfo• Cross-platform, free, extremely mature and frequently updated

• GUI for casual use, command-line for scripting and automation

• Features customizable reporting and metadata extraction capabilities

• XML output can be piped to other processes and DAMs


Metadata Toolkit

Adobe Creative Suite metadata tools• Custom File Info panels

• Custom metadata schemas

• Custom Flash palettes

• Script Events Manager and Photoshop scripting

• BridgeTalk inter-application scripting


Adobe CS – File Info Panels

Adobe Creative Suite – File Info Panels• Many creatives live and die by the Adobe product stack

• Did you know there’s a data entry interface built in to every CS

product?

• More importantly, did you know it’s insanely customizable?


Adobe CS – File Info PanelsOut of the box, they are one-size-fits-all, but…

•Easily customized for simple panels; tweaking an

XML file is enough to get started

•The real power is in completely custom panels

• Flash based

• XMP enabled

• Network connected

• The UI sky is the limit

• Build and maintain centrally for easy updates

• Modal


Adobe CS – Flash Palettes

• Build your own custom UI within Photoshop

• Uses existing Flash/ActionScript skills

• Modeless – floats in a palette, like other Photoshop elements

• Integrates XMP, Photoshop automation, Web services


Adobe CS – Script Events Manager• Hooks scripts and actions to Photoshop events

• Start, New, Open, Save, Close, Print, and Export

• Enables workflow policy enforcement at the creative application level

• Validate or apply metadata on open or save

• Boilerplate metadata from templates; dynamic metadata (eg., time stamps for reporting)

• Apply watermarks during printing

• Apply scripts or Photoshop actions based on embedded metadata

• Extensive JavaScript and native scripting APIs

• The unsung hero of Photoshop workflow automation


Great Tools, So Now What?Solving metadata problems

• Collecting (better) metadata within a creative workflow

• Improving embedded metadata extraction in your DAM

• Enforcing embedded metadata policies at distribution

• Generating better (or any) proxies/previews*

Leveraging metadata to streamline workflow• Preflighting assets during ingest for formatting and naming compliance

• Integrating your DAM with desktop tools

* Yes, proxies are metadata


Metadata in Creative Workflow• Critical metadata is often best known by those creating it

• Adding metadata after the fact can be costly and inconsistent

• Metadata capture is often seen as an onerous chore, best avoided

• Your job is to provide tools that…

• Make it easier to capture and contribute controlled metadata so that

• ...you can enforce policies about what data is captured, without making

enemies.


Metadata in Creative Workflow• A Flash palette or file info panel as a view into your DAM

• Presenting key information about an asset, such as workflow status, job info, or editorial

comments, directly within Photoshop

• Leveraging the controlled vocabularies in your DAM to quickly tag assets from within

Photoshop

• Automatically populating metadata in an asset through a lookup

• Script Events Manager scripts as a compliance tool

• Check metadata, filenaming, and other criteria during opening, saving


Improving Metadata ExtractionDAMs often don’t extract the metadata you need. No one wants to enter it.

Most of the time they shouldn’t have to.

• Much of the needed metadata can be gleaned

• Tools can extract hidden metadata

• Scripts can provide context and workflow data (username, timestamp, approval status etc.)

• The more metadata you can extract for the content and context, the happier

everyone is


Improving Metadata Extractionexiftool, ImageMagick, MediaInfo and ffmpeg can all provide extremely

robust metadata extraction capabilities, depending on your file formats and

metadata needs

• Run as part of an ingest process from within the DAM

• Run on a schedule, batch extracting metadata and inserting into your DB

• Run within the creative workflow, extracting and summarizing key bits


Enforcing Embedded Metadata Policies

• Embedded metadata is everywhere, in every file

• Embedded production metadata may bloat your distribution file sizes

• …And most of it has no value in distribution/publishing anyway

• Embedded metadata can be very revealing – too revealing

• Do you know what information is being leaked out of your DAM when

you distribute assets?



The embedded metadata whitelisting process…

•For each of your major file types and distribution processes,

determine what embedded fields should be preserved, e.g.:

• Copyright metadata

• Asset identifiers

• Captions, transcriptions, keywords, other blessed descriptors



• Add a step into your distribution processes – whether it’s within the

DAM, or simply a Folder Action attached to an FTP directory

• Using exiftool, delete all metadata *except* the fields you’ve

consciously chosen to preserve• This way, you don’t have to keep up with what might be in there, selectively deleting; just

preserve what you know you want. It’s as simple as this:

exiftool -all= --Credit --XMP:Description


Generating (Better) Proxies*• File formats change constantly, your DAM may not

• Your DAM can store any file, but may not generate useful proxies

• Your DAM is customizable, one way or another

• There are tools out there that can help your DAM create better proxies

*proxies are metadata, if you squint just right**

** I guarantee we’re not the only people here today who will abuse the term “metadata”, just the first


Generating (Better) Proxies*

Digital Photography Proxy Issues•Everyone shoots RAW, but no two vendors interpret that RAW file the same way

•Your creatives spend time adjusting their RAW files (in Lightroom, or PhotoMechanic, etc) to

get them looking just right

•When those RAW files are ingested into your DAM, they look nothing like the edits your

creatives made – if they can be displayed at all

•So you’ve resorted to workarounds, like generating a companion TIFF, negating much of the

benefit of the RAW workflow (or at least adding complexity)



Digital Photography Proxy Solutions•The DNG workflow

• DNG repackages the RAW data into a universal container

• DNG still has all the flexibility of a RAW file, but just one file format to support

• DNG editors can maintain a JPEG preview, embedded in the DNG

• The JPEG preview reflects the exact look that was created in the editor – just like creating that companion TIFF,

but without the extra file

• JPEG previews can be full size, for high-resolution, edit-correct previews

• JPEG previews are easy to extract from a DNG, and insert into your DAM



Digital Photography Proxy Solutions•The DNG workflow

• Use the Adobe DNG Converter tool to automate DNG conversion processes, or…

• Have the RAW editor convert to DNG while downloading files from the camera/cards

• If your DAM doesn’t already extract previews from DNGs, you can still..

• Use exiftool or dcraw to extract the JPEG preview and insert into your DAM

•Don’t want to go DNG, but at least want to generate viewable previews of your obscure camera file formats?

• dcraw can generate previews of nearly any camera file format you can find

• It may not match the exact tone and color the creative is looking for, but it’s something!



Video Proxy Challenges•You’ve got a DAM with basic video format support, but it can’t properly preview some formats

•You don’t have a commercial video conversion product like FlipFactory

•You’re increasingly getting video assets in a huge range of formats, from Canon 5dMk2 to

iPhone files, to that obscure format your security cameras use



Video Proxy Solutions•ffmpeg• Supports reading and writing dozens of audio and video container formats, codecs, and

importantly the exponentially large mix of the two

• Can provide extensive format reporting

• Can generate streamable proxies in the most popular formats, such as flv and mp4


Ingest Preflighting with Metadata• You have specific file format requirements in certain workflows, but your DAM

can’t see enough about the file to tell a TIFF from a TIFF.

• You want to check your ingested PDFs for compliance – are fonts

embedded? What PDF profile(s) are the files compatible with?

• Your DAM is customizable (have we said that before?)


Ingest Preflighting with Metadata• ImageMagick’s “describe” tool in verbose mode can tell you everything you

want to know about your image file formats, including PDFs

• Jhove can provide validation of certain formats, including validating against

many of the most common PDF and TIFF profiles

• ffmpeg can provide detailed analysis of your audio and video file formats


Conclusion• Format Support: Free tools can support new formats your DAM can’t

• Workflow automation: Free tools can make your DAM less “visible” (intrusive)

• Metadata Extraction: Free tools can improve your DAM’s searchability

• Metadata Processing: Free tools can prevent you from releasing private info

• Preflighting: Free tools can ensure compliance and quality standards

• Format Conversion: Free tools can fix problems in your asset library

Bottom Line: Free Tools Can Add Value to Your Existing DAM

Metadata & Interoperability: Free Tools

Data & Analytics

Transcript of Metadata & Interoperability: Free Tools