BackgroundBackground

• Mission performance requirements for critical products and the need to keep operations costs low drive the need for automation. Historically, missions have, with low inheritance, developed unique capabilities to manage production of critical instrument products.

• Development on a project specific system for Cassini (CDRS) was approximately 2.5 FTEs.

• MER developed a similar system for 1.0 FTE.

What is MATIS?What is MATIS?• MATIS – Multi-mission Automated Task Invocation

Subsystem

• A distributed data processing framework for automated program execution. It is a workflow manager:– Execute programs in a specific order and under specific

conditions– Synchronized program execution

• Leveraging off lessons learned and skill sets from Cassini and MER, MATIS development over FY06-FY07 was approximately 2.0 FTEs. Adaptation effort varies from project to project, but it’s on the order of a couple months.

Current ApplicationsCurrent Applications

Phoenix Diviner Mars Climate Sounder MSL

Programs that the MATIS executes:

Level 1 product generation

Product delivery Quick look of EDRs

Level 1b product generation

Report generation Product delivery Product

accountability

Level 2 product generation

EDR and RDR product generation

Product delivery Product

accountability Quick look of EDRs

Event Trigger Listen to the directory for files in the directory that match the given condition

Listen to the directory for files in the directory that match the given condition

Listen to the directory for files in the directory that match the given condition

Messaging on the JMS bus notifying the system that a file is ready to be processed

#Process Definitions

4 3 3 4+

Initial Adaptation Effort (Dev)

9 weeks 6 weeks 4 weeks TBD

Code Size 5K (Logical SLOC) 2K (Logical SLOC) 1.4K (Logical SLOC) TBD

Slope Maps

Radiometrically Correct

Geometrically Correct

Stereo Disparity

3-D XYZ

Range

Surface Normal

Reachability

Surface Rough

MATIS - Phoenix

"Courtesy of D. Alexander"

Automated Generation of RDRsAutomated Generation of RDRs

Phoenix RequirementsPhoenix Requirements• The Operational Product Generation Subsystem (OPGS), part of the

Ground Data System (GDS), is responsible for the generation of:- Experiment Data Records (EDR) for 6 instruments, including all

cameras- Reduced Data Records (RDR)

• The higher level products, or the Reduced Data Records (RDR), are used for daily planning which has to be completed within very tight timing requirement. Timing requirements:- Produce and deliver the lowest level products, EDRs within 60

seconds- Produce higher imaging products, RDRs, within 30 minutes.

Included in the RDRs, are some CPU intensive products which may take up to 5 minutes each.

Phoenix AdaptationPhoenix Adaptation

• Met all timing requirements

• Phoenix pipeline - 40 parallel streams (including EDRs, RDRs, mosaic and terrain meshes) distributed among 4 machines (need not be distributed evenly)

• Demonstration

MATIS OverviewMATIS Overview

• Core – Logging– Fail-safe capability– Automated parallel

processing– Distributed processing– Load balancing– Execution of the programs– Interface definitions for the

project specific modules– Scalable– Configurable

• Adaptation – What to run– When to run– Where to run – How to run

Workflow Manager Framework Provides common functionalities in the Core part,

allowing the Adaptation effort to be small.

MATIS ArchitectureMATIS Architecture

Host a

MATIS Persistent Store Controller

Host b

Service Factory

Host c

Service Factory

Host d

Service Factory

Host e

Objects outlined in red are project-specific.

Database

Pipeline Design

Process Definition (PD)

Process Definition (PD)

What?

Scheduler

When?

LoadBalancer

Where?

RunTask RunTask RunTask

How?

• Processing high data volumes• Supporting rapid product turnaround• Distributing programs across multiple hosts• Keeping operational costs low• Automating steps that need to be repeated• Performing different tasks based on different

conditions

When is it Useful to use MATIS?When is it Useful to use MATIS?