1

Distributed Operations for the Mars Exploration RoverMission with the Science Activity PlannerJustin V. Wick, John L. Callas, Jeffrey S. Norris, Mark W. Powell, Marsette A. Vona, III

Jet Propulsion Laboratory, Pasadena CaliforniaJustin.Wick@cornell.edu

Abstract— The unprecedented endurance of both the Spiritand Opportunity rovers during the Mars Exploration RoverMission (MER) brought with it many unexpectedchallenges. Scientists, many of whom had planned onstaying at the Jet Propulsion Laboratory (JPL) in Pasadena,CA for 90 days, were eager to return to their families andhome institutions. This created a need for the rapidconversion of a mission-planning tool, the Science ActivityPlanner (SAP), from a centralized application usable onlywithin JPL, to a distributed system capable of allowingscientists to continue collaborating from locations aroundthe world.

Rather than changing SAP itself, the rapid conversion wasfacilitated by a collection of software utilities that emulatedthe internal JPL software environment and providedefficient, automated information propagation. During thisprocess many lessons were learned about scientificcollaboration in a concurrent environment, use of existingserver-client software in rapid systems development, and theeffect of system latency on end-user usage patterns.

Switching to a distributed mode of operations also saved aconsiderable amount of money, and increased the number ofspecialists able to actively contribute to mission research. Long-term planetary exploration missions of the future willbuild upon the distributed operations model used by MER.

TABLE OF CONTENTS

1. INTRODUCTION ............................................ 12. CENTRALIZED OPERATIONS............................ 13. MOVING TO DISTRIBUTED OPERATIONS............ 24. ARCHITECTURE AND IMPLEMENTATION........... 55. TECHNICAL CHALLENGES .............................. 66. MISSION IMPACT .......................................... 67. RECOMMENDATIONS FOR THE FUTURE............ 78. CONCLUSION................................................ 8REFERENCES.................................................... 8BIOGRAPHY..................................................... 8

1. INTRODUCTION

The Mars Exploration Rover Mission has been anunqualified success. At the time of this writing, both theSpirit and Opportunity rovers have exceeded theiroperational lifetimes by a factor of three. Both roverscontinue to roam Mars, returning a wealth of valuable newinformation to Earth.\

The unprecedented length of the Mars Exploration Roversmission created many challenges for mission planners.

Although the original architecture of the mission planningsystem was intended to be distributed in nature[1], budgetconstraints did not allow for the development of thiscapability. As a result, the planning software was designedsuch that it was heavily reliant upon internal computingresources at JPL, making it unusable at remote sites.

In March 2004, as the primary mission for both rovers drewto a close, it became evident that both rovers were likely tocontinue operating long past their original 90 sol lifetimes. Faced with the reality that mission scientists would shortlybegin departing from JPL to their respective researchinstitutions, the decision was made to redesign the systemto accommodate the participation of scientists at remotesites. Since the Science Activity Planner was the primarytool used by scientists in the high level planning process,an effort was begun to adapt SAP for use outside of JPLand to provide a collaborative framework for scientists tooperate it.

2. CENTRALIZED OPERATIONS

During the centralized phase of the MER mission, scientistswere gathered in a single building at JPL and used theplanning and analysis software in a specially configuredcomputing environment, which was called the FlightOperations System. The Flight Operations System was theplatform for the Ground Data System (GDS) software thatdrove the data processing and planning processes for themission.

The Science Activity Planner (SAP) is a GDS tool that isused to perform dual roles facilitating manual science andengineering-level data analysis, and planning the dailyactions of each rover in a coarse, high level fashion. Thistactical decision-making process is similar to that practicedin the FIDO field tests [2]. SAP provides functionalityallowing users to view and manipulate images captured bythe rover in what is known as the Downlink Browser (figure2). There, objects can be examined, and points in spaceknown as “targets” can be marked for use in creatingplanned activities. The Uplink Browser (figure 3) allowsusers to create “activities” (a high level description of anaction the rover should take, such as acquiring an image ordriving) and “observations” (groups of related activities).

Each day the tactical process starts with a science meetingduring which scientists are briefed about the currentsituation. After this, the scientists break up into “themegroups” such as “atmospheric” or “soils” or “long termplanning” and work in parallel, using SAP to construct

2

sequences of instructions for the rover that reflect theirscientific goals. During this several hour period, the data isanalyzed within SAP, points are designated as targets forthe rover’s actions, and the potential plans are put together. After this time, the final plan is debated and assembled inthe Science Operations Working Group (SOWG) meeting. Possible scientific observations from each group are rankedaccording to importance. After a structured debate, theaccepted observations are arranged together in SAP to meetthe daily energy, time, and bandwidth budgets that havebeen established by the engineering team. The final mergedplan is then delivered for further refinement and processingdownstream to be converted to the actual sequence ofinstructions sent to the rover.

A homogeneous computing environment consisting ofcustom-built workstations running the Linux operatingsystem facilitated collaboration within the system. Therewas a central Network File System server (the OSS) for eachrover (MER-A and MER-B) and also a central SQL databaseserver for each. Because all downlinked data and planninginformation were kept in these two central repositories,collaboration was simple. When a scientist saved a plan file,it was immediately available to all others to be analyzed andmerged. Target designation, critical to the planningprocess, was also synchronized with low latency (~2 sec) viathe central SQL server. All science workstations wereguaranteed to have access to the exact same set of data.

3. MOVING TO DISTRIBUTED OPERATIONS

Due to budget and lifestyle constraints, the mission wasshifted to a new, more distributed mission architecture. Thecost of keeping relevant scientists on location in Pasadenawas prohibitive, and many of the scientists and engineershad family elsewhere in the world for whom they wereresponsible. Because of this, the decision was made tocreate an environment in which scientists could tacticallyplan with SAP at remote sites. This software environmentwould have to:

Make planning-relevant downlink data availableto the remote scientists in a timely fashion.

Allow scientists to interactively share targetsdesignations.

Facilitate the sharing of plan files that containthe scientific observations for the day.

Dynamically create indexing metadata ofavailable data products to make them available in SAP.

Maintain operational security through use ofencryption, authentication, and firewalls.

It was decided that the best possible action was to closelyreplicate the JPL software environment, rather than changeSAP itself. SAP expects a highly structured file systemdatabase containing images, range data, three dimensionalmeshes, spectral data records, coordinate frame information,planning constraints, and plan files. Because no availablenetwork file system server was fast enough to be used bySAP interactively, the relevant data sets would have to bemirrored locally. This also meant that the indexing of that

data (which is how SAP knows what information isavailable on the file system) would also have to be donelocally. Also, the sharing of targets and plans presented achallenge, since the servers hosting the plans and targetswere not accessible outside of JPL.

Data Synchronization

The first matter was to arrange for the data to be delivered tothe remote SAP workstations. SAP expects data to exist ina highly structured, hierarchical system of folders,numbering well over a million for each rover. This filesystem database is known as the Operational SoftwareSystem, or OSS. The folders separate data by sol (martianday), instrument, and data type. The job of the datasynchronization subsystem was to replicate the internal filesystem database of downlinked data on client workstationsaround the world.

The first solution to this problem that was developedutilized an open source program known as RSYNC, whichcan synchronize files and directories recursively betweenmachines through a secure SSH tunnel. A daemon wascreated that repeatedly synchronized the directories for recentsols with a central server. The central server itself was to befilled with the SAP-relevant data from the operational NFSservers.

The problem with this approach was that it relies heavily onpolling, and tens of thousands of files and directories had tobe recursively compared. It was decided that it would placetoo much of a load on the server to have an acceptably lowlatency for data delivery. Worse, overloading issues werealready a severe problem on the operational NFS server, andit was decided that this solution would most likelyexacerbate the situation.

We then decided to create a second data synchronizationsolution, utilizing the JPL Multi-mission Image ProcessingLab’s (MIPL) File Exchange Interface (FEI). FEI is asystem that MIPL uses to automatically push out data toremote sites, such as research institutions or museums. While it supports polling and client-initiated downloading,it also has an event-driven server-push mode that relies onthe “subscriptions” of a client to a set of file types. FEIwas chosen because it has a very low latency (~2 secondswithin the JPL network) and is very well load balanced.

The main problem associated with this approach was thatFEI does not keep track of the path in the file system to thedirectory where a particular file came from – this data wouldhave to be reconstructed. In addition to this, FEI alsocontains a large number of files that cannot be used by SAPand are not relevant for tactical planning. Because of thelow bandwidth at many remote sites, the files would haveto be filtered for relevance prior to downloading. Thesystem also needed to allow for the gathering of archivedfiles from specific sols of interest – a feature not supportedby FEI.

The final solution was to have two methods of getting files– an automatic subscription program, and a manual archived

3

file retrieval program. Both programs used a filter todetermine whether or not a given file was desired based onits relevance (and in the case of archival data, whether or notit fell within a specified range of sols). Also, a script wasassembled that could sort the files into their final locationsbased solely on the file names. This was made possible bythe fact that the file names systematically encode the datatype, instrument name, time acquired, and the rover fromwhich the data was obtained.

Each workstation established a connection with the FEIserver, and signed up to be “notified” when files in arelevant “filetype” were made available. This notificationwas pushed from the server to the client, at which time theclient decided, based on the filename, if the file wasdesirable. If the file was wanted, it was retrieved from theserver and then sorted into the file system. This system haslatencies on the order of minutes or less, and has nearly ideabandwidth use (the server/client messages are very short).

Obtaining access to archival data was somewhat lessstraightforward. That program, given a rover designation (Aor B, for Spirit or Opportunity) and a desired range of sols,downloads an entire roster of all available files in relevantfiletypes. It then filters the names of files to find those filesthat fall into the specified range of sols, and also do notcurrently exist on the local file system. This roster listingprocess is very inefficient and takes several minutes,however downloading the requisite data can take hours, sothe overhead is acceptable.

The final step in the data synchronization process is theData State Manager Daemon – a daemon process that scansavailable downlinked data products and creates acomprehensive index of what data is available. Every thirtyseconds the most recent sols are scanned (and occasionallyolder sols, according to a probabilistic algorithm) to see ifnew data has been made available. When new data isdiscovered, it is processed and incorporated into the index,making it available for SAP. A nearly identical process isrun at JPL, where the cost of all open SAP instancesscanning each sol would have been prohibitive.

Target Synchronization

Target synchronization was another vital component of thedistributed SAP system. At JPL, targets were synchronizedbetween machines by storing them in a central SQL server. The various SAP instances would poll the server every twoseconds, checking timestamps in the database to see if newtargets had been created, or if old ones had been modified. There were no security issues because the database was notaccessible from the outside world, and all individuals usingcomputers that could access the database were cleared todesignate targets.

In a distributed setup, however, everything changed. It wasno longer possible to make the central JPL target serveravailable to machines outside of JPL for security reasons;however, each remote site had to be able to see the sametargets as users at JPL with minimal latency. Moreover,there had to be a method to take targets from outside JPL

and import them into the internal JPL server. This entireprocess was required to be as low-latency and automatic aspossible, while maintaining operational security.

The solution we chose was that there should be a secondary,“external” SQL server that would be accessible to authorizedmachines outside of JPL. A script at JPL forwardedchanges and new additions to the JPL internal targetdatabase out to the external server every few seconds. Because of the nature of the database, it was acceptable fortargets to exist in the external database but not in theinternal database without causing any problems. Plan files,however, contain reference to targets (to decide where todrive, or aim a camera, etc). If a plan were brought intoJPL that referenced an external target, that target would haveto be manually imported by a script at JPL. That scriptwould then have to extract a static copy of the target fromthe plan file text. This process was considered securebecause it required a human in the loop to verify that thetarget was valid. Also, the external server was protected bya strict firewall that only allowed access from a set ofsecured university computers that were certified as part ofthe planning process. The data from JPL was encryptedusing an SSH tunnel with public key authentication.

A final consideration for target sharing was the complicationthat was caused by SAP’s use of MySQL database polling.The newly changed entries in the JPL target database had tohave a timestamp in the remote database that would causethe remote SAP clients to notice the change. Due tovarious internal details of the SAP client and MySQLservers, these timestamps had to be adjusted into the futurebefore being sent to the external targets server. This causednew targets to be downloaded repeatedly by clients, but thebandwidth that this used was acceptably small.

Plan Sharing

The issues associated with plan sharing were similar tothose of target synchronization in that the central server (inthis case, the internal NFS server at JPL) was not accessibleto the outside world. Also, there were similar securityconcerns. Plan files coming out of JPL automatically werenot considered to be a security issue. However no one fromoutside JPL could be able to insert a plan file into thenormal planning directories inside JPL.

The solution chosen was that there should be tworepositories for plan files, one inside JPL (the NFS server)and one outside JPL. These two repositories wouldautomatically synchronize, however no user outside JPLcould be allowed to write a file that would propagate to anormal planning directory inside JPL. Instead, usersoutside JPL would have to place plans into special“external” directories. The planning directories inside JPLfor each sol had names such as “apxs” or “soil”, etc, brokendown by group, and each containing an additional named“working” directory. The planning directories weremodified by adding another directory named “external” ineach subgroup directory. A user outside JPL could submithis plan to the central external plan server, but only if itresided inside an “external” directory.

4

Figu

re 1

– D

ata

flow

in D

istr

ibut

ed S

AP

. B

lue

arro

ws

indi

cate

dat

a pr

oduc

ts.

Gre

en i

ndic

ates

pla

ns.

Ora

nge

indi

cate

s ta

rget

s.

Proc

esse

s m

arke

d in

red

requ

ire

hum

an in

terv

entio

n.

rsyn

c

OSS

Plan

s

Loca

l O

SS

Subm

it-pl

an

SAP

Was

hing

ton

Univ

ersi

ty

DSM

SFi

rew

all

JPL

Fire

wal

lW

ashi

ngto

n Un

iver

sity

Fire

wal

l

Rem

ote

Site

JPL

MSA Pl

ansu

bmitd

int

Staging

Plan

sync

d

Plan

sync

d

RSVP SA

P

ext

ext

int

ext

Rem

ote

Fire

wal

l

targ

etsy

ncd

Targ

ets

Impo

rt-ta

rget

int /ext

int /ext

int

Staging int/ext

int

ext

ext

int

ext

FEI

MIP

L

Inco

npu

shou

td

Targ

ets

Sapf

eid

int/e

xtrs

vp

int

int

ext

ext

RSVP

Data

OPG

S DA

TA

INCO

NSVF

int

RSVP

int ext

DSM

dindex

data

data

data

ext

5

The majority of the synchronizations were automatic. JPL’sNFS server was considered the canonical source for“internal” plans. Every 30 seconds the next 5 sols worth ofinternal plans were sent to the external server. Every 30seconds or so, those same sols were synchronized from theexternal server to the SAP workstations at each institution. However, because there was no single canonical source forplans created at an institution, it was decided thatsubmission of an “external” plan to the central server wouldbe a manual process. Once an “external” plan wassubmitted to the central server, within 30 seconds it wouldbe copied to the same directory on the JPL NFS server, tobe seen by those at JPL. This is how planned observationsthat were created outside of JPL could become part of thefinal plan at the SOWG meeting held at JPL.

4. ARCHITECTURE AND IMPLEMENTATION

Figure 1 illustrates the overall architecture of the system. The left side of the diagram represents the portion of thesystem running at JPL. The lower right corner of thediagram shows the servers at Washington University of St.Louis. Finally, the upper right represents each individualDistributed SAP workstation. The data flow is illustratedby colored arrows: blue for downlink data, green for plans,and yellow for targets. The cylindrical shapes representservers, and the named rectangles signify a process orcollection of processes that are logically grouped together. A name in red signifies that the process requires a humanintervention. Whether or not a target or plan beingtransferred was created inside or outside JPL is indicated byan “int” or “ext” label on the associated arrow. Namesending in “d” refer to “daemon” processes that runconstantly in the background. RSVP is an engineeringlevel planning program that is used by some scientistsremotely, and uses much of the same data as SAP.

Downlinked Data

To understand how the system works, one should firstexamine the downlink data flow (blue arrows). The Multi-mission Image Processing Laboratory (MIPL) is the sourceof all processed imagery used in this system. MIPL, alongwith the “Inconpushoutd” – a daemon that pushes out initialconditions of the rover for a sol, the planning constraints,and coordinate frame information – supplies the FEI serverwith the files that are needed for the use of SAP outside ofJPL.

After the files are sent to the FEI server, the clients arenotified of the newly available files through the “Sapfeid”, adaemon that handles all of the processes that wait for newfiles. If the files are deemed relevant, they are downloadedfrom the server into a temporary directory, and then stored in their proper location in the local OSS (the local set offolders that hold the data for each sol). Once the new datais noticed by the DSMd (Data State Manager daemon), it isthen indexed. After that the data can be accessed by SAP.

Target Data

The target data flow is more symmetric – a target canoriginate either at JPL or at an external workstation. SAPinstances at JPL create targets in the internal database. AJPL computer running the “targetsyncd” – the targetsynchronization daemon – takes newly generated targets andsends them out to the centralized target server atWashington University. Every time an external SAP clientopens a plan from a given sol, it fetches the targetsassociated with that sol from the central server. The SAPclient also maintains a polling thread that keeps looking fornew targets being made on that sol.

A computer external to JPL can create a target in theexternal database, making it available to all other externalSAP instances. If the target needs to be used inside JPL, anexternal plan file is saved and then submitted to the planserver; a copy arrives at the JPL OSS. A person, either atJPL or logged in remotely then runs the import-targetscript, giving it the plan file, and the name of the target tobe imported. The import-target program reads the targetdata from the plan file and then enters it into the local JPLdatabase. Any open internal SAP instances can then see thenew target, and it can be used in the final, official plan.

Planning Data

The final component of the system is the shared planningdata flow (green arrows). Just like shared targets, there aretwo separate places where plans can be generated – internalto JPL by SAP, or external to JPL by SAP. Inside JPL,they are kept in special directories on the OSS. Anautomated process called Plansubmitd polls the OSS every30 seconds to check for new plans, or newly modifiedplans, and uploads them to the external planning server atWashington University. A similar process called Plansyncdpolls the server for new or newly modified external plans tobe imported. Plansyncd imports all changed plans to astaging area, but only copies external plans to the actualOSS for security reasons. This prevents anything submittedto the external server from affecting the internal planswithout intervention from a human at JPL.

The right side of this data flow concerns the remote sites. If a plan is created or modified at a remote site, and the userwants to share it with the rest of the distributed SAP users,the “submit-plan” script is run. This sends the file to theserver (overwriting any older version of that file if itpreviously existed). Also, a slightly different version of thePlansyncd is running in the background. It is identical tothe JPL version, except that it copies both internal andexternal plans to the local OSS.

Programming Languages Used

All of the daemon programs were written in Perl 5, andutilized utility shell scripts. The import-target program is acombination of a Perl frontend and a Java backend. Perlwas used because the system is tied heavily to theunderlying OS, and it made invocation of Unix commands

6

and file manipulation particularly easy. Also a largeamount of the work done by these programs involved textparsing.

5. TECHNICAL CHALLENGES

The technical challenges of this project were many andvaried. Most of the challenges involved reliablecommunications between all of the parts of the system,atomicity of transactions, and server load. Also, out ofnecessity, many parts of the system used software in waysthat were not originally intended.

The most common technical challenge of the entire projectwas the large set of problems created by repeated polling offile systems and servers. Because the MER GDS has nocentralized, common event-driven architecture, most of thecomponents of the distributed system use some form ofpolling to handle propagated changes. Polling itself is nota significant challenge in software development. However,the efficiency of the polling was a severe limiting factor inwhat design choices were available, and it forced us to usenondeterministic algorithms for some of the less importantparts of the system.

Our data indexing process, the Data State Manager (DSM),needed to poll tens of thousands of subdirectories of the filesystem every thirty seconds. This grew to the point whereit was untenable, so a compromise was made in thesystem’s design. Instead of scanning all sol data directoriesevery 30 seconds, it would scan only the three most recentfor new data constantly. The older directories would have aprobability of being scanned each 30 second sweep such thatabout 95% of all sols would be scanned in a given 24 hourperiod. The use of nondeterministic algorithms wasconsidered safe because older sols tended not to changeoften, and their changes tended not to be important.

Another example where polling was a bottleneck was thePlan Synchronization Daemon. The Plan SynchronizationDaemon (plansyncd) relied heavily on polling of a centralserver. Plansyncd used the RSYNC client tunneled throughSSH, and RSYNC only permits one directory to berecursively synchronized per connection. Because of thisand the fact that the first five upcoming sols had to besynchronized every thirty seconds, each requiring a separateconnection, the SSH authentication server on the centralplanning server became intolerably slow. While plans stillpropagated, it was at a reduced rate, and often connectionsto the server were rejected due to the overload. As of thiswriting, we plan to replace this polling process with amanual process due to the incredible load it places on theserver.

A different issue encountered was reliable communicationsthrough a highly heterogeneous network environment. There were a lot of very complicated firewalls involved –two levels at JPL, at least two at Washington University ofSaint Louis, and usually between one and two firewalls atother institutions. SSH tunneling made communications

through these firewalls possible, however this requiredauthentication keys to be distributed. Network failures werenot entirely uncommon, and temporary workarounds had tobe set up in the event that a server was not reachable. Server load and reliability was often the deciding factor forthe success of the Distributed SAP system.

One of the biggest causes of bugs was the relativeheterogeneity of systems running SAP outside of JPL. Inside JPL the software was run exclusively on Red HatLinux 7.3 boxes, all of which contained identical processorsand graphics cards. Outside of JPL, Red Hat Linux 7.3, 8,9, Red Hat Enterprise Linux 3, and Fedora Core 1 were inuse. This was a problem because it required differentsystems to use different versions of the FEI client, whichwas not fully tested on Fedora Core 1. Also, newer Linuxdistributions shipped version 5.8 of Perl, which has subtlydifferent semantics for a few very important operations, suchas regular expression matching. This lead to a few bugsinvolving data delivery, which were very difficult to trackdown.

Last but not least was the fact that some softwarecomponents of the system were being used in ways thattheir creators had not intended. This sometimes put thesystem into odd states requiring manual intervention. TheFEI server system was not designed, for instance, to notifyclients if a file that already existed on the server wasmodified, only when new files were added. So, whencertain important configuration files had to be pushed out,they had to be removed and then added to FEI. Also, FEIhad no method of filtering files based on the sol theybelong to, or specific details of the file type. This had to beimplemented in one of the more complicated Perl programsthat we created. The same goes for the lack of file systemmetadata preservation in FEI. The files had to be sorted bya Perl program, based solely on the name of the file – a factthat precluded the sorting of certain types of files accurately.

6. MISSION IMPACT

The impact of the Distributed Science Activity Planner onthe MER mission was very significant. By allowingscientists to analyze data and collaboratively plan at remoteinstitutions, Distributed SAP was a primary enabling factorin the feasibility of the distributed operations architecture.

Transitioning to distributed operations has saved aconsiderable amount of money during the extended mission. The principal expense change with the implementation ofremote operations was the savings on travel expense. Approximately $250 per day was expensed to the project fora typical visiting scientist on MER to cover travel-relatedexpenses. During the period of remote operations, it isestimated that about 13 outside scientists were necessary tooperate each rover – 26 in total. The cost of this team’sresidence at JPL for operations is estimated atapproximately $200,000 per month of dual-rover operations.The current implementation of remote operations for MER

7

saves the MER Project at minimum this amount eachmonth.

There are also some modest remote operations relatedexpenses. These expenses included development cost forthe remote operations infrastructure and the on-goingmaintenance of this infrastructure. However, since most ofthe hardware was existing equipment at JPL that wasrelocated to remote sites and the maintenance engineers arealready on staff, the added cost for remote operations wassmall relative to the overall travel cost savings.

The MER science team has adapted quite well to remoteoperations. The team has consistently been able to producethe detailed activity plans and associated sequences on timefor each rover on every sol during the period of remoteoperations. This is likely due to the extensive training thescience team received prior to the beginning of rover surfaceoperations - the intense co-located operational period of theprime mission phase (90 sols) which precisely honed eachteam members operational skills. It is also aided by thesimilarity of the remote operations infrastructure to theprime mission operational infrastructure (e.g., tools andprocedures, etc.).

According to MER Principle Investigator Steve Squyres,there has been no loss in capability of the science team dueto moving to the distributed missions architecture. Hereports that the system is highly robust – planning atCornell University continued even during a power failurevia a laptop and cellular phone modem. Having thescientists at home with their family and collogues has beengood for morale, and it has enabled the team to successfullyretain its members in the long term.[3]

There are still challenges in using the distributedarchitecture. Communication is more difficult when peopleare not in the same room. Also, a significant amount oftime was spent emailing screen shots back and forth, due tothe fact that many mission computer programs were notdesigned to be collaborative over a distance. Much of thecommunications difficulties were mitigated by the use ofteleconferencing equipment, web cameras, Virtual NetworkComputing, and SSH. There are aspects to the system thatneed improved, however, the net impact of moving to adistributed architecture is overwhelmingly positive.

7. RECOMMENDATIONS FOR THE FUTURE

Future missions will be more ambitious and will requiresignificantly more complex supporting technology. Theamount of data returned will skyrocket as novelcommunications infrastructures are utilized. Planning willlikely become more specialized and more involved as larger,more sophisticated science payloads are sent to space. Also,the trend towards greater international cooperation in spaceexploration continues to diversify the geographicaldistribution of labor.

Consequently, future missions are likely to be partially orfully distributed in their ground support architecture. Thisrequires a new breed of operations software – softwaredesigned from the ground up to support collaboration frommultiple locations. This software will have to deal withissues of versioning, concurrency, data propagation, contentsynchronization, and effective communications among teammembers. To support its development, we make thefollowing recommendations:

1) Design operations software as fundamentallydistributed – from the beginning. The single mostchallenging aspect of the MER distributed operationseffort was coercing centralized software into functioningproperly in a distributed environment. The performanceof the system was adversely affected by manycompromises made to meet the requirements of theoriginal software, and potential robustness wassacrificed due to the need of ad-hoc solutions.

2) Avoid polling by using a common event notificationsystem. The largest obstacle to good performance inour system was the lack of a common event notificationsystem. Almost all processes were forced to rely onpolling, which was inefficient, high in latency, andconsumed enormous system resources. It was also asource of unnecessary complexity. A common eventnotification system would solve this problem bymaking processes aware of new or newly changedresources. The File Exchange Interface includes asubset of this desired functionality. This is the mostsignificant feature that the current architecture lacks.

3) Distributed planning software should operate in ahigh-concurrency environment. Many parts of theplanning process are by nature collaborative, as is thescientific process itself. Plan sharing in DistributedSAP was problematic because it was not designed to behighly concurrent. Systems should be designed toallow multiple users to edit a resource simultaneously,or prevent resource conflicts. This might be bestaccomplished using notions of resource ownership, or a“copy on write” model. Operations should betransactional and reversible where possible.

4) Minimize latency effects in collaborativefunctionality. The latency involved in the sharedplanning subsystem, while low, is still high enough todiscourage its use in the interactive part of a planningsession. It is important to use techniques, such asoptimistic locking in order to reduce the inevitableeffects of latency in this kind of global system.

5) Utilize decentralized architectures for informationpropagation where possible. As the volume ofmission information increases, it will become morecritical to use a less centralized method for distributingstatic information (data products). We suggest the useof swarming downloads (as demonstrated by theBitTorrent protocol) or localized synchronization ofcaches (swarm caching). The ability for operations

8

software to obtain data from a localized peer shouldincrease effective throughput significantly, and alsolower latency. The network topology in this operationsmodel consists of small, interconnected “clusters,”which is ideal for swarming transfers.

6) Use SQL databases where possible. SQL databasesare an industry standard, and they have successfullysolved many problems relating to concurrency, locking,transactions, data propagation, and querying. The useof SQL databases for targets was critical for the successof Distributed SAP. To aid in the use of SQL, objectoriented software can use what is known as an Object-Relational Bridge (ORB), a piece of software thatconverts object graphs in memory to a databaserepresentation, and back. Distributed SAP uses CastorJava Data Objects to perform this function.

7) Use XML or ASCII Text for metadata. Lack ofintegration with specialized science analysis andplanning software lead to extra work and reducedcapabilities for MER scientists. The use ofplatform/language neutral standard formats wouldgreatly increase the interoperability of mission software.

8. CONCLUSION

The Distributed Science Activity Planner has contributed tothe success of distributed operation for the Mars ExplorationRover mission. Scientists were able to analyze data andplan from their home institution, and collaborate with otherscientists around the world. The distributed operationsarchitecture has enabled a large science team to operateSpirit and Opportunity well beyond the original missionlifetime as they continue to return valuable scientificinformation to earth. Distributed MER operations will serveas a model for missions into the future.

REFERENCES

[1] Robert Steinke, Paul G. Backes, and Jeffrey S.Norris. Distributed mission operations with the multi-mission encrypted communication system. In ProceedingsIEEE Aerospace Conference, Big Sky, Montana, March 9-16 2002.

[2] Paul G. Backes , Jeffrey S. Norris , Mark W.Powell , Marsette A. Vona , Robert Steinke , and JustinWick. The Science Activity Planner for the MarsExploration Rover Mission: FIDO Field Test Results. InProceedings IEEE Aerospace Conference, Big Sky,Montana, March 8-15 2003.

[3] Personal correspondence with Professor SteveSquyres, December 8th, 2004.

BIOGRAPHY

Justin Wick worked as a member of theoperations staff of the Mars ExplorationRover mission at the Jet PropulsionLaboratory during 2004. He designedand implemented the DistributedScience Activity Planner system toenable collaboration between scientistsat remote locations. Prior to the missionJustin spent four years pursuing abachelors of science from Cornell

University in Applied and Engineering Physics. While notworking on MER, Justin parallelized severalMagnetohydrodynamic simulation codes for theoreticalastrophysics, created a kinematics simulation system for theCornell RoboCup robotic soccer team, and lead thecomputer science team on the Cornell Snake Arm roboticsproject.

Justin currently lives in Ithaca, NY, working for MaasDigital, creating custom computer graphics algorithms forhigh definition IMAX films. He plans to pursue a career incomputational physics.

Dr. John L. Callas received hisBachelor's degree in Engineering fromTufts University in 1981 and hisMasters and Ph.D. in Physics fromBrown University in 1983 and 1987,respectively. After completing hisdoctorate in elementary particle physics

in 1987, he joined the Jet Propulsion Laboratory inPasadena, California to work on advanced spacecraftpropulsion, which included such futuristic concepts aselectric, nuclear and antimatter propulsion.

In 1989 he began work supporting the exploration of Marswith the Mars Observer mission and has since worked onseven Mars missions. In 2000, Dr. Callas was asked tojoin the Mars Exploration Rover (MER) Project as theScience Manager. Dr. Callas continues as the ScienceManager for the highly successful Spirit and Opportunityrovers. Recently, Dr. Callas has begun serving as aMission Manager on MER, as an additional duty, as therovers continue their great success on the surface of Mars. In addition to his Mars work, Dr. Callas is involved in thedevelopment of instrumentation for astrophysics andplanetary science, and teaches mathematics at Pasadena CityCollege as an adjunct faculty member. In his spare time, hementors students interested in science and works withschools classrooms on science projects.

9

Jeffrey S. Norris is a computerscientist and member of the technicalstaff of the Mobility Systems ConceptDevelopment section at the JetPropulsion Laboratory. At JPL, hiswork is focused in the areas ofdistributed operations for Mars roversand landers, secure data distribution,and science data visualization.

Currently, he is a software engineer on the MarsExploration Rover ground data systems and missionoperation systems teams. Jeff received his Bachelor’s andMaster’s degrees in Electrical Engineering and ComputerScience from MIT. While an undergraduate, he worked atthe MIT Media Laboratory on data visualization andmedia transport protocols. He completed his Master’sthesis on face detection and recognition at the MITArtificial Intelligence Laboratory. He lives with his wife,Kamala, in La Crescenta, California.

Mark W. Powell has been amember of the technical staff in theMobility Systems ConceptDevelopment section at the JetPropulsion Laboratory, Pasadena,CA, since 2001. He received hisB.S.C.S. in 1992, M.S.C.S in1997, and Ph.D. in ComputerScience and Engineering in 2000from the University of South

Florida, Tampa. His dissertation work was in the area ofadvanced illumination modeling, color and range imageprocessing applied to robotics and medical imaging. AtJPL his area of focus is science data visualization andscience planning for telerobotics. He is currently servingas a software and systems engineer, contributing to thedevelopment of science planning software for the 2003Mars Exploration Rover mission and the JPL MarsTechnology Program Field Integrated Design andOperations (FIDO) rover task. He, his wife Nina, anddaughters Gwendolyn and Jacquelyn live in Tujunga, CA.

Marsette A. Vona, III is currentlya PhD candidate in ComputerScience at Massachusetts Instituteof Technology. He is a formermember of the technical staff of theMobility Systems ConceptDevelopment Section at the JetPropulsion Laboratory. At JPL, hiswork was focused in the areas ofhigh-performance interactive 3D

data visualization for planetary exploration, user-interfacedesign for science data analysis software, and Javasoftware architecture for large, resource-intensiveapplications. Marsette received a B.A. in 1999 fromDartmouth College in Computer Science andEngineering, where he developed new types of hardwareand algorithms for self-reconfigurable modular robots. Hecompleted his M.S. in 2001 at MIT’s Precision MotionControl lab, where he developed a high-resolution

interferometric metrology system for a new type ofrobotic grinding machine. Marsette was awarded theComputing Research Association OutstandingUndergraduate Award in 1999 for his research in Self-Reconfigurable Robotics.

ACKNOWLEDGEMENTS

The research described in this paper was carried out at theJet Propulsion Laboratory, California Institute ofTechnology, under a contract with the NationalAeronautics and Space Administration. We would like tothank Professor Steve Sqyures, Mary Mulvanerton,Daniel Crotty, Rupert Scammell, and Adam Wick fortheir generous assistance in the preparation of this paper.

10

Figu

re 2

– U

plin

k B

row

ser.

The

Upl

ink

Bro

wse

r al

low

s us

ers

to e

dit

plan

s.

Tw

o pl

ans

are

curr

ently

ope

n, a

long

wit

h a

reso

urce

ana

lysi

s gr

aph.

T

hede

tails

dia

log

box

cont

ains

add

ition

al s

etti

ngs

for

a pl

anne

d pa

nora

mic

cam

era

obse

rvat

ion.

T

he u

pper

left

con

tain

s a

file

tree

of

avai

labl

e pl

an f

iles.

T

helo

wer

left

con

tain

s a

list

of

avai

labl

e ac

tivity

type

s.

11

Figu

re 3

– D

ownl

ink

Bro

wse

r. T

he D

ownl

ink

Bro

wse

r al

low

s us

ers

to v

iew

im

ages

and

cre

ate

targ

ets.

In

thi

s fi

gure

, an

equ

icyl

indr

ical

pro

ject

ion

of a

navi

gatio

nal c

amer

a m

osai

c is

bei

ng c

onst

rast

str

etch

ed.

Red

cir

cles

ind

icat

e ta

rget

s, a

nd t

he y

ello

w t

rape

zoid

is

the

pred

icte

d “f

ootp

rint

” of

a p

lann

edpa

nora

mic

cam

era

obse

rvat

ion.

T

he y

ello

w r

ecta

ngle

is a

mea

suri

ng to

ol.