Knowledge-Driven Agile Sensor-Mission Assignment

ACITA 2009

, Konrad Borowiecki ,

Geeth de Mel , Wamberto Vasconcelos,

Amotz Bar-Noy , Matthew P. Johnson ,

Tom La Porta , Hosam Rowaihy

Knowledge-Driven Agile Sensor-Mission Assignment

Alun Preece, Diego Pizzocaro

http://users.cs.cf.ac.uk/D.Pizzocaro [email protected]

mailto:[email protected]


Extensions

Outline

Previous work

Sensor-Mission Assignment problem

Implementation

1. Task Representation

Ontology-based matching

Conclusion & Future


2. Asset Allocation



Main problem

Sensor-Mission Assignment

is the problem of assigning sensing assets to missions to cover the information needs (ISR)

of individual tasks in each mission.




Sensors (or Sensing assets)

Simple sensors

Platforms




Sensors (or Sensing assets) Missions

Simple sensors composed by different TASKS:

e.g. Peace Support mission

TASK 1

Detectvehicle

TASK 4

Area Surveillance

TASK 3

Area Surveillance

TASK 2

Identifypeople

Platforms




Scenario




Scenario

• A network of heterogeneous sensing assets:




Scenario


- Support multiple tasks competing for bundles of sensors

TASK 7

LocalizeJeep


TASK 3

DetectPeople

TASK 5

DetectHelicopter

TASK 1

Identifypeople

TASK 6

IdentifyTank

TASK 4

DetectAircraft

TASK 2

DetectGround Vehicle



Scenario



TASK 7

LocalizeJeep


TASK 3

DetectPeople

TASK 5

DetectHelicopter

TASK 1

Identifypeople

TASK 6

IdentifyTank

TASK 4

DetectAircraft

- Sensors are scarce and in high demand.

TASK 2




Scenario



TASK 7

LocalizeJeep


TASK 3

DetectPeople

TASK 5

DetectHelicopter

TASK 1

Identifypeople

TASK 6

IdentifyTank

TASK 4

DetectAircraft


- Highly dynamic (sensor failures, change of plan)

TASK 2




Scenario




TASK 1

Identifypeople


- Highly dynamic (sensor failures, change of plan)

Where to send that particular bundle?

TASK 2





Problem formulation

• We need schemes to assign bundles of assets to

the task they best serve.demonstrates how the core knowledge-based approach can beused to drive asset allocation, using the CASS combinatorialauction algorithm as an illustration. In Section VI we reviewthe status of our illustration-of-concept application, SAM(Sensor Assignment to Missions), and discuss a variety ofroles this kind of tool can play in supporting the process ofsensor-mission assignment.

While other papers have presented earlier or incompleteparts of our knowledge-driven approach to sensor-missionassignment (notably [6], [8], [9]) and resource allocation(notably [10], [7]) this is the first paper to show how theseelements can provide an integrated solution.

II. SENSOR-MISSION ASSIGNMENT PROBLEMFORMULATION

We formulate the sensor-mission assignment problem as agraph; an example is shown in Figure 1. We distinguish threekinds of node:

• Tasks denote the entities that are competing for availableassets.1 In our approach, the only important feature ofa task is its information requirements — these are whatdrive the asset matching and allocation processes — sothe task nodes represent information requirements.

• Bundles are collections of individual assets (sensors andplatforms). An arc between a bundle and a task indicatesthat the bundle is able to meet (at least to some extent)that information requirement (task). Each bundle has aunique bundle type, described in more detail below. Eachbundle is composed of several assets and, depending onits type, may be suitable for several tasks.

• Assets represent the individual sensor and platform re-sources. An arc between an asset and a bundle meansthat that asset can be assigned to that bundle. Assetsmay be suitable for assignment to several bundles, againdepending on the bundle type. Although not shown in thefigure, an asset may be assigned to bundles of differenttypes.

In these terms, a solution to the sensor-mission assignmentproblem is an assignment of bundles to tasks, subject to theconstraints: each task can have at most one bundle assignedto it, a bundle may be assigned to at most one task, each assetmay be assigned to at most one bundle. Thus, an assignmentis a matching in the sub bipartite graph (B, T ) and is a semimatching in the sub bipartite graph (A, B). Note that the thinarcs in Figure 1 show possible assignments, while the bold arcsshow one actual assignment. The figure is drawn with tasks onthe left and assets on the right to emphasise that, although theassignment arcs go right to left, solutions to the assignmentproblem are constructed left to right, that is, starting with tasks.

To illustrate this with a concrete example (originallyfrom [5]): in a peace support operation [4], UK and USbases have been established to detect/deter insurgent activ-ity on a border. The bases rely on a main supply route

1In much of our work, we consider these to be synonymous with “missions”(e.g. [8], [7]).

T1

T2

Tasks Bundles Assets

A1

A5

A6

A4

A2

A3

B1

B4

B2

B3

B5

BT1

BT2

Fig. 1. Example sensor-mission assignment problem as a graph

(MSR) which must be surveilled and protected. Surveillingthe border will likely involve, among other things, detectionof suspicious vehicle activity near it: vehicle detection canbe formalized as an information requirement task T1. Thismay be accomplished by a variety of means, depending onthe kinds of assets available. We assume these include flyinga UAV over the border to gather IMINT, or using acousticsensing to identify types of vehicles likely to be used byinsurgents. Further, we assume that a single UAV can coverthe area of interest (AoI), but that at least two acousticarrays will be needed. Each of these options is representedas a type of bundle: BT1 = !{UAV, IMINT-sensor}", BT2 =!{AcousticArray}, {AcousticArray}". Assume there are multipleUAV assets available (A1 and A3) but only one functioningIMINT payload (A2) that can be mounted on either A1 orA3. Assuming there are three acoustic arrays deployed in thearea (A4, A5, A6), then we can meet the requirements of T1

by assigning UAV-IMINT bundles {A1, A2} or {A3, A2} orone of the pairs of acoustic arrays (e.g. {A4, A5}, {A4, A6},{A5, A6}).

However, it is likely that there will be other tasks, potentiallyin competition for these assets; for example, a requirementto detect vehicles posing a potential threat to the MSR (T2)may be accomplished by the same types of bundle as T1

but, because the areas of interest (MSR and border) do notintersect, it will not be possible to share assets between thesetasks. So, for example, if we assign the UAV to T1 then we willhave to satisfy T2 by means of acoustic intelligence (ACINT).

We argue that, for a specific problem instance, constructionof the task-bundle-asset graph requires a significant amountof domain knowledge. Specifically, need to know which typesof bundle are suitable for which tasks, and which types ofasset can be collected into bundles. In our original work [6] —summarised in the next section — this knowledge is expressedin the form of a set of domain ontologies, allowing us toapply a reasoning procedure to derive the graph for a given

• Tasks: the entities competing for assets.

Each task is associated with Information Requirements.

• Bundles: collections of assets.

Each bundle has a unique Bundle Type (BT).

• Assets: individual sensors and platforms.




Problem formulation

• We need schemes to assign bundles of assets to

the task they best serve.demonstrates how the core knowledge-based approach can beused to drive asset allocation, using the CASS combinatorialauction algorithm as an illustration. In Section VI we reviewthe status of our illustration-of-concept application, SAM(Sensor Assignment to Missions), and discuss a variety ofroles this kind of tool can play in supporting the process ofsensor-mission assignment.

While other papers have presented earlier or incompleteparts of our knowledge-driven approach to sensor-missionassignment (notably [6], [8], [9]) and resource allocation(notably [10], [7]) this is the first paper to show how theseelements can provide an integrated solution.

II. SENSOR-MISSION ASSIGNMENT PROBLEMFORMULATION

We formulate the sensor-mission assignment problem as agraph; an example is shown in Figure 1. We distinguish threekinds of node:

• Tasks denote the entities that are competing for availableassets.1 In our approach, the only important feature ofa task is its information requirements — these are whatdrive the asset matching and allocation processes — sothe task nodes represent information requirements.

• Bundles are collections of individual assets (sensors andplatforms). An arc between a bundle and a task indicatesthat the bundle is able to meet (at least to some extent)that information requirement (task). Each bundle has aunique bundle type, described in more detail below. Eachbundle is composed of several assets and, depending onits type, may be suitable for several tasks.

• Assets represent the individual sensor and platform re-sources. An arc between an asset and a bundle meansthat that asset can be assigned to that bundle. Assetsmay be suitable for assignment to several bundles, againdepending on the bundle type. Although not shown in thefigure, an asset may be assigned to bundles of differenttypes.

In these terms, a solution to the sensor-mission assignmentproblem is an assignment of bundles to tasks, subject to theconstraints: each task can have at most one bundle assignedto it, a bundle may be assigned to at most one task, each assetmay be assigned to at most one bundle. Thus, an assignmentis a matching in the sub bipartite graph (B, T ) and is a semimatching in the sub bipartite graph (A, B). Note that the thinarcs in Figure 1 show possible assignments, while the bold arcsshow one actual assignment. The figure is drawn with tasks onthe left and assets on the right to emphasise that, although theassignment arcs go right to left, solutions to the assignmentproblem are constructed left to right, that is, starting with tasks.

To illustrate this with a concrete example (originallyfrom [5]): in a peace support operation [4], UK and USbases have been established to detect/deter insurgent activ-ity on a border. The bases rely on a main supply route

1In much of our work, we consider these to be synonymous with “missions”(e.g. [8], [7]).

T1

T2

Tasks Bundles Assets

A1

A5

A6

A4

A2

A3

B1

B4

B2

B3

B5

BT1

BT2

Fig. 1. Example sensor-mission assignment problem as a graph

(MSR) which must be surveilled and protected. Surveillingthe border will likely involve, among other things, detectionof suspicious vehicle activity near it: vehicle detection canbe formalized as an information requirement task T1. Thismay be accomplished by a variety of means, depending onthe kinds of assets available. We assume these include flyinga UAV over the border to gather IMINT, or using acousticsensing to identify types of vehicles likely to be used byinsurgents. Further, we assume that a single UAV can coverthe area of interest (AoI), but that at least two acousticarrays will be needed. Each of these options is representedas a type of bundle: BT1 = !{UAV, IMINT-sensor}", BT2 =!{AcousticArray}, {AcousticArray}". Assume there are multipleUAV assets available (A1 and A3) but only one functioningIMINT payload (A2) that can be mounted on either A1 orA3. Assuming there are three acoustic arrays deployed in thearea (A4, A5, A6), then we can meet the requirements of T1

by assigning UAV-IMINT bundles {A1, A2} or {A3, A2} orone of the pairs of acoustic arrays (e.g. {A4, A5}, {A4, A6},{A5, A6}).

However, it is likely that there will be other tasks, potentiallyin competition for these assets; for example, a requirementto detect vehicles posing a potential threat to the MSR (T2)may be accomplished by the same types of bundle as T1

but, because the areas of interest (MSR and border) do notintersect, it will not be possible to share assets between thesetasks. So, for example, if we assign the UAV to T1 then we willhave to satisfy T2 by means of acoustic intelligence (ACINT).

We argue that, for a specific problem instance, constructionof the task-bundle-asset graph requires a significant amountof domain knowledge. Specifically, need to know which typesof bundle are suitable for which tasks, and which types ofasset can be collected into bundles. In our original work [6] —summarised in the next section — this knowledge is expressedin the form of a set of domain ontologies, allowing us toapply a reasoning procedure to derive the graph for a given

• Tasks: the entities competing for assets.

Each task is associated with Information Requirements.

• Bundles: collections of assets.

Each bundle has a unique Bundle Type (BT).

• Assets: individual sensors and platforms.

• GOAL: an assignment of bundles to tasks, such that

1. One-to-One matching between tasks-bundles

2. One-to-Many matching between bundles-assets




Example

• Mission based on an ITA scenario: Peace support operation

TASK BUNDLE

BT1 = {(UAV, DaylightTV)}

BT2 = {(AcousticArray), (AcousticArray)}

SENSOR

Predator (UAV)

DaylightTV

Acoustic array 1

Acoustic array 2

Reaper (UAV)

!"#$%&'(&)*+,-&"./(,%&.0&"1#+)2,11,#"./11,("3&"4./5.6+&&7&89.*5.6,::#7;+#89.!5.<#+#$,&7=,89.>5.'&.2&%[email protected];17#"7&%#1?.9..

/5! <;+)B#CD.9.25.65.E#F"1#"D.9.G5.H;.6#+4;I.9.J5.K#$;,FCI..

8L;+',M.N",-&+1,4C9.N!O.7#"4;74.&3;,%P.;5'5Q+&&7&R7157;+',M5;75S=O.?N",-&+1,4C.#T./U&+'&&"9.N!..

DL,4C.N",-&+1,4C.#T.B&$.V#+=9.N0/O.I6&""1C%-;",;.04;4&.N",-&+1,4C9.N0/.

Knowledge representation and reasoning techniques can support sensor-mission assignment, proceeding from a high-level

specification of information requirements, to the allocation of assets such as sensors and platforms. In our previous work, we showed

how assets can be matched to mission tasks by formalising the military missions and means framework in terms of an ontology, and using this ontology to drive a matchmaking procdure. The work reported here extends the earlier approach in two important ways: (1)

by providing a richer and more realistic way for a user to specify their information requirements, and (2) by using the results of the

semantic matchmaking process to define the search space for efficient asset allocation algorithms.

The Task-Bundle-Asset model defines the search space relating what bundles of assets are required for different types of task.

Higher-level task representations allow

knowledge base to offer widest range of

ISTAR solution types."

Task: detect vehicles

Bundle 1: UAV

+IMINT

Bundle 2: acoustic

motes

Tasks denote the entities that are competing for

available assets. In our approach, the only important

feature of a task is its information requirements."

Bundles are collections of individual assets (sensors

and platforms). An arc between a bundle and a task

indicates that the bundle is able to meet (at least to

some extent) that information requirement (task). Each

bundle has a unique bundle type. "

Assets represent the individual sensor and platform

resources. An arc between an asset and a bundle

means that that asset can be assigned to that bundle."

A solution to the sensor-mission assignment problem

is an assignment of bundles to tasks, subject to the

constraints: each task can have at most one bundle

assigned to it, a bundle may be assigned to at most

one task, each asset may be assigned to at most one

bundle. !

Note that the thin arcs in the example show possible

assignments, while the bold arcs show one actual

assignment."

Hybrid ontology+rules-based reasoning:

Rule-based representation of (extended) NIIRS scale allows tasks of form:

detect/identify/distinguish <set of detectables>

Proves extensibility of our original ontology-based approach.

How it works: NIIRS-based rules infer basic capabilities (IMINT, ACINT, etc) needed for high-level tasks. We then apply matchmaking using Missions & Means Framework as before, to give us bundle types.

Each bundle type is an intensional definition of a set of bundles of assets that can

satisfy a task (to some extent). We generate candidate bundles, and find an

allocation using the CASS combinatorial auction algorithm.

The assignment problem can be formulated as an auction in which the bidders

are tasks T1,...,Tm and the items are assets A1,...,An. Each task is associated with

a utility demand dj, indicating the amount of sensing resources needed, and a

profit pj, indicating the importance of the task.

Each task places a bid, equal to its own profit, for any set of assets which satisfy

the task's demand and respect the task's budget constraint. We compute the

value that bidder Tj obtains if it receives the bundle B of assets using:

We are now able to demonstrate

full integration of hybrid reasoning

with optimization problem-solving, in our prototype SAM tool!

Where:

uj is utility of B to Tj

wj is cost of B to Tj

T is satisfaction threshold

! UK and US bases established to surveil the border

! Bases rely on a Main Supply Route (MSR)




Example


TASK BUNDLE



SENSOR

Predator (UAV)

DaylightTV

Acoustic array 1

Acoustic array 2

Reaper (UAV)

TASK 1

Detectvehicle

!"#$%&'(&)*+,-&"./(,%&.0&"1#+)2,11,#"./11,("3&"4./5.6+&&7&89.*5.6,::#7;+#89.!5.<#+#$,&7=,89.>5.'&.2&%[email protected];17#"7&%#1?.9..

/5! <;+)B#CD.9.25.65.E#F"1#"D.9.G5.H;.6#+4;I.9.J5.K#$;,FCI..













Bundle 1: UAV

+IMINT

Bundle 2: acoustic

motes

















bundle. !



assignment."



















Where:









Example


TASK BUNDLE



SENSOR

Predator (UAV)

DaylightTV

Acoustic array 1

Acoustic array 2

Reaper (UAV)

TASK 1

Detectvehicle

!"#$%&'(&)*+,-&"./(,%&.0&"1#+)2,11,#"./11,("3&"4./5.6+&&7&89.*5.6,::#7;+#89.!5.<#+#$,&7=,89.>5.'&.2&%[email protected];17#"7&%#1?.9..

/5! <;+)B#CD.9.25.65.E#F"1#"D.9.G5.H;.6#+4;I.9.J5.K#$;,FCI..













Bundle 1: UAV

+IMINT

Bundle 2: acoustic

motes

















bundle. !



assignment."



















Where:




B1

B2

B3





Ontology-basedmatching

matches types of assets to types of tasks,based on sensing capabilities provided and required.


Previous Work

TASK

AerialImageryIntelligence

SENSING ASSET TYPES




Methodology


• Semantic matchmaking to evaluate the fitness-for-purpose of collections of asset

types to the information requirements of a task.




Methodology




• We use ontologies for:




Methodology





Specify the information

requirements of a task.

Specify the sensing capabilities

provided by assets.

ISR ontologies: tasks, sensors,

etc




Methodology





Specify the information

requirements of a task.

Specify the sensing capabilities

provided by assets.

Compare the two of them.

ISR ontologies: tasks, sensors,

etc

MMF ontology




MMF ontology

• Based on the pre-existent Missions and Means Framework (MMF)

! MMF is an informal framework to help human planners determine the

capabilities required to accomplish a military mission. (developed by US ARL)

! In the military domain very precise definitions of capabilities required:

! ISR requirements

Intelligence

Surveillance Reconnaisance




MMF ontology (contd)

• Main concepts and relations in MMF ontology (implemented in OWL DL):

Task Capability

Operation

Mission

Asset

Platform System

Sensor

comprises toAccomplish

comprises toAccomplish

toPerform

is-a

is-a

is-a

mounts

attachedTo

requires

providesallocatedTo

interferesWith

entails





• Given a task T, with a set of ISR requirements: RT = {R1, R2, ...}

• Product of matching is a (BT) Bundle Type:

bundle of assets that can satisfy the requirements of a task.








• Steps to build a BT:

1. generate a (PC) Platform Configuration = (P , S)

where P is a single platform type ,

S = {S1, ... , Sm} is a set of sensor types mountable on P (at the same time).












2. generate a (PK) Package Configuration = { PC1 , PC2 , ... }

The aggregate capabilities of PK have to semantically match RT












2. generate a (PK) Package Configuration = { PC1 , PC2 , ... }

The aggregate capabilities of PK have to semantically match RT

3. Create BT by adding cardinality constraints to each PC in the PK:

e.g. : “2 UAVs with a DaylightTV each” ! BT1 = { (UAV, DaylightTV), (UAV, DaylightTV)}




Limitation

• This approach is conceptually simple, extensible and well founded (MMF)

• Limitation:

Task requirements have to be specified at a low level

! Over-reliant on the ontology of INT types

E.g. RT = { ACINT, IMINT }

trivially determines the need

for acoustic and imagery sensing.

• Need for higher-level representation of ISR tasks.

Fig. 3. Sample concept taxonomies relevant to the ISR domain: ISR tasksand INT types

their properties (e.g. [14], [15], [16]). There are also severalwell-known structured descriptions of tasks in the militarymissions context, most notably the Universal Joint Task List(UJTL)[17]. Based largely on these pre-existing sensor andtask ontologies, we have identified a collection of concepthierarchies relevant to the ISR domain. Portions of two ofthese (defining ISR tasks and various types of intelligenceinformation — “INT” types) are shown in Figure 3.

In these terms, we define a bundle type to be an intensionaldefinition of a set of bundles of assets that can satisfy atask. The essential part of a bundle type is the specificationof required sensor and platform types needed to providethe capabilities to satisfy the task. For this, we define aplatform configuration ! = !P,S", where P is a type ofplatform, and S = {S1, . . . , Sm} is a set of sensor typesthat can be mounted on P simultaneously. Given a task Twith a set of required ISR capabilities CT = {C1, . . . , Cn},a single platform configuration is a valid match for T ifthe combined capabilities of P and S satisfy CT . Formally,V(T ) = {!1, . . . ,!n} is the set of valid matches for task Tiff !P,S" # V(T ) $ (%Ci # CT )((P & 'provides.Ci)((Si #S & 'provides.Ci)).

More commonly, the requirements of a task will not besatisfied by a single platform configuration. We define a pack-age configuration ! = {!1, . . . ,!n}, where !i is a platformconfiguration. ! is a valid match for T if collectively theplatforms and sensors in {!1, . . . ,!n} satisfy the capabilitiesin CT , and ! is minimal with respect to CT (there does notexist any subset of ! that is a valid match for T ).

Bundle types are created by post-processing the packageconfigurations, to add cardinality constraints. This is currentlydone using pre-defined configuration knowledge, for example:

• at least 1 UAV with at least 1 Camera• at least 2 AcousticArrays with exactly 2 ACINTSensorsWhile this approach was conceptually simple, extensible (in

terms of the modularity of the domain-specific ontologies),

and well-founded (on MMF), it was rather limited in that itrequired task capabilities to be specified at a low level in orderto drive the matching procedure. Specifically, the approach wasover-reliant on the ontology of INT types shown in Figure 3 todetermine the type of sensor required; for example, a choiceof ACINT or IMINT in CT would determine — rather trivially— the need for acoustic or imagery sensing). We sought ahigher-level representation of ISR tasks, as described in thenext section.

IV. SPECIFIYING TASKS

Our main requirements for a higher-level representation oftasks were:

• That the task should as much as possible specify onlywhat is the information requirement, and avoid sayinganything about how it should be obtained. The reasonsfor this requirement were to avoid the main flaw in ourearlier approach, and allow a greater degree of flexibilityin allocating assets to tasks.

• That the representation of tasks should be familiar topotential users of the approach, not only to make any toolbased on the approach easier to use, but also to maximisethe acceptability of our approach (the same reason thatwe adopted MMF as the underlying framework for thetask-asset matching).

Following guidance from subject-matter experts, we based ourrepresentation on the National Imagery Interpretability RatingScale (NIIRS) for various kinds of imagery intelligence[18].NIIRS is a well-established way to determine the kinds of data(for example, visible imaging, or radar) that are interpretableto answer particular information requirements (detecting, iden-tifying and distinguishing various kinds of thing). The NIIRSscale defines ratings on a ten point scale (0–9) for variouskinds of imagery data; for example, detection of vehicles ofparticular types is achievable by Visible NIIRS 4 and RadarNIIRS 6. This is useful because sensor/platform combinationscan then be rated in terms of their NIIRS values, allowinganalysts to seek data that are suitable for their tasks. From ourpoint of view, NIIRS supports the kind of task-asset matchingwe established as described in the previous section.

We define a task as a quadruple, !NC, DS,A, T ", where:• NC is one of three basic “capabilities” defined in NIIRS:

detect (find or discover the presence or existence of anentity), distinguish-between (determine that two detectedentities are of different types), and identify (name anobject by type);

• DS is a set of “detectable” types of entity (for example,people, vehicles, or installations at various levels ofspecificity);

• A defines an area-of-interest;• T defines a period of time.

It is worth noting that, in accordance with our earlier work,we continue to refer to this representation as a “task” repre-sentation. In NIIRS, and in ISR more generally, these kindsof description are referred to as “capabilities”. In a sense,



Task Representation


Extension 1

specify only WHAT is the information requirement,

and avoid saying HOW it should be obtained.

T = {ACINT, IMINT} T = {“Detect Vechicle”}




• Approach based on National Imagery Interpretability Rating Scale (NIIRS)

! determines the kinds of data that are interpretable

to answer an information requirement:

E.g. Visible, Radar, etc.

! defines ratings on a ten point scale (0–9)

NIIRS

T= {“Detect Vehicles”}

Visible NIIRS 4




• Approach based on National Imagery Interpretability Rating Scale (NIIRS)

! determines the kinds of data that are interpretable

to answer an information requirement:

E.g. Visible, Radar, etc.

! defines ratings on a ten point scale (0–9)

NIIRS

T= {“Detect Vehicles”}

Visible NIIRS 4

• Platform Configurations can be rated with NIIRS values

“UAV with DaylightTV-camera” ↔ Visible NIIRS 4

• Therefore “UAV with DaylightTVcamera” matches “Detect Vehicles”

! NIIRS can support the task-asset matching




Hybrid reasoning

• To integrate NIIRS in sensor-task matching:

! We formalized it as a set of rules which allow for:

We adopt an HYBRID REASONING approach: ontology & rule-based reasoning

! Reasoning steps:

1) Specify high-level task:

2) NIIRS rule-based system infers basic capabilities:

3) Ontology-based matchmaking returns the bundle types:

Task specification

Detect/Identify/Distinguish <set of detectables>




Hybrid reasoning




! Reasoning steps:




as our approach makes clear, they can be regarded as high-level capabilities, from which lower-level capabilities can beinferred. However, we prefer to locate these representations inour ontology as specialisations of the Task class rather thanthe Capability class to avoid confusion. Note also that, in thecurrent implementation, we omit T as we assume that all tasksare required to happen in the same mission period.

!"#$%&'($)*+,-./0'&*+1

!0#$20&+$)*+3(4'1$*5$6('(3'0"7(1 !3#$8&+(1$*5$9/0+14*/'0'&*+

Fig. 4. A portion of our ontology of “detectable” types of entity from NIIRS

Full details of the representation of the NIIRS scale, and ourrules for performing reasoning with it, are given in [9]. Thevarious “detectable” types of entity are defined in an ontology,drawn from the entities appearing in the military version ofthe NIIRS documentation. A portion of this ontology is shownin Figure 4. Using this ontology, we formalised the variousNIIRS interpretation tasks as a set of clauses of the followingform: !NC, DS, FS, C,NT,NR", where:

• NC is a NIIRS “capability” as above (one of: detect,distinguish-between, identify);

• DS is a set of detectables, as above;• FS is a set of more specific features of the detectable

entities (for example, the roads or guard posts of a base,the runways of an airport, or the piers and warehouses ofa port) — these features are also defined in the ontologyof “detectables”;

• C is a context, defining the preconditions that must holdfor the clause to apply (for example, detection of a shipin the context of open water, or identification of vehiclesin a known motor pool); and

• NT is the type of imaging from the NIIRS scale (forexample, Visible or Infrared); and

• NR is a NIIRS rating (for example, Visible-6 or Infrared-4).

Some examples:• !detect, {Port}, {}, {}, Visible, Visible-1": a port can be

detected using visible imagery of rating 1;• !detect, {Port}, {Pier, Warehouse}, {}, Radar, Radar-1":

a port can be detected based on the presence of piersand warehouses using radar imagery of rating 1;

• !distinguish-between, {Taxiway, Runway}, {}, {Airfield},Visible, Visible-1": taxiways and runways can bedistinguished at an airfield using visible imagery ofrating 1;

• !detect, {Vehicle}, {}, {Motor-Pool}, Radar, Radar-4": ve-hicles can be detected in a known motor pool using radarimagery of rating 4;

The basic reasoning procedure for using these clauses is asfollows (there are additional features described in [9]:

1) Given a task T = !NC, DS,A, T ", find all clauses!NC !, DS!, FS,C, NT, NR" where NC = NC ! andDS # DS! or DS # FS;

2) For each clause where C is not empty, check that all theconditions in C hold, normally by referencing mappinginformation for area A, and if they do not hold, discardthe clause;

3) Gather all of the NT and NR elements from the re-maining clauses to form the set of required capabilities,CT ;

4) Because higher ratings of a given NIIRS type entaillower ones, we reduce CT so that it contains only thehighest rating of each NIIRS type.

We then use our previous approach to matching, describedin Section III, with two important modifications:

• We have added the relevant NIIRS capabilities to thesensor and platform types. Because of the way NIIRS isdefined, we associate the NIIRS imaging types NT withsensors (for example, Visible or Radar) and the NIIRSratings NR with platforms (for example, Visible-1 orRadar-4). Thus, an inferred platform configuration mustsatisfy both the required imaging type and the rating.

• Because NIIRS gives us alternative ways to satisfy a task,each different NR and its corresponding NT is matcheddisjunctively; so, for example, if CT contains Visible-1and Radar-4, we generate separate sets of valid matchesfor each, and then aggregate them into a complete set ofvalid matches.

An important limitation of this approach is the publishedNIIRS scale is restricted to imagery intelligence. However,in principle, there is no reason why it cannot be extendedto cover other types of intelligence also. For our purposes,we have extended our knowledge base where we know ofnon-imagery approaches that have been shown - at leastin controlled trials - to have particular capabilities. For ex-ample, following [19] we introduced clauses that indicateacoustic data can be used to detect vehicles (note thatwe leave the rating essentially unspecified by using zero):!detect, {Vehicle}, {}, {}, Acoustic, Acoustic-0"

In this section, we have shown how we extended our originalmatching approach to support higher-level and more realistictask representations. In the next section, we address the issueof asset allocation based on the output from the matchingprocess.

Task specification





Hybrid reasoning




! Reasoning steps:



!"#$%&'($)*+,-./0'&*+1

!0#$20&+$)*+3(4'1$*5$6('(3'0"7(1 !3#$8&+(1$*5$9/0+14*/'0'&*+
























Task specification


T = {“Detect Vehicles”}




Hybrid reasoning




! Reasoning steps:




!"#$%&'($)*+,-./0'&*+1

!0#$20&+$)*+3(4'1$*5$6('(3'0"7(1 !3#$8&+(1$*5$9/0+14*/'0'&*+
























Task specification



RT = {ACINT-level0, IMINT-level4}




Hybrid reasoning




! Reasoning steps:





!"#$%&'($)*+,-./0'&*+1

!0#$20&+$)*+3(4'1$*5$6('(3'0"7(1 !3#$8&+(1$*5$9/0+14*/'0'&*+
























Task specification



RT = {ACINT-level0, IMINT-level4}

BT1 = {UAV, DaylightTV} BT2 = {AcousticArray, AcousticArray}




Advantages & Limitations

• Higher-level task representations allow a wider range of assets to satisfy a task.

• Limitation: NIIRS scale is restricted to imagery intelligence (IMINT).

! How did we include acoustic intelligence?

• There is no reason why NIIRS cannot be extended to other types

• We introduced a number of rules for ACINT based on literature

• Still the need for NIIRS to be “officially” extended!

TASK 1

Detectvehicle

!"#$%&'(&)*+,-&"./(,%&.0&"1#+)2,11,#"./11,("3&"4./5.6+&&7&89.*5.6,::#7;+#89.!5.<#+#$,&7=,89.>5.'&.2&%[email protected];17#"7&%#1?.9..

/5! <;+)B#CD.9.25.65.E#F"1#"D.9.G5.H;.6#+4;I.9.J5.K#$;,FCI..













Bundle 1: UAV

+IMINT

Bundle 2: acoustic

motes

















bundle. !



assignment."



















Where:




“IMINT” Bundle

“ACINT” Bundle



Asset Allocationassign bundles instances to competing tasks,

based on recommended bundle types.


Extension 2

XTask 1

XTask 3

XTask 2

XTask 4

XTask 4




Allocation model

• A set of tasks competing for the exclusive usage of sensing assets.

! For each task:

d = utility demand

w = budget p = priority

• A set of sensors to be grouped into bundles based on bundle types.

! For each bundle-task pair:

e = joint utility to a task c = joint cost to a task

• Goal: an allocation of sensor bundles that maximizes the total profit.

S1

S2

S3

S4

B1

B2

Sensors(sensing assets)

Bundles

T1

T2

Tasks

(p1, d1, w1)e11, c11

e12, c

12

(p2, d2, w2)




Allocation model

• A set of tasks competing for the exclusive usage of sensing assets.

! For each task:

d = utility demand

w = budget p = priority

• A set of sensors to be grouped into bundles based on bundle types.

! For each bundle-task pair:

e = joint utility to a task c = joint cost to a task

• Goal: an allocation of sensor bundles that maximizes the total profit.

• The profit is a function of the utility cumulated by the task.

S1

S2

S3

S4

B1

B2

Sensors(sensing assets)

Bundles

T1

T2

Tasks

(p1, d1, w1)e11, c11

e12, c

12

(p2, d2, w2)




Combinatorial auction

• This can be seen as a combinatorial auction

! bidders: tasks

! items: sensors

! tasks bids for bundles of sensors

• Many algorithms have been proposed:

we use CASS (Combinatorial Auction Structured Search)




Combinatorial auction

• This can be seen as a combinatorial auction

! bidders: tasks

! items: sensors

! tasks bids for bundles of sensors

• Many algorithms have been proposed:

we use CASS (Combinatorial Auction Structured Search)

• Bundle generation issue:

! In the worst case we might have 2N bundles (with N = number of sensors)

! The REASONING step improves the bundle generation:

Reduce the number of

generated bundles

Prune the set of bids

placed by tasks

Generate only bundles

matching bundle types



ImplementationTwo prototypes of Top-to-bottom applications

to solve the Sensor-Mission assignment.





SAM - base station

• Level of use:

! Base station

! Operational analyst

Section IV. A screenshot of the new user interface is shown inFigure 5. As before, a user logs-in as a member of a coalition(in our example, we have a US/UK coalition). The user isable to select one or more areas-of-interest on a map (leftpanel) and, for each, to select multiple ISR tasks (top-rightpanel) using our NIIRS-based representation. (As we noted inSection IV our approach is currently simplified to assume thatall tasks are required at the same time; this could easily beextended.)

Fig. 5. Setting an area-of-interest and selecting tasks using the SAMapplication

Fig. 6. Selecting an asset bundle manually using the SAM application

SAM is implemented as a Web application in Java. TheNIIRS-based reasoning to infer capabilities from the selectedtasks is implemented as a rule-based system in Java (fordetails, see [9]. The matching procedure is implemented usingthe Pellet reasoner [26]. Before performing reasoning, theSAM application queries inventory catalogues to determinewhat types of platforms and sensors are actually available,so it can recommend package configurations including themost specific types that are potentially deployable. Once thepackage configurations have been found, SAM retrieves all

asset instances of the appropriate types from the inventorycatalogues, and generates all asset bundles specified by usingthe package configurations as bundle types, together withadditional cardinality information as noted at the end ofSection III. Because the number of assets is small in ourdemonstration, the set of bundles is currently presented tothe user, to make the choice of preferred bundle. This stepis shown in Figure 6.

In doing this, SAM takes into account access policies onthese resources, including ownership (note that the figureshows UK/US ownership of assets in the bottom-right panel)and whether the user has sufficient privileges to task thoseassets (shown by the “lock” icon next to the asset types).While simple, this mechanism is intended as an expansionpoint to allow the incorporation of more sophisticated accesspolicies in future, such as those described in [4]. Once anasset has been selected, SAM allows the user to “subscribe”to available data feeds from that asset, using the ITA SensorFabric [27]. Additionally, if an asset fails, SAM allows theuser to “backtrack” and make a different choice of bundle,and then obtain data that satisfies their task in some otherway. For example, the user might initially choose to satisfythe vehicle detection task (Figure 5) by imagery data to beobtained by a UAV (Figure 6); then, if the UAV fails for somereason, they could opt instead to solve the task using acousticdata from, for example, an acoustic array.

While the current version of SAM demonstrates the conceptof knowledge-driven sensor-mission assignment, it does notyet incorporate automated asset assignment, as it assumesan inventory with a relatively small number of assets. Thecombinatorial auction mechanism is currently implementedseparately. Integrating these two components requires a num-ber of choices to be made, most importantly:

• Users need a way to specify mission/task “profit” in away that is meaningful to them. Potentially this could bedone using a simple indication of priority. In principle,it ought to be part of a more general plan representationformalism. An open question here is whether priority canin some cases be inferred from the context of the task(for example, the type of mission: hostage rescue wouldnormally take priority over environmental monitoring).

• Further research is necessary to determine where are themost appropriate “choice points” for user intervention.It is clearly only feasible for users to select bundleinstances when there are very few choices available. Inmore typical cases, it would be more sensible to allowthe user to select (or rank) alternative bundle types (orpackage configurations), this giving a (perhaps “soft”)preference as to how they would prefer to obtain theirdata.

• SAM is intended to be used in a distributed fashion, asexplained in [5]. Multiple users in a coalition submit theirtasks using instances of SAM, and we need to examinealternative ways to apply the allocation algorithms inthis context. We have investigated distributed versus cen-tralised algorithms in the context of homogeneous sensor

Section IV. A screenshot of the new user interface is shown inFigure 5. As before, a user logs-in as a member of a coalition(in our example, we have a US/UK coalition). The user isable to select one or more areas-of-interest on a map (leftpanel) and, for each, to select multiple ISR tasks (top-rightpanel) using our NIIRS-based representation. (As we noted inSection IV our approach is currently simplified to assume thatall tasks are required at the same time; this could easily beextended.)

Fig. 5. Setting an area-of-interest and selecting tasks using the SAMapplication

Fig. 6. Selecting an asset bundle manually using the SAM application

SAM is implemented as a Web application in Java. TheNIIRS-based reasoning to infer capabilities from the selectedtasks is implemented as a rule-based system in Java (fordetails, see [9]. The matching procedure is implemented usingthe Pellet reasoner [26]. Before performing reasoning, theSAM application queries inventory catalogues to determinewhat types of platforms and sensors are actually available,so it can recommend package configurations including themost specific types that are potentially deployable. Once thepackage configurations have been found, SAM retrieves all

asset instances of the appropriate types from the inventorycatalogues, and generates all asset bundles specified by usingthe package configurations as bundle types, together withadditional cardinality information as noted at the end ofSection III. Because the number of assets is small in ourdemonstration, the set of bundles is currently presented tothe user, to make the choice of preferred bundle. This stepis shown in Figure 6.

In doing this, SAM takes into account access policies onthese resources, including ownership (note that the figureshows UK/US ownership of assets in the bottom-right panel)and whether the user has sufficient privileges to task thoseassets (shown by the “lock” icon next to the asset types).While simple, this mechanism is intended as an expansionpoint to allow the incorporation of more sophisticated accesspolicies in future, such as those described in [4]. Once anasset has been selected, SAM allows the user to “subscribe”to available data feeds from that asset, using the ITA SensorFabric [27]. Additionally, if an asset fails, SAM allows theuser to “backtrack” and make a different choice of bundle,and then obtain data that satisfies their task in some otherway. For example, the user might initially choose to satisfythe vehicle detection task (Figure 5) by imagery data to beobtained by a UAV (Figure 6); then, if the UAV fails for somereason, they could opt instead to solve the task using acousticdata from, for example, an acoustic array.

While the current version of SAM demonstrates the conceptof knowledge-driven sensor-mission assignment, it does notyet incorporate automated asset assignment, as it assumesan inventory with a relatively small number of assets. Thecombinatorial auction mechanism is currently implementedseparately. Integrating these two components requires a num-ber of choices to be made, most importantly:

• Users need a way to specify mission/task “profit” in away that is meaningful to them. Potentially this could bedone using a simple indication of priority. In principle,it ought to be part of a more general plan representationformalism. An open question here is whether priority canin some cases be inferred from the context of the task(for example, the type of mission: hostage rescue wouldnormally take priority over environmental monitoring).

• Further research is necessary to determine where are themost appropriate “choice points” for user intervention.It is clearly only feasible for users to select bundleinstances when there are very few choices available. Inmore typical cases, it would be more sensible to allowthe user to select (or rank) alternative bundle types (orpackage configurations), this giving a (perhaps “soft”)preference as to how they would prefer to obtain theirdata.

• SAM is intended to be used in a distributed fashion, asexplained in [5]. Multiple users in a coalition submit theirtasks using instances of SAM, and we need to examinealternative ways to apply the allocation algorithms inthis context. We have investigated distributed versus cen-tralised algorithms in the context of homogeneous sensor

• Features:

" High-level task representation

" Subscribe bundles manually




SAM - mobile

• Level of use:

! Mobile user (on the field)

! Not an expert

! Time constraint




SAM - mobile

• Level of use:


! Not an expert

! Time constraint

• Features:





SAM - mobile

• Level of use:


! Not an expert

! Time constraint

• Features:


" Automatic asset allocation

Satisfied Unsatisfied




Conclusion & Future

• Future:

- Assets or bundle of assets might be shared between tasks

- Most appropriate “choice points” for user intervention (human-in-the-loop)

- Information delivery to a mobile user (bandwidth constraints)

• We extended our original ontology-based approach to provide:

! a richer and more realistic specification of task requirements (using NIIRS)

! automatic asset allocation (using combinatorial auctions)

• We also showed these new features in a centralized and a mobile application.




Thanks for listening!

Questions?



Knowledge-Driven Agile Sensor-Mission Assignment

Technology

Transcript of Knowledge-Driven Agile Sensor-Mission Assignment