High Performance High Performance Cluster Computing Cluster Computing with PHOTOMOD HPC with PHOTOMOD HPC EditionEdition

Mikhail Drakin Head of Software Development Department

Racurs

September 2011, Tossa de Mar, Spain

11th International Scientific and Technical Conference From Imagery to Map: Digital Photogrammetric Technologies

In BriefIn Brief

Distributed processing of remote sensing data – the key to performance and efficiency

Parallelizing trends in modern hardware

Distributed processing in “conventional” PHOTOMOD:pros and cons

PHOTOMOD HPC Edition for computing clusters

PHOTOMOD Conveyor – the solution for on-line satellite monitoring

RSD – Ideal for Parallel ProcessingRSD – Ideal for Parallel Processing

Most operations on remote sensing data satisfy the conditionsfor effective parallelization – can be easily split into numerous tasks, which:

Are similar in natureand/or

Take source data bits independent from other tasksand/or

Produce data bits independent from output of other tasksand/or

Are performed automatically

RSD – Ideal for Parallel ProcessingRSD – Ideal for Parallel Processing

Common operations, implemented in automatic distributed mode in PHOTOMOD:

Tie points measurement DTM calculation (pickets and “dense” DEM) Orthorectification Mosaics

including sequence of distributed task sets:

• Cutlines creation

• Tie points searching

• Final mosaic building

+ a bunch of service operations (image format conversion, pan-sharpening, DEM reprojection, etc).

Parallel processing helps saving precious time,reducing days of work to hours

Parallel Hardware TrendsParallel Hardware Trends

The same trend at all scales: performance gainedby increasing number of identical computing units.This allows building applications scalable over a wide range

Increasing number of cores in off-the-shelf CPUs

Larger local networks

Supercomputers as clusters of tens to thousands similar nodes

““Conventional” PHOTOMODConventional” PHOTOMOD

Distributed processing first implemented in version 4.4.

Each new version supports several new operations in distributed mode.

Suited for medium-scale parallelization:

Distribution of tasks within single workstation to effectively utilize allprocessor cores

Distribution of tasks between several workstations to employ LAN resources of a department

Ideology: “one server – several workstations”

““Conventional” PHOTOMODConventional” PHOTOMOD

Drawbacks of distributed processing in standard PHOTOMOD:

Targeted at interactive work (flexible, but multiple GUI options complicate automation)

Not optimized for centralizedadministering

Implies storing configurationdata, logs, etc. locally on the workstation

Standard PHOTOMOD does not rely on consistent LAN structure and fixed hardware configuration, thus sacrificing some performance for flexibility.

Hardware SideHardware Side

Massive photogrammetric processing involves:

Sophisticatedcomputingalgorithms

Intensivedata

exchange

Hugestored data

volume

CPU

x K units?

Net

x M Gbps? x N TB?

Storage

Hardware configuration must be carefully planned and balanced to avoid bottlenecks

PHOTOMOD HPC Edition: Key FeaturesPHOTOMOD HPC Edition: Key Features

PHOTOMOD HPC Edition – next level of distributed processing in PHOTOMOD

Ideology: “one workstation – multiple servers”

Oriented at coherent high-end hardware and software complex

Centralized administering, “lessmessage boxes”

No modifications on computing nodes – all installation, configuration, logs storage, etc organized on a dedicated server

Customization for particular customer needs and computing environment is implied.

ApplicationsApplications

Fully automatic or semi-automatic solutions based on PHOTOMOD HPC Edition can pay off in processing of any kind of digital imagery,including:

Conventional aerial survey and rapidly emerging UAV imaging projects, with constantly increasing productivity and positioning quality, which allows fully automatic processing, especially with properly marked ground control points.

Rapid creation of quality orthomaps from HR and VHR satellite images for online monitoring (emergency management in situation centers, reconnaissance, etc).

CounterpartsCounterparts

Examples of counterpart products:

VisionMap A3 Ground Processing System

PCI GeoImaging Accelerator (GXL)

Spot Image Pixel Factory

Photogrammetric systems with distributed processing (LPS, Inpho, etc)

“Counterparts” here is not exactly equal to “Competitors”: each high-end product has specific niche.

PHOTOMOD HPC Edition: DistinctionsPHOTOMOD HPC Edition: Distinctions

Distributed processing subsystem is native for the platform andconstantly improved for specific applied needs

The system can be easily adopted to any hardware meetingbasic requirements, not tied to some specific set of devices. The simplest cluster, which can bring significant productivity benefits with PHOTOMOD HPC, is probably 4 computing nodes (Windows HPC Server) with carefully designed storage system (e.g. FreeBSD-based), starting from $30-40 thousand

A computing cluster with HPC Edition can be tightly integrated with “conventional” interactive PHOTOMOD workstations, taking all advantages and eliminating drawbacks mentioned earlier

The system is initially positioned as a platform for building customized solutions

Testing in KazakhstanTesting in Kazakhstan

Image courtesy of SmartCom

December 2010:PHOTOMOD HPC was successfully tested on KazNTU computing cluster

• 10.9 TFLOPS

• 128 computing nodes

• NetApp storage system

• Windows HPC Server OS

The experiment:Orthomosaic creation• Output GSD 0.16 m • Area 1289 sq. km• DEM cell size 1.6 m • Breaklines 858• Total output files 156 GB

Processing time: 3 hours

PHOTOMOD ConveyorPHOTOMOD Conveyor

Example of HPC Edition customization for specific application

Collaborative project by Racurs and CTT Group (Russia) for heavy-duty mosaic production from VHR satellite data: hardware + software complex solution

+

PHOTOMOD ConveyorPHOTOMOD Conveyor

Fully automatic workflow, including:

• Creation of the processing project from source satellite products

• Adjustment of the project (by RPC)

• Orthorectification by existing DEM

• Exclusion of images with undue cloud coverage

• Building cutlines

• Searching for tiepoints to eliminate residual discrepancies along seams

• Radiometric balancing

• Final stitching into seamless mosaic, saved into standard sheets set

1 000 000 sq. km per day of 1:25 000 mosaic from GeoEye, DG or Alos

FutureFuture

Distributed GPU computing in cluster environment

Extensive set of automatic functions for data processing and enhancement

Library of fully automatic workflows for UAV processing, DTM creation and filtering, etc.

Advanced control over scheduling for convenient sharing of a single cluster between several workgroups

Integration with GIS systems, databases and geoportals

Any modifications and customizations requested by customers

Thank you for attentionThank you for attention!!