Massive galaxies in massive datasets M. Bernardi, J. Hyde and E. Tundo
REAL TIME MANAGEMENT OF MASSIVE 2D DATASETS
description
Transcript of REAL TIME MANAGEMENT OF MASSIVE 2D DATASETS
Gunnar Misund 1
REAL TIME MANAGEMENT
OF MASSIVE 2D DATASETS
Gunnar Misund
Gunnar Misund 2
Shuttle Radar Topography Mission (Mission to Earth)
• Landing - May 2000 • 18 terabytes of raw data• 2 years of post processing• Virtual Earth: 3D model of 80% of the
continental area, 30m mesh• 20m horizontal resolution, 4m vertical
Gunnar Misund 3
ONE WORLD – ONE MAP
• On-the-fly generation of user defined maps in real time, typically via Internet servers
• Any combination of layers• Any selection, from global to street level views• Any resolution, from large graphical desktop displays to small
PDA/cellular screens• Frequently updated in formation• Such servers already present, e.g.
http://tiger.census.gov/cgibin/mapsurfer• ...BUT: Still more to do
Gunnar Misund 4
Mostly DCW data: coasts, rivers, political boundariesCanada: Elevation contours, roads, utilities etc. (”upgraded” DCW)US coast: 1:70.000Total ca 30.000.000 points
Gunnar Misund 5
Gunnar Misund 6
Gunnar Misund 7
Gunnar Misund 8
Gunnar Misund 9
Gunnar Misund 10
Gunnar Misund 11
Gunnar Misund 12
Gunnar Misund 13
Gunnar Misund 14
Gunnar Misund 15
N50 1:50.000 map sheetCa 180.000 points
Gunnar Misund 16
Gunnar Misund 17
Gunnar Misund 18
Gunnar Misund 19
Gunnar Misund 20
Gunnar Misund 21
Approximation: Simplification vs. Data Reduction
Simplification (smoothing): Reducing the (visual) complexity of a geometric object
Data reduction (thinning): Reducing the amount of data (often 2D/3D) needed to represent a geometric object within a given tolerance
Mostly treated as two aspects of same phenomenon
Gunnar Misund 22
Cartographic Generalization
Road and river network, 3 different scales:
Gunnar Misund 23
Level of Detail (LOD)
Gunnar Misund 24
MASSIVE MAPS SERVERS – SOME REQUIREMENTS I
• Efficient storage:– The size of the database should propotional with the size of
the dataset– Multiple representations should be avoided, prone to
inconstency problems• Efficient retrieval:
– Efficient window query– Efficient approximation of the data in the query window– The combined query/approximation requests must run in
sublinear time– Can’t afford to inspect every point in the data set– Should be close to logaritmic order– Must run in external memory
• Efficient maintaince:– Removals, additions and modifications must run in
sublinear time
Gunnar Misund 25
MASSIVE MAPS SERVERS – SOME REQUIREMENTS II
• Generalization:– Selection, aggregation and possible deformations should
be performed more or less automatically• Topology preservation:
– Elevation contours must not cross, road networks have to remain consistent after a query process
• Scalability:– Operations should be decomposable:
• Spatial partitioning allow for parallell methods
– Should facilitate fusion of data from heterogenous sources• Implementation:
– Simple methods are easy to implement and maintain
Gunnar Misund 26
GLOBAL QUERY WINDOW, VARYING DATA DENSITY
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
0 10000000 20000000 30000000 40000000
# POINTS IN DATA SET
# P
OIN
TS
INS
PE
CT
ED
/ #
PO
INT
S R
ET
RIE
VE
D
600 x 400 RESOLUTION
1200 x 800 RESOLUTION
Gunnar Misund 27
LOCAL QUERY WINDOW, FIXED DATA DENSITY
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
0 10000000 20000000 30000000 40000000
# POINTS IN DATA SET
# P
OIN
TS
INS
PE
CT
ED
/ #
PO
INT
S R
ET
RIE
VE
D
600 x 400 RESOLUTION
1200 x 800 RESOLUTION
Gunnar Misund 28
1W1M: ONE WORLD – ONE MAP
• Long term project, coordinated from HiØ– Provide free access for all internet users to a virtual map
with global coverage• Gateway
– Consumer side of 1W1M– Retrieval of customized maps for any area, in any
resolution– Free of charge– ”Common” users will receive a graphic depiction as the
result of the query (e.g. a JPEG image)– Producers are have access to fully functional GIS data
• Clearinghouse– Producer side of 1W1M– Any party can submit public domain geodata– Approval of submissions based on ”peer review”
Gunnar Misund 29
COMMENTS
• One World – One Map solutions are technologically within reach
• New user demands, new sources of data and new technology calls for – new geodata models– rethinking of the generalization consept– distributed and integrated storage and retrival systems– increased focus on standards and integration issue