8/14/2019 billptest

1/20

Bit Sequential (bSQ) Data Modeland

Peano Count Trees (P-trees)

Department of Computer Science North Dakota State University, USA

(the bSQ and P-tree technology is patented by NDSU)

8/14/2019 billptest

2/20

8/14/2019 billptest

3/20

RSI dataset example

TIFF image Yield Map

RSI data can be viewed as collection of pixels.Each pixel has a value for each feature attribute

For example, the RSI dataset above has 320 rows and 320 columns of pixels(102,400 pixels) and 4 feature attributes (B,G,R,Y). The (B,G,R) feature

bands are in the TIFF image and the Y feature is color coded in the YieldMa .

8/14/2019 billptest

4/20

Spatial Data Formats

Existing formats BSQ (Band Sequential) BIL (Band Interleaved by Line) BIP (Band Interleaved by Pixel)

New format bSQ (bit Sequential)

8/14/2019 billptest

5/20

Spatial Data Formats (Cont.)

BAND-1254 127(1111 1110) (0111 1111)

14 193(0000 1110) (1100 0001)

BAND-237 240(0010 0101) (1111 0000)

200 19(1100 1000) (0001 0011)

BSQ format (2 files)

Band 1: 254 127 14 193Band 2: 37 240 200 19

8/14/2019 billptest

6/20

Spatial Data Formats (Cont.)

BAND-1254 127(1111 1110) (0111 1111)

14 193(0000 1110) (1100 0001)

BAND-237 240(0010 0101) (1111 0000)

200 19(1100 1000) (0001 0011)

BSQ format (2 files)

Band 1: 254 127 14 193Band 2: 37 240 200 19

BIL format (1 file)

254 127 37 24014 193 200 19

8/14/2019 billptest

7/20

Spatial Data Formats (Cont.)

BAND-1254 127(1111 1110) (0111 1111)

14 193(0000 1110) (1100 0001)

BAND-237 240(0010 0101) (1111 0000)

200 19(1100 1000) (0001 0011)

BSQ format (2 files)

Band 1: 254 127 14 193Band 2: 37 240 200 19

BIL format (1 file)

254 127 37 24014 193 200 19

BIP format (1 file)

254 37 127 24014 200 193 19

8/14/2019 billptest

8/20

Spatial Data Formats (Cont.)

BAND-1254 127(1111 1110) (0111 1111)

14 193(0000 1110) (1100 0001)

BAND-237 240(0010 0101) (1111 0000)

200 19(1100 1000) (0001 0011)

BSQ format (2 files)

Band 1: 254 127 14 193Band 2: 37 240 200 19

BIL format (1 file)

254 127 37 24014 193 200 19

BIP format (1 file)

254 37 127 24014 200 193 19

bSQ format (16 files)B11 B12 B13 B14 B15 B16 B17 B18 B21 B22 B23 B24 B25 B26 B27 B281 1 1 1 1 1 1 0 0 0 1 0 0 1 0 10 1 1 1 1 1 1 1 1 1 1 1 0 0 0 00 0 0 0 1 1 1 0 1 1 0 0 1 0 0 01 1 0 0 0 0 0 1 0 0 0 1 0 0 1 1

8/14/2019 billptest

9/20

8/14/2019 billptest

10/20

The tabular formats (inverted list)

BSQ and bSQ are tabular formats BSQ consist of a separate table for each feature band bSQ consist of a separate table for each bit of each band

One can view it this way:

The data set is initially one relation or table, R(K 1,..,K k , A 1, A 2,, A n) where K 1,..,K k are the structure attributes and each A i is afeature attribute.

The structure attributes of a 2-D image are the X and Y coordinates of the pixels (rows).

The feature attributes are the bands, B,G,R, NIR, In BSQ we separate each feature into a separate file and suppress the

structure attributes altogether (under the assumption that the pixels arealways arranged in raster order.

In bSQ we separate each bit of each feature into a separate file (same raster order assumption)

8/14/2019 billptest

11/20

Peano Count Tree (P-tree)

P-tree represents spatial bSQ data bit-by-bit ina recursive quadrant-by-quadrant arrangement.An P-tree is a lossless representation of the

original data.A P-tree is a compressed structure.A P-tree is count pre-computed.

8/14/2019 billptest

12/20

An example of Ptree

Peano or Z-orderingPure (Pure-1/Pure-0) quadrantRoot Count

LevelFan-outQID (Quadrant ID)

1 1 1 1 1 1 0 01 1 1 1 1 0 0 0

1 1 1 1 1 1 0 0

1 1 1 1 1 1 1 0

1 1 1 1 1 1 1 11 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1

0 1 1 1 1 1 1 1

55

16 8 15 16

3 0 4 1 4 4 3 4

1 1 1 0 0 0 1 0 1 1 0 1

16 16

55

0 4 4 4 4

158

1 1 1 0

3

0 0 1 0

1

1 1

3

0 1

8/14/2019 billptest

13/20

55

16 8 15 16

3 0 4 1 4 4 3 4

1 1 1 0 0 0 1 0 1 1 0 1

An example of Ptree

Peano or Z-orderingPure (Pure-1/Pure-0) quadrantRoot Count

LevelFan-outQID (Quadrant ID)

1 1 1 1 1 1 0 01 1 1 1 1 0 0 0

1 1 1 1 1 1 0 0

1 1 1 1 1 1 1 0

1 1 1 1 1 1 1 11 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1

0 1 1 1 1 1 1 1

0 1 2 3

111

( 7, 1 ) ( 111, 001 ) 10.10.11

2

3

2 . 2 . 3

001

Level-0

Level-3

Level-2

Level-1

8/14/2019 billptest

14/20

P-tree variation PM-tree

Peano Mask tree (PM-tree) uses mask instead of count.1 denotes pure-1, 0 denotes pure-0 and m denotes mixed.It provides an efficient way for ANDing.

1 1 1 1 1 1 0 01 1 1 1 1 0 0 0

1 1 1 1 1 1 0 0

1 1 1 1 1 1 1 0

1 1 1 1 1 1 1 11 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1

0 1 1 1 1 1 1 1

m

1 m m 1

m 0 1 m 1 1 m 1

1 1 1 0 0 0 1 0 1 1 0 1

8/14/2019 billptest

15/20

Ptree Algebra

AndOr Complement

Other (XOR, etc)Ptree: 55

____________/ / \ \___________ / ___ / \___ \

/ / \ \16 ____8__ _15__ 16

/ / | \ / | \ \3 0 4 1 4 4 3 4

//|\ //|\ //|\1110 0010 1101

Complement: 9____________/ / \ \___________

/ ___ / \___ \/ / \ \

0 ____8__ __1__ 0/ / | \ / | \ \

1 4 0 3 0 0 1 0//|\ //|\ //|\

0001 1101 0010

8/14/2019 billptest

16/20

8/14/2019 billptest

17/20

Alternative forms for Ptrees (all lossless)

P1: 0

______/ / \ \______ / / \ \

/ / \ \1 0 0 1

/ / \ \ / / \ \0 0 1 0 1 1 0 1

//|\ //|\ //|\1110 0010 1101

P0: 0

______/ / \ \______ / / \ \

/ / \ \0 0 0 0

/ / \ \ / / \ \0 1 0 0 0 0 0 0

//|\ //|\ //|\0001 1101 0010

PNZ (=P0) 1

________ / / \ \___ / ____ / \ \

/ / \ \

1 1 1 1/ / \ \ / / \ \

1 0 1 1 1 1 1 1//|\ //|\ //|\

1110 0010 1101

1 means quadrant is pure-1, 0 otherwise(pure0 if no sub-tree ptrs, otherwise mixed)

1 means quadrant is pure-0, 0otherwise

1 means quadrant is Not pure-Zero, 0otherwise (Note: PM = PNZ XOR P1 )

P1V (as a table):qid vector[ ] 1001[01] 0010[10] 1101[01.00] 1110[01.11] 0010[10.10] 1101

P0V:qid vector[ ] 0000[01] 0100[10] 0000[01.00] 0001[01.11] 1101[10.10] 0010

PNZV:qid vector[ ] 1111[01] 1011[10] 1111[01.00] 1110[01.11] 0010[10.10] 1101

Vector forms (A table entry for each mixed inode containing its qid and its children bit-vector ; Eliminate need for subtree pointers)

00 01

00

10 11

0001 0110 1011 11

Since there is noqid=[01.01] in thetable we know itspure0, not mixed

8/14/2019 billptest

18/20

E

B11 B13B12

8/14/2019 billptest

19/20

Examp e

PNZP1V 11qid NZ P1[ ] 1111 1001[01] 1011 0010[10] 1111 1101[01.00] 1110 1110[01.11] 0010 0010

[10.10] 1101 1101

PNZP1V 12qid NZ P1[ ] 1010 1000[10] 1111 1110[10.11] 0111

PNZP1V 13qid NZ P1[ ] 0111 0001[01] 1111 1110[10] 1110 0110[01.11] 0110[10.00] 1000

Redundant! Since nomixed at this level.

1 1 1 1 1 1 0 01 1 1 1 1 0 0 0

1 1 1 1 1 1 0 0

1 1 1 1 1 1 1 0

1 1 1 1 1 1 1 11 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1

0 1 1 1 1 1 1 1

B11 B13B12

1 1 1 1 0 0 0 01 1 1 1 0 0 0 0

1 1 1 1 0 0 0 0

1 1 1 1 0 0 0 0

1 1 1 1 0 0 0 01 1 1 1 0 0 0 0

1 1 0 1 0 0 0 0

1 1 1 1 0 0 0 0

0 0 0 0 1 1 1 10 0 0 0 1 1 1 1

0 0 0 0 1 1 0 1

0 0 0 0 1 1 1 0

1 0 1 1 1 1 1 10 0 1 1 1 1 1 1

1 1 0 0 1 1 1 1

1 1 0 0 1 1 1 1

6 6 6 6 5 5 1 16 6 6 6 5 1 1 1

6 6 6 6 5 5 0 1

6 6 6 6 5 5 5 0

7 6 7 7 5 5 5 56 6 7 7 5 5 5 5

7 7 4 6 5 5 5 5

3 7 6 6 5 5 5 5

Blue Band, B 1, with 3-bit values

RC(P 1,6)=RC(P 1,110 )=RC(P 11^P 12^P 13). At [ ]: P1 1,6 =P 11^P 12^PNZ 13 (since P 13 is P0V 13=PNZ 13). Calculate COUNT=P1Cnt*4 level

PNZ 1,6=PNZ 11^PNZ 12^P1 13 (since PN1 13 = P1 13). Recursively get P1Cnt*4 level from each mixed child (PMV 1,6=P0 1,6^ P1 1,6).

L-2 P1[ ]: 1001^1000^1000=1000, contrib 1*4 2=16, PNZ[ ]=1111^1010^1110 = 1010, P1[ ]=1000, PM[ ]=0010

L-1 P1[10]: 1101^1110^0001=0000, contrib 0*4 1=0, PNZ[10]=1111^1111^1001=1001, P1[10]=0000, PM[10]=1001L-0 P1[10.00]: 1111 1111^0111=0111, contrib 3*4 0 =3; Level-0 P1[10.11]: 1111^0111^1111=0111, contrib 3*4 0 =3 TOT=22

i ib d ?qid NZ P1

id NZ P1

id NZ P1P P P

8/14/2019 billptest

20/20

Distributed P trees?qid NZ P1[ ] 1111 1001[01] 1011 0010[10] 1111 1101[01.00] 1110[01.11] 0010[10.10] 1101

qid NZ P1[ ] 1010 1000[10] 1111 1110[10.11] 0111

qid NZ P1[ ] 0111 0001[01] 1111 1110[10] 1110 0110[01.11] 0110[10.00] 1000

P 11 P 12 P 13

Assume a 5-computer cluster; NodeC, Node 00 , Node 01, Node 10, Node 11 .

Send to Node ij if qid ends in ij: Bp qid NZ P1 0011[01.00] 111013[10.00] 1000Bp qid NZ P1 C

11[ ] 1111 100112[ ] 1010 100013[ ] 0111 0001

Distributed Request : Send request, RC(P 1,6) to NodeC (if Root Count for a subquadrant only, send to that qid, eg, [10] RC(P 1,6)

NodeC does: P1[ ]: 1001^1000^1000=1000. Count=1*4 2=16. PNZ[ ]=1111^1010^1110=1010. P1[ ]=1000. PM[ ]=0010.

Send request RC(P 1,6) to Node 10 .

Node 10 does: P1[10]: 1101^1110^0001=0000. Count=0. PNZ[10]=1111^1111^1001=1001. P1[10]=0000. PM[10]=1001.

Send request RC(P 1,6) to Nodes 00 Node 11

Node 00 does : P1[10.00]: 1111 1111^0111=0111, contribution is 3*4 0 =3 Send (3) to NodeC (all nodes return counts to C)

Node 11 does : P1[10.11]: 1111^0111^1111=0111, contribution is 3*40

=3 Sends (3) to NodeC

Bp qid NZ P1 0111[01] 1011 001013[01] 1111 1110

Bp qid NZ P1 1011[10] 1111 110111[10.10] 110112[10] 1111 111013[10] 1110 0110

Bp qid NZ P1 1111[01.11] 001012[10.11] 011113[01.11] 0110

All nodes return counts to the original root (in this case, NodeC)