integrating Data for Analysis,Anonymization, and SHaring

Supported by the NIH Grant U54HL108460 to the University of California, San Diego

Input(.bam/.sam)

Sequence Other Fields Quality value

Matched reads

Mismatchedread

Use another reference and

shorten the read length

K-means clustering

Encode aligning

information

Encoder

Output1 Output2 Output4

Encoder

Delta codeHuffman code Gamma code LZW codeetc.

Output3

Fast Reference based Alignment tool

Reads length> threshold?

Yes

No

HUGO: Hierarchical mUlti-reference Genome cOmpression tool for aligned short reads Pinghao Li,1 Xiaoqian Jiang,2 Shuang Wang,2 Jihoon Kim,2 Hongkai Xiong,1 and Lucila Ohno-Machado2

1EE Department, Shanghai Jiaotong University, Shanghai, China2Division of Biomedical Informatics, University of California–San Diego, La Jolla, California, USA

Introduction HUGO framework

Experimental Results

Summary of Conclusions Storage and transmission are important challenges in the use of large sequencing ‘Big Data’. We developed a novel compression technique, the HUGO framework, for compressing aligned reads. Our method also presents an innovative way of hierarchically matching gradually shortened reads in order to make full use of available reference genomes. Our experiments compared the performance of our algorithm with other state-of-the-art compression algorithms, such as CRAM, to which ours was superior, and Samcomp, which had similar compression performance.

Short-read sequencing is becoming the standard of practice for the study of structural variants associated with disease. However, with the growth of sequence data largely surpassing reasonable storage capability, the biomedical community is challenged with the management, transfer, archiving, and storage of sequence data.

We developed Hierarchical mUlti-reference Genome cOmpression (HUGO) [1], a novel compression algorithm for aligned reads in the Sequence Alignment/Map (SAM) format. We first aligned short reads against a reference genome and stored exactly mapped reads for compression. For the inexact mapped or unmapped reads, we realigned them against different reference genomes using an adaptive scheme by gradually shortening the read length. Regarding the base quality value, we offer lossy and lossless compression mechanisms. The lossy compression mechanism for the base quality values uses k-means clustering, where a user can adjust the balance between decompression quality and compression rate. The lossless compression can be produced by setting k (the number of clusters) to the number of different quality values.

Ref

eren

ces

Image source: http://www.ncbi.nlm.nih.gov/Traces/sra/i/g.png

# Name Description1 QNAME Query NAME of the read or the read pair2 FLAG Bitwise FLAG (pairing, strand, mate strand, etc.)3 RNAME Reference sequence NAME4 POS 1-based leftmost POSition of clipped alignment5 MAPQ MAPping Quality (Phred-scaled)6 CIGAR Extended CIGAR string (operations: MIDNSHP)7 MRNM Mate Reference NaMe (‘=’ if same as RNAME)8 MPOS 1-based leftmost Mate POSition9 ISIZE Inferred Insert SIZE10 SEQ Query SEQuence on the same strand as the reference11 QUAL Query QUALity (ASCII-33=Phred base quality)12 OPT Optional fields.

Methodology

Child

Reference from Mother

Reference from Father

EMR: exact mapped readIMR: inexact mapped reads(with less than 4 mismatches) UMR: unmapped reads(with more 4 mismatches)

5 10 15 20 30 40 45 510

50

100

150

K

MB

/MA

PE

Compressed sizeError

The compression using k-mean clustering followed by bzip2, where the quantization error is measured by Mean Absolute Percentage Error (MAPE)

ID Sequence Name BAM size SAM size1 NA12878chrom20 356MB 1.58GB2 HG00096chrom11 661MB 2.65GB3 HG00103chrom11 717MB 2.91GB4 HG01028chrom11 964MB 3.95GB5 NA06984chrom11 1.19GB 5.16GB6 NA06985chrom11 2.33GB 9.41GB

1 20

2

4

6

8

10

Sequence

Com

pres

sion

Rat

io

HUGO with lossy compression

bzip2CRAMSAMcompHUGOHUGOL(30)

HUGOL(20)

HUGOL(10)

HUGOL(1)

HUGO lossy

SAMcompCRAM

bzip2 bzip2HUGO lossless

HUGO lossy

SAMcompHUGO lossless

1 2 3 4 5 60

5

10

15

20

Sequence

Mem

ory

usag

e in

GB

Encoding memory usage

CRAMSamcompHUGO

1 2 3 4 5 60

2

4

6

8

Sequence

Mem

ory

usag

e in

GB

Decoding memory usage

CRAMSamcompHUGO

Encoding Memory usage

Decoding Memory usage

ID Program name Reference BAM size Compressed size

3

bzip2 hg19

717 MB

720MBCRAM[2] hg19 453.6MB

Samcomp[3] hg19 349MBHUGO hg19 392.5MBHUGO hg19, HuRef 390.2MB

4

bzip2 hg19

964 MB

967 MBCRAM hg19 585MB

Samcomp hg19 480MBHUGO hg19 548.1MBHUGO hg19, HuRef 545.3MB

5

bzip2 hg19

1.19 GB

1.192GBCRAM hg19 736.8MB

Samcomp hg19 538MBHUGO hg19 653.1MBHUGO hg19, HuRef 650.2MB

6

bzip2 hg19

2.33 GB

2.34GBCRAM hg19 1570MB

Samcomp hg19 1247MBHUGO hg19 1496MBHUGO hg19, HuRef 1491MB

HUGO Lossless with multi-reference

[1] Li, P., Jiang, X., Wang, S., Kim, J., Xiong, H., Ohno-Machado, L.. HUGO: Hierarchical mUlti-reference Genome cOmpression for aligned reads. Journal of the American Medical Informatics Association, 2013;0:1–11. doi:10.1136/amiajnl-2013-002147[2] Fritz MH-Y, Leinonen R, Cochrane G, et al. Efficient storage of high throughput DNA sequencing data using reference-based compression. Genome Res 2011;21:734–40.[3] Bonfield JK, Mahoney MV. Compression of FASTQ and SAM format sequencing data. PloS ONE 2013;8:e59190.