Future Architectures for genomics

Guy Coates

Wellcome Trust Sanger Institute

gmpc@sanger.ac.uk

The Sanger Institute

Funded by Wellcome Trust.2nd largest research charity in the world.

~700 employees.

Based in Hinxton Genome Campus, Cambridge, UK.

Large scale genomic research.Sequenced 1/3 of the human genome. (largest single contributor).

Large scale sequencing with an impact on human and animal health.

Data is freely available.Websites, ftp, direct database access, programmatic APIs.Some restrictions for potentially identifiable data.

My team:Scientific computing systems architects.

The not quite so scary graph

Peak Yearly capillary sequencing: 30 Gbase

Current weekly sequencing:7-10 Tbases. Not increasing as aggressively as the historical trend.

New CPU Architectures

MRSA Outbreak

MRSA:Antibiotic resistant bacteria.

Carried by many people with no symptoms.

But can cause serious diseases (boils, sepsis, necrotising fasciitis).

Bad news in hospitals (14 patients presented absesses during the outbreak)

Infection control via traditional methodsCompare antibiotic resistence profiles of samples.

Epidemiology detective work to find common factors.

Close wards for deep cleans.

3 different outbreaks of MRSA over 6 months

Sequencing Advantages

Sequence samples from infections:Compare sequence and build phylogenetic trees.

Identified extra cases, missed by traditional screening.Traced infection to a staff member (asymptomatic).

Real Time information: culture sequence informatics within 48 hrs.

Aids, not replaces, current infection control protocols.

P25

P22

P21

P24

+

P8

P9

P13

P10

P17

P14, 23

P7

P20

P26

P6

P1

P4, 16, and 19

P2, 3

P5

P15

P11, 12

H

H

H

H

H

H

H

H

H

H

H

H(2)

H(2)

H(5)

F

Loss of

ermC

plasmid

Informatics in a hospital setting?

Can we produce lab friendly informatics as well as sequencing?Low power, low footprint compute.

Informatics was run in our datacentre (20,000 cores, 20 PB storage)Can we do it on something you can plug into a 13AMP socket in the lab?

ARM for Bioinformatics

Not as stupid as it first seems:

Bioinformatics code:Single threaded, integer, not floating point dominated.

Bacteria have small genomes:3 Mbases (MRSA) vs 3 Gbases (Human)

May not need 64bit memory address space for some problems.

Compute Farm Memory footprint

Some jobs do have large memory footprints:But large number of jobs below 1.5GB.

Intel vs ARM

ARM: Calxeda Energy Core

4 core 1.0 GHz ARM A9, 4GB RAM.

Intel:IBM HS22

6 core Xeon X5650 (Westmere) 2.67 GHz, 36 GB RAM

(Not the latest or greatest, but what we had available)

OS: Ubuntu 12.04 LTSKernel 2.6.32 on Intel, 3.6 on ARM.

gcc 4.6.3

Porting code:Surprisingly easy.

R / Python / Perl: Interpreted languages worked in our favour!

C just worked.

Bioinformatics Codes

Standard set of bioinformatics code:ARM ~5x slower than the Intel.

Right ballpark to be more efficient on performance/watt.

Some exceptions:Exonerate 2.2: 250x slower.Pointer chasing?

Hmmer v3: uses sse2 on Intel.

SNP calling Pipeline

Looks for single base changes in DNA sequenceC, java with perl glue.

Data sizes:2 x 40 Mbyte sequence file files.

3 Mbyte reference

Intel is 4.2 speed of the ARM310 vs 1326 seconds.

MAP(BWA)Sort(samtools)SNP call(mpileup)

Further Investigations

Scaling tests:ARM A9 showed poorer scaling than Intel as we loaded the cores up.

Kernel related?

Explore compiler / JVM opts:Built with the package defaults (typically gcc / -O2).

gcc vs icc etc.

Lots of change in the ecosystem:
ARM A15 vs ARM 64 vs Atom etc etc..

Conclusions

Initial tests look promising.Code runs.

Is ARM cost effective?Numbers look to be in the right ballpark wrt performance / Watt.going from datasheet power numbers.

We don't know yet on price/performance.

We need to see real production systems so we can get real power / $ numbers.

Object Storage

Storage for Sequencing

Sequencing produces lots of data.Lots of data + lots of people quickly becomes un-manageble.

We use iRODS to manage data.Stores data + *metadata* in a storage agnostic system.

Alignments(200GB)Sequencing data flow.

Sequencer

Processing/QC

analysisdatastore

Structured data(databases)

InternetVariation data(1GB)Feature(3MB)Raw data(10 TB)Sequence(500GB)Unstructured(Flat files)Sequencing the start of most analysisPeople = Umanaged dataData in wrong placeDuplicatedNobody can find anythingInc systems:Backups/securityCapacity planning?

Alignments(200GB)Sequencing data flow.

Sequencer

Processing/QC

analysisdatastore

Structured data(databases)

InternetVariation data(1GB)Feature(3MB)Raw data(10 TB)Sequence(500GB)

Unmanaged Pbytes!Unstructured(Flat files)Sequencing the start of most analysisPeople = Umanaged dataData in wrong placeDuplicatedNobody can find anythingInc systems:Backups/securityCapacity planning?

iRODS Data Management

ICATCataloguedatabaseRule EngineImplements policiesIrods ServerData on diskUser interfaceWebDAV, icommands,fuse

Irods ServerData in database

Irods ServerData in S3

Sanger Implementation

Storage5PB Storage (2 x 2.5 PB).

Data stored on standard posix filesystems at the backend.

Mixture of vendors and filesystem sizes.

40TB 200TB chunks.

Database:Oracle 10g RAC.

Replicated:One copy in two sections of our datacentre.(probably move 1 off site this year)

Federated:Split system to isolate research teams from one another.

Still single namespace.

User interaction:Via CLI tools (think command line ftp) or via C API.

Archival system, separate from our HPC lustre systems.

Irods Filesystem Issues

We have lots of filesystems behind irods.60 filesystems.

Filesystems need TLC.They get full.

Sometimes they go wrong.Can you fsck a 200TB filesystem?

Is there an alternative storage backend that is simpler or cheaper?

Object Stores

System for storing objects (files)put and get symantics.

Not POSIX:Fewer features, but simpler to implement; should be more scalable and robust.

No directory structure; just a set of object Ids.You need to implement your own organisational schema on top of the object store.

Lots of alternativesCommercial, open source, hardware or software based.

Different approaches to data integrity / DR.Replication vs erasure coding.

No standard APIs.Amazon S3 defacto API.

Lowest common donominator: (S3 currently does not support seek operations, which is important if we are dealing with large structured files.)

Object store and iRODS

Object stores are interesting, but very different from our POSIX world.

Conceptually object is a good fit for iRODS.

Transparency:iRODS is storage back-end agnostic.

Putting object store behind irods makes it transparent to the end user.

Provides a good organisational schema.Searchable metadata.

(Potentially) simplifies storage administration.

Questions we need to answer

How does iRODS replication and object store replication interact?iRODS knows how to replicate objects.

Most (all?) object stores have replication / erasure coding mechaisms.

What is the right level to do the replication at?

How are seek operations handled?We can currently pull records out of BAM files without having to download the entire file.

Will this still work on object store.

Data localityImportant in multi-site / federated irods installations.

If I get an object from irods, I'd like to talk to the storage elements nearest to me on the network

Many ways to potentially tackle this:Loadbalancers, proxies and other network tricks.

Make irods aware of the object store topology.

Not clear what the best mechanism will be.

Acknowledgements

My Team:Pete Clapham(ARM & iRODS)

James Beal

Helen Brimmer

Karl FreundCalxeda

Whole-genome sequencing for analysis of an outbreak of meticillin-resistant Staphylococcus aureus: a descriptive study

Lancet Infect Dis. 2013 February; 13(2): 130136.

doi: 10.1016/S1473-3099(12)70268-2

Click to edit the title text formatClick to edit Master title style

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level

Seventh Outline LevelClick to edit Master text stylesSecond levelThird levelFourth levelFifth level

SpeedupARMIntel

exonerate15.9

blastn17.4

blastp15.3

tblastn15

SpeedupIntelARM

4.271