An Introduction to CAMERA and Underlying Technologies Philip Papadopoulos University of California,...
-
Upload
lillian-hess -
Category
Documents
-
view
217 -
download
3
Transcript of An Introduction to CAMERA and Underlying Technologies Philip Papadopoulos University of California,...
An Introduction to CAMERA and Underlying Technologies
Philip Papadopoulos
University of California, San Diego
San Diego Supercomputer Center
California Institute of Telecommunications and Information Technology (Calit2)
PI Larry Smarr
Announced 17 Jan 2006. Public Release 13 March 2007$24.5M Over Seven Years
DNA Basics for Non-Biologists
• Nucleotide bases of DNA– ACTG (Adenine, Cytosine, Guanine, Thymine)– A Sequence of Bases Forms One Side of a DNA
Strand– Complementary Bases form the other side of
DNA – A matches T (pair)
– C matches G (pair)
• During cell replication, DNA is “unzipped” . The complementary side can then be replicated perfectly
• Human DNA is about 3 billion base pairs on 26 Chromosomes
Bases Amino Acids
• Triplets of nucleotide bases are called codons and define amino acids.– Amino acids are the basic building blocks of proteins– There are 20 amino acids, but 4^3 = 64 nucleotide combinations.– Many amino acids have multiple codons– Special codons (called start and stop codons) assist in DNA translation
during cell replication.
• Reading Frames of: GGGAAACCC– This raw sequence could be read as
– GGGAAACCC (GGG AAA CCC) (Glycine, Lysine, Proline)
– GGAAACCC (GGA AAC) (Glycine, Asparagine)
– GAAACCC (GAA ACC) (Glutamic Acid, Threonine)
Sequencing Tidbits
• The Institute for Genomic Research (TIGR) sequenced the genome of the bacterium Haemophilus influenzae in 1995 using shotgun sequencing
– 1.8 Million Base Pairs (Human: 3 Billion)
• Sequencing does NOT tell you what function a particular gene plays
• It is believed that only ~1.5% of human chromosome codes for expressed characteristics– The non-coding portions contain our genetic history– Unknown what function the rest our DNA plays
Most of Evolutionary Time Was in the Microbial World
You Are
Here
Source: Carl Woese, et al
Tree of Life Derived from 16S rRNA Sequences
Marine Genome Sequencing Project – Measuring the Genetic Diversity of Ocean Microbes
Sorcerer II Data Will Double Number of Proteins in GenBank!
Need Ocean Data
Some CAMERA Goals
• Provide an infrastructure where scientists from around the world can perform analysis on genetic communities– Global Ocean Sampling (GOS) is the initial large data set
– ~ 8.5 Billion base pairs of raw Reads
– Metadata is available for samples– Saline, Temperature, Geographic Location, Water Depth, Time of Day …
– Other metadata will be correlated with samples (e.g. MODIS Satellite)
• Allow others to search and compare input sequences against CAMERA data.
• Overall provide a resource dedicated to metagenomics – Support new datasets – Support new analysis tools and web services
Global Ocean Survey (GOS) Sequences are Largely Bacterial
Source: Shibu Yooseph, et al. (PLOS Biology in press 2006)
~3 Million Previously Known
Sequences
~5.6 Million GOS
Sequences
Reason for CAMERA
• The Global Ocean Survey (GOS) is a huge influx of sequence data
• Factors that interrelate microbes and microbial communities are not well known
• Significant analysis requires large resources– All-to-all comparisons – Integration of other environmental (meta) data (weather,
temperature, salinity,…) is essential
• Raw Sequence Data sets are mid-sized– Current set of GOS Raw Reads is about 100GB (FASTA
Files)
Calit2 CAMERA ProductionCompute and Storage Complex is On-Line
512 Processors ~5 Teraflops
~ 200 Terabytes Storage
User Map – 03 May 2007
• Site in production on 13 March 2007• More than 500 Registered users from around the globe (~10 new users/day)
Flat FileServerFarm
W E
B P
OR
TA
L
TraditionalUser
Response
Request
DedicatedCompute Farm(100s of CPUs)
TeraGrid: Cyberinfrastructure Backplane(scheduled activities, e.g. all by all comparison)
(10000s of CPUs)
Web(other service)
Local Cluster
LocalEnvironment
DirectAccess LambdaCnxns
Data-BaseFarm
10 GigE Fabric
Calit2’s Direct Access Core Architecture CAMERA’s Metagenomics Server Complex
Source: Phil Papadopoulos, SDSC, Calit2+
We
b S
erv
ice
s
Sargasso Sea Data
Sorcerer II Expedition (GOS)
JGI Community Sequencing Project
Moore Marine Microbial Project
NASA and NOAA Satellite Data
Community Microbial Metagenomics Data
Calit2 CAMERA ProductionCompute and Storage Complex is On-Line
Compute Nodes
1 and
10 Gb
it/sS
witch
ing
200 TB
File S
torag
e10 G
bit/s N
etwo
rk
Web
, Ap
plicatio
n, D
BS
ervers
Global Elements
• Data location – Storage Resource Broker Meta data catalog
• Data-type aggregation, cross-correlation, integration – BIRN Data Mediator
• Identity Management– Use Grid Security Infrastructure (GSI) Public Key
System – Integrated Grid Accounts Management Architecture
(GAMA) from SDSC for ease-of-use and Single Sign On
• Portal Services– Based on GridSphere – Small Dedicated Compute Cluster (32 nodes)
Cluster Nodes and File Servers
Logical Layout of Servers
Web Server
Portal Server
(Tomcat)
Single Sign-onServer
Postgres Database
GAMAServer
Blast Master (Jboss)
Cluster Frontend
Single Sign On Layer
Public Net
Private Net
An Incomplete List of Software Components
• Postgres Database• Apache Tomcat• Jboss Servlet Container• Google Web Toolkit• Sun Grid Engine• GAMA (Grid Accounting and Management Architecture)/GSI from Globus• OPAL (Grid/Web Services Wrapper)• GridSphere Portlet Container• CAMERA Registration Portal• Venter Application Portal• NCBI Blast, MPIBlast, ClustalW, MrBayes, CDHit, and host of other Bio
Software• Ergatis Workflow Engine• Jforums• Drupl• All Integrated with Rocks … Single Person Deployment
OptIPortal– Another Rocks Cluster Termination Device for the OptIPuter Global Backplane
• 20 Dual CPU Nodes, 20 24” Monitors, ~$50,000• 1/4 Teraflop, 5 Terabyte Storage, 45 Mega Pixels--Nice PC!• Scalable Adaptive Graphics Environment ( SAGE) Jason Leigh, EVL-UIC
Source: Phil Papadopoulos SDSC, Calit2
Use of OptIPortal to Interactively View Microbial Genome
Source: Raj Singh, UCSD
Acidobacteria bacterium Ellin345 (NCBI)Soil Bacterium 5.6 Mb
15,000 x 15,000 Pixels
Use of OptIPortal to Interactively View Microbial Genome
Source: Raj Singh, UCSDAcidobacteria bacterium Ellin345 (NCBI)
Soil Bacterium 5.6 Mb
15,000 x 15,000 Pixels
A Look at Networking
Introduction to QuartziteAn Experimental Network
Sunlight (10 Gigabit) Campus/WAN
Using a Lambda Network for CAMERA
• Many community databases – Protein Databank (PDB)
– GenBank
– SwissProt
• Support only web or web services interfaces– New analysis/programs need access to raw databases/files
– Usually, groups make a point-in-time copy of the database
– We call this a data “fork”– Updates are not processed– Papers published with point-in-time data out of date by months or
years• CAMERA “Direct Connect” will allow us to provide a high-speed connection
to the backend servers– Try to eliminate data forking
– Copies of CAMERA data is inevitable– Need mechanisms that allow others to keep their copies in synch with
CAMERA
UCSD Quartzite Core at Completion (Year 5 of OptIPuter)
QuartziteCore
CalREN-HPRResearch
Cloud
Campus Research Cloud
GigE Switch with Dual 10GigE Upliks
.....To cluster nodes
GigE Switch with Dual 10GigE Upliks
.....To cluster nodes
GigE Switch with Dual 10GigE Upliks
.....To cluster nodes
GigE
10GigE
...To othernodes
Quartzite Communications Core Year 3 (DWDM)
GlimmerGlass 128 port OOO
Juniper T320
4 GigE 4 pair fiber
Wavelength Selective
Switch(Lucent)
To 10GigE cluster node interfaces
..... To 10GigE cluster node interfaces and
other switches
Force10 E1200
32 10GigE
• Funded 15 Sep 2004
• Physical HW to Enable Optiputer and Other Campus Networking Research
• Hybrid Network Instrument
Reconfigurable Network and
Enpoints
25 | AT&T Labs, October 2007
4x4 Wavelength Cross-Connect:
• All integrated optics (except optical amplifiers)
– 4 1x4 WSS modules
– 4 4x1 passive optical combiners
• 4 x 40x 40Gbps = 6.4Tbps switching capacity
– currently using central 8
1x4 WSS
1x4 WSS
1x4 WSS
1x4 WSS
4x4 WXC rack
WSSs
combiners
OpticalAmps
26 | AT&T Labs, October 2007
WXC performance demonstration:
1x4 WSS
1x4 WSS
1x4 WSS
1x4 WSS
ASE source
4x1 switch OS
A8 lasers at centre of C-Band at 100GHz spacinguse ASE source to illustrate wide bandwidth1.use external 4x1 switch to scan WXC ports2.alter switch states of WSS1 and WSS3shown in movie on next page
WSS1
WSS2
WSS3
WSS4
1 1 2 3 42 2 3 4 13 3/1 4 1/3 24 4 1 2 35 1 2 3 46 2 3 4 11 3/1 4 1/3 28 4 1 2 3
27 | AT&T Labs, October 2007
WXC performance demonstration:
What Does it Cost to Drive the Network
• Dominant cost is DWDM optics
• Construction of Multiplexers is Simple, and not expensive ~ $250/Channel/End
Channel 31 Channel 32 Channel 33 Channel 34
10Gbps Switch X 4Per Side (optional)
XFP Switch Module X 4 Per Side (optional)
XFP DWDM Optics X 4 Per Side
Used in Host or Switch
SC to LC Fiber 2M X 5 Per Side
DWDM MuxTransmit X 1 Per
Side
DWDM DeMuxReceive X 1 Per Side
1 Fiber Pair
Corning 1U Rack Containing DWDM
Mux / DeMux + SC to SC couplers, 1 Per side
Layer 1 – Four Channel DWDM
1)OpticsSFP/XFP Optics Costs
DWDM Optics from AACTelecom
10Gbps Luminent XFP DWDM per unit (ZR 80Km) OC-192 and 10GE compatible
3500 US
10Gbps Luminent (assembled in US) XFP DWDM per Unit (ER 40Km) OC-192 and 10GE compatible
2900 US
1 Gbps SFP DWDM per Unit (80KM model)OC-48 compliantand 1 GE compatible
1220 US
10Gbps non-DWDM 1310nm (LR 10Km model)
700 US
10Gbps capable switch
SMC8748L2 (A0707505)+ EXP MOD-10G (A0707506) from Dell
Switch2 x 10Gbps XFP ports, 48 x 1Gbps Copper
1700 US
10 Gbps module (holds XFP)
300 US
2) Optional - Layer 2 Switch (10Gbps capable)
DWDM Mux DeMux (SC connector type)
4, 8 , 16 channel = DWDM-100From oemarket.com
4 Channel (31,32,33,34)
560 US
8 Channel 880 US
16 Channel 1600 (approx) US
3) DWDM Mux DeMux
Corning Mux DeMux container -1U rack mount
Corning PCH-01U from Ed Carlin Graybar
1 U (sufficient for 4, 8 or 16 channel)
200 US
2 sets of SC to SC adaptors
100 US (approx)
Fiber Patch Cables, Single Mode
From Ed Carlin Graybar
2M, SC to LC connector type
30 US (approx) each
4) Corning Rack Mount, Couplers, Fiber
Complete Solution
DWDM to Copper Media Converter
From Carl Stelling at Aaxeon.com
SFP pluggable DWDM to copper media converter
150 US each, not including DWDM optics (just converter)
5) Optional- DWDM Media Converter
Quartzite State Nov 2007
• Core Packet Switch with 68 10 GigE ports (More than ½ Terabit)• Approximately 30 Channels Lit• 64-port All-Optical Glimmerglass Switch - All Fiber into Quartzite is
switchable• 4 port x 8 Lambda DWDM switch at Lucent (On site at Calit2 in Dec)• 4 Channel DWDM Between Calit2 and SDSC
– One channel is used for 10Gigabit Production to BIRN Data Racks.
• Ordered, but waiting for fulfillment• 20 Mux/Demux (8 C-band DWDM Channels + 1 1310 (LR) Passband)• 32 DWDM XFPS (Channel 40-43 – will fill out rest of channels in 2008)