www.nottingham.ac.uk/adac

Obtaining, QC, mapping and analysis of NGS data. Richard Emes Associate Professor & Reader in Bioinformatics. School of Veterinary Medicine and Science Director Advanced Data Analysis Centre richard.emes@nottingham.ac.uk www.nottingham.ac.uk/adac @rdemes @ADAC_UoN

2

What is ADAC? The University of Nottingham Advanced Data Analysis Centre (ADAC). •  Bioinformatics and data analysis support. Why is this important? •  Complex data underpins much current research. •  Innovative analysis can prompt new discoveries. •  Excellent research can often be stalled due to a lack of expertise in conducting data

analysis, availability or cost of inclusion of diverse specialists.

Why ADAC? •  ADAC supports high-class research by providing analysts with expertise in a range of

bioinformatics and computer science disciplines. •  Flexible support

•  Consultancy, collaboration, bespoke analysis. •  Leadership from recognized experts in the fields of bioinformatics and computer

science. •  Track record of funding

•  Pivotal role in collaborations funded by amongst others Zoetis, BBSRC, NERC and Technology Strategy Board.

•  ADAC is conducting transcriptome analysis for a multinational FP7 funded project (EU Prohealth).

http://www.nottingham.ac.uk/adac/ Enquiries: adac@nottingham.ac.uk or richard.emes@nottingham.ac.uk @ADAC_UoN or @rdemes

Current Areas of expertise relevant here:

•  Transcriptomics (Microarray, NGS)

•  Comparative genomics (eukaryotic, prokaryotic)

•  Identification of biomarkers from genetic and epigenetic datasets

•  Artificial intelligence for decision support

•  Machine Learning

•  Integration of complex datasets

•  Data Management

@HWI-_FC_20BTNAAXX:2:1:215:593 ACAGTGCATGACATGCATAGCAGCATAGACTAC +HWI-_FC_20BTNAAXX:2:1:215:593 GhhhhhhhhhhhUhhEGhhhGhhhhhhhhhhhhh

Some common terms

•  Library: collection of molecules. This is the “complexity” of what you sequence. •  Flowcell: slide where sequencing is attached to a solid platform. •  Lane: unique sequencing unit of the flowcell. •  Reads: Raw sequence of bases and imputed quality scores. •  Fragment: Original molecule being sequenced (fragment of genome/gene). Ie

PE are reads form the same fragment. •  Cluster: DNA bound to slide, local amplification of product to amplify signal to

measure fluorescence. •  Mapping: Finds where your sequence matches to a reference. Importantly

gives a probability that this is the correct location.

Illumina sequencing

Illumina sequencing

What coverage do I need?

Obtaining NGS Data •  Short Read Archive (SRA) •  European Nucleotide Archive (ENA)

Obtaining NGS Data •  Short Read Archive (SRA) •  Will need to convert to FastQ using sra toolkit

Deciphering a fastq file @HWI-_FC_20BTNAAXX:2:1:215:593#0/1 ACAGTGCATGACATGCATAGCAGCATAGACTAC +HWI-_FC_20BTNAAXX:2:1:215:593#0/1 GhhhhhhhhhhhUhhEGhhhGhhhhhhhhhhhhh Header: @HWI-_FC_20BTNAAXX:2:1:215:593#0/1 HWI-_FC_20BTNAAXX instrument identifier 2 flowcell lane 1 tile number in flowcell lane 215 x - coordinate of cluster in the tile 593 y - coordinate of cluster in the tile #0 index of multiplexed samples /1 member of pair /1 or /2 if Paired end

Deciphering a fastq file

@HWI-_FC_20BTNAAXX:2:1:215:593#0/1 ACAGTGCATGACATGCATAGCAGCATAGACTAC +HWI-_FC_20BTNAAXX:2:1:215:593#0/1 GhhhhhhhhhhhUhhEGhhhGhhhhhhhhhhhhh Sequence: ACAGTGCATGACATGCATAGCAGCATAGACTAC Quality Header: +HWI-_FC_20BTNAAXX:2:1:215:593 Quality: GhhhhhhhhhhhUhhEGhhhGhhhhhhhhhhhhh

Deciphering a fastq file Quality: GhhhhhhhhhhhUhhEGhhhGhhhhhhhhhhhhh Sanger encoding = ASCII table lookup – 33 Solexa encoding = ASCII table lookup – 64 G = 71 – 64 = 7 h = 104- 64 = 40

Deciphering a fastq file Quality: GhhhhhhhhhhhUhhEGhhhGhhhhhhhhhhhhh

SNP Calling •  Genotyping: identifying variants in a single genome

(i.e. from each parent) •  SNP Calling: identifying variants between individual genomes

ACGTGCAGCATAGCA?CGACATCGACATACGC TGCACGTCGTATCGT?GCTGTAGCTGTATGCG ****A******* ***A***** **T****** *****T********** ******T******** ACGTGCAGCATAGCATCGACATCGACATACGC TGCACGTCGTATCGTAGCTGTAGCTGTATGCG

Sample Genome(s) Reads Reference Genome

INDEL Calling •  INDEL: Insertion/deletion

ACGTGCAGCATAGCA???CGACATCGACATACGC TGCACGTCGTATCGT???GCTGTAGCTGTATGCG ****ACG******* ***ACG***** **---******** *****---********** ******---******** ACGTGCAGCATAGCACGTCGACATCGACATACGC TGCACGTCGTATCGTGCAGCTGTAGCTGTATGCG

Sample Genome(s) Reads Reference Genome

A pipeline for SNP identification

•  Quality Control –  FastQC, Fastx toolkit

•  Trimming: –  Sickle, Trimgalore, Trimomatic, Cutadapt

•  Mapping –  BWA, Bowtie, Stampy

•  Remove Duplicates –  Picard tools, Samtools rmdup

•  Call SNPs / INDELS –  Samtoools mpileup, VarScan, GATK, Many

others!

Galaxy: https://usegalaxy.org/

Toolbox Workflows

Visualize the data •  FASTQC •  Stand Alone or non-interactive

–  Basic Statistics module, includes: •  Filename: The original filename of the file which was analyzed •  Encoding: Says which ASCII encoding of quality values was found in this file. •  Total Sequences: A count of the total number of sequences processed. •  Sequence Length: Provides the length of the shortest and longest sequence in

the set. If all sequences are the same length only one value is reported.

Visualize the data •  FASTQC: Per Base Sequence Quality:

Red line = Median quality Yellow box = IQR Whiskers = 10%-90% Blue line = Mean quality If the lower quartile for any base is less than 10, or if the median for any base is less than 25. If the lower quartile for any base is less than 5 or if the median for any base is less than 20.

Visualize the data •  FASTQC: Per Sequence Quality Scores:

If the most frequently observed mean quality is below 27 - this equates to a 0.2% error rate. If the most frequently observed mean quality is below 20 - this equates to a 1% error rate.

Visualize the data •  FASTQC: Per Base Sequence Content

Proportion of each base position in a file for which ATCG DNA bases has been called. If the difference between A and T, or G and C is greater than 10% in any position. If the difference between A and T, or G and C is greater than 20% in any position. Possibly adapters or affect of trimming.

Visualize the data •  FASTQC: Sequence Length Distribution

Distribution of fragment sizes in the file. If all sequences are not the same length. If any of the sequences have zero length.

Visualize the data •  FASTQC: Duplicate Sequences

Degree of duplication within first 200,000 reads of file. Distribution of duplication levels in dataset If non-unique sequences make up more than 20% of the total. If non-unique sequences make up more than 50% of the total.

Visualize the data •  FASTQC: Overrepresented Sequences

•  FASTQC: Adapter Content

Lists all of the sequences which make up more than 0.1% of the total. If any sequence is found to represent more than 0.1% of the total. If any sequence is found to represent more than 1% of the total. To know if your library contains a significant amount of adapter in order to be able to assess whether you need to adapter trim or not. If any sequence is present in more than 5% of all reads. If any sequence is present in more than 10% of all reads.

Cut adapters

-f = the type of file (in this case fastq) -q CUTOFF, Trim low-quality ends from reads before adapter removal. -a ADAPTER, Sequence of an adapter that was ligated to the 3' end. The adapter itself and anything that follows is trimmed. -m 100 minimum length of reads following adapter removal. Reads less than 100 will be discarded --discard-untrimmed any reads without an adapter will be discarded. -o output file also in fastq format.

cutadapt -f fastq -q 20 -a AGATCGGAAGAG -m 100 --discard-untrimmed -o SNP.test.trimmed.fastq SNP.test.fastq

Cut adapters Galaxy: Fastx_clipper from fastx_toolkit

Quality filters

Quality filters

Fastx_toolkit -q = Minimum quality score to keep. -p = Minimum percent of bases in a read that must have [-q] quality. - i input file (output of adapter trimming step) -v verbose -Q quality encoding -o output file also in fastq format.

fastq_quality_filter -q 20 -p 70 -i SNP.test.trimmed.fastq -v –Q64 -o SNP.test.trimmed.QC.fastq

Quality filters •  Galaxy: filter by quality

Align to genome

The problem •  Generally a large genome (Human > 3Gb) •  Large number of short reads The solution •  Index genome into hash of kmers or short sequences •  Use efficient aligners

–  Large number of aligners available. •  Common aligners: Bowtie1/2, BWA, Stampy

Align to genome Example Bowtie 2 alignment Build index -f fasta formatted genome file ./bowtie.index.files/chr17 output location for index files Galaxy: select pre-built index when using bowtie or BWA

bowtie2-build -f ./genome/chr17.fa ./bowtie.index.files/chr17

Align to genome

Align to reference -p number of processors --end-to-end alignment is not local -k 1 number of positions read is allowed to align k = 1 means all non-uniquely mapping reads are discarded -x path to indexed genome file to align reads to. -U reads are unpaired (in this case -S output in SAM format

bowtie2 -p 4 --end-to-end -k 1 -x ./bowtie.index.files/chr17 -U SNP.test.trimmed.QC.fastq -S SNP.test.trimmed.QC.fastq.sam

Alignment file formats SAM - Sequence Alignment/Map format. (BAM is a binary compressed equivalent)

–  TAB-delimited text format –  header section (optional)

@HD = Header, VN[format version], SO[sorting order] @SQ = Reference sequence line @PG = Program, ID[program ID], CL[command line]

@HD VN:1.0 SO:unsorted @SQ SN:chr17 LN:81195210 @PG ID:bowtie2 PN:bowtie2 VN:2.2.1 CL:"/home/rde/tools/bowtie2-2.2.1/bowtie2-align-s --wrapper basic-0 -p 4 --end-to-end -k 1 -x ./bowtie.index.files/chr17 -S SNP.test.trimmed.QC.fastq.sam -U SNP.test.trimmed.QC.fastq”

Alignment file formats •  SAM - Sequence Alignment/Map format.

–  TAB-delimited text format –  alignment section mandatory 11 columns –  Optional fields –  For details http://samtools.github.io/hts-specs/SAMv1.pdf –  Samflags http://picard.sourceforge.net/explain-flags.html

instrument_name:100:flowcellID:1:1:32575:625 0

chr17 72387574 255 141M * 0 0

GGAAGAGCTGGGACCAGGCCCAGCAATTACCTCACCATGGTTGGGTGCAACCAAGTGGGGCAACTCTTTGGCCAGAAAGCAAAAGTCTTTTTAGCTTCAATGTAGGCCATTCTGGGTCCCAGACCCACAGCTTTGGACATT

Dgga^h`hedac`hcc`c^_f`ChggeDfC`e`_h_ddgd^_`be^bcg^acchC`^^bCdagfbgf_`^dc^^cD`dbha^Dbc`fb^d`CdaD^``dDb_edffdDDh_chceabh`heecd_h`gb_b^Ch`Dd_^c` AS:i:0 XN:i:0 XM:i:0 XO:i:0 XG:i:0 NM:i:0 MD:Z:141 YT:Z:UU

1  Query template 2  Bitwise flag 3  Reference Name 4  1-based leftmost position 5  Mapping Quality 6  CIGAR string 7  Reference Name of mate 8  Position of mate 9  Observed template length 10  Sequence 11  Phred-scaled Quality

Remove duplicates

•  Duplicate reads generated as artifact in the library generation step results in false confidence in variants.

•  samtools [rmdup], Picard tools [MarkDuplicates]

samtools view -bS SNP.test.trimmed.QC.fastq.sam -o SNP.test.trimmed.QC.fastq.bam samtools rmdup -s SNP.test.trimmed.QC.fastq.bam SNP.test.trimmed.QC.fastq.rmdup.bam

Remove duplicates

Call Variants •  Identify regions in alignment where sequence differs

****A******* ***A***** **T****** *****T********** ******T******** ACGTGCAGCATAGCATCGACATCGACATACGC

Reads Reference Genome

Call Variants samtools sort SNP.test.trimmed.QC.fastq.rmdup.bam SNP.test.trimmed.QC.fastq.rmdup.sorted samtools index SNP.test.trimmed.QC.fastq.rmdup.sorted.bam samtools faidx ./genome/chr17.fa samtools mpileup -f ./genome/chr17.fa SNP.test.trimmed.QC.fastq.rmdup.sorted.bam | java -jar ./tool/VarScan.v2.3.6.jar mpileup2snp --output-vcf –strand-filter 0

samtools mpileup -f faidx indexed reference sequence file VarScan mpileup2snp or mpileup2indel --min-coverage Minimum read depth at a position to make a call [8] --min-reads2 Minimum supporting reads at a position to call variants [2] --min-avg-qual Minimum base quality at a position to count a read [15] --min-var-freq Minimum variant allele frequency threshold [0.01] --strand-filter Ignore variants with >90% support on one strand [1]

Call Variants

Call Variants

VCF Variant Call Format file

•  Header text marked with ## •  Column headings marked with # •  Mandatory columns

–  CHROM Chromosome –  POS Position of variant start –  ID Unique variant ID –  REF Reference Allele –  ALT Alternate non-reference alleles (comma separated) –  QUAL Phred quality score –  FILTER Filtering information –  INFO User annotation

Visualize: IGV

So Many SNPS – So What? •  Get gene •  Functional Analysis to identify key candidates

Identify Homologues

Locate variants

Identify Ontologies

Pathway and interaction

analysis

Locate SNPs on structure

Compare to current data

•  Functional Analysis to identify key candidates A step by step example (don’t do this with lots of variants!) •  Get gene •  Modify bases as shown in VCF file. •  BLASTx to identify reading frame. •  Produce mRNA, and encoded peptide sequence fasta files (provided) •  Determine variant positions in mRNA & peptide sequences

Compare to current data

•  dbSNP –  SNP already known?

•  Repositories such as Ensembl, UCSC –  In splice variant? –  In known regions, domain etc?

•  Variant effect predictor (more later)

•  Visualize the position in genome – Ensembl/UCSC –  Add custom track using GFF/bed file

•  Coding/non-coding –  Synonymous / non-synonymous –  Codon usage

•  Locate in relation to known domains –  Pfam –  SMART

•  Repeat regions

Locate variants

•  For single genes –  Search for available information PubMed, interpro

etc •  For multiple genes

–  BLAST2GO –  DAVID

Identify Ontologies

•  Pathway analysis –  Understand the process of your gene. Does it make

biological sense? –  IPA –  DAVID –  Webgestalt

•  Interaction analysis –  BioGRID –  STRING –  PSICQUIC

Pathway and interaction

analysis

•  Structure prediction –  BLAST of PDB –  Predict structure

•  PSIPRED – 2° structure •  ITASSER – 3° structure •  Phyre – 3° structure

–  Locate in 3D •  Swiss PDB viewer

Locate SNPs on structure

•  Predict effect of SNPs •  Suspect •  VEP

Locate SNPs on structure

Arg Ser

Identify Homologues

Emes R.D. Inferring function from homology. in Methods in Molecular Biology 453: Humana Press 2008.

Links and websites 1 •  Many at http://emeslab.wordpress.com/useful-links/ •  SRA: http://www.ncbi.nlm.nih.gov/sra •  SRA toolkit http://eutils.ncbi.nih.gov/Traces/sra/?view=software •  ENA: http://www.ebi.ac.uk/ena/ •  Galaxy: https://usegalaxy.org/ •  FASTQC: www.bioinformatics.babraham.ac.uk/projects/fastqc/ •  Fastx_toolkit: http://hannonlab.cshl.edu/fastx_toolkit/ •  Bowtie 1: http://bowtie-bio.sourceforge.net/index.shtml •  Bowtie 2: http://bowtie-bio.sourceforge.net/bowtie2/index.shtml •  BWA: http://bio-bwa.sourceforge.net/ •  Stampy: http://www.well.ox.ac.uk/project-stampy •  SAMtools: http://samtools.sourceforge.net •  PicardTools: http://picard.sourceforge.net •  VarScan: http://varscan.sourceforge.net •  IGV: https://www.broadinstitute.org/software/igv/

Links and websites 2 •  dbSNP: http://www.ncbi.nlm.nih.gov/SNP/ •  Ensembl: http://www.ensembl.org/index.html •  UCSC Genome Browser: https://genome.ucsc.edu/ •  Pfam: http://pfam.xfam.org/ •  SMART: http://smart.embl-heidelberg.de/ •  GO: http://www.geneontology.org/ •  BLAST2GO: http://www.blast2go.com/b2ghome •  IPA: http://www.ingenuity.com/products/login •  DAVID: http://david.abcc.ncifcrf.gov/ •  Webgestalt: http://bioinfo.vanderbilt.edu/webgestalt/ •  BioGRID: http://thebiogrid.org/ •  PSICQUIC: http://www.ebi.ac.uk/Tools/webservices/psicquic/view/main.xhtml •  ITASSER: http://zhanglab.ccmb.med.umich.edu/I-TASSER/ •  Phyre: http://www.sbg.bio.ic.ac.uk/phyre2/ •  SuSPect: http://www.sbg.bio.ic.ac.uk/~suspect/ •  PSIPRED: http://bioinf.cs.ucl.ac.uk/psipred/ •  SwissPDB: http://spdbv.vital-it.ch/

Richard Emes Associate Professor & Reader in Bioinformatics. School of Veterinary Medicine and Science Director Advanced Data Analysis Centre richard.emes@nottingham.ac.uk www.nottingham.ac.uk/adac @rdemes @ADAC_UoN

61