Tensor Contraction for Parsimonious Deep Netstensorlab.cms.caltech.edu/users/...presentation.pdf ·...

12
Tensor Contraction for Parsimonious Deep Nets Jean Kossaifi, Aran Khanna, Zachary C. Lipton, Tommaso Furlanello and Anima Anandkumar AMAZON AI

Transcript of Tensor Contraction for Parsimonious Deep Netstensorlab.cms.caltech.edu/users/...presentation.pdf ·...

Page 1: Tensor Contraction for Parsimonious Deep Netstensorlab.cms.caltech.edu/users/...presentation.pdf · •Potential space savings •Performance improvement. Tensor Contraction •Tensor

Tensor Contraction for Parsimonious Deep Nets

Jean Kossaifi, Aran Khanna, Zachary C. Lipton, TommasoFurlanello and Anima Anandkumar

AMAZON AI

Page 2: Tensor Contraction for Parsimonious Deep Netstensorlab.cms.caltech.edu/users/...presentation.pdf · •Potential space savings •Performance improvement. Tensor Contraction •Tensor

Traditionalapproaches

• DATA=>CONV=>RELU=>POOL=>Activationtensor• Flatteningloosesinformation• Canweleveragedirectlytheactivationtensorbeforetheflattening?• Potentialspacesavings• Performanceimprovement

Page 3: Tensor Contraction for Parsimonious Deep Netstensorlab.cms.caltech.edu/users/...presentation.pdf · •Potential space savings •Performance improvement. Tensor Contraction •Tensor

TensorContraction

• Tensorcontraction:contractalongeachmodetoobtainalowrank,compacttensor

Page 4: Tensor Contraction for Parsimonious Deep Netstensorlab.cms.caltech.edu/users/...presentation.pdf · •Potential space savings •Performance improvement. Tensor Contraction •Tensor

TensorContractionLayers(TCL)

• Takeactivationtensorasinput

• FeeditthroughaTCL

• Outputalow-rankactivationtensor

Page 5: Tensor Contraction for Parsimonious Deep Netstensorlab.cms.caltech.edu/users/...presentation.pdf · •Potential space savings •Performance improvement. Tensor Contraction •Tensor

TensorContractionLayers(TCL)

• Compactrepresentationlessparameters

• Measuredintermsofpercentageofspacesavingsinthefullyconnectedlayers:

1–𝑛𝑇𝐶𝐿

𝑛𝑜𝑟𝑖𝑔𝑖𝑛𝑎𝑙

• Similarandsometimesbetterperformance

Page 6: Tensor Contraction for Parsimonious Deep Netstensorlab.cms.caltech.edu/users/...presentation.pdf · •Potential space savings •Performance improvement. Tensor Contraction •Tensor

PerformanceoftheTCL

• Trainedend-to-end

• OnImageNetwithVGG:• 65.9%spacesavings• performancedropof0.6%only

• OnImageNetwithAlexNet:• 56.6%spacesavings• Performanceimprovementof0.5%

Page 7: Tensor Contraction for Parsimonious Deep Netstensorlab.cms.caltech.edu/users/...presentation.pdf · •Potential space savings •Performance improvement. Tensor Contraction •Tensor

Low-ranktensorregression

TensorRegressionNetworks,J.Kossaifi,Z.C.Lipton,A.Khanna,T.Furlanello andA.Anandkumar,ArXiv pre-publication

Page 8: Tensor Contraction for Parsimonious Deep Netstensorlab.cms.caltech.edu/users/...presentation.pdf · •Potential space savings •Performance improvement. Tensor Contraction •Tensor

Performanceandrank

Page 9: Tensor Contraction for Parsimonious Deep Netstensorlab.cms.caltech.edu/users/...presentation.pdf · •Potential space savings •Performance improvement. Tensor Contraction •Tensor

PerformanceoftheTRL

TRLrank Performance(in%)Top-1 Top-1 Spacesavings

baseline 77.1 93.4 0

(200,1,1,200) 77.1 93.2 68.2

(150,1,1,150) 76 92.9 76.6

(100,1,100) 74.6 91.7 84.6

(50,1,1,50) 73.6 91 92.4

ResultsonImageNetwithaResNet-101

• 92.4%spacesavings,4%decreaseinTop-1accuracy• 68.2%spacesavings,nodecreaseinTop-1accuracy

Page 10: Tensor Contraction for Parsimonious Deep Netstensorlab.cms.caltech.edu/users/...presentation.pdf · •Potential space savings •Performance improvement. Tensor Contraction •Tensor

Implementation

• MXNet asaDeepLearningframeworkhttp://mxnet.io/

• TensorLy fortensoroperationshttps://tensorly.github.io

• Comingsoon:mxnet backendforTensorLytensoroperationonGPUandCPU

Page 11: Tensor Contraction for Parsimonious Deep Netstensorlab.cms.caltech.edu/users/...presentation.pdf · •Potential space savings •Performance improvement. Tensor Contraction •Tensor

Conclusionandfuturework• TensorcontractionandtensorregressionforDeepNeuralNets• Addasanadditionallayerorreplaceone• Lessparameters• Similarorevenbetterperformance

Futurework• Fastertensoroperations• Exploremoretensoroperations/networksarchitectures

Page 12: Tensor Contraction for Parsimonious Deep Netstensorlab.cms.caltech.edu/users/...presentation.pdf · •Potential space savings •Performance improvement. Tensor Contraction •Tensor

Mahalo!

Anyquestion?

[email protected]@JeanKossaifi


Tensor Contraction Engine Manual2.3 Usage of the Algebraic Expression Engine AEE TheAEEisintendedtogeneratethealgebraicexpressionfromCoupled ClusterorManyBodyPerturbationTheoryusingonlyoperatorsandtheir

Tensor Contraction Engine Manual2.3 Usage of the Algebraic Expression Engine AEE TheAEEisintendedtogeneratethealgebraicexpressionfromCoupled ClusterorManyBodyPerturbationTheoryusingonlyoperatorsandtheir

High-Performance Tensor Contraction without BLASsc16.supercomputing.org/sc-archive/tech_poster/... · High-Performance Tensor Contraction without BLAS Tensor computations – in particular

High-Performance Tensor Contraction without BLASsc16.supercomputing.org/sc-archive/tech_poster/... · High-Performance Tensor Contraction without BLAS Tensor computations – in particular

Tensor contraction engine & extensible many-electron theory module in NWChem So Hirata Pacific Northwest National Laboratory MSS group meeting (24 Oct,

Tensor contraction engine & extensible many-electron theory module in NWChem So Hirata Pacific Northwest National Laboratory MSS group meeting (24 Oct,

Parsimonious Discovery of Synergistic Drug Combinations

Parsimonious Discovery of Synergistic Drug Combinations

Parsimonious Charge Deconvolution for Native Mass Spectrometry

Parsimonious Charge Deconvolution for Native Mass Spectrometry

Tensor Electric Assembly Tools - CloudCart · TENSOR ASSEMBLY TOOLS Tensor DL ... Tensor DS/DL Tensor SII/SL Tensor ST/STB Tensor STR ETO ETC FS HAD ETV ETP ETD 1 …

Tensor Electric Assembly Tools - CloudCart · TENSOR ASSEMBLY TOOLS Tensor DL ... Tensor DS/DL Tensor SII/SL Tensor ST/STB Tensor STR ETO ETC FS HAD ETV ETP ETD 1 …

Assessment of Human Skeletal Muscle Contraction and Force ... · Diffusion, Tensor, Skeletal Muscle, Contraction, Muscle Force 1. Introduction Diffusion tensor imaging (DTI) is a

Assessment of Human Skeletal Muscle Contraction and Force ... · Diffusion, Tensor, Skeletal Muscle, Contraction, Muscle Force 1. Introduction Diffusion tensor imaging (DTI) is a

Role of Tensors in Deep Learning - ANU College of ...users.cecs.anu.edu.au/~koniusz/tensors-cvpr17/present/anandkumar_anima... · Tensor Contraction Extends the notion of matrix product

Role of Tensors in Deep Learning - ANU College of ...users.cecs.anu.edu.au/~koniusz/tensors-cvpr17/present/anandkumar_anima... · Tensor Contraction Extends the notion of matrix product

1 03 - tensor calculus 03 - tensor calculus - tensor analysis.

1 03 - tensor calculus 03 - tensor calculus - tensor analysis.

Cyclops Tensor Framework: reducing communication and ...sor Framework (CTF), a distributed tensor contraction library. We expose the generality of this framework via an elegant in-terface

Cyclops Tensor Framework: reducing communication and ...sor Framework (CTF), a distributed tensor contraction library. We expose the generality of this framework via an elegant in-terface

Parsimonious models of precipitation phase derived from ...

Parsimonious models of precipitation phase derived from ...

Tensor Contraction with Extended BLAS Kernels on CPU and …users.wfu.edu › ballard › SIAM-ALA18 › shi.pdfWhy do we need tensor contraction? Tensor Contraction-Motivation h:

Tensor Contraction with Extended BLAS Kernels on CPU and …users.wfu.edu › ballard › SIAM-ALA18 › shi.pdfWhy do we need tensor contraction? Tensor Contraction-Motivation h:

Exploring the possibilities of parsimonious nitrogen ...

Exploring the possibilities of parsimonious nitrogen ...

tensor algebra - invariants 03 - tensor calculus - tensor ... · 03 - tensor calculus - tensor analysis tensor calculus 2 tensor algebra - invariants ¥ ... ¥ consider scalar,vector

tensor algebra - invariants 03 - tensor calculus - tensor ... · 03 - tensor calculus - tensor analysis tensor calculus 2 tensor algebra - invariants ¥ ... ¥ consider scalar,vector

Simulating the generation, evolution and fate of electronic … · 2013-10-08 · Anna Krylov: Efficient tensor contraction algorithms and codes for wave-function methods Development

Simulating the generation, evolution and fate of electronic … · 2013-10-08 · Anna Krylov: Efficient tensor contraction algorithms and codes for wave-function methods Development

Synthesizing Parsimonious Inexact Circuits through ...al4/lingamneni-tecs13a.pdf · 93 Synthesizing Parsimonious Inexact Circuits through Probabilistic Design Techniques AVINASH LINGAMNENI,

Synthesizing Parsimonious Inexact Circuits through ...al4/lingamneni-tecs13a.pdf · 93 Synthesizing Parsimonious Inexact Circuits through Probabilistic Design Techniques AVINASH LINGAMNENI,

Tensor fields - sorbonne-universite · EXAMPLES OF APPLICATIONS • Gradient tensor • Stress tensor • Diffusion tensor • Metric/Curvature tensor • Structure tensor 6. REPRESENTATION

Tensor fields - sorbonne-universite · EXAMPLES OF APPLICATIONS • Gradient tensor • Stress tensor • Diffusion tensor • Metric/Curvature tensor • Structure tensor 6. REPRESENTATION

Parsimonious Tensor Response Regressionani.stat.fsu.edu/~henry/TensorResponseEnv.pdfmer, the tensor predictor regression, focuses on understanding the change of a clinical outcome

Parsimonious Tensor Response Regressionani.stat.fsu.edu/~henry/TensorResponseEnv.pdfmer, the tensor predictor regression, focuses on understanding the change of a clinical outcome

MitochondrialCalciumUptakeRegulatesRapidCalcium TransientsinSkeletalMuscleduringExcitation-Contraction … · MitochondrialCalciumUptakeRegulatesRapidCalcium TransientsinSkeletalMuscleduringExcitation-Contraction

MitochondrialCalciumUptakeRegulatesRapidCalcium TransientsinSkeletalMuscleduringExcitation-Contraction … · MitochondrialCalciumUptakeRegulatesRapidCalcium TransientsinSkeletalMuscleduringExcitation-Contraction

Using MLIR to Optimize Tensor Contractionsmhall/mlir4hpc/lcpc-MLIR-saday.pdf · ♦Tensor contraction: high-dimension analog of matrix multiplication §Direct implementation as above

Using MLIR to Optimize Tensor Contractionsmhall/mlir4hpc/lcpc-MLIR-saday.pdf · ♦Tensor contraction: high-dimension analog of matrix multiplication §Direct implementation as above