TSpace Research Repository tspace.library.utoronto.ca

Deep Learning with Microfluidics for

Biotechnology

Jason Riordon, Dušan Sovilj, Scott Sanner, David Sinton, and Edmond W. K. Young

Version Post-print/Accepted Manuscript

Citation (published version)

Riordon, Jason, et al. ""Deep Learning with Microfluidics for Biotechnology."" Trends in Biotechnology (2018).

Copyright/License This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0

International License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.

How to cite TSpace items

Always cite the published version, so the author(s) will receive recognition through services that track citation counts, e.g. Scopus. If you need to cite the page number of the author manuscript from TSpace

because you cannot access the published version, then cite the TSpace version in addition to the published version using the permanent URI (handle) found on the record page.

This article was made openly accessible by U of T Faculty. Please tell us how this access benefits you. Your story matters.

1

Deep Learning with Microfluidics for Biotechnology

Jason Riordon, Dušan Sovilj, Scott Sanner1, David Sinton2 & Edmond W. K. Young3*

Department of Mechanical and Industrial Engineering, University of Toronto, 5 King’s College

Road, Toronto, ON, Canada, M5S 3G8

*Correspondence: eyoung@mie.utoronto.ca (E.W.K. Young)

1http://d3m.mie.utoronto.ca/ 2 http://www.sintonlab.com/ 3 http://ibmt.mie.utoronto.ca/

Keywords: Deep Learning, Machine Learning, Microfluidics, Lab-on-a-Chip

Abstract: Advances in high-throughput and multiplexed microfluidics have rewarded

biotechnology researchers with vast amounts of data. However, our ability to analyze complex

data effectively has lagged. Over the last few years, deep artificial neural networks leveraging

modern graphics processing units have enabled the rapid analysis of structured input data -

sequences, images, videos - to predict complex outputs with unprecedented accuracy. While

there have been early successes in flow cytometry, for example, the extensive potential of pairing

microfluidics (to acquire data) and deep learning (to analyze data) to tackle biotechnology

challenges remains largely untapped. Here we provide a roadmap to integrating deep learning

and microfluidics in biotechnology labs that matches computational architectures to problem

types, and provide an outlook on emerging opportunities.

2

The Challenge of Processing Data from Microfluidics

Over the past two decades, microfluidics (see Glossary) has shown great promise in enhancing

biotechnology applications by leveraging small sample volumes, short reaction times,

parallelization and fluid manipulation at sample-relevant scales [1]. Major milestones in the field

are indicated in Figure 1. By demonstrating an ability to efficiently perform a wide range of

functions within biotechnology laboratories, including DNA and RNA sequencing [2,3], single-

cell omics [4,5], antimicrobial resistance screening [6] and drug discovery [7], microfluidic

technologies have revolutionized the way we approach experimental biology and biomedical

research. Microfluidic technologies have also demonstrated an ability to capture, align, and

manipulate single cells for cell sorting and flow-based cytometry [8–10], mass [11] and volume

[12] sensing, phenotyping [13], cell fusion [14], cell capture (e.g., circulating tumor cells

[15,16]), and cell movement (e.g., sperm motility [17–20]). Microfluidic high-throughput

screening using droplets benefit from rapid reaction times, high sensitivity and low cost of

reagents [21]. Microfluidics has also enabled the capture and monitoring of zebrafish embryos

[22,23], the high-throughput and rapid imaging of the nematode worm Caenorhabditis elegans

[24], and the ordering and orientation of Drosophila embryos [25].

While microfluidic applications in biotechnology vary widely, the real product in all cases is

data. For example, a typical time-lapse experiment can readily generate >100 GB of data (e.g.,

100 cells × 4 images/cell (1 brightfield + 3 fluorescence bands) × 6 time points/hr × 48 hrs ×

1 MB/image). However, the generation of data using high-throughput, highly parallelized

microfluidic systems, has far outpaced researchers’ abilities to effectively process this data – an

analysis bottleneck.

3

Machine learning (Box 1) is a class of artificial intelligence-based methods that enable

algorithms to learn without direct programming. While traditional machine learning has long

offered data processing capabilities, the advent of deep learning methods represents a step-

increase in the ability to handle structured data such as images or sequences. Deep learning

architectures can now exploit structured data applicable to a variety of research fields [26,27],

and leverage inherent data structures. Deep learning has achieved impressive recent gains in

analyzing a variety of data types: images [28–31], natural language translation [32–34], speech

data [35,36], text documents [37,38] and computational biology [39,40]. These major gains have

been powered by increased computational power from GPUs, open-source frameworks (e.g.,

TensorFlow), and distributed computing.

Traditional machine learning has already been paired with microfluidics for biotechnology

applications, for example in disease detection within liquid biopsies as comprehensively

reviewed by Ko and colleagues [41]. This marriage between traditional machine learning and

microfluidics has yielded (i) improvements in analysis methods including single-cell lipid

screening [42], cancer screening [43,44] and cell counting [45], and (ii) advances in microfluidic

design and flow modeling including predicting water-in-oil emulsion sizes [46]. Applications

that combine deep learning with microfluidics, however, are only beginning to emerge, with

label-free cell classification as a prime example. Cells have been identified using either deep

learning architectures that process pre-extracted features [47] or use raw images as inputs and

fully leverage a deep network’s ability to extract relevant features for improved prediction [48–

50]. Deep learning methods have also led to well-defined flows by tuning channel geometry [51].

While these deep learning demonstrations indicate potential, we see far broader biotechnology

applicability in the near future.

4

In this perspective we outline the many opportunities presented by the marriage of deep

learning (to analyze data) and microfluidics (to generate data) for biotechnology applications.

We provide a roadmap to integrating deep learning strategies and microfluidics applications,

carefully pairing problem types to artificial neural network architectures - Figure 2 (Key

Figure). We begin with advances in processing simple unstructured data for which there is

much precedence in biotechnology, and transition to complex sequential and image data for

which there are abundant opportunities. We provide practical implementation guidelines for

biotechnology researchers and research leaders new to deep learning, including a summary of

how neural networks function and tips on getting started (Box 2). We end with an outlook on

emerging opportunities that will have a profound impact on biotechnology research in the near

and medium term.

Deep Learning Architectures for Microfluidic Challenges

Deep neural architectures have been developed and applied to a wide range of challenges,

including single-molecule science [52], computational biology [40], and biomedicine [53,54].

Here we highlight strategies that have successfully been applied within the microfluidics

community, and where such approaches could be applied in the near future. Most relevant

architecture categories (input-to-output data types) are listed alongside existing microfluidic

applications in Figure 2. We progress from the simple unstructured-to-unstructured case to the

more complex image-to-image combination, and highlight microfluidic achievements and

opportunities.

Unstructured-to-Unstructured Neural Networks – e.g., for Classifying Cells Based on

Manually-Extracted Cell Traits

5

Unstructured-to-unstructured neural networks represent the simplest neural network case – a

type of architecture that typically falls into the traditional machine learning category, but where

deep learning can often be beneficial. In a typical biotechnology application, a neural network

designed to handle unstructured inputs could be used to classify flowing cells within a

microfluidic channel, where the input x would be a vector of cell traits (e.g., circularity,

perimeter and major axis length) and the output y would be a class (e.g., white blood cell or

colon cancer cell) as in the label-free cell segmentation and classification example in Figure 3A

by Chen and colleagues (simplified – only three traits are shown for clarity). “Label-free” here

refers to physical labels such as fluorophores rather than feature labels as used in machine

learning terminology. The authors used time-stretch quantitative phase microscopy [47] to create

a rich hyperdimensional feature space of cell traits, trained their network using supervised

learning, and achieved an accuracy and consistency in cell classification with their deep neural

network surpassing that of traditional machine learning approaches, including logistic regression,

naive Bayes and support vector machines.

Whereas the unstructured-to-unstructured example above utilized a deep network, most

microfluidic work using simple unstructured-to-unstructured networks have not necessitated

deep learning. Various imaging modalities (e.g., light microscopy [55], digital holographic

microscopy [44]) have been used to image individual cells, and various cell features (e.g., cell

size, perimeter, eccentricity, image intensities [56]) have been used as inputs in supervised

learning algorithms to achieve, for example, label-free cell classification [44]. Traditional

machine learning algorithms have also been used to classify and identify disease biomarkers

[57]. An excellent example of unstructured input (i.e., cell features extracted from images) to

unstructured output is from the Lu group at Georgia Tech, who used microfluidic capture arrays

6

to position and orient C. elegans for imaging synaptic puncta patterns within the organism [58].

The authors discovered highly subtle but important phenotypic differences between worms that

revealed genetic differences that were previously hidden. For additional biotechnology examples

and applications of machine learning, we refer the reader to an excellent review by Vasilevich

and colleagues [59].

Sequence-to-Unstructured Neural Networks – e.g., for Microfluidic Soft Sensor

Characterization

Sequential data is prevalent in microfluidics – from measuring an electrical signal to

characterizing a droplet generator, microfluidic measurements are often produced as part of a

time series. When data has a sequential structure, there are alternative deep network architectures

that can better reflect and exploit the sequences. These deep neural networks are generally

referred to as recurrent neural networks (RNNs) and are shown in Figure 3B-C. The first case

of interest is sequence-to-unstructured, whereby an input sequence is assigned a single output

value (Figure 3B). In this architecture, recurrent weights connect the hidden layer to itself and

permit training through a gradient descent technique known as backpropagation-through-time

[60]. The example shown in Figure 3B is reproduced (with simplification) from Han and

colleagues [61], where a RNN was used to characterize a microfluidic soft sensor. An analog

voltage was measured over time at the ends of a microchannel filled with a liquid metal. The

signal was monitored as a pressure stimulus was applied at various locations along the channel.

The RNN was trained to not only identify the pressure applied, but also discern the location of

stimulus along the channel.

Sequence-to-Sequence Neural Networks – e.g., for DNA Base Calling

7

Another case of interest is sequence-to-sequence (Figure 3C). Here, rather than classify an entire

sequence of events as corresponding to a given class, the output itself is a sequence - for example

in DNA base calling applications where current events are identified in order [62]. Boža and

colleagues demonstrated an open-source DNA base caller with a RNN structure to segment

current change events from MinION nanopore data into DNA base pairs [62]. Such an

architecture also has great potential in applications where the growth of a cell (via either volume

[12] or mass [63]) is monitored over extended periods, and any individual size measurement

would likely benefit from taking previous measurements in consideration. Given the strong

correlation between pulses (i.e. a cell that is growing), using an RNN would likely substantially

increase measurement accuracy. In tagging problems such as this example, each element of the

input sequence is annotated, i.e. every pulse amplitude corresponds to the passage of a cell, or

each current event corresponds to a certain base pair. Architectures have also been developed for

cases where the output sequence is not the same length as the input sequence, for example in

language translation (where an output sentence need not have the same number of words as an

input text). To tackle this challenge, a more powerful architecture is employed [64], namely the

encoder-decoder model. In this approach, there are two separate RNNs for two distinct phases.

The first network is tasked with reading the input sequence and producing an encoded state that

is a concise lower-dimensional representation of the same sequence, namely the encoder part.

The decoder RNN is then modified to have outputs passed further down the sequence. This

encoder-decoder architecture is now the backbone of most sequence-to-sequence learning

models in language translation and speech-to-text transcription, and has begun to impact

biomedical research, as evidenced by recent work on predicting DNA-binding proteins from

primary sequences [65]. Since training RNNs is computationally expensive, a significant effort

8

has gone into developing more efficient structures and adapting them to the temporal domain.

Recently, several works have explored using CNN-based sequence-to-sequence learning [66]

and reported comparable or improved results over RNN-based models with a substantial

decrease in training times.

As identified in a recent review by Angerer and colleagues, single-cell transcriptomics

has thus far relied on simpler models, but stands to benefit significantly from deep learning

approaches [67]. Deng and colleagues recently demonstrated scScope, a deep learning

architecture capable of identifying cell-type composition from single-cell RNA sequence gene-

expression profiles, leveraging a recurrent network structure [68]. Angermueller and colleagues

developed DeepCpG, a deep learning architecture capable of predicting single-cell methylation

states, here using a bidirectional gated recurrent network, a variant of the RNN [40]. Not all

networks dealing with genomics data rely on RNNs: DeepVariant, a network that first translates

genomic data into images, was developed by researchers at Verily to analyse genetic variability

within genomes using CNNs [69]. For further reading on how deep learning stands to transform

genetics and biological networks, we refer the reader to a thoughtful review by Camacho and

colleagues [70]. We also refer the reader to an excellent review on the multi-omics of single cells

by Bock and colleagues [71].

Image-to-Unstructured Neural Networks – e.g., for Classifying Cells Based on Images

Directly

Deep networks have also been developed to handle spatially-distributed data, or images – useful

for example in classifying cells directly, without requiring prior manual extraction of traits.

Image analysis is key to most microfluidic experiments – multiplexing, rapid throughput and the

9

planar geometry of many microfluidic networks lead to the production of vast quantities of

images. Image-to-unstructured neural networks offer the promise of handling such data directly,

without requiring time-consuming pre-extraction of relevant features. Further, by embedding

feature extraction layers within a deep network, the selection process is no longer subject to

human biases. Thus, it is not only a deep network’s ability to accelerate classification tasks (by

incorporating the feature-extraction step), but its ability to predict what features are relevant that

make deep learning approaches so powerful.

Convolutional neural networks (CNNs) are the backbone of all image-based deep

learning (Figure 4). Their architecture is such that nodes are restricted to connect to a portion of

the input image. Convolutional blocks (or filters shown as light green layers, Figure 4) operate

on all parts of an image by sliding a small window region along an image, outputting the

weighted sum of pixel values for that filter within the region, and applying a nonlinear

transformation. These convolutional layers can be combined with pooling (subsampling) layers,

which extract the most dominant values in the feature maps and reduce their resolution (dark

green layers, Figure 4). The process is repeated several times until a specific filter map

resolution size is achieved. In summary, initial stages of the pipeline are designed to tackle

spatial data, and convolutions act as special feature detectors (such as edges or lines) [72],

thereby teaching the network to associate proximate pixels in space. The output of these layers is

then a low-dimensional embedded representation of an image that constitutes a far better

representation of the image content than other feature-extraction methods [72,73]. For image-to-

unstructured applications specifically, for example in classification tasks, a fully connected stack

of layers is used at the end of the network (Figure 4A). Such an architecture has been applied in

several label-free microfluidic cell cytometry applications [48–50]. In the flow cytometry

10

example by Heo and colleagues (Figure 4A), a CNN is used to classify a binary population of

lymphocytes and red blood cells at high-throughput [48]. In the second example, Stoecklein and

colleagues show how precise flow profiles can be generated using clever pillar distributions

along the channel. Interestingly, solving the “what flow shape will result from a given

geometry?” problem is easily solved by a computational flow model, whereas deep learning can

be used to solve the much more difficult and inverse problem, “what geometry is required to

produce a desired flow shape?” [51]. Recently, an image-to-unstructured architecture has also

been used to quantify bacterial growth within microfluidic culture systems [74]. Kim and

colleagues used a CNN to calculate the concentration of Pseudomonas aeruginosa within on-

chip microfluidic cultures using only culture images as inputs.

Image-to-Image Neural Networks – e.g., for Cell Segmentation

Image-to-image applications include predicting the next frame in a video, predicting the depth of

an image, or most notably segmenting images – where cell contours, for example, can be learned

and applied to produce fully segmented images. In the nerve cell segmentation application in

Figure 4B by Zaimi and colleagues, nerve cell images are segmented into regions marking axon

(blue), myelin (red) and background (black) [75] using a CNN tailored for such a purpose. In a

semantic image segmentation problem, the goal is to map each pixel within the input image to

one of many classes present in the corpus (e.g., membrane, nucleus, and cytoplasm) [76]. Here,

many classifications are required (one classification per pixel rather than one classification per

image). Two broad approaches have been proposed using fully convolutional networks. The first

approach is to use an encoder-decoder, whereby a reverse series of operations (upsampling and

deconvolutions) are performed to reconstruct a segmented image from its embedding in the

11

initial convolutional and pooling layers [73,77,78] (Figure 4B). The dashed arrows indicate a

popular “U-Net” modification, where a connected network can be applied at different scales and

subsequently combined [79,80]. In the microscopy field, the U-Net has been successfully applied

to segment and count cells [77], and employed by Zaimi and colleagues in Figure 4B (simplified

here) [75]. Alternatively, a spatial pyramidal pooling approach can also be used [81–83]. Spatial

pyramid pooling focuses on capturing context at several scales from the image-level feature.

That is, given the low-level representations, several filters at different resolutions are applied in

order to capture objects at multiple scales. These filtered representations are joined together and

passed to the final convolutional layer. Spatial pyramidal methods produce a representation that

explicitly tackles different scales with specific filters placed in the pyramidal component.

Although both encoder-decoder and pyramidal approaches are applicable, a pyramidal

architecture separates the tasks of image downscaling and convolution, which often leads to

better segmentation at multiple image scales.

Hybrid Approaches: Video

The above examples represent common microfluidic problem types and associated well-

developed deep learning architectures. This list should not be construed as being comprehensive,

but rather an overview of the most common deep learning approaches for biotechnology

applications. Notably, combinations of the above architectures can also be used in conjunction,

for example to analyze videos (i.e., image sequences). One recent example is in the identification

of hematopoietic lineage by Buggenthin and colleagues, where a combination of CNN and RNN

architectures were used to predict single-cell lineage [84]. In this case, a CNN was applied to

brightfield images of stem cells to extract local image features, and these vectors were then fed

into an RNN to take temporal information into account (i.e. analyze the next frame in a video by

12

considering the previous frame). In this way, 5,922 cells were imaged over 400 sequential

frames, resulting in ~2.5 million image patches. The authors demonstrated that their algorithm

could predict cell type after differentiation, and up to three generations before conventional

molecular markers were observed. An alternative approach to processing video involves

compression into static images in a manner that preserves key features. For example, Yu and

colleagues recently used such a video compression approach to analyze phenotypic antimicrobial

susceptibility [85]. Videos of bacteria within microfluidic channels were compressed into two

sets of static images, each capturing either cell morphology or motion trace. These compressed

images were then fed into a CNN, and bacterial cells inhibited by an antibiotic were successfully

differentiated from cells that were uninhibited by an antibiotic.

Emerging Opportunities

So far, we have mapped the most promising pairings of deep learning architectures and

microfluidics for biotechnology applications. Below we provide an outlook on intriguing

opportunities that stem from the marriage of deep learning and microfluidics.

Organ-on-a-Chip and AI-Autonomous Living Systems

Organ-on-a-chip (OOC) systems are engineered devices that rely on the integration of advanced

microfabrication, microfluidics, living cells, and biomaterials to create human tissue constructs

that accurately mimic the structure, function, and microenvironment of real human tissue. OOC

systems have already been developed for many organs of the body, and have demonstrated

immense potential as test platforms for drug development and towards personalized medicine

[86,87]. Based on the current pace of progress, we anticipate that OOC systems will mimic real

tissues with increasing accuracy and complexity, and any deep learning application that can be

13

proposed for real tissues can conceivably be proposed for OOC systems. First, as OOC systems

continue to evolve, microfluidic tissue-level structures will be cultured, maintained, and

monitored on-chip, and could be studied over large sample regions of the OOC tissue to detect

spatial heterogeneity in a manner similar to modern histopathology analysis [88]. Thus, we

anticipate the generation of enormous datasets of images and videos of living cells, tissues, and

organs residing on-chip within their respective in vitro microenvironments and undergoing

changes in behaviour and function, and we believe this data will provide a rich dataset for deep

learning architectures. Indeed, the application of deep learning for histopathology has recently

been shown using CNNs to organize and classify histomorphologic information [89], and thus

can be conceivably extended to OOC-derived tissue constructs given that the most common data

output structure to date for OOCs is fluorescence microscopic images.

Second, considering the emerging vision of the person-on-a-chip (or body-on-a-chip

[90]) for clinical applications and personalized medicine [91], there is a potential role for deep

networks and artificial intelligence in control of the entire multi-organ system. Such multi-organ

microphysiological systems have recently been reported [90], demonstrating control of flow

partitioning to the various organs to mimic human physiology. One can imagine deep networks

monitoring individual organs, metering communication, providing real-time control of multiple

organs-on-chip systems, and working towards a fully “trained” and AI-guided multi-organ

microphysiological system that essentially regulates itself. Ironically, a learning AI-driven multi-

organ system may in fact be more fundamentally natural, or self-guided, than human-controlled

organ microsystems - a rather intriguing proposition. Thus, we envision a clear and immediate

role here for deep learning in accelerating data analysis from OOC systems, and an intriguing

14

medium- to long-term role in the integrated control and self-regulation of multiple organs or

tissues making up a larger living whole.

Deep Learning-Powered Experiment Design and Control

While the majority of near-term deep learning applications will focus on a post-experiment data

analysis role, there is growing potential for deep learning in designing microfluidic systems and

controlling systems during experiments. The synthetic biology company Zymergen, for example,

controls thousands of parallel microwell-based microbial culture experiments where the fluidic

decisions (e.g. what to inject and when to do it) are made autonomously via AI. In that case the

machine learning analytics and fluid experimental system work together autonomously toward

the goal of genetically engineering microorganisms to produce useful chemicals [92]. Neural

networks have also been applied to predict the outcome of organic chemistry reactions [93]. In

addition to cultivation optimization, we see near-term opportunity in applying deep learning to

fields with deeply complex and convoluted factors. For example, an increasing awareness of

climate change and pollution is motivating a growing emphasis on elucidating the role of

environmental stressors on microbiota [94,95]. Quantifying the response of complex systems to

multiple stressors – a biotechnology grand challenge – will require high numbers of parallel

experiments and roles for deep learning in both experiment planning and post-experiment

analysis. Early work in this area by Nguyen and colleagues showed how an aerogel-based gas

diffusion platform could be used to evaluate the role of various stressors (temperature, light, food

supply, various pollutants) on microalgal growth [94]. Lambert and colleagues developed a

chemotaxis assay to study bacterial communities, enabling the study of complex chemical

interactions at an organism-relevant microfluidic scale [95]. Sifting the resulting data is currently

a bottleneck, and despite the large numbers of tests possible in microsystems, the number of

15

variables is still too high for brute force. These recent approaches point to the opportunity for

small scale environmental toxicology testing, as well the need for deep learning guidance both in

experiment planning (increasing efficiency) and post-experiment analysis.

Globally Distributed Microfluidics with Cloud-Based Deep Learning

Inexpensive microfluidic tests such as paper-based assays [96] can provide rapid results in high

numbers and at high frequency worldwide. Particularly when paired with imaging and data

transmission capabilities of now ubiquitous smartphones [97], paper-based tests can uniquely

provide real-time, globally distributed analytical data ideally suited to deep learning algorithms.

We see this combination being particularly powerful in the medium-term in microfluidic point-

of-care diagnostics [98] and food safety [99]. In diagnostics, data generated from low-cost

pathogen-detecting paper microfluidic devices from millions of globally distributed users could

be paired with deep learning algorithms to track, predict and ultimately contain outbreaks. In

addition to the detection and prediction of an rapidly evolving outbreak, there may be an

additional role for microfluidics in a targeted distributed response. For example, the local

production of antibodies, therapeutics and vaccines is possible via hydration of freeze-dried cell-

free transcription machinery [100]. This vision leverages deep learning powered analysis and

prediction to turn distributed local microfluidics-based detection into a coordinated and effective

local response. Analogously in food safety, microfluidic systems could be applied to test and

monitor food quality and safety throughout the food production chain, feeding data-hungry deep

learning strategies to contain and ultimately prevent contamination. Particularly in supply chains,

microfluidic (data acquisition) and deep learning (analysis) will likely be further combined with

cloud-based distributed ledger systems known as blockchain. In both diagnostics and food safety,

cloud-based deep learning algorithms are an ideal partner to low-cost distributed testing.

16

Concluding Remarks

Deep learning and microfluidics represent an ideal marriage of experimental and analytical

throughput. The pairing will only strengthen as technologies in both fields advance. In essence,

the massive amount of data recovered from highly parallelized microfluidic systems represents

the ideal biotechnology application for today’s modern deep learning algorithms. It is also likely

that the integration of these approaches within the biotechnology research workflow will

synergistically accelerate research in powerful ways (see Outstanding Questions). For example,

the training of high performance generic image classifiers (e.g., human, chair, car) has enabled

retraining for medical image classification tasks with substantially reduced data requirements

[101]. Similar architecture reuse may accelerate progress and reduce data requirements for deep

learning in biotechnology. While the adoption of any new technology within a lab presents

challenges and costs, the opportunity for a union of microfluidics and deep learning is clear, and

for many biotechnology applications the barriers to entry are now relatively low. We hope this

roadmap demystifies deep learning, highlights its tremendous potential, and encourages rapid

implementation to the benefit of biotechnology research.

Acknowledgements

The authors gratefully acknowledge support from the Natural Sciences and Engineering Council

of Canada, the Discovery Grants program, an E.W.R. Steacie Memorial Fellowship and the

Canada Research Chairs program. The authors also thankfully acknowledge support from the

Canadian Institutes of Health Research Collaborative Health Research Projects program.

17

Box 1: Deep Learning Basics

What is a neural network?

A neural network is a type of machine learning architecture where a structured nonlinear

function y=f(x) is learned to map input x into output y. The essence of neural networks is in the

multiple layers of simple algebraic operations used to compute function f [26,102]. Each node

within a hidden layer is computed as a weighted linear combination of all nodes within the

previous layer, followed by a nonlinear transform (e.g. a sigmoidal function or a rectified linear

unit). Next, the output from this layer is computed as a weighted linear combination of all nodes

within the hidden layer in a similar fashion. Each layer’s outputs are fed into the next layer, until

a series of network outputs are generated. The power and versatility of such a network comes

from simply combining many of these simple operations together. In a process called supervised

learning, a labeled data set with input x and output y pairs can be used to train a neural network

by optimization of the weights. At each iteration of the code, the predicted outputs y (predicted

cell classes) are compared to known values (e.g. known cell classes), and the error calculated

(e.g. squared error or cross-entropy). New weights are assigned and back-propagated using a

gradient descent algorithm to minimize the error [60,103]. After multiple iterations, the neural

network is trained and capable of making predictions on new test datasets.

Deep Learning vs. Traditional Machine Learning

While traditional machine learning methods including neural networks have been active areas of

research for decades, only in the past 10 years have deep neural networks (i.e., usually

considered to be neural networks with three or more hidden layers) started to significantly

outperform other methods. While deep neural networks were traditionally very hard and time-

consuming to train, the advent of large-scale data and storage, fast GPU-based computation, and

18

advances in training methods have combined to make possible deep learning performance

breakthroughs on a variety of tasks including image recognition, speech recognition, and

language translation. In recent years, deep convolutional neural networks as deep as 152 layers

have offered state-of-the-art performance on image recognition tasks [104]. Critically, unlike

many previous machine learning methods, deep learning methods naturally and efficiently

exploit the sequential and spatial structure of data, which is the key innovation of deep learning.

Box 2: Getting Started with Deep Learning

When to go Deep?

Certainly many biotechnology applications could benefit from traditional machine learning

methods since many common problems require simple classification or regression tasks with

simply unstructured inputs (e.g. cell parameters, not sequences or images). This simplified

approach, however, fails to leverage the most compelling attribute of deep learning: the ability to

learn complex features or correlations of features not known a priori. A deep learning algorithm

can discover features relevant for prediction that describe the image far better than a limited set

of established features, or features that a human observer deems important. Deep learning can

thus represent a significant improvement in efficacy, particularly if there are complex nonlinear

relationships, and is essential in the case of sequential or image inputs.

Is Big Data Required?

How much data is required varies based on how many classes are being trained, how different

these classes are, and whether tricks can be used to augment the data. Extensive training can in

some cases be avoided by using a pre-trained network (termed transfer learning) [101]. Recently,

Gopakumar and colleagues showed how a CNN pre-trained on the ImageNet database [105]

19

(>106 labelled everyday images such as dogs, trees, and cars) could be re-trained to predict cell

class at reasonable accuracy with as few as ~30 cell images [50]. Another popular approach is to

enlarge a small dataset by augmenting the original set with modified images (e.g., flipping,

rotating, defocusing, and noise addition). In short, even a small amount of data is sufficient to get

started with deep learning implementation.

How do I Get Started?

Computer engineering is producing hardware solutions well-adapted for deep learning

computing requirements, yielding graphics processing units (GPUs) optimized for parallel data

processing. Training networks still generally requires days for datasets with ~108 samples, for

example, while more typical biotechnology applications with hundreds or thousands of samples

can be processed in under an hour on a single commercial GPU. Biotechnology labs that widely

embrace deep learning may additionally consider hardware clusters or external services for

computing. A student with basic programming skills can quickly get up to speed via a deep

learning introduction course, or equivalent online course (e.g., Andrew Ng’s deep learning

course: www.coursera.org/learn/machine-learning). In summary, the barriers to entry into deep

learning are quickly fading, and we encourage biotechnology researchers to leverage the potent

combination of microfluidics and deep learning.

20

References

1 Sackmann, E.K. et al. (2014) The present and future role of microfluidics in biomedical research. Nature 507, 181–189

2 Lan, F. et al. (2017) Single-cell genome sequencing at ultra-high-throughput with microfluidic droplet barcoding. Nat. Biotechnol. 35, 640–646

3 Zilionis, R. et al. (2016) Single-cell barcoding and sequencing using droplet microfluidics. Nat. Protoc. 12, 44–73

4 Su, Y. et al. (2017) Single cell proteomics in biomedicine: High-dimensional data acquisition, visualization, and analysis. PROTEOMICS 17, 1600267

5 Caen, O. et al. (2017) Microfluidics as a Strategic Player to Decipher Single-Cell Omics? Trends Biotechnol. 35, 713–727

6 Liu, Z. et al. (2017) Microfluidics for Combating Antimicrobial Resistance. Trends Biotechnol. 35, 1129–1139

7 Dittrich, P.S. and Manz, A. (2006) Lab-on-a-chip: microfluidics in drug discovery. Nat. Rev. Drug Discov. 5, 210–218

8 Wolff, A. et al. (2003) Integrating advanced functionality in a microfabricated high-throughput fluorescent-activated cell sorter. Lab. Chip 3, 22–27

9 Gossett, D.R. et al. (2012) Hydrodynamic stretching of single cells for large population mechanical phenotyping. Proc. Natl. Acad. Sci. U. S. A. 109, 7630–7635

10 Mazutis, L. et al. (2013) Single-cell analysis and sorting using droplet-based microfluidics. Nat. Protoc. 8, 870–891

11 Cermak, N. et al. (2016) High-throughput measurement of single-cell growth rates using serial microfluidic mass sensor arrays. Nat. Biotechnol. 34, 1052–1059

12 Riordon, J. et al. (2014) Quantifying the volume of single cells continuously using a microfluidic pressure-driven trap with media exchange. Biomicrofluidics 8, 011101

13 Wang, B.L. et al. (2014) Microfluidic high-throughput culturing of single cells for selection based on extracellular metabolite production or consumption. Nat. Biotechnol. 32, 473–478

14 Skelley, A. et al. (2009) Microfluidic control of cell pairing and fusion. Nat. Methods 6, 147–152

15 Nagrath, S. et al. (2007) Isolation of rare circulating tumour cells in cancer patients by microchip technology. Nature 450, 1235–1239

16 Sarioglu, A.F. et al. (2015) A microfluidic device for label-free, physical capture of circulating tumor cell clusters. Nat Methods 12, 685–691

17 Cho, B.S. et al. (2003) Passively driven integrated microfluidic system for separation of motile sperm. Anal. Chem. 75, 1671–1675

18 Nosrati, R. et al. (2014) Rapid selection of sperm with high DNA integrity. Lab. Chip 14, 1142–1150

19 Knowlton, S.M. et al. (2015) Microfluidics for sperm research. Trends Biotechnol. 33, 221–229

20 Nosrati, R. et al. (2017) Microfluidics for sperm analysis and selection. Nat. Rev. Urol. advance online publication,

21 Sesen, M. et al. (2017) Droplet control technologies for microfluidic high throughput screening (μHTS). Lab. Chip 17, 2372–2394

21

22 Wielhouwer, E.M. et al. (2011) Zebrafish embryo development in a microfluidic flow-through system. Lab. Chip 11, 1815–1824

23 Choudhury, D. et al. (2012) Fish and Chips: a microfluidic perfusion platform for monitoring zebrafish development. Lab. Chip 12, 892–900

24 Chung, K. et al. (2008) Automated on-chip rapid microscopy, phenotyping and sorting of C. elegans. Nat. METHODS 5, 637–643

25 Chung, K. et al. (2011) A microfluidic array for large-scale ordering and orientation of embryos. Nat. Methods 8, 171-U103

26 LeCun, Y. et al. (2015) Deep learning. Nature 521, 436–444 27 Bengio, Y. et al. (2013) Representation Learning: A Review and New Perspectives.

IEEE Trans. Pattern Anal. Mach. Intell. 35, 1798–1828 28 Krizhevsky, A. et al. (2012) , Imagenet classification with deep convolutional neural

networks. , in Advances in neural information processing systems, pp. 1097–1105 29 Simonyan, K. and Zisserman, A. (2014) Very deep convolutional networks for large-

scale image recognition. ArXiv Prepr. ArXiv14091556 30 Szegedy, C. et al. (2016) , Rethinking the inception architecture for computer vision. ,

in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826

31 Redmon, J. et al. (2016) , You Only Look Once: Unified, Real-Time Object Detection. , pp. 779–788

32 Luong, M.-T. et al. (2015) Effective approaches to attention-based neural machine translation. ArXiv Prepr. ArXiv150804025

33 Sutskever, I. et al. (2014) , Sequence to sequence learning with neural networks. , in Advances in neural information processing systems, pp. 3104–3112

34 Cho, K. et al. (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. ArXiv Prepr. ArXiv14061078

35 Bahdanau, D. et al. (2016) , End-to-end attention-based large vocabulary speech recognition. , in Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on, pp. 4945–4949

36 Graves, A. et al. (2013) , Speech recognition with deep recurrent neural networks. , in Acoustics, speech and signal processing (icassp), 2013 ieee international conference on, pp. 6645–6649

37 Joulin, A. et al. (2016) Bag of tricks for efficient text classification. ArXiv Prepr. ArXiv160701759

38 Jaderberg, M. et al. (2016) Reading Text in the Wild with Convolutional Neural Networks. Int. J. Comput. Vis. 116, 1–20

39 Zhou, J. and Troyanskaya, O.G. (2015) Predicting effects of noncoding variants with deep learning–based sequence model. Nat. Methods 12, 931–934

40 Angermueller, C. et al. (2016) Deep learning for computational biology. Mol. Syst. Biol. 12, 878

41 Ko, J. et al. (2018) Machine learning to detect signatures of disease in liquid biopsies – a user’s guide. Lab. Chip 18, 395–405

42 Guo, B. et al. (2017) High-throughput, label-free, single-cell, microalgal lipid screening by machine-learning-equipped optofluidic time-stretch quantitative phase microscopy. Cytometry A 91, 494–502

22

43 Ko, J. et al. (2017) Combining Machine Learning and Nanofluidic Technology To Diagnose Pancreatic Cancer Using Exosomes. ACS Nano 11, 11182–11193

44 Singh, D.K. et al. (2017) Label-free, high-throughput holographic screening and enumeration of tumor cells in blood. Lab. Chip 17, 2920–2932

45 Huang, X. et al. (2016) Machine Learning Based Single-Frame Super-Resolution Processing for Lensless Blood Cell Counting. Sensors 16,

46 Mahdi, Y. and Daoud, K. (2017) Microdroplet size prediction in microfluidic systems via artificial neural network modeling for water-in-oil emulsion formulation. J. Dispers. Sci. Technol. 38, 1501–1508

47 Chen, C.L. et al. (2016) Deep Learning in Label-free Cell Classification. Sci. Rep. 6, 21471

48 Heo, Y.J. et al. (2017) Real-time Image Processing for Microscopy-based Label-free Imaging Flow Cytometry in a Microfluidic Chip. Sci. Rep. 7,

49 Van Valen, D.A. et al. (2016) Deep Learning Automates the Quantitative Analysis of Individual Cells in Live-Cell Imaging Experiments. PLOS Comput. Biol. 12, e1005177

50 Gopakumar, G. et al. (2017) Cytopathological image analysis using deep-learning networks in microfluidic microscopy. J. Opt. Soc. Am. A 34, 111

51 Stoecklein, D. et al. (2017) Deep Learning for Flow Sculpting: Insights into Efficient Learning using Scientific Simulation Data. Sci. Rep. 7, 46368

52 Albrecht, T. et al. (2017) Deep learning for single-molecule science. Nanotechnology 28, 423001

53 Ching, T. et al. (2017) Opportunities And Obstacles For Deep Learning In Biology And Medicine. bioRxiv

54 Mamoshina, P. et al. (2016) Applications of Deep Learning in Biomedicine. Mol. Pharm. 13, 1445–1454

55 Das, D.K. et al. (2013) Machine learning approach for automated screening of malaria parasite using light microscopic images. MICRON 45, 97–106

56 Mirsky, S.K. et al. (2017) Automated Analysis of Individual Sperm Cells Using Stain-Free Interferometric Phase Microscopy and Machine Learning. Cytometry A 91A, 893–900

57 Ko, J. et al. (2017) Combining Machine Learning and Nanofluidic Technology To Diagnose Pancreatic Cancer Using Exosomes. ACS NANO 11, 11182–11193

58 San-Miguel, A. et al. (2016) Deep phenotyping unveils hidden traits and genetic relations in subtle mutants. Nat. Commun. 7,

59 Vasilevich, A.S. et al. (2017) How Not To Drown in Data: A Guide for Biomaterial Engineers. Trends Biotechnol. 35, 743–755

60 Werbos, P.J. (1990) Backpropagation through time: what it does and how to do it. Proc. IEEE 78, 1550–1560

61 Han, S. et al. (2018) Use of Deep Learning for Characterization of Microfluidic Soft Sensors. IEEE Robot. Autom. Lett.

62 Boža, V. et al. (2017) DeepNano: Deep recurrent neural networks for base calling in MinION nanopore reads. PloS One 12, e0178751

63 Godin, M. et al. (2010) Using buoyant mass to measure the growth of single cells. Nat. Methods 7, 387–390

64 Jozefowicz, R. et al. (2016) Exploring the limits of language modeling. ArXiv Prepr. ArXiv160202410

23

65 Qu, Y.-H. et al. (2017) On the prediction of DNA-binding proteins only from primary sequences: A deep learning approach. PLOS ONE 12, e0188129

66 Vaswani, A. et al. (2017) , Attention is all you need. , in Advances in Neural Information Processing Systems, pp. 6000–6010

67 Angerer, P. et al. (2017) Single cells make big data: New challenges and opportunities in transcriptomics. Curr. Opin. Syst. Biol. 4, 85–91

68 Deng, Y. et al. (2018) Massive single-cell RNA-seq analysis and imputation via deep learning. DOI: 10.1101/315556

69 Webb, S. (2018) Deep learning for biology. Nature 554, 555–557 70 Camacho, D.M. et al. (2018) Next-Generation Machine Learning for Biological

Networks. Cell 173, 1581–1592 71 Bock, C. et al. (2016) Multi-Omics of Single Cells: Strategies and Applications. Trends

Biotechnol. 34, 605–608 72 LeCun, Y. and Bengio, Y. (1995) Convolutional networks for images, speech, and time

series. Handb. Brain Theory Neural Netw. 3361, 1995 73 Zeiler, M.D. et al. (2011) , Adaptive deconvolutional networks for mid and high level

feature learning. , in Computer Vision (ICCV), 2011 IEEE International Conference on, pp. 2018–2025

74 Kim, K. et al. (2018) Visual Estimation of Bacterial Growth Level in Microfluidic Culture Systems. Sensors 18,

75 Zaimi, A. et al. (2018) AxonDeepSeg: automatic axon and myelin segmentation from microscopy data using convolutional neural networks. Sci. Rep. 8, 3816

76 Long, J. et al. (2015) , Fully convolutional networks for semantic segmentation. , in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431–3440

77 Ronneberger, O. et al. (2015) , U-net: Convolutional networks for biomedical image segmentation. , in International Conference on Medical image computing and computer-assisted intervention, pp. 234–241

78 Badrinarayanan, V. et al. (2017) SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39, 2481–2495

79 Farabet, C. (2013) , Towards real-time image understanding with convolutional networks. , PhD Thesis, Université Paris-Est

80 Lin, G. et al. (2016) , Efficient Piecewise Training of Deep Structured Models for Semantic Segmentation. , in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3194–3203

81 Zhao, H. et al. (2017) , Pyramid Scene Parsing Network. , in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6230–6239

82 Liu, W. et al. (2015) Parsenet: Looking wider to see better. ArXiv Prepr. ArXiv150604579

83 Chen, L.-C. et al. (2016) Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. ArXiv Prepr. ArXiv160600915

84 Buggenthin, F. et al. (2017) Prospective identification of hematopoietic lineage choice by deep learning. Nat. Methods 14, 403–406

24

85 Yu, H. et al. (2018) Phenotypic Antimicrobial Susceptibility Testing with Deep Learning Video Microscopy. Anal. Chem. 90, 6314–6322

86 Bhatia, S.N. and Ingber, D.E. (2014) Microfluidic organs-on-chips. Nat. Biotechnol. 32, 760–772

87 Ahadian, S. et al. (2018) Organ-On-A-Chip Platforms: A Convergence of Advanced Materials, Cells, and Microscale Technologies. Adv. Healthc. Mater. 7, 1700506

88 Lu, C. et al. (2017) An oral cavity squamous cell carcinoma quantitative histomorphometric-based image classifier of nuclear morphology can risk stratify patients for disease-specific survival. Mod. Pathol. 30, 1655–1665

89 Faust, K. et al. (2018) Visualizing histopathologic deep learning classification and anomaly detection using nonlinear feature space dimensionality reduction. BMC Bioinformatics 19,

90 Edington, C.D. et al. (2018) Interconnected Microphysiological Systems for Quantitative Biology and Pharmacology Studies. Sci. Rep. 8, 4530

91 Zhang, B. et al. (2016) Biodegradable scaffold with built-in vasculature for organ-on-a-chip engineering and direct surgical anastomosis. Nat. Mater. 15, 669–678

92 Check Hayden, E. (2015) Synthetic biology lures Silicon Valley investors. Nat. News 527, 19

93 Wei, J.N. et al. (2016) Neural Networks for the Prediction of Organic Chemistry Reactions. ACS Cent. Sci. 2, 725–732

94 Nguyen, B. et al. (2018) A Platform for High-Throughput Assessments of Environmental Multistressors. Adv. Sci.

95 Lambert, B.S. et al. (2017) A microfluidics-based in situ chemotaxis assay to study the behaviour of aquatic microbial communities. Nat. Microbiol. 2, 1344–1349

96 Gong, M.M. and Sinton, D. (2017) Turning the Page: Advancing Paper-Based Microfluidics for Broad Diagnostic Application. Chem. Rev. 117, 8447–8480

97 Erickson, D. et al. (2014) Smartphone technology can be transformative to the deployment of lab-on-chip diagnostics. Lab. Chip 14, 3159–3164

98 Pandey, C.M. et al. (2017) Microfluidics Based Point‐of‐Care Diagnostics. Biotechnol. J. 13, 1700047

99 Weng, X. and Neethirajan, S. (2017) Ensuring food safety: Quality monitoring using microfluidics. Trends Food Sci. Technol. 65, 10–22

100 Pardee, K. et al. (2016) Rapid, Low-Cost Detection of Zika Virus Using Programmable Biomolecular Components. Cell 165, 1255–1266

101 Tajbakhsh, N. et al. (2016) Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning? IEEE Trans. Med. Imaging 35, 1299–1312

102 Krogh, A. (2008) What are artificial neural networks? Nat. Biotechnol. 26, 195–197 103 Rumelhart, D.E. et al. (1986) Learning representations by back-propagating errors.

Nature 323, 533–536 104 He, K. et al. (2016) , Deep Residual Learning for Image Recognition. , in 2016 IEEE

Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, pp. 770–778

105 Deng, J. et al. (2009) , ImageNet: A large-scale hierarchical image database. , in 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255

106 Kilby, J.S. 23-Jun-(1964) Miniaturized electronic circuits. , US3138743 A

25

107 Aspray, W. (1997) The Intel 4004 microprocessor: What constituted invention? IEEE Ann. Hist. Comput. 19, 4–15

108 Terry, S.C. et al. (1979) A gas chromatographic air analyzer fabricated on a silicon wafer. IEEE Trans. Electron Devices 26, 1880–1886

109 Duffy, D.C. et al. (1998) Rapid Prototyping of Microfluidic Systems in Poly(dimethylsiloxane). Anal. Chem. 70, 4974–4984

110 Rosenblatt, F. (1958) The perceptron: a probabilistic model for information storage and organization in the brain. Psychol. Rev. 65, 386

111 Jordan, M.I. (1997) Serial order: A parallel distributed processing approach. In Advances in psychology 121pp. 471–495, Elsevier

112 Abadi, M. et al. (2016) Tensorflow: Large-scale machine learning on heterogeneous distributed systems. ArXiv Prepr. ArXiv160304467

113 Rampasek, L. and Goldenberg, A. (2016) TensorFlow: Biology’s Gateway to Deep Learning? Cell Syst. 2, 12–14

26

Figure Legends

Text Box 1 Figure. Neural network basics. (A) Neural network weights adjusted via back-

propagation. (B) Neural network with three hidden layers.

27

Figure 1. Deep learning with microfluidics. (A) A brief history of deep learning and

microfluidics. Microfluidics emerged from MEMS technologies in the 1990s. (Top): 1958 - first

integrated circuit by Kilby [106]; 1971 - first commercially available microprocessor (Intel)

[107]; 1979 - first lab on a chip [108]; 1998 - Demonstration of rapid prototyping with PDMS

[109]. Concurrently, artificial intelligence and machine learning algorithms have been

progressing over a similar time period (Bottom): 1957 – first perceptron [110]; 1974 –

introduction of backpropagation within neural networks by Werbos [60], and in 1986

popularized by Rumelhart, Hinton & Williams [103]; 1986 - introduction of the recurrent neural

network (RNN) by Jordan [111]; 2012 – demonstration of a foundational convolutional neural

network (CNN), AlexNet, developed by Krizhevsky, Hinton and Sutskever [28]. RNNs and

CNNs were at first limited by data and computational power, but in the last few years have

gained massive popularity by leveraging fast GPUs, frameworks such as TensorFlow [112,113]

and distributed computing.

28

Key Figure: Figure 2. Mapping microfluidic applications to deep learning architectures.

Example applications are paired with deep learning architectures, from the simplest

unstructured-to-unstructured case to the more complex image-to-image case. Each example is

described in detail in corresponding sections.

29

Figure 3. Deep learning architectures: unstructured data and structured sequences as

inputs. (A) Unstructured-to-unstructured application and neural network architecture. Flow

cytometry images (left) are reproduced from “Deep Learning in Label-free Cell Classification”

by Chen and colleagues [47], and licensed under CC BY 4.0. Images were modified for clarity.

Grey circles represent nodes and arrows depict connections between nodes (light grey dashed

arrows) or between layers (solid black arrows). Layers are color-coded with the input layer in

30

blue, hidden layers in green and output layers in red. (B) Sequence-to-unstructured application

and recurrent neural network (RNN) architecture. Microfluidic soft sensor characterization

example (left) images modified for clarity from ref. [61]. A single recurrent neural network layer

is shown with progressive shading to show progression through time – nodes are not only fed

new current values as inputs, but also their previous values. (C) Sequence-to-sequence

application and recurrent neural network architecture. DNA base calling example image from

“DeepNano: Deep recurrent neural networks for base calling in MinION nanopore reads” by

Boža and colleagues [62], and licensed under CC BY 4.0. Image modified and inset schematic

added for clarity. The deep learning architecture schematics (right) are simplified from

references to denote network principles of operation, and do not represent an exact

representation.

31

Figure 4. Deep learning architectures: images as inputs. (a) Image-to-unstructured

application and convolutional neural network architecture. Flow cytometry cell classification

application and images (top left) are reproduced from “Real-time Image Processing for

Microscopy-based Label-free Imaging Flow Cytometry in a Microfluidic Chip” by Heo and

colleagues [48], and licensed under CC BY 4.0. Images modified for clarity. Convolutional

layers are light green, pooling layers are dark green, and different filters at the same scale (i.e.,

channels) are shown as vertical planar slices. Images reproduced from “Deep Learning for Flow

Sculpting: Insights into Efficient Learning using Scientific Simulation Data” by Stoecklein and

colleagues [51], and licensed under CC BY 4.0. Images were modified for clarity. (c) Image-to-

image cell segmentation application and CNN architecture. An SEM image of a rat spinal cord is

segmented into one of three classes: axon (blue), myelin (red) and background (black) using a

32

modified U-Net architecture, here simplified for clarity [51]. Images reproduced from

“AxonDeepSeg: automatic axon and myelin segmentation from microscopy data using

convolutional neural networks” by Zaimi and colleagues [75], and licensed under CC BY 4.0.

Images were modified for clarity. The deep learning architecture schematics (right) are

simplified from above references to denote network principles of operation, and do not represent

an exact representation.