I. Systems Biology of Circadian Rhythms
II. Neural Network Capacity
P. BaldiUniversity of California, Irvine
Department of Computer ScienceInstitute for Genomics and BioinformaticsCenter for Machine Learning and Intelligent Systems
1
Circadian (and Other) Rhythms are Pervasive in Biological Systems
Circadian (≈24 hours)
Ultradian (<24 hours)
Seasonal (>24 hours)
EEG activity during sleep
Sleep/wake
Strumwasser, F. (1960) Some physiological principles governing hibernation. Bulletin of the Museum of Comparative Zoology, 124, Harvard University
Hibernation
EEG activity during sleep
2
Circadian Rhythms are Self-Perpetuating
Observation: plant leaves continued to fold rhythmically,
even in constant darkness
Observation: Plant leaves continued to fold rhythmically but rhythms slightly deviant from the
24-hour time span. This was indicative of endogenous, free-running clock
Jean-Jacques d'Ortous de Mairan
Augustin Pyramus de Candolle
Mimosa pudica
3
4
5
Astronauts in Caves: Michel Siffre
Circadian Clocks are Tethered to the Environment via the SCN
6
7
Cell, 155, 7, 1464-1478, 2013.8
Autonomous Core Clock
9
Consequences of the Clock Breaking Down
• Sleep disorders (FASPS, etc.)• Depression• Obesity/metabolic disorders• Aging
10
Towards Personalized Medicine
11
Zhang et al. PNAS, 11, 45, 2014.
BIO_CYCLE
• Given a circadian time series for a transcript, metabolite, protein etc determine if it is periodic or not with some statistical significance.
• If periodic, estimate the period, the phase, the amplitude.
12
BIO_CYCLE Architecture
13
3 Hidden Layers 100 Units Each
Sigmoidal: Periodic/AperiodicLinear: Period
Additional Computation of Amplitudes, Phases, p-values, q-values
Sample of Training Signals
14Training set 1-10M examples
Evaluation
15
BIO_TIME
• Given measurements for transcripts, metabolites, proteins etc taken at a single time point, determine the time (or phase).
• Initially use only core clock transcripts in mouse.
16
17
Circadian AutoencoderNeural Network
BIO_TIME Architecture
18
10-16 Core Clock Genes
Cos=U1/sqrt(U12+U22)Sin=U2/sqrt(U12+U22)
BIO_TIME EVALUATION
• For WT, predicts time with 90 minutes fairly robustly across tissues.
• Used to impute a time for all mouse Gene Expression Omnibus (GEO) experiments.
• Challenges for extending to other species.
19
Examples of High-Throughput Experiments
20
S. Masri, T. Papagiannakopoulos, K. Kinouchi, Y. Liu, M. Cervantes, P. Baldi, T. Jacks, and P. Sassone-Corsi. Lung Adenocarcinoma Distally Rewires Hepatic Circadian Homeostasis. Cell, 165, 4, 896—909, (2016).
S. Masri, P. Rigor, M. Cervantes, N. Ceglia, C. Sebastian, C. Xiao,M. Roqueta-Rivera, C. Deng, T. F. Osborne, R. Mostoslavsky, P. Baldi, and P. Sassone-Corsi. Partitioning Circadian Transcription by SIRT6 Leads to Segregated Control of Cellular Metabolism. Cell, 158, 3, 659—672, (2014).
K. L. Eckel-Mahan, V. R. Patel, S. de Mateo, N. J. Ceglia, S. Sahar, S. Dilag, K. A. Dyar, R. Orozco-Solis, P. Baldi, and Paolo Sassone-Corsi. Reprogramming of the Circadian Clock by Nutritional Challenge. Cell, 155, 7, 1464-1478, (2013).
M. M. Bellet, E. Deriu, J. Liu, B. Grimaldi, C. Blaschitz, M. Zeller, R. A. Edwards, S. Sahar, S.Dandekar, P. Baldi, M. D.George, M. Raffatellu, and P. Sassone-Corsi. The Circadian Clock Regulates the Host Response to Salmonella. PNAS, 110, 24, 9897-9902, (2013).
K. L. Eckel-Mahan, V. R. Patel, K. S.Vignola, R. P. Mohney, P. Baldi, and P. Sassone-Corsi. Coordination of Metabolome and Transcriptome by the Circadian Clock. PNAS, 109 (14) 5541-5546, (2012).
http://circadiomics.igb.uci.eduNature Methods 9, 8, 772-773, 2012.Nuclei Acids Research, Web Server Issue, in press, (2018).
21
Cell, 155, 7, 1464-1478, 2013.22
23
Trends in Cell Biology, 24, 329-331, 2104.Bioinformatics, 31, 19, 2015.
At p = 0.05, 68% oscillate.95% with more recent data sets.
24
At p = 0.05, 67% oscillate.
25
26
Main Findings1. The core clock contains only a dozen genes.2. In any tissue/condition 10% (± 5%) of transcripts
or metabolites oscillates. 3. The overlap across tissues/conditions is small
(2%).4. Genetic or environmental perturbations result in
massive changes:– Amplitude changes (including suppression)– Phase changes– New oscillations
27
Explanation
1. In general, molecular species in isolation do not oscillate
2. Loops of interacting (regulatory, metabolic, PPI) species can oscillate
3. Many oscillator loops in the cell4. Why do they tend to have an intrinsic period
of 24h?
28
29
Physical objects have intrinsic vibration frequencies….
30
3.5x109x365= 1.3 x 1012
More like: 2 x 1012
(period has increased due to tidal effects) 31
Number of Revolutions since the Origin of Life
Cyanobacteria
32
Network of Coupled Circadian Oscillators: Spectrum of Models
• At one extreme, completely centralized. The core clock controls all the oscillators.
• At the other extreme, completely decentralized. The oscillators compete and self-organize.
• Biology is somewhere in between. Where?
33
34
Circadian Regulatory Control (CRC)
• TF protein coding transcript if and only if:1) TF and transcript are circadian at some p-
value (BIO_CYCLE);2) TF has binding sites in the promoter of
transcript (MotifMap, MotifMap-RNA);3) TF and transcript have the “right” phase
lag;Similarly for RBPs (taking the introns or UTRs of the target transcript).
35
Empirical Distribution of Lags
36
Tables showing the ranking of circadian TFs and RBPs by CRC E-score in different tissue types. The leftmost table shows ranking in mouse transcriptome across all datasets.
RBPs are labeled in red while TFs are labeled in black. Core clock TFs have been removed from the listing.
37
38
39
Highly enriched in olfactory GPCRs.
40
Highly enriched in olfactory GPCRs.
Hierarchical Organization
• Core Clock at the apex (level 0)• Level 1 (35%, distance 1)• Level 2 (70%, distance 2)• Level 3 (80%, distance 3)• Fan out decreases with distance.• There is feedback between levels.• Most of cellular reprogramming must occur at level 1.• Small set of transcripts that do no oscillate in any
experiment: highly enriched in olfactory GPCRs.
41
42
Summary• Roughly 10% of all molecular species oscillate in any
cell/tissue/condition with small overlaps beyond the core clock.• Genetic, epigenetic, and environmental conditions (e.g. diet) have a
profound effect on which species oscillate and lead to cellular reprogramming.
• Tools: BIO_CYCLE, BIO_TIME, CircadiOmics.• Coupled-circadian-oscillator networks provides a general
framework.• Hierarchical organization emanating at the core clock. • Precision medicine: monitor and optimize health by monitoring
and optimizing oscillations. • Precision medicine: New diagnostic tools. Optimize timing of
therapeutic interventions (drugs). • Even a 5% increase in efficacy could have significant impact.
43
II. Neural Network Capacity
44
Neural Network Capacity
• h = target function (typically known from examples)• A = class of hypothesis or approximating functions (typically
associated with a NN architecture)
h
A
Neural Network Capacity
• h = target function (typically known from examples)• A = class of hypothesis or approximating functions (typically
associated with a NN architecture)
h
A
C(A) = log2 |A|
Neural Network Capacity• Can we compute C(A) for specific, interesting, neural
networks?
h
A
C(A) = log2 |A|
Neural Network Capacity
• Assume neural networks of linear or polynomial threshold gates (Boolean functions)
f = sign
Threshold Gates
• Linear Threshold Gatesy = sign [∑i wi xi]
• Polynomial Threshold Gatesy = sign [Pd(x)]
• Variations:– Homogenous– Binary weights– Positive weights
Network Capacity
• Given a network of linear or polynomial threshold gates, |A| is finite.
• We define the capacity as:
C(network) = log2 |A|=log2(#number of Boolean functions that can be
implemented by the network)
ANDORNOTGEQLEQSINGLE
PARITYCONNECTEDPAIR
22𝑁𝑁
?
Capacity of a Single Linear Threshold Gate
Capacity Of Linear Threshold Gates
C[LTG(N)] ≤ N2
T. Cover 1965
Capacity Of Linear Threshold Gates
C[LTG(N)] ≤ N2
cN2 ≤ C[LTG(N)] (c<1) T. Cover 1965
S. Muroga (1965)
Capacity Of Linear Threshold GatesC[LTG(N)] ≤ N2
cN2 ≤ C[LTG(N)] (c<1) T. Cover 1965
S. Muroga (1965)
C[LTG(N)] = N2 (1 + o(1))
Yu. A. Zuev (1989)
ANDORNOTGEQLEQSINGLE
PARITYCONNECTEDPAIR
22𝑁𝑁
2𝑁𝑁2
Capacity of a Single Linear Threshold Gate
Capacity Of Polynomial Threshold Gates
C[PTG(N,d)] ≤ 𝑁𝑁𝑑𝑑+1
𝑑𝑑!P.B. 1988
Capacity Of Polynomial Threshold Gates
C[PTG(N,d)] ≤ 𝑁𝑁𝑑𝑑+1
𝑑𝑑!P.B. 1988
𝑁𝑁𝑑𝑑 + 1 ≤ C[PTG(N,d)] M. Saks 1993
Capacity Of Polynomial Threshold Gates
C[PTG(N,d)] ≤ 𝑁𝑁𝑑𝑑+1
𝑑𝑑!P.B. 1988
𝑁𝑁𝑑𝑑 + 1 ≤ C[PTG(N,d)] M. Saks 199
C[PTG(N,d)] = 𝑁𝑁𝑑𝑑+1
𝑑𝑑!(1 + o(1))
P.B. and R.V. 2018
Additional Results
• Binary weights (wi= -1 or +1):
C(Binary-Weight LTG) = N
• Positive weights (wi ≥ 0): C(Positive-Weight LTG) = N2 – N
• ReLUC(ReLU) = N2 + N
𝑑𝑑 = 1 𝑁𝑁2
𝑑𝑑 = 2 𝑁𝑁3/2
⁄𝑁𝑁𝑑𝑑+1 𝑑𝑑!
𝑑𝑑 = 1𝑁𝑁
𝑑𝑑 = 1 𝑁𝑁2−𝑁𝑁
Linear threshold functions with binary weights
Linear threshold functions with positive weights
Linear threshold functions (d=1)
Polynomial threshold functions of degree d
All Boolean functions of N variables 2N
ReLU N2 + N
What about Networks?
• Focus on LTG but everything can be extended to PTG.• Fully connected RNN with N linear threshold gates:
C(RNN) = N3
• More generally, for any neural network NN with N linear threshold gates:
C(NN) ≤ ∑ capacities = ∑ (fan-ini)2 ≤ N (max fan-ini)2
Neural Network Capacity
• The capacity satisfies: C(NN) ≤ ∑ capacities
• The capacity of a polynomial-size network can be expressed as a polynomial. The capacity of a network with N LTG units is at most N3.
• Can we get estimates on the capacity?• Can we get lower bounds on the capacity?• Can we compare different architectures?
Neural Network Capacitywith Single Hidden-Layer
N
M
Theorem: C[N,M,1] = MN2 (1 + o(1))
For instance, if M = α N: C ≈ α N3
Conclusions• Precise definition of capacity C=log2[#functions].• C [PTG(N,d)] = Nd+1/d! (1+o(1)).• C[LTG(N,d)] = N2(1+o(1)).• Extensions to special cases (binary weights, positive
weights).• The capacity of a fully connected network with N units
is N3.• The capacity of feedforward networks can be
estimated. It is a low degree polynomial in the relevant variables (fan-ins, layer sizes).
• C[N,M,1] = MN2(1+o(1))
Conclusions
• The capacity can be used to compare different architectures.
• Ongoing work: deep networks versus shallow networks, finite size versus asymptotic.
THANK YOU
66
A
67
Diet Change
Normal Chow (2221) (1517) High-Fat (1110)
68
Clock-Bmal1 form a complex
And bind to tandem E-boxes in the promoter of Upp2
Driving the rhythmic expression of Upp2
And in turn the rhythmic expression of Uracil and Uridine
Example:
PNAS, 109 (14) 5541-5546, 2012. 69
Coupled-Circadian-Oscillators Framework
• Many oscillatory loops• Intrinsic periodicity close to 24 h (evolution)• Coupled-oscillators• Many coupling mechanisms:
– ≈10% of genes are in a directed loop containing Clock or Bmal1
– ≈60% of genes are within two hops from Clock or Bmal1
– odd/even loops• Amplitude limited by energy balance
(homeostasis)
70
71
Highly enriched in olfactory GPCRs.
72
Highly enriched in olfactory GPCRs.
73
Highly enriched in olfactory GPCRs.
Top Related