Digital yet Deliberately Random: Synthesizing Logical Computation on Stochastic Bit Streams Ph.D....

Digital yet Deliberately Random:Synthesizing Logical Computation

on Stochastic Bit Streams

Digital yet Deliberately Random:Synthesizing Logical Computation

on Stochastic Bit Streams

Ph.D. CandidateElectrical & Computer Engineering

University of MinnesotaAdvisor: Marc D. Riedel

Weikang Qian

A

B

C

Ph.D. Final DefenseJuly 25, 2011

Hierarchy of Modern Digital Systems

NMOS PMOSPhysical Level

Logical Level

System Level

Application Level

Challenges: Physical Level

Transistor Scaling: Approaching Physical Limit

Increasing concerns about variability and errors.

Gordon Moore

Challenges: Physical Level

Emerging Device Technologies

• Randomness in connections• High defect rates

Carbon Nanotubes Nanowire Crossbar

Challenges: Application Level

Future: machine learning, pattern recognition, data mining, …

Today: process data Future: comprehend data

Is she Lady Gaga?

These applications are probabilistic: we look for an answer with high probability of being true

Deterministic Paradigm

Deterministic Encoding• Arithmetic unit: numbers

encoded by binary radix• Control unit: instructions

defined as deterministic sets of zeros and ones

Physical Level

Logical Level

System Level

Application Level

Probabilistic, Random

Probabilistic Application

Physical Level

Logical Level

System Level

Application Level

Deterministic Encoding

Hard to maintainaccuracy

Unnecessary

Stochastic Encoding

Stochastic ParadigmDeterministic Paradigm

Synthesizing Logic that Computes on Stochastic Bit Streams

1,1,0,1,0,1,1,0,…

1,0,0,0,1,1,0,0,…

combinationallogic

0,1,1,0,1,0,1,0,…

0,1,1,0,1,0,0,0,…

1,0,0,0,0,0,1,0,…1,0,1,1,0,1,1,1,…

Applicable to arbitrary arithmetic functions

Gamma CorrectionFunction

Synthesizing Logic that Generates Probabilities

0.40.6

0.50.3 0.7

0.40.5 0.2

0.14

0.50.50.4

0.6 0.30.15

0.85

0.119

Transform a source set of probabilities to a target setentirely through combinational logic

0,0,0,0,1,0,1,0,0,0, …

0.119combinationallogic

Probability:0.4

0.5

Outline

• Preliminaries

• Synthesizing Logic that Computes on Stochastic Bit Streams

• Synthesizing Logic that Generates Probabilities

• Future Work

Logical Computation On Sequences of Random Bits

1,1,0,1,0,1,1,0,…

1,0,0,0,1,1,0,0,…

combinationallogic

0,1,1,0,1,0,1,0,…

0,1,1,0,1,0,0,0,…

1,0,0,0,0,0,1,0,…

1,0,1,1,0,1,1,1,…

Representing a Value by a Sequence of Random Bits

A real value x in [0, 1] is represented by a sequence of random bits, each of which has probability x of being one and probability of 1 − x of being zero.

0,1,0,1,1,0,0

x = 3/7

Serial versus Parallel

0,1,0,1,1,0, 0

Stochastic Bit Streamsx = 3/7

Probabilistic Bundles

x = 3/7

0101100

Stochastic Bit Streams as Inputs/Outputs

1,1,0,1,0,1,1,0

1,0,0,0,1,1,0,0

combinationallogic

0,1,1,0,1,0,1,0

0,1,1,0,1,0,0,0

1,0,0,0,0,0,1,0

1,0,1,1,0,1,1,1

4/8

3/8

2/8

6/8

3/8

5/8

Probability values are the input and output signals.

AND

A

BC

A Single AND Gate Performs Multiplication!

1,1,0,0,0,0,1,01,1,0,1,0,1,1,1

1,1,0,0,1,0,1,0

a = 6/8

b = 4/8

c = 3/8

Assume two input bit streams are independent

3/8 = 6/8 • 4/8

A Conventional Multiplier

HAa1HA b1

a0b1

FA a0b2

a2 b0 a1 b0

a1FA b2

a2 b1

HAFA

a0 b0

c0

c1

c2

c3c4c5

a2 b2

a2 a1 a0 b2 b1 b0

c2 c1 c0c5 c4 c3

a b

c

• HA: Half adder, 2 basic gates (AND and XOR)

• FA: Full adder, 5 basic gates (AND, OR, and XOR)

In total 30 gates!

Error due to Stochastic Variance

• Effect of error is small.

• Error can be reduced by increasing the bit length.

• Target at applications that can tolerate small errors, e.g., image processing.

x = P(X = 1) = 2/5

0,1,0,0,1,1,0,1,0,0Ideal:

1,0,1,0,1,0,0,1,1,0Practical:

1/2

Precision versus Bit Length

• Binary radix encoding– Positional and compact:

To represent 2n different values, need n bits

• Stochastic encoding– Uniform and not compact:

To represent 2n different values, need 2n bits

For applications that can tolerate small errors, we don’t need a large n.

Example: (0101100010) → 0.4

Example: (1001)2 → 9

Fault Tolerance

• Stochastic Encoding– A bit flip does not substantially change the probability:

1010111001 → 1010011001

• Binary Radix Encoding– A bit flip in the most significant bit causes a huge

change in the value:

(1010)2 → (0010)2

0.6 0.5

10 2

Comparison of Encoding

Spectrum of Encoding

Binary Radix Encoding Stochastic Encoding

Binary Radix Encoding Stochastic Encoding

Circuit Area Large Small

FaultTolerance

Bad Good

Delay Short Long

(Positional)(Uniform,

Long Stream)

(Not compact,Long Stream)

(Compact,Efficient)

(Positional, Weighted) (Uniform)

Outline

• Preliminaries



• Future Work

Prior Work: Stochastic Bit Streams

• Gaines showed how to implement basic operations such as addition and multiplication (in 1969).

• Brown and Card showed how to implement the tanh function and the linear gain function (in 2001).

• Gaudet and Rapley implemented low-density parity-check (LDPC) decoder (in 2003).

Contributions

• Showed what kind of functions can be implemented by combinational logic operating on stochastic bit streams– Must be an arithmetic polynomial

• Proposed a general method to synthesize arbitrary polynomial functions.

• Generalized the above method to synthesize arbitrary non-polynomial functions via approximation.

Gamma CorrectionFunction

Mathematical Model

combinationallogic

X2

X1

Xn

IndependentRandomBooleanVariables

YRandomBooleanVariable

?

Mathematical Model

F is a polynomial on x1, …, xn with integer coefficients and degree no more than one.

A

BC

MUX

S

0

1

Example: Multiplexer

Implementing General Polynomials

combinationallogic

IndependentProbabilities F(x1, …, x5)

Special Polynomial

x2

x1

x5

x3

x4

tt

c1

tc0

g(t) = t2 – 0.8t + 0.8

General Polynomial

• t represents variable probabilities.

• ci’s represent constant probabilities.

A

BC

MUX

S

0

1

Set s = a = t, b = 0.8

c = t2 – 0.8t + 0.8c = sa + b – sb

Can we implement polynomial with real coefficients and degree more than one on stochastic bit streams?

General Polynomial

g(t) = 1.2t2 – t3

The Problem: Synthesizing Circuit

combinationallogic

IndependentProbabilities

tt

c1

tc0

Target

? ?• Illustrate with univariate polynomials, but can be

generalized to multivariate polynomials

Synthesizing Circuit to Implement Polynomial

Power-Form Polynomial

Bernstein basis polynomial Bernstein polynomial of degree 2

Bernstein coefficient

Step 1: Convert the polynomial into a Bernstein form.


Power-Form Polynomial

Step 2: Elevate the Bernstein polynomial until all coefficients are in the unit interval.

less than 0All coefficients in unit interval

Step 3: Implement the Bernstein polynomial with all coefficients in the unit interval by “generalized multiplexing.”

Step 1: Convert the polynomial into a Bernstein form.

0,0,0,1,1,0,1,1 (1/2)

1,0,1,1,0,0,1,0 (1/2)

1,1,0,1,1,0,0,0 (1/2)

1,0,1,1,0,1,1,0 (5/8)

+X1

X2

X3

2,1,1,3,2,0,2,1

0,0,0,0,0,0,0,0 (0)

0,0,1,0,0,0,0,0 (1/8)

1,1,1,1,1,1,1,1 (1)

MUX 0,0,0,1,0,1,0,0 (1/4)

Z0

Z1

Z2

Z3

Y

0

1

2

3


Power-Form Polynomial (Evaluate on t = 1/2)

P(Xi=1) = t (= 1/2)

P(Zi=1) = bi,3

(BernsteinCoefficient)

g(1/2) = 1/4

Independent

Generalized Multiplexing

+X1X2

Xn

MUX

Z0

Z1

Zn

Y

0

1

n

ƩXi

(0 ≤ bi,n ≤ 1)

P(Xi=1) = t

P(Zi=1) = bi,n

Independent

Binomial distribution

Bernstein basispolynomial

Non-Polynomial Functions

Find a Bernstein polynomial with coefficients in the unit interval that approximates the non-polynomial g(t).

Find real values to minimize

subject to

Solved by quadratic programming

Example: Gamma Correction Function

Coefficients of degree-6 Bernstein polynomial approximation:

b0,6 = 0.0955, b1,6 = 0.7207, b2,6 = 0.3476, b3,6 = 0.9988,b4,6 = 0.7017, b5,6 = 0.9695, b6,6 = 0.9939

Hardware Cost Comparison

• Compare conventional implementation to stochastic implementation of polynomial functions.

• Mapped onto FPGA (counting the number of LUTs)• Conventional implementation: 10-bit binary radix• Stochastic implementation: bit stream of length 210

Fault-Tolerance Comparison

Conventional v.s. Stochastic implementation of Gamma correction function with noise injection

Conventional Implementation

Stochastic Implementation

1% 2% 10%

Outline

• Preliminaries



• Future Work

Generating Stochastic Bit Streams

• A premise for logical computation on stochastic bit streams

• Many other probabilistic applications

– Monte Carlo simulation

– In test of digital circuits: generate weighted random patterns

P(Left) = 0.4 P(Right) = 0.6

General Random Bit Generators

RandomSource

Constant Value

<

R

pdf of R

C

R

pdf of R

Cprobability to be one

R

If R < C, output a one;If R ≥ C, output a zero.

1,0,1,…

Comparator

Types of Random Sources

RandomSource

Constant Value

<

• Pseudorandom Number Generator

• Physical Random Source

Linear Feedback Shift Register

(expensive)

Thermal Noises

(cheap)

Challenge withPhysical Random Sources

RandomSource

Constant Value

<

R

pdf of R

C

cheap

Voltage Regulators

expensive

Suppose many different probabilities are needed:{0.2, 0.78, 0.2549, 0.43, 0.671, 0.012, 0.82, …}.

It is costly to generate them directly.(many expensive constant values required.)

expensive

C1 C2

RandomSource

<RandomSource

Constant Value

<

Opportunity with Physical Random Sources

R

pdf of R

C

cheap

expensive<

RandomSource 1,1,0,0,0, …

0,1,0,1,0, …0,0,1,0,1, …

• Independent• Same probability

Solution

When we need many different probabilities:

{0.2, 0.78, 0.2549, 0.43, 0.671, 0.012, 0.82, …}

• Generate a few source probabilities directly from random bit generators.

• Synthesize combinational logic to generate other probabilities.

Probability: Probability of a signal being logical one

Basic Problem

Random Bit Generators

LogicCircuit

q2

q1

q3

q4

Set S of Input Probabilities

{p1 , p2}

Other Probabilities

Needed

• Synthesize Logic Circuit?

• Choose Set S ?

……

p1

p1

p1

p2p2

p2

Set S of Input Probabilities

{p1 , p2}

p1

p1

p1

p2p2

p2

Independent

(|S| small)

Example

0.6

0.2LogicCircuit

x zP(x = 1) = 0.4 P(z = 1) = 0.6

1,0,1,1,0,1,0,0,0,0 0,1,0,0,1,0,1,1,1,1

P(z = 1) = P(x = 0)

AND

xy z

P(x = 1) = 0.4

P(z = 1) = 0.20,1,0,1,0,0,1,1,0,0

0,0,0,1,0,0,1,0,0,0

P(z = 1) = P(x = 1) P(y = 1)

1,0,1,1,0,0,1,0,0,1P(y = 1) = 0.5

0.4

0.50.4

0.5

……

Generating Decimal Probabilities

LogicCircuit

q2

q1

q3

q4

Arbitrary Decimal Probabilities

|S| Small!

Choose Set S = {p1, p2, p3}

• Found Set S for

|S| = 2 |S| = 1

p1

p2

p1

p2p3

p3

Independent

……

…

Generating Decimal Probabilities

Theorem: With S = {0.4, 0.5}, we can synthesize arbitrary decimal output probabilities.

• Constructive proof.• Derived a synthesis algorithm.

Algorithm

0.40.5

0.6 0.7

0.50.35

0.40.86

0.50.5

0.430.785

0.60750.50.4

0.757

AND

ANDAND

AND

AND

AND

AND

(Black dots are inverters)

×0.50.86

1 −0.14×0.4

0.35×0.5

0.71 −×0.5

0.30.61 −

0.4

×0.4 0.6075 1 −0.3925

×0.50.785 0.215

1 − ×0.5 0.430.7571 −

0.243

Example: Synthesize q = 0.757 from S = {0.4, 0.5}

For a probability value with n digits, need at most 3n AND gates.

Implementation: Reduce Circuit Depth

0.40.5

0.6 0.7

0.50.35

0.40.86

0.50.5

0.430.785

0.60750.50.4

0.757

AND

ANDAND

AND

AND

AND

AND

Balancing

• Logic Level Optimization: Balancing

ANDAND

FaninCone

ab

... AND

FaninCone ...

ANDa

b

Before Balancing After Balancing

(a and b are primary inputs)

Factorization of Fractions

• High Level Optimization: Factorization of Fractions

0.50.5

0.20.98

0.49

0.4

ANDAND

0.5

AND

ANDAND

0.4

0.25

0.1 0.5

0.49AND

AND

AND

0.5

0.5

0.4

0.40.7

0.7

Example: Synthesize q = 0.49 from S = {0.4, 0.5}

Basic

Factor

(Black dots are inverters)

0.49 = 0.7 x 0.7

Outline

• Preliminaries



• Future Work

Encoding

Spectrum of Encoding

Binary Radix Encoding(Compact, Positional)

Stochastic Encoding(Not compact, Uniform)?

Possible encodings in the middle with theadvantages of both?

Logic Optimization

Minimize the area cost of circuit for probabilistic computation

Example: Generate 0.3 from source probabilities 0.4 and 0.5

better!

0.4

0.5

AND0.5

AND0.25

0.75

0.3a

bc

y

AND

0.4 0.6

0.5

0.3ab

y

Challenges of Optimization

0.4

0.5

AND0.5

AND0.25

0.75

0.3a

bc

y

AND

0.4 0.6

0.5

0.3ab

y

AND

ac

c

yAND

ORb

Traditional Logic Synthesis•Manipulate the representations of Boolean functions

Logic Synthesis for Probabilistic Computation

(Implement the same function y = ac + bc)

The Boolean functions can be different!

ab

yOR

ANDc

Emerging Nanotechnologies

VDD

Nanowire Crossbar (Idealized)

A

VDD

AA1

A2

A3

A4

• Opportunities: High density of bits/interconnects• Challenges: Inherent structural randomness; High defect rates

A collection of inverters with shuffled outputs!

Nanowire Crossbar Array

VDD

VDD VDD

A1

A2

A3

A4

B1

B2

B3

B4

A4B3

A1B2

A2B4

A3B1

A

B

C

Shuffled AND

Inputs to AND gates are shuffled

Shuffled AND: Multiplication

A

B

C

c = P(C=1) = P(A=1)•P(B=1) = a • b

Inputs to AND gates are shuffled

010110

x = 3/6

Multiplication

Probabilistic Bundles0011 0

0

1

1

Other Emerging Technologies

– Carbon nanotubes

– Molecular switches

– DNA

Thank You!

Digital yet Deliberately Random: Synthesizing Logical Computation on Stochastic Bit Streams Ph.D....

Documents

Transcript of Digital yet Deliberately Random: Synthesizing Logical Computation on Stochastic Bit Streams Ph.D....