What’s A Bit?

What’s A Bit?

Noah MendelsohnTufts UniversityEmail: [email protected]: http://www.cs.tufts.edu/~noah

COMP 40: Machine Structure and

Assembly Language Programming (Fall 2016)

mailto:[email protected]

© 2010 Noah Mendelsohn

Topics

What is information? What’s a bit? How do computer memories

store information?

© 2010 Noah Mendelsohn3

The History ofInformation Theory


Claude Shannon

1948: Claude Shannon publishes: A mathematical theory of communication*

* http://www.alcatel-lucent.com/bstj/vol27-1948/articles/bstj27-3-379.pdf Photo by Tekniska Museet

http://www.alcatel-lucent.com/bstj/vol27-1948/articles/bstj27-3-379.pdf


Questions

We can weigh things that have mass We can determine the volume of solid object or a liquid We can measure the height of the walls in this room Can we measure information? Can we distinguish more information from less? What units could we use?

5


Intuition

There is more information in the library of congress than there is in a single word of text…

…but how can we prove that rigorously?


Crucial insight

Whenever two parties communicate:– We can view the communication as answering one or more questions– Example: you and I are deciding whether to have dinner. We agree in advance

that I am going to phone you and give you the shortest possible message to convey the answer. I will say “yes” if we’re having dinner and “no” if not.

– Harder example: we also need to decide whether we’re going to the movies. This time, we agree in advance that I will say “yes, yes” for movie and dinner, “yes, no” for dinner only, “no, yes” for movie only, and “no, no” for staying home.

7

This is profound……communication is answering questions!


Crucial insight

Whenever two parties communicate:– We can view the communication as answering one or more questions– Example: you and I are deciding whether to have dinner. We agree in advance

that I am going to phone you and give you the shortest possible message to convey the answer. I will say “yes” if we’re having dinner and “no” if not.

– Harder example: we also need to decide whether we’re going to the movies. This time, we agree in advance that I will say “yes, yes” for movie and dinner, “yes, no” for dinner only, “no, yes” for movie only, and “no, no” for staying home.

8

This is profound……communication is choosing among possibilites!


Measuring information

Just saying “yes” or “no” isn’t enough… …we have to agree on what the choices are The more choices we have to make, the more “yes” or “no”

answers we’ll have to communicate

9

We define the ability to convey a single yes/no answer as a bit

We define the amount of information as the number of yes/no questions to be answered

Shannon’s paper introduced the term bit! (atributing it to

co-worker John Tukey)


How many bits for Beethoven’s 9th Symphony?

If you and I agree in advance that we are choosing between only two recordings that we both have, then:– We can choose between them with 1 bit!

If we agree in advance only that it is some digital sound recording, then:– We need enough bits so that you can choose the

intended sound wave from all possible such 70+ minute recordings*

10

* By the way, the compact disc format was chosen to have enough bits to encode Beethoven’s 9th , at 44KHz x 16bits/sample x 2 channels estimated at 74 minutes. Approximately 5,872,025,600 bits


Things to notice We always have to agree in advance what the possible

choices are:– Whether we’re having dinner or not– Which of N sound wave forms I want you to reproduce

We always have to agree on which answers (bit values) corresponds to which choices

We can use any labels we like for the bit values, e.g.– [yes] will mean Beethoven– [yes yes] will mean dinner and movie

Or…– [true] will mean Beethoven– [true false] will mean dinner and no movie

11


What if we want to encode numbers? We always have to agree in advance what the possible

choices are:– Whether we’re having dinner or not– Which of N sound wave forms I want you to reproduce– Which of N numbers I’ve stored in a computers’s memory

Question: What are good labels for encoding numbers?


Let’s try some labels for encoding numbers

Encoding Number encoded[no, no, no] 0

[no, no, yes] 1

[no, yes, no] 2

[no, yes, yes] 3

[yes, no, no] 4

[yes, no, yes] 5

[yes, yes, no] 6

[yes, yes, yes] 7

13



Encoding Number encoded[false, false, false] 0

[false, false, true] 1

[false, true, false] 2

[false, true, true] 3

[true, false, false 4

[true, false, true] 5

[true, true, false] 6

[true, true, true] 7

14

Any two labels will do……but do you notice a pattern?



Encoding Number encoded[0,0,0] 0

[0,0,1] 1

[0,1,0] 2

[0,1,1] 3

[1,0,0] 4

[1,0,1] 5

[1,1,0] 6

[1,1,1] 7

15

Hey, that’s the binary representation of the number!


Encoding numbers in a computer memory How many bits do I need if we need to encode which of 8 values are in

the memory?– 1 bit: 0 or 1 [two choices]– 2 bits: 00, 01, 10, 11 [four choices]– 3 bits: 000, 001, 010, 011, 100, 101, 110, 111 [8 choices]

Number_of_choices = 2N-bits

16

N-Bits = log2 (number_of_choices)


Encoding numbers in a computer memory

How many bits do I need if we need to encode which of 8 values are in the memory?– 1 bit: 0 or 1 [two choices]– 2 bits: 00, 01, 10, 11 [four choices]– 3 bits: 000, 001, 010, 011, 100, 101, 110, 111 [8 choices]

Number_of_choices = 2N-bits

As we said, those are binary numbers! 1 x 22 + 0 x 21 + 1 x 20

So…that’s why we label the states zero and one, because we can play this game to assign bit patterns to binary encodings of numbers

17

= 5

N-Bits = log2 (number_of_choices)

Our hardware has instructions to do very efficient arithmetic on these binary representations of numbers


Note: we will discuss negative numbers, numbers with fractions, very large and very small numbers, and arithmetic on all of these, at a later time


Software structures model real world objects and concepts

Number Students Bank statements Photographic images Sound recordings Etc.

19

These things aren’t bits!!They don’t live in computers, but…


Software structures model real world objects and concepts

Numbers Students Bank statements Photographic images Sound recordings Etc.

20

We build data structures that model them...we agree which bit patterns represent them


What we’ve learned so far…

Bits encode yes/no choices To communicate, we agree in advance on which bit patterns

represent which choices More information: more choices…which means more bits! We can store any information in a computer memory as

long as we agree on which bit patterns represent which choice

If we label the bit states 0 and 1, then binary numbers are an obvious representation for the integers

We choose other encodings for characters (e.g. ASCII), photos (pixel on/off), music (digitized wave amplitude)

21


How Do We Build Bits into Computers?


Building a bit in hardware

We need hardware that can be in a choice of two states Computer main memory history:

– 1940’s: spots on a TV tube; sound pressure waves in a mercury delay line; vacuum tubes

– 1950’s: rotating magnetic drum; vacuum tubes– 1950s – 1970s: tiny magnetizeable iron donuts (core memory)– 1970s – present: charges on a capacitor driving a transistor

Computer bulk storage– Magnetizeable tape– Magnetizeable disk– Transistors holding charge or solid state magnetic devices

23

These vary in cost/size/speed – all encode bits


Technology for Storing Bits

24

Relay

Thyratrons&

Vacuum Tubes*


Technology for Storing Bits

25

Punch Cards

Transistors Core Memory

Limited Integration:

Magnetic Tape

Integrated circuit USB Key


Binary Numbers


Learn your binary numbers

27

N 2N

0 1

1 2

2 4

3 8

4 16

5 32

6 64

7 128

N 2N

8 256

9 512

10 1024

11 2048

12 4096

13 8192

14 16384

15 32768

16 65536

220 ~= 1M 230 ~= 1B 232 ~= 4B 264 = HUGE


Another way to think about it

28

0816

1011 = 11 (decimal)


Another way to think about it

29

0816

1011


Another way to think about binary numbers

30

0816

1011



31

0816

1011



32

0816

1011

The binary representation encodes a binary search for the number!


Characters, Numbers and Puns


Is it a character or a number?

34

#include <stdio.h>#include <stdlib.h>

int main(int argc, char *argv[]){

(void)argc;(void)argv;

unsigned char var1 = 'N';unsigned char var2 = 'O';unsigned char var3 = 'A';unsigned char var4 = 'H';

/* %c prints as character %u prints unsigned integer */

printf("The number for %c is %u\n", var1, var1);printf("The number for %c is %u\n", var2, var2);printf("The number for %c is %u\n", var3, var3);printf("The number for %c is %u\n", var4, var4);

exit(EXIT_SUCCESS);}

This program prints:

The number for N is 78The number for O is 79The number for A is 65The number for H is 72

On day 1, we noted that characters have number codes (ASCII, Unicode)

That’s why C character types are actually numeric

types.


What does this program print?

35

#include <stdio.h>#include <stdlib.h>

intmain(int argc, char *argv[]) {

char c = 'A';

while (c <= 'Z') {putchar(c);c = c + 1;

}

putchar('\n');

exit(EXIT_SUCCESS);}

ABCDEFGHIJKLMNOPQRSTUVWXYZ

PunsCharacters are numbersBoth are bits

Much of modern computing is based on this amazing formulation..this is why you can compute using the same hardware on characters, numbers, image color and brightness, sounds, etc., etc.


The Structure ofComputer Memories


0 1 1 0 0 1 1 0 1 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 1 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 1 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 1 1 1 0 0 1 1 0

32 bits


The logical structure of computer memory

0 1 1 0 0 1 1 0 1 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 1 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 1 1 1 0 0 1 1 0 0 1 1 0 0 1 1 0 1 1 1 0 0 1 1 0

Can we get a C pointer to a bit in memory?

Pointers (on most modern machines) are to whole bytes

0 1 2 3 4 5 6 7

NO!

Addr:


Why byte addressing?

Can address more memory with smaller pointers Not too big, not too small 256 values: about right for holding the characters Western

cultures need (ASCII) – one character fits in one byte 8 is a power of 2 … we’ll see advantages later Unfortunately: we need multiple byte representations for

non-alphabetic languages (Chinese, Japanese Kanji etc.) – we deal with that in software

39

What’s the largest integer we can store in a byte?


Computers can work efficiently with bigger words

40

Byte

• C has types for these• The hardware has instructions to directly manipulate these• The memory system moves these efficiently (in parallel)

Sizes vary with machine architecture: these are for AMD 64

BYTE (8) SHORT (16) INT (32)

LONG (64)

POINTER (64)

Byte Byte Byte Byte Byte Byte Byte


Review


Review

Bits encode choices We can thus choose a representation for information in bits We can interpret the same bit values in different ways

(number 66 or ASCII letter C) If we call the bit states 0 & 1: then we easily get binary

numbers We know how to implement bit stores in hardware and to

compute with them We generally address bytes not bits We often use words of types like integer…these are useful,

and the machine handles them efficiently

42


Abstractions -- again


Abstractions Are Layered at Every Level in our Systems• “Real world” concepts modeled in Hanson ADTs & C Types

• Hanson ADTs implemented in C Types• Soon: bits & bytes used encode machine instructions

• Words, Bytes and bits used to implement C types• Bytes grouped in hardware to make words (int, long, etc.)

• True/false bits grouped to make bytes• Information modeled as true/false bits

• True/false bits encoded in charges on transistors


An Aside on Information Theory


We’ve over-simplified the story a little

What we’ve said about bits and choices is true However:

– Many encodings are wasteful…I.e. values of the bits are somewhat predictable– Example: for each day of the year: [1=There was a hurricane, 0=No hurricane]

…we know most bits will be zero– Can you find a better encoding?

To really measure information: we use the smallest possible encoding

Also: Shannon didn’t just measure information…he predicted how reliably you could send it through a noisy transmission line

46

Still: what we’ve studied here is a great start on thinking about bits and information, which are the foundations for

modern digital computing.

What’s A Bit?

Documents

Transcript of What’s A Bit?