LOS TRICERAFLOPSAdrian Buckley

Evan Defibaugh

John Duhamel

Alex Kaiser

Darnell Parker

Craig Riggins

T Q

~Q

Project Objectives

- Relay real time Nintendo 64 session data between multiple hosts for network play

- Provide an interface for a Microsoft Kinect frontend in place of tradition controller

- Utilize non-obtrusive electronics- Minimize cost and maximize functionality- Power consumption is a secondary concern

System Architecture

Client 1 Client 2Server

Nintendo 64

ServerClient 1 Client 2

Player 1 Television

Player 2 Television

Player 1Controller Signal

Player 2Controller Signal

Player 2Controller Signal

Player 1Controller Signal

N64 RCASignal

N64 RCASignal

N64 RCASignalN64 RCA

SignalN64 RCA

Signal

Client Architecture

Regular N64 Controllers

TelevisionARM Cortex

MCU

N64 Controller Signal

KinectUSB Signal

DigitalRCA Signal

DigitalController Signal

Ethernet IO

Microsoft Kinect

Analog RCASignal

MCU takes input from 4 N64 controllers Microsoft Kinect can track 2 people

Server Architecture

ARM Cortex MCU

DigitalController Signal

DigitalRCA Signal

Ethernet IO

Nintendo 64

N64 ControllerSignal

AnalogRCA Signal

Can communicate with 1-4 clients Moves controller signal from client to N64

and RCA signal from N64 to client

Unified Hardware

N64 Controllers / Microsoft Kinect Television

ARM Cortex MCU

Controller Signal

RCA Signal

Controller Signal

Ethernet IO

RCA Signal

Nintendo 64

ControllerSignal

RCA Signal

1 hardware architecture will be able to operate as either the client or the server

Advanced System Architecture

Client 2Client 1 / Server

Nintendo 64

Client 1 / Server

Client 2

Player 1 Television

Player 2 Television

Player 2Controller Signal

Player 2Controller Signal

Player 1Controller Signal

N64 RCASignal

N64 RCASignalN64 RCA

SignalN64 RCA

Signal

Kinect Integrated Peripheral

Kinect Processor Linux Microcontroller

XBOX Kinect ProcessorLinux

Microcontroller

Kinect Sensor

I/O2 Cameras (RGB and IR Depth)IR Output4 Array MicrophoneUSB Port

Possible Implementations

ROS: BeagleBone VS C# 2010: Netduino VS C# 2010: PC

Implementing Kinect via ROS on Beaglebone

ROS (Robot Operating System) is an open source project focusing on interfacing microprocessors with robots (in our case the Kinect and a peripheral Linux microcontroller)

Kinect Processor

Control Node

The Netduino operates on Microsoft Visual Studios C# 2010 via Net. Micro workforce

Implementing Kinect via VS C# 2010 on Netduino

Minor Hiccups in Progression

Using the Robot OS ROS tutorial has

incorrect/missing info

Open source, thus compatibility issues from node to node

Unintuitive to navigate

Using the Netduino• Netduino requires

Net. Micro framework, where Kinect requires Net. Framework

• Trouble converting from standard to micro

There is no problems implementing Kinect via PC

Last Resort and Additional info

As a last resort the Kinect can be routed through a Windows PC for processing then sent to microcontroller.

More ROS infohttp://www.ros.org/wiki/kinect

(Install ROS)

http://www.ros.org/wiki/ROS/Tutorials

(ROS Tutorial)

http://beagleboard.org/bone

(Beaglebone Description)

http://www.ros.org/browse/list.php

(ROS software)

More Net. Micro workforcehttp://docs.com/BTN4 (Installing OpenNI)

http://www.netduino.com/netduino/schematic.pdf (Netduino Schematics)

http://informatix.miloush.net/Microframework/Articles/WpfEmulators.aspx

(.net micro install)

http://informatix.miloush.net/microframework/Articles/Reflector.aspx (.net micro reflector)

http://netduino.com/netduino/specs.htm

(Netduino Specs)

Gesture and Speech Algorithms

Joint Data is returned in X, Y, Z, and W (confidence) data

Evaluate position of joints relative to another and trigger button pressed event

For audio grammar files must be written Compare text output of audio and set trigger

for the controller or the sensor as being used Output control trigger and button triggers

Kinect Processor Interface

The Processor will output triggers in the form of buttons and sensor/controller triggers for both players one and two

The triggers will be relayed to the Linux microcontroller that will convert button trigger to N64 controller output format and use the control trigger to decide if forwarding the N64 controller data or sensor converted data

N64 Controller Tutorial

How the N64 Controller Sends Its Data

The N64 controller only has three pins: Ground, data, and VCC. It takes a +3.6V supply.

The N64 uses a protocol in which bits are sent serially to and from the controller as 4µs-wide pulses. ‘Low’ is 3 µs low and 1 µs high, and ‘High' is 1 µs low and 3 µs high.

N64 Data Polling

The N64 continually polls the controller using an 8-bit command (0x01), and the controller reports the state of its buttons and analog joystick with a 32-bit response: 16 bits for the buttons, signed 8 bits for horizontal joystick position, and signed 8 for vertical. Each bit packet ends with a stop bit (1).

Button Configuration

The buttons are mapped to their respective bits as follows:

Sample Joystick Positions and respective bits:

Bit 0 1 2 3 4 5 6 7

Button A B Z Start D-Up D-Down D-Left D-Right

Bit 8 9 10 11 12 13 14 15

Button Null Null L R C-Up C-Down C-Left C-Right

Left Extreme 1 0 0 0 0 0 0 1

Left Off Center 1 1 1 1 1 1 1 1

Right Off Center 0 0 0 0 0 0 0 1

Right Extreme 0 1 1 1 1 1 1 1

Up Off Center 0 0 0 0 0 0 0 1

Down Off Center 1 1 1 1 1 1 1 1

Joystick Off-Center Left & Down

The Translator Box The Kinect will take player movement/sound data and

send over button and trigger data to the “translator box,” which then converts that data into the 64-bit struct format shown below.

There will be a physical master switch on the translator box, to switch between Kinect control and N64 controller control. When this switch is set to “Kinect,” the translator will await trigger and button data from the Kinect to process.

If the switch is set to “N64,” the translator will need to keep track of one or two controllers, simultaneously converting their button presses into the shown format, but with the “More Buttons” bit and time delay consistently set to zero.

Button Data Joystick Data Which Player? More Buttons? Time Delay

16 bits 16 bits 1 bit (0 or 1) 1 bit Int (30-bit)

Server to Translator to N64 Another Translator Box is needed to translate the

Server’s data into N64 controller data for the N64. This translator will read in the controller data, and send

the correct button sequence via serial ports to the N64, correctly timed. If the N64 polls the “controller,” but the translator has not received button data from the server, the translator will automatically respond with a 32-bit ‘0’ analog stream and continue doing so until new button data appears.

Audio/Visual Signal

The video output of the Nintendo 64 carries visual information via an analog signal, plus two synchronizing signals for aligning the video signal on a television.

The audio output, also analog, is split into two signals for stereo speakers in a television.

Audio/Visual Waveforms

Video

Audio

Audio/Visual Sampling

The audio and visual output of the Nintendo 64 must be digitized and formatted for input into the server for transport over the internet.

Once the audio/visual information is sent over the internet it must be converted back into an analog signal to be displayed on a player's television.

A Series of Tubes – Hardware

Initial design and testing will be done on an Arduino with an Ethernet shield.Ardiuno is easy to program to send and receive

data packets. After getting a fully functional system with the

Premade hardware we will move on to a custom PCB with a Linux OS.

Our Internet Objectives

We need the data received from the server to be real timeIf there is missing data we just move to the next

data packet.We know that the receiver is ready to receive

packets.

TCP/UDP

TCP/IP uses an established connection between sender and receiver.This requires an acknowledge signal from the

receiver which takes time.

UDP instead sends packets in a continuous stream and doesn’t care if packets are missing on the receiver side.Instead of waiting for acknowledgement UDP just

tags the packet with the correct receiver IP address and sends it on its way.

UDP in depth

UDP sends data with a header, and within the header a 2byte portion telling the receiver how long the data section will be.This 2byte section allows for a maximum of 64Kb

section of data which is more than enough for sending the n64 signal and frames of an RCA signal.

The other parts of the header are receiver IP and port number.

Putting it all Together

Using UDP because we need a real time connection and don’t mind dropped packets.

Using a static IP for our server so that we have something to aim at for our UDP packets.

Using Arduino and Ethernet shield to prototype and onto custome PCB once all kinks are worked out.

ID Task Name Duration Start Finish Predecessors

1 Capstone Project 79 days Mon 1/16/12 Thu 5/3/12

2 Idea Development 5 days Mon 1/16/12 Fri 1/20/12

3 Documentation 73 days Mon 1/23/12 W ed 5/2/12 2

4 Functional Specification 5 days Mon 1/23/12 Fri 1/27/12

5 Funct. Req. Document 28 days Mon 1/30/12 Wed 3/7/12 4

6 Block Diagrams 20 days Mon 1/23/12 Fri 2/17/12

7 Block Diagrams for PDR 6 days Mon 1/23/12 Mon 1/30/12

8 Block Diagrams for CDR 14 days Tue 1/31/12 Fri 2/17/12 7

9 Build User's Manual 73 days Mon 1/23/12 W ed 5/2/12

10 Preliminary version for PDR 6 days Mon 1/23/12 Mon 1/30/12 2

11 Revised version for CDR 19 days Tue 1/31/12 Fri 2/24/12 10

12 Final version for Design Expo 48 days Mon 2/27/12 Wed 5/2/12 11

13 System Engineering 28 days Mon 1/30/12 W ed 3/7/12

14 Kinect Design 21 days Mon 1/30/12 Mon 2/27/12 4

15 N64 Signal Design 21 days Mon 1/30/12 Mon 2/27/12 4

16 Ethernet Design 21 days Mon 1/30/12 Mon 2/27/12 4

17 RCA Design 28 days Mon 1/30/12 Wed 3/7/12 4

18 Integration 37 days Tue 2/28/12 W ed 4/18/12

19 Kinect/N64 Integration 14 days Tue 3/6/12 Wed 4/4/12 15

20 N64 to Server Integration 14 days Tue 2/28/12 Fri 3/16/12 15

21 Server/RCA Integration 14 days Tue 2/28/12 Fri 3/16/12 15

22 PCB Review 30 days Thu 3/8/12 Wed 4/18/12 13

23 Beta Testing 11 days Thu 4/19/12 Thu 5/3/12 22

24

25 Initial Design Objectives Due 1 day Mon 1/23/12 Mon 1/23/12 2

26 Preliminary Design Review 1 day Tue 1/31/12 Tue 1/31/12 10,7,2

27 Critical Design Review 1 day Mon 2/27/12 Mon 2/27/12 11,8

28 Milestone 1 1 day Tue 3/20/12 Tue 3/20/12 13

29 Milestone 2 1 day Tue 4/10/12 Tue 4/10/12 21

30 Technical Reference Manual 73 days Mon 1/23/12 Wed 5/2/12

31 Capstone Expo 1 day Thu 5/3/12 Thu 5/3/12 23,12,18,13,3

11 14 17 20 23 26 29 1 4 7 10 13 16 19 22 25 28 31 3 6 9 12 15 18 21 24 27 1 4 7 10 13 16 19 22 25 28 31 3 6 9 12 15 18 21 24 27 30 3 6 9December 2011 January 2012 February 2012 March 2012 April 2012 May 2012

Project Schedule

Longest Design Period

11 Days for Testing

Technical Reference Manual

37 Days for Integration

Division of LaborTask Adrian Evan John Alex Darnell Craig

Program Mgmt & Documentation

P S S

Kinect Programming S P S

N64 Translation S P S

Ethernet Interface P P

Controller Programming S S P

Audio/Video Signal Conversion

S P S

N64 Gameplay Action P P P P P P

P = Primary S = Secondary

Advanced Parts ListHardware Cost Quantity Extended Cost Possible Substitutions

Television Provided 2 ProvidedXBOX Kinect Provided 2 ProvidedNintendo 64 Provided 2 ProvidedNintendo 64 Controller Provided 3 ProvidedNintendo 64 Controller $19 4 $76Nintendo 64 Controller Cable Extension $10 2 $20Nintendo 64 ROMS Provided 7 ProvidedNetduino $60 2 $120Beagle Bone $90 3 $270 Beagle BoardModem Provided 1 ProvidedParts for Printed Circuit Board $50 1 $50XBOX Kinect / Nintendo 64 Shield $14 2 $28RCA Cable Provided 1 Provided Coax CableCase $10 1 $10PCB Development / Revisions $500 TBD $500

TOTAL ESTIMATED PROJECT COST = $1,074

*NOTE* – ALL EXPENSES PAID TRIP TO LAS VEGAS FOR “ELECTRONICS SEMINAR” NOT INCLUDED

GoalsPrimary:• Control Nintendo 64 game exclusively via an XBOX Kinect

Secondary:• Have one N64 controller communicate with one N64

console over the internet• Control the N64 game using either the Kinect or the N64

controller remotely

Tertiary:• Play a complete N64 game with 2 players at one location

using the Kinect, and the other 2 players at a different location using N64 controllers

• Allow functionality with a variety of different controllers

Milestone 1 Objectives

Kinect Translation • Kinect translator sends simple gesture data to

the Linux µController, in correct format

Nintendo 64• Send properly timed Nintendo 64 analog

signal from the µController to the N64 console

Network Configuration• Send data packets across the network to and

from each user using the µController

Audio/Video• Convert analog input into a digital code

Milestone 2 Objectives

Kinect Determine ideal Kinect gestures for

satisfactory user interface Transmit verbal commands through Kinect

Network Configuration Send and receive N64 and audio/video

signals across server

Audio/Video Output signal from server to television

Risks and Contingency Plan

Risks Syncing the N64 timing sequence with the Netduino

microcontroller’s internal timer Busy internet path/lost data packets Delay between Kinect / N64 input and television

output Data corruption or potential injury resulting from

falling or player collisions while playing with a motion sensitive controller

Solutions Use only N64 controllers (no Kinect) for game control Leave out server / direct connection from

microprocessor to N64 User plays video game at own risk!!

Questions???