NEStalgia Demonstration Video and Project Files

Working on NEStalgia has been a blast, and we've learned so much this past semester. Check out a video taken of our final demonstration below, as well as a link to download a zip file with our final report. Our final report contains a detailed document describing the project with pictures, schematics, and source code.

Download NEStalgia Report, Schematics, Source Code

Penn's Engineering Dean Checks Out NEStalgia!

A few days after our initial demo of NEStalgia, we were invited to show off our project to Eduardo Glandt, the Dean of Penn's Engineering school. It was a huge honor for us, and we think he really enjoyed our project. Check out a picture of us with the Dean below:

Love, Loss, and Redemption

Over the past week, we have accomplished a lot. The first thing we did was insert a Firefly node into the chamber of the toy gun. This node contained a sensor board with a 3-axis accelerometer, which would allow us to detect changes in acceleration.

Toy gun with Firefly node inside

To ensure that the Firefly node would fit in the toy gun, we desoldered the default attached battery pack and soldered new wires to the Firefly that led to a more compact AAA battery pack (stolen from a small flashlight).

Firefly node inside toy gun with compact AAA battery pack stolen from flashlight (4.5 V regulated to 3.3 V by Firefly)

We also wired a trigger circuit consisting of a low-asserting switch to one of the GPIO pins of the Firefly node in the gun. This would allow us to detect "trigger presses" and output the A button accordingly.

Complete toy gun with Firefly, compact AAA battery pack, trigger switch circuit, and gyroscope (read on)

This setup did not work as well as we expected. Part of this was due to our misunderstanding of how an accelerometer actually works. Accelerometers measure acceleration, not an integrated velocity or position value. This means that the accelerometer's output will only change when we initially move the gun in a direction, and then also when the gun stops moving (deceleration). This is counter-intuitive when playing a game like Cabal. A simple swing of the gun to the right would initially move the character right and then move the character left because of the deceleration of the movement.

After some brainstorming, we decided to try a different approach for character movement. Instead of using the toy gun for both character movement and crosshair movement, we implemented two more Firefly nodes that were strapped to our ankles. A tap of either the left or right foot would move the character in that direction. This proved to work much better than simply moving the entire toy gun left or right and relying on accelerometer data.

Using a Firefly node on each foot to detect "foot taps" for character movement

However, the crosshair movement was still largely inaccurate. To remedy this, we decided to try a gyroscope. Unlike an accelerometer, a gyroscope is not affected by the slowing down or stopping of a gun swing. A gyroscope's output changes when it detects a change in angle about one of its two primary axes. Using the gyroscope was much more accurate than using the accelerometer.

Soldering a ribbon cable to the Sparkfun IDG1215 2-axis gyroscope breakout board

However, when we finally strapped the two Firefly nodes to our feet and played the game with the gun, we realized something fundamentally important about our project. It seemed that we had lost sight of our original goal, which was to encourage fun physical interactivity with video games. We realized that the two Firefly leg nodes took a considerable amount of foot tapping and quickly became more work than fun. And it was at that point that we decided to radically change our original design concept. Instead of using our legs to control character movement, we decided to use our heads.

A fellow classmate tries out our head-mounted control system

By strapping a single Firefly to our foreheads, we were able to detect changes in acceleration in the X and Z axes. In other words, by looking up, down, left, or right, you could move the character on screen in that respective direction.

Evan makes a funny face while wearing the head-mounted Firefly

It might look odd, but playing the games with a Firefly mounted on the forehead turned out to be extremely fun. And, in conjunction with using the gyroscope to detect gun rotations instead of the accelerometer, we found that we had created a very impressive wireless physical interaction system for the Nintendo Entertainment System.

Our class watches a student play Super Mario Bros. 3 using our head-mounted control system

Today is Friday, April 30th. From 11 AM to 1 PM we will be demoing NEStalgia to the general public. We'll let you know how it goes, but from our initial results, we're very excited and very proud of the devices we've created.

The complete NEStalgia wireless controller system

Reading Keyboard Input

This is just a brief post - Kevin's posts and videos are usually clear and succinct enough not to warrant additional cruft by me :) For those who watched the game play video in the keyboard input post, you might have noticed that Kevin was pressing the key repeatedly before the input was accepted.

Due to Nano-RK's task based style, getting user input from the keyboard needs to be a bit of a polling hack. This is because C input methods such as getchar() are blocking methods, meaning that once called the program will not continue until the function has returned. Unfortunately Nano-RK sees this blocking as the task being in use and will not preempt it, so it never switches to the idle task. This causes the task reading user input to eventually violate its period allotted, as well as the additional reserve.

To solve this problem we loop, waiting for the signal in uart_rx_signal, which is registered with the system before. Once the signal is set, and data is ready essentially in the buffer, we then call getchar(), which should return almost immediately with the typed character.

Until we work out exactly how the uart_rx_signal is set and tighten up the timings a little bit, we will continue to experience this lag and missed keystrokes when checking for keyboard input. Fortunately however, using the keyboard for input was only a midway point to prove that we could send user-defined input to the NES. We can now safely put the keyboard aside, since the Firefly node attached to the NES will now be receiving all of its inputs from its corresponding Firefly node in the toy gun.

Hacking the Toy Gun

Today we started taking apart one of the toy guns that will eventually become the wireless controller for the NES.

Toy gun that will soon become a wireless NES controller

The toy gun is really 4 plastic parts that are held together by many screws.  Two of the parts are actually glued together at the ring of the front of the flashlight, which is located below the barrel of the toy gun.  To disassemble the gun, we unscrewed all of the screws and used a saw to cut off the glued-on ring of the flashlight.

Inside of toy gun after flashlight ring was cut off with saw

Once the ring was removed, we had easy access to all four pieces of the toy gun as well as the underlying circuit that powers the speaker and flashlight.  The gun is actually perfect for this project because it provides an easy-access panel to the battery compartment, which we can gut out and replace with the Firefly node.  This will allow for easy programming of the node while it is embedded in the toy gun.

Using a Dremel to round off the edges where we cut off the flashlight ring

Lastly, in an effort to improve the aesthetics of the gun were we had sawed off the flashlight ring, we used a Dremel to round off the edges.  During the next week, we'll embed the Firefly node in the toy gun and program it to wirelessly transmit accelerometer data to the receiver node connected to the NES.

Hardcore Hackers

Writing to the NES

Today we successfully used the computer keyboard via serial terminal to control the NES.  We tested everything using Super Mario Bros. 3.

We ran into some trouble in trying to use the Firefly to mimic the 4021 parallel/serial shift register IC that's in the NES controller.  Apparently the Atmega chip that the Firefly uses is just too fast to achieve the correct division you would need to send data out to the NES at the correct frequency (83.33 KHz).

Using a USB TV Tuner to view the NES game on a laptop

We then attempted to use SPI (Serial Peripheral Interface) with the Firefly as a slave to the NES.  That way the NES could provide the correct clock frequency and latch signal and the Firefly could output data accordingly.  However, to accomplish this on the Firefly you must de-solder a resistor and switch one of the reset lines.  Long story short, we were asked to find another solution.

Stripping the wires of an NES controller

So we decided to simply stop trying to reinvent the wheel.  We de-soldered the 4021 chip from the NES controller and connected 8 GPIO pins from the Firefly to the 8 parallel input pins of the chip.  Finally, after more than a week of head-scratching, we were able to control the NES using the PC keyboard.  Check the video below to see our setup in action.  

This week's goal will be to figure out wireless transmission so we can efficiently transmit the ADC information from the Firefly node in the gun to the receiver Firefly node connected to the NES. We'll also post pictures of what the inside of the toy gun looks like.

NES Controller

Just a brief update today. This past week we ordered two used NES controllers, and the first one came in today. We plan to cut the wire on the controller, discard the controller, strip the wires, and connect them straight to the Firefly node.

Another solution for this project could be to simply connect an NES controller to the NES, open the controller, and solder wires to the same traces that the NES buttons connect to. That way we wouldn't have to mimic the programming of the NES controller's onboard serial chip, and you could technically still play the NES with the controller if you wanted to.

However, since this is a design project, we will first attempt to create an interface to the NES that only consists of the plug and the Firefly node. If that fails, we might consider the alternative solution.

Purchase of Toy Gun

We went to Kmart last Saturday and purchased two "RuffStuff Electronic Blasters Combat M16" toy guns. The guns have a flashlight under the barrel that lights when you lightly pull the trigger. The toy guns also play gun sounds from an embedded speaker when you pull the trigger all the way.

We plan to take the toy guns apart and insert a Firefly microcontroller node into the wide part of the barrel shown below.

Interfacing the NES Controller with the Firefly Microcontroller

Today we successfully interfaced the NES controller with the Firefly microcontroller.

Following the instructions here and here, we programmed three GPIO (General Purpose Input/Output) pins on the Firefly to mimic the NES in retrieving the states of the buttons.

An NES controller contains seven wires. Two of the wires quite literally do nothing. They're not even connected to the 4021 serial IC in the controller. The other five wires are +5 V, Ground, Latch, Pulse, and Data. The NES performs the following operations in order to read the button states from the controller:

1. Every 16.667 milliseconds (60 Hz), the NES sets the Latch line high for 12 us and then low again.
2. Six us later, the NES sends out eight 12 us period pulses on the pulse line.
3. The NES controller sends out the states of the buttons for each pulse in the following order: A, B, Select, Start, Up, Down, Left Right.

It's that simple. Check out the diagram below for a more visual explanation.

The majority of the GPIO pins on the Firefly are hidden under the mini-SD card slot, so eventually we'll probably have to de-solder the mini-SD card slot to access them.

In the mean time, we used the Firefly programmer/debugger board which provides additional access to the GPIO pins. For this experiment, we used NRK_DEBUG_0, NRK_DEBUG_1, and NRK_DEBUG_2. Note the "NRK_" prefix. We omitted that prefix initially, which prevented our program from working for several hours.

NES Controller plug connected to breadboard

Firefly programmer/debugger (left) connected to Firefly node (right)

Firefly node in action

Finally, we'll leave you with a video of our program that prints out to the serial terminal what buttons are being pressed on the NES controller. The next step of the project will be to interface the Firefly directly with the NES game console so that we can play the NES using the keyboard of the computer.

Introductions!

Hello interested readers! This is a blog for the design and implementation of a Wireless Rifle Controller for the Nintendo Entertainment System, specifically for the NES video game "Cabal." This project is being completed for the Spring 2010 ESE 350 (Microcontrollers) class at the University of Pennsylvania.

So what exactly are we doing? Our idea is to create an alternative user interface for a classic Nintendo game: Cabal. We would like to provide the users with a more natural interface instead of the classic Nintendo controller (Directional pad, A, B, Start, Select).

We plan to take a stock toy gun and mount a microcontroller on or inside of it. This will allow the user to physically strafe in front of the display to strafe the character on screen, providing a more in-depth gaming experience. Right now our intention is to use the Firefly microcontroller to achieve this, most likely one on each gun and a receiver node at the NES controller input as well.

An added benefit of using this input device would be that it is wireless, allowing the user freedom of movement when playing. A possible “reach” function would be to add a small vibrating motor to give force feedback to the user when shooting.

Check out our proposal presentation slides here.

We'll post periodically with updates about the project. In the mean time, check out this video of Cabal in action: