First attempt at an IR hand, head, body, etc. tracker / dataglove using the Wiimote or a simple camera

The Wiimote as IR tracker

We discovered that the Wiimote contains a fast and accurate tracker for IR light sources. The Wiimote looks for small, bright IR light sources, and reports the position and estimated radius of the sources it finds. It can track up to four marker spots simultaneously. We want to try and build useable tracker applications with it. The following video served as inspiration: Finger tracking using Wiimote by Johnny Lee, Carnegie Mellon University.

See also: 3D tracking using 2 Wiimotes

The basic idea is that, in contrast with normal usage, the Wiimote functions as a fixed camera tracking a moving light source, rather than as a moving mouse-like device tracking a fixed light source (the fixed light source that comes with the Wiimote is called the "sensor bar", which is attached to your TV). A stereo image from two Wiimotes would even enable 3D tracking. It is possible to use four Wiimotes simultaneously, and maybe more.

Connecting the Wiimote to a PC

I connected the Wiimote to a Linux PC by means of a Bluetooth USB dongle. I enabled the USB and Bluetooth kernel options in the Linux 2.6 kernel, and it worked without problems. As there is no standard for Bluetooth USB dongles, you should ensure the particular dongle you want to use is supported by Linux. A lot of dongles are supported though. I use a Conceptronic CBT200U2A.

The Wiimote speaks the HID (Human Interface Device) protocol, also used for Bluetooth keyboards, mice, etc. I used the CWiid library to talk to the remote. CWiid has an easy C interface for writing your own Wiimote software.

Positioning the Wiimote

The Wiimote can be placed next to the computer screen, but I think that having the tracking center correspond to the center of the screen will be significantly more user-friendly. You control the user interface by hovering your hands above or in front of the keyboard. Switching between keyboard and tracker control should be even easier than switching between keyboard and mouse.

The Wiimote could be attached to the top of the computer screen pointing downwards. You'd need something like a regular webcam holder, though the Wiimote is both larger and heavier than a typical webcam. One might also use a regular tripod, placed behind the screen (this would work better for TFT screens). An alternative is to hang it from the ceiling. Another is to have the Wiimote stand upright below the screen, and use a near horizontal mirror, either placed just above the Wii or attached to the top of the screen, to make the Wii see your hands properly.

I used a small 5x10 cm mirror attached to a piece of metal wire, held into place by a stack of books. This mirror configuration manages to track your hands in a 20cm or so radius right below and in front of the wiimote. The result looks something like this:

Wiimote technical considerations I found the following:

A cheap mini camera as IR tracker

A second discovery I made is that a cheap video camera with low image quality and even a blurry image is well able to track a light source position and size up to very high accuracies. I tested this with a common mini video camera connected to a TV card. This setup has an advantage over webcams, as it does not compress that data in any way. You usually don't want to compress image data that you want to use for image processing. The camera looks like this:

Actually the picture shows an identical-looking variant of the same series. There are also many other types of cheap video camera. This one is a B/W camera with IR (that is, it does not have an IR blocking filter), which saves us the trouble of having to remove the IR filter. The LEDs on both sides of the lens are in fact IR LEDs used to illuminate the scene in the dark. This type of mini camera is easily attached to the keyboard:

The camera is taped to a piece of 1mm aluminium sheet. The sheet is attached to the keyboard by means of a clamp. The sheet can be bent, so you can bend the camera into the proper angle (horizontal or slightly upwards). In this position it can track your hands above and in front of the keyboard. Note that the LEDs are taped off and the piece of film in front of the lens acts as IR filter.

With a very simple algorithm I managed to get position resolutions up to 1/100 pixel (that's something like 1/100 mm at the use distance!) and distance resolutions (based on size) of 1/10 mm and smaller. This is enough to track the 3D position of a light source without having to revert to stereo interpretation and other fancy techniques.

The camera I'm using produces significant noise when there are low light levels, because it amplifies the signal. This happens in typical cases when tracking light sources. This means that the 1/100 pixel accuracy cannot be achieved, and with the noise we're at something like 1/10 or 1/20 pixel, which is still pretty good.

Tracking two light sources Tracking two light sources with the camera also turned out to be fairly easy too. An algorithm first determines the orientation of the outlying pixels in the scene (outlying wrt the light center), which is used to estimate the dividing line between the two assumed light sources. The radius and position of each source can be computed in a way similar to the single light source case.

The availability of two 3D positions can be used to determine 5 DOF or 6DOF of certain trackable objects (5DOF: stylus with two leds; 6DOF: flat surface, as long as oriented towards camera).

Tracking things with trackable marker spots

Things that we would like to track:

Basically there are two ways to obtain a trackable marker spot: one is to have a light source (lamp or LED) placed on the location to be tracked, the other is to place reflective material on the location and shine on it with a bright lamp or LED array. Here is a preliminary comparison of the two techniques:

Aspect Light source on target Reflective material on target
Visibility of light source The light source is allowed to emit some visible light (but not too much). We can use incandescent bulbs or LEDs. The light source must not emit any visible light, to prevent annoyance. This means we have to use LEDs.
Portability Light source must be as small as possible, and power source should be small and close to the light source. Reflective material can be made very lightweight and non-intrusive.
Angle of light emission Problematic. LEDs typically have narrow angle, and most light sources distribute their light unevenly. A lens or diffuser may help. Well suited. A "white" reflector can be made to reflect light at 180 degrees or even 360 degrees (such as white tape wrapped around a finger)
Size of marker spot Can be made very small. Small LEDs or lamps make very small and bright light sources Likely cannot be made very small.

Light source considerations

We consider two types of light sources here: IR LEDs and incandescent light bulbs.

Most IR LEDs have similarly high light yield and emit at a wavelength of 840-940 nm. Most have a narrow angle of emittance. Voltage drop is typically around 1.4 volts. LEDs need to be connected to the power source in series with a resistor of around 100 ohm to stabilise the current. This means the supply voltage will have to be at least twice the LED voltage drop. So, to use them as on-target light sources, you will need at least two or three AAA batteries or a 9V battery.

Incandescent light bulbs have their peak of emittance around 1000 nm. In fact, they are better IR light sources than they are visible-light sources. For our experiments, a small lamp (possibly covered by a cheap IR filter) may be a good choice for on-target light sources. Suitable low-voltage lamps are available and can be connected directly to a single AAA battery.

Tracking experiments

First dataglove model using on-target light sources

For on-target light sources, I chose small 1.2 V, 0.24 W light bulbs, taken from a 220V Christmas light strand with 200 bulbs. When connected to a single AAA battery they emit enough light to be tracked by the Wiimote, and should last for several hours. The lamps are placed on top of the fingernails using duct tape rings which can be slipped around the fingers. This way you can use your fingers almost unrestrainedly, while a Wiimote looking at your hands from above should be well able to pick up the locations of the fingertips by tracking the bulb filaments (which emit nearly 180 degrees in all directions). The battery is worn using a Velcro wrist strap. This is what the result looks like:

The light sources can be made nearly invisible too, by wrapping a piece of black camera (negative or slide) film around them. The film acts as a cheap IR filter (see for example this graph). I tried covering the lights with some film, and the light becomes much dimmer to the eye (though not quite invisible), while the Wiimote performance did not seem to degrade.

I played around with this glove, and it seems to work pretty well, if you manage to place the Wiimote so that it is at close distance (50cm or less) and it sees your hands from above. The Wiimote cannot handle the light bulbs when it sees them from the top rather than the side. The top of the bulb (which is a sort of cone shape) acts as a kind of lens, which is not suitable for the Wiimote at this brightness and distance (you get multiple small light sources). LEDs may have a similar problem, since the distribution of light emitted from the top is not always very regular, that is, even within the specified angle of emittance. One problem with this glove is that the thumb cannot be detected properly because the light shines sideways rather than upwards, and is often obscured when moving your hands into specific positions. A finger tracker without the thumb may work well, though.

Second dataglove model using an IR LED

I still found that the mini bulb light sources show up a bit flickery on the Wiimote's tracker, even at short distances. So, I tried to make a better glove model. I made a similar dataglove with an IR LED (type LD 274). It has a angle of radiance of only about 10 degrees. The light source was made more suitable by using a bit of toilet paper wrapped around the LED to act as diffuser. The diffuseness and brightness of the resulting light source was checked with help of a cheap cellphone with two layers of film taped over its camera, effectively turning the phone into an IR camera (check out this photo of a daylight scene made with the phone). The power source is two 1.2V batteries. The LED is soldered to a resistor so you get the following electrical circuit: (this was calculated using the led circuit wizard; resistor is on the safe side)

The result looks like this:

With help of the diffuser, this LED version of the dataglove works better than the lamp version, as I found the Wiimote can track it to larger distances.

Stylus using two small light bulbs

This stylus is meant to be used to track 5DOF with the camera and the two-light-source algorithm. It consists of two light bulbs attached to the ends of a small section of PVC pipe, within which a battery is placed. The light bulbs are covered with cotton tissue to obtain large, even light sources.

In contrast to finger-mounted light sources, the stylus can be rotated in any orientation over a large angle with only small movements of the fingers that hold it. When light sources are mounted on one or two fingers, rotation of the wrist is required to produce some of the rotations, making the required hand movement larger and more tiring.

The stylus can be tracked reasonably well, when it is held in one hand, with the orientation of the hand itself being fixed, and only the fingers manipulating the stylus's orientation. The camera position can then be chosen so that the hand does not obscure the stylus. One major problem with the stylus is that it is easy to accidentally obscure one light source by the stylus itself. This happens inadvertently and seems difficult to prevent.

Head tracker using LEDs mounted on safety goggles

Safety goggles proved to be stable enough to hold the LEDs and the corresponding two AAA batteries. The LEDs can be placed one above each eye (obtaining 3D eye position) or one LED can be placed in the middle (for head movement only).

I used the latter configuration to obtain a useable head-tracked input method. The basic idea is that a person has a natural tendency to move his head in such a way as to look "around" the model as desired. The model can be rotated as would be expected from those corresponding head movements by means of the single-light-source head tracker, with some assumptions. This can be used as a sort of model rotation input method, however, the head movements required to rotate the model would be quite large. I used this basic method, only with the speed of the rotation being amplified a number of times. This retains enough naturalness to be intuitive, and it enables very small head movements to be used to rotate the model.

See the head tracker in action!

Dataglove model using reflective spots?

I also want to try the reflective spot scheme using a black glove with white or reflective fingertips. For that we can purchase a LED array light source. We'd need something like a LED array with about 25-100 LEDs, which are available starting from 20 euros.

Head tracker using black anaglyph glasses with white markers?

Tracking position and orientation of the head may be possible by having marker spots on the head or a pair of glasses. This enables automatic rotation of a 3D model by rotating the head. This may make for extra realism and user-fiendlyness for stereoscopic viewing.