Eyepatch: Camera based interaction

Eyepatch is a design tool for camera based interaction. It allows designers to extract useful information from video input without any specialized computer vision programming. The extracted data can be sent as a live stream to other rapid prototyping tools like Flash and Visual Basic. Data is extracted from video using classifiers. In an example, a designer building an interactive whiteboard application has mounted a camera above the white board. The objective of this is to know when someone is in front of the board and whether or not they are looking at it. she can do this using some built-in classifiers.

Face detection classifier and background subtraction classifier – eye patch shows the detected regions. After activating an output, such as XML over TCP, eyepatch begins streaming the output over a network socket. A TelNet terminal can be opened to show the formatted output steam. This XML data can be read into flash with a short snippet of action script.

In many cases, users need to train their own classifiers that are tailored to their application and its context. They can do this by selecting examples from recorded videos and training a classifier from those examples. Eyepatch provides seven different types of classifiers.

The colour classifier: extracts a hue histogram from the captured examples. When a trained colour classifier is run on an input frame, it computes a back projection image in which the intensity of each pixel corresponds to its probability of coming from the hue distribution of the training set. The brightest contiguous regions fromt he image are then output from the classifier. This can be used to track objects that are distinctively coloured.

The brightness classifier: works in a similar fashion. It looks at the brightness distribution of the examples and finds regions of the image of a similar brightness. This can be used to track laser pointers or flashlights.

The shape classifier: uses canny edge detection followed by contour matching, using ParaWise geometrical histograms. In an example, the user can train a recognizer for a USB key. After adding it to the example set (highlighting the area) the contour of the object is extracted and when the classifier is run on input frame, it finds any contours in the frame that closely resemble the contours in the examples. This works best against simple backgrounds and objects with distinctive outer contours.

SIFT classifier: Users can also train classifiers based on scale and varient feature transforms (SIFTs). These classifiers have better invariance to object pose, scale and illumination. In an example, the SIFT features are extracted from the bottle logo (highlighting). SIFT can be more reliable that shape matching in situations when the background is messy and the object has less clearly defined contours. SIFT works well for recognizing specific objects but for training recognizers for a general class of objects, such as faces or cars, we provide a classifier based on  the adaboost classifier: A machine learning algorithm that builds a boosted cascade of classifier based on simple features.

In an example, the user is training a car recognizer. She selects a collection of positive recognition examples by left clicking and dragging the mouse and negative examples by right clicking. In general, the more examples that are added to the training set, the better the classifier that will be produced. Extra examples can also be loaded from the collections of static images.

Motion classifiers: Dynamic events can be recognized with motion classifiers which use segmented motion-history images to characterize motion. In an example, someone is waving his hand back and forth. After selecting a portion of the video, in which the hand is moving to the right, the system detects motion to the right. The motion components of the input frame are compared against the motion directions in the examples. And regions that match are output from the classifier. Here, the classifier that was just trained to recognize motion to the right is run on live video. And you can see that the only motion to the right is shown in the classifier output frame in the image below.

Gesture Classifier: Sometimes, instantaneous motion direction is not enough and the user wishes to recognize patterns of motion over time. This could be accomplished with the Gesture Classifier which uses blob detection followed by motion trajectory matching using the condensation algorithm. Upon switching into gesture mode, the system quickly runs through the video, performs blob detection and captures the motion trajectories of the detected blobs. The user can then scroll through the video and see these motion trajectories as green trails overlaid on the video. A particular motion trajectory can be added to the set of training examples by highlighting that range of video frames and clicking “grab range” (button). In an example, the user trains two different gesture types: an up and down motion and a circular motion. When the train classifier is run on a sequence of input frames, the system shows the match probabilities for each gesture.

Eyepatch uses interactive machine learning to allow designers to create, test and refine their own customizes computer vision algorithms without writing any specialized code. It’s diversity of classification strategies makes it adaptable to a wide variety of camera-based applications.

Video

Diamonds Edge

Video

This is a tool where the designer can draw on paper, transfer it to the digital world and then back to paper again. It is a collaborative design tool.

**Pictionaire**

Pictionaire

Pictionaire is a large interactive table designed to support collocated, creative collaboration. The standing high table measures 4×6, a size that encourages group work. The table senses direct touch, as well as location and orientation of objects placed on it. A high resolution digital still camera captures photographs from above.

Interaction techniques

Users can create digital copies of physical artifacts with a drag off gesture. To annotate, users can drag digital items from table, onto paper-sized whiteboard slates. Substituting paper for whiteboards, Pictionaire can act as a light table to create hand drawn copies. Users drag images into their notebook and trace relevant parts manually. Overhead image capture and projection also enable spotlight highlighting on physical artifacts.

Wireless keyboards and mice are supported by the table – enabling efficient text input and image search. To organize visual material, Pictionaire offers collection containers.

Evaluation

Sixteen design professionals evaluated the table in open-ended sessions. In addition, eight graduate students completed a thirty minute brainstorming task in pairs. The evaluation revealed that the combination of a large table form factor, general purpose image capture, and digital image management supported a wide variety of creative work styles.

Video

As the size of interactive tables increases and people begin to investigate their use for productivity tasks, it becomes beneficial to consider augmenting table tops with mice and key boards. Pictionaire augments and interactive table top with multiple keyboards and mice. These peripheral devices are tracked by the table and it is demonstrated how their combined input facilitates novel interaction opportunities.

Keyboard and table interaction techniques

Several methods for establishing and visualizing the target for text entry are offered.

Link – by – locking

Dragging an object such  as a text box, until it is adjacent to the key board is one way of establishing keyboard focus. Another way to establish focus is to pace the keyboard on top of an object such as a text box. If one begins typing when the keyboard’s focus has not been established, a contextual command prompt appears adjacent to the keyboard. The commands meaning can then be changed by soft buttons projected alongside the keyboard. For example, the user can invoke an image search on the web. The location and position of keyboards on the surface relative to each other might impact the semantics of the text that is entered. For example, distance between keyboards is used to switch between individual and joint queries. Whether joint queries are conjunctive or dysjunctive is expressed by the angle between the two involved keyboards. An open angle broadens the search and a narrow one restricts it.

Collaborative Editing

In text-editing mode, placing keyboards side by side allows them to collaboratively edit a single block of text .

Mouse and table interaction techniques

A leader line can help one locate their mouse cursor on the large surface. The mouse also enables users to interact with objects out of reach, such a rotating objects by rotating the mouse itself or scaling the object using the scroll wheel. Alternatively mouse cursors can be treated as touch inputs in a multi-touch system. Then traditional actions, such as two-fingered resizing, can be achieved using either multiple mice or combinations of fingers and mice.

Proximity information can be used to associate and dissociate mice and keyboards. Or, one can use the mouse’s cursor to click on either a physical or digital object in order to associate it with the mouse.

Video

Brainstorm

Ideas

Outpost programme – Maybe could bring back the old ways of the graphic design process. Maybe use the physical tools from back in the day to do the work.

Maybe use a physical roller with sensor device in order to flatten an image in photoshop?

Using a flexible french curve to draw/produce lines and curves.

In order to bring back the old items that are forgotten about and unknown to the new generation of graphic designers. To be used as an informational device, not really for convenience and ease as the new generation of graphic design has already done that.

The Designers’ Outpost: A Tangible Interface for Collaborative Web Site Design

Stanford HCI

From studies into web design, it was found that pens, paper, walls and tablets were often used for explaining, developing and communicating ideas during the early phases of design. In one common early phase practice, designers collect ideas about what should be in a website onto post-it notes and arrange them on a wall into categories. This technique is often called affinity diagramming. These collaborative paper based practices inspired the designer’s outpost. A tangible user interface that combines the affordances of paper and large physical work spaces with the advantages of electronic media to support information design. Users have the same fundamental capabilities as in a paper based system. One can create new pages by writing on post-it notes and organize a site by physically moving post-it notes around on the board. Paper and physical world becomes an input device for the electronic world.

This interaction is enabled by a touch-sensitive smart board, augmented with a robust computer vision system. Employing a rear-mounted video camera for interactively locating physical media and a front mounted high resolution camera for capturing the contents of the media.

Users can write on a note with a standard pen and add it to the board. The vision system recognizes it and updates its internal understanding of the board. To link a pair of notes, the user draws a line from one note to another with the board stylus. To delete a note and its associated links, the user pulls the note off the board. To move a note and its links, the user picks it up and places it at it’s new location. This provides for a  lightweight means of keeping the electronic data and the physical object coupled. Tapping a note invokes an electronic context menu, enabling the manipulation of the electronic properties and body by physical objects.

In addition to being a space for interacting with physical post-it notes, Outpost is also an electronic white board, supporting free-form drawing using board styli.

A physical move tool provides a means of interacting with the system after the physical content has become electronic, retaining haptic direct manipulation. Working like a normal white board eraser, the Outpost eraser removes ink on the board. It operates semantically, deleting each stroke it passes over.

Users can press save to save the boar state to disk. Through its electronic capture of designs, our system supports the transition from this early representation to later electronic tools such as Denim. Denim is a sketch-based tool supporting information and navigation design of websites. Denim supports sketching input, allows design at different refinement levels, and unifies the levels through zooming. Opening and outpost information architecture in Denim offers users the ability to edit the information architecture, specify page level details and create the navigational structure for a website.

The designer’s outpost offers users the best of both the physical and the virtual worlds. It supports the physical wall-scale design practices for creating information architectures, augmenting them with electronic interactions for structuring the information and transitioning to other tools.

Potential structure for paper

• Definition of TUI
• Origin – why?
• Ubiquitous computing
• More in depth definition of TUI – can result in new digital works…
• Emerging field of research – Tangible media group MIT
• Mention projects 2-3 (general TUI’s)
• Creative individual – graphic designer – before and after comps – Aspects lost along the way? interview…
• Specific TUI’s for a graphic designer
• Suitable design for this individual – What  kind of TUI could help a graphic designer – is there a need? can new work come out of this?

3-4000 words

Proposal for masters programme

Exemplar

Exemplar is a programming by demonstration tool to explore and prototype interactions involving sensors. For interaction designers, connecting sensor input data to application logic remains an exercise in patience and trial and error testing because current tool support is geared towards experts and heavy analysis.

With exemplar, a designer first demonstrates a sensor-based interaction. He then reviews and edits a visual representation of the action and tests his author behavior by performing the action again.

Scenario:

Dan is an interaction designer who wants to prototype a smart bicycle helmet with automatic blinkers such that tilting the helmet left or right causes the associated blinkers to signal. He picks a 3-d accelerometer and connects it to a compatible hardware interface. Data from all connected inputs is shown simultaneously in the exemplar software. Previews light up when sensor data changes rapidly to attract attention. This overview allows him to discover pertinent signals. Dan brings one accelerometer output into focus. Then demonstrates examples of the action he wants to recognize. He reviews and marks up the sensor signal to define the regions he wants to recognize. Here exemplar generalizes from the examples by calculating thresholds within the highlighted regions. In addition to drawing regions, Dan can also use a foot pedal to define examples while he performs them. Dan can immediately test his design to see if it works. Exemplar shows a visualization of sensor signal regions that match the demonstration.

Exemplar will then generate events based on matching regions in two-ways. First, the software can send system mouse and keyboard events to remote control over applications or Exemplar can send events to the d-tools prototyping environment to generate hardware output.

Dan can refine the learn interaction model through direct manipulation. He can apply filters to calibrate ad transform the incoming sensor data.

Exemplar can generalize from the examples, by calculating thresholds within the highlighted region, or by applying a pattern matching algorithm to capture complex interaction. For threshold events he can drag minimum and maximum threshold values, and adjust hysteresis as well as minimum diration parameters graphically. In pattern matching mode, Exemplar displays and error graph that shows a distance metric between the current input and the learned example. Dan can drag a threshold line to determine the maximum acceptable error. When the error line falls below the threshold, the corresponding event is fired.

Experts can add their own filters written in java to extend the library of signal transformations. Interactions can also be authored for events that depend on multiple sensors simultaneously (Accelerometer and Bend sensor: Event is fired only if both sensors match). Finally Exemplar also allows designers to output continuous values corresponding to filtered input signals.

A first use study with twelve participants showed how exemplar can enable novices and experienced designers alike, to create interaction designs involving sensors. Participants brainstormed, proto-typed, tested and refined their own functional game controllers in less than thirty minutes.

Exemplar has been released to the community as open source.

Video

Sensor based interaction

Authoring a sensor-based interaction comprises three steps: choosing and connecting the appropriate hardware, creating application logic, and specifying the relationship between sensor values and application logic.

This paper contributes techniques for enabling a wider audience of designers and application programmers to turn
raw sensor data into useful events for interaction design through programming by demonstration. It introduces a
rapid prototyping tool, Exemplar.

** linked with next post “Exemplar”**

Click to access hartmann-chi2007.pdf

CopyCAD

With CopyCAD users can copy 2d shapes from real world objects, which are captured by a camera. The copied shapes are then projected directly where the real objects were. These shapes can then be edited, scaled, copied, deleted using multitouch and pen gestures. New shapes can be added and sketches can be captured using any pen. Then the design can be machined directly on the table using a variety of materials.

T(ether)

T(ether) is a novel spatially aware display that supports intuitive interaction with volumetric data. The display acts as a window affording users a perspective view of three- dimensional data through tracking of head position and orientation. T(ether) creates a 1:1 mapping between real and virtual coordinate space allowing immersive exploration of the joint domain. Our system creates a shared workspace in which co-located or remote users can collaborate in both the real and virtual worlds. The system allows input through capacitive touch on the display and a motion-tracked glove. When placed behind the display, the user’s hand extends into the virtual world, enabling the user to interact with objects directly.

It is a spatially and body aware window for collaborative editing and animation of 3D virtual objects. Through T(ether) three dimensional objects can be viewed and edited with unprecedented ease with other people. A simple pinch gesture through a motion tracked glove can be used to create and manipulate virtual objects.

Object manipulation; shape selection – sphere, mesh cube…; hand tracking.

All the objects in the scene can be animated by key frames. Above the screen, interactions allow users to scrub through time using the pinch gesture.

T(ether) allows multiple devices to connect simultaneously, allowing mass collaboration.

T(ether) uses Vicon motion capture cameras to track the position and orientation of tablets, user heads and hands. Server-side synchronization was coded using NodeJS and tablet-side code uses Cinder. The synchronization server forwards tag location to each of the tablets over wifi, which in turn renders the scene. Touch events on each tablet are broadcasted to all other tablets using the synchronization server.

Video