The Virtual Mirror A handheld mirror simulation Alexandre R.J. François and Elaine Kang

Introduction

The perception of the world reflected through a mirror depends on the viewer's position with respect to the mirror and the 3-D geometry of the world (see video of a real mirror).

In order to simulate a real mirror on a computer screen, images of the observed world, consistent with the viewer's position, must be synthesized and displayed in real-time.

Our system is build around a flat LCD screen manipulated by the user, a single camera fixed on the screen, and a magnetic tracking device. The continuous input video stream and tracker data are used to synthesize, in real-time, a continuous video stream displayed on the LCD screen. The synthesized video stream is a close approximation of what the user would see on the screen surface if it were a real mirror.

Problem fornulation and geometric analysis

When a person looks into a mirror, they see what they would see from the symmetrical (virtual) point of view, as shown on the figure below.

A camera whose position and orientation are fixed with respect to the mirror produces a video stream that corresponds to this particular viewpoint.

In order to synthesize the images corresponding to the user's viewpoint from the images captured by the camera, we must relate them geometrically. Such a transform cannot be formulated without assumptions or knowledge about the world geometry. In our case, the focus of attention is the user's own reflection in the mirror. We thus model the world as a plane parallel to the mirror, at the user's distance from the mirror. We have shown that under these assumptions, the transform relating the camera view and the user view:

• depends only on the relative positions of the user viewpoint and the camera center, and
• is reduced to a scaling and a translation.

The transform is exact when the user viewpoint lies on the camera axis. This model provides a straightforward and efficient way of synthesizing the viewer's image stream from the camera stream, and provides a good approximation when the viewer's position remains close to the camera axis.

System integration

We have implemented the mirror image transform as a module in the Modular Flow Scheduling Middleware (MFSM), an open source implementation of IMSC's Software Architecture for Immersipresence. It defines and implements a framework for distributed parallel processing of data streams in real-time. It is also designed to facilitate complex system integration. In particular, its modularity dramatically reduces development and integration time.

The flow graph for the virtual mirror application is shown below.

The actual virtual mirror system (see picture) comprises:

• a handheld thetered device, composed of a 10" LCD flat screen fitted with a camera and a magnetic tracker sensor,
• a fixed processing unit,
• a fixed magnetic tracker transmitter, and
• a second magnetic tracker sensor placed on the user.

Real-time camera and tracker inputs are used to synthesize the virtual mirror stream and display it in real-time on the LCD screen. The result is a very convincing handheld mirror simulation.

Demonstrations

Real mirror: MPEG1 352x240 (~2MB)

No mirror simulation (simple image flip): The dynamics pan, tilt and field of view are not correct... MPEG4 320x240 (~3MB)

Virtual mirror: Look for realistic pan, tilt and field of view dynamics...
MPEG1 352x240 (~6MB), MPEG4 640x480 (~6MB)

Perspectives

The Virtual Mirror project was initiated as an enabling technology for the Virtual Daguerreotype project for the "Lost and Found: Rediscovering Early Photographic Processes" Exhibit Museum Application. Other mirror settings, such as a large (simulated) mirror on a wall, are currently being designed.

The design and construction of realistic virtual mirrors represents an original experiment in Human-Computer Interface, that opens the door to various interactive experiments in entertainment, arts and communication. An example is the Virtual Daguerreotype project.

Use of video analysis and graphics techniques will allow to explore and interfere with what has always been a private, solitary act of a person looking in a mirror.

Related publications

ICME 2003
SIGGRAPH 2002
ICAR 2003

Invention disclosure: Virtual Mirrors, Alexandre R.J. François and Gérard G. Medioni, file #3283, USC Office of Technology Licensing.

Acknowledgements

Umberto Malesci (IMSC Undergraduate Research Program)

Margaret Lazzari (Fine Arts, USC),
Margaret McLaughlin (Annenberg School for Communications, USC)
Gérard Medioni (Computer Science, USC)

This research was funded in part by a grant from the USC Arts Initiative, and by the Integrated Media Systems Center, a National Science Foundation Engineering Research Center, Cooperative Agreement No. EEC-9529152. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect those of the National Science Fundation.