Conceptual Overview

A 3D video recorder first records 2D video streams from several synchronized digital video cameras and stores pre-processed images to disk. An off-line processing stage combines these images and enhances them with 3D information. The new data set is then stored as 3D video to disk. This can either be done using progressive or flat coding schemes. Furthermore, 3D video can be encoded in image-space (i.e., encoding of images) or in object-space (i.e., encoding of 3D point samples). A 3D video player decodes and renders 3D videos from a storage device (i.e., hard-disk or DVD) or from the network in real-time using a point-based rendering framework.

As opposed to a traditional 2D home video recorder comprising only two stages - recording and playback - a 3D video recorder features an additional stage: processing. It is clear that recording and playing need to be carried out in real-time. There are no hard real-time constraints for the off-line processing stage. Spending 60 times more time on off-line processing than on on-line decompression is still acceptable. A 3D replay application in a broadcasting studio has stronger time constraints, since the replay needs to be available for broadcast only 10-20 seconds after the live action. The ultimate goal of a 3D video recorder is to process 3D information within these time limits.

Interaction

For pre-recorded data, the 3D video player provides interaction features known from video cassette recorders, like variable-speed forward, reverse, and slow motion. 3D video playback can be enhanced with novel 3D video effects such as freeze-and-rotate and arbitrary scaling.

We can randomly access and decode individual frames of the 3D video. Furthermore, every frame can be retrieved at different quality levels, reaching from a coarse approximation to a very detailed representation. During regular playback, we read frame after frame and control the quality level such that the player adheres to the frame-rate of the original 2D video sequence. The quality level is thus determined by the decoder’s performance and rendering complexity. When the sequence is paused, the decoder can completely decode the current frame at the highest quality level. Fast-forward is implemented by decoding only a coarse representation, by using larger frame increments, or by a combination of both. Fast-reverse is realized in a corresponding way, we just need to decrement the requested frame number instead of incrementing it in between frames. Slow motion can simply be realized by slowed-down playback, i.e., by decoding higher quality levels than the player supports in real-time. High-quality slow motion, however, requires additional point-based shape interpolation between consecutive frames, or, the use of high-speed cameras. The 3D video player implements a virtual trackball and hence arbitrary navigation and scaling is possible and follows the popular interaction metaphors from other graphics renderers.

Special effects, such as freeze-and-rotate, can easily be achieved by playing a sequence, pausing, rotating the viewpoint, and continuing playback again. In case the system is used for editing a 2D video from a 3D video sequence, the virtual camera path and the frame increments can be configured in a script file.