Conceptual Overview
The real-time system is built on the concept of 3D video fragments. Differential update schemes
are
used to exploit the spatio-temporal coherence of the video
streams of multiple cameras. These updates are issued by operators such as
inserts and deletes, which account for changes in the input video images. We
also introduce a novel concept for camera control which dynamically selects
the set of relevant cameras for reconstruction, and adapts to the processing
load and rendering platform.
We acquire images from multiple calibrated video
cameras. The images are processed to segment foreground from
background. By means of dynamic camera control we determine a
set of active cameras from which we generate 3D point samples as
well as a set of supporting cameras delivering additional data
to improve the 3D reconstruction. Using inter-frame prediction
in image space we generate a stream of differential operators
which dynamically update point sample attributes including
position or color. We thus avoid to recompute the full 3D
representation in each frame. The dynamic point samples are
rendered by an efficient point splatting scheme and are
composited with a virtual scene. In a final stage we apply
deferred operations like visual effects by running them directly
on the graphics hardware.
|