Video-based Rendering

Marcel Germann, Tiberiu Popa, Richard Keiser (LiberoVision AG), Remo Ziegler (LiberoVision AG), Markus Gross

Video-based rendering aims to generate virtual views of a real wold scene that was recorded by one or more video cameras. The goal is to achieve as realistic as possible images based on only the camera input, e.g. from standard TV cameras. We developed novel representation and rendering methods that result in images visually not distinguishable from original camera images. Together with our collaborator, LiberoVision AG, we show a successful application for this: virtual replays of sports events (see image).

Virtual view of a soccer game
(Images courtesy of Teleclub and LiberoVision AG)

Our research focuses on the pose estimation, representation and rendering of the human body. In our current work, we achieved the following:

• Articulated billboards: We developed a novel representation for free-viewpoint video of human characters based on multiple input video streams. The basic idea is to approximate the articulated 3D shape of the human body using a subdivision into textured billboards along the skeleton structure. Billboards are clustered to fans such that each skeleton bone contains one billboard per source camera. We call this representation articulated billboards.
• Construction: We developed a semi-automatic, data-driven algorithm to construct articulated billboards, which handles even challenging acquisition scenarios characterized by sparse camera positioning, inaccurate camera calibration, low video resolution, or occlusions in the scene. First, for each input view, a 2D pose estimation based on image silhouettes, motion capture data, and temporal video coherence is used to create a segmentation mask for each body part. Then, from the 2D poses and the segmentation, the actual articulated billboard model is constructed by a 3D joint optimization and compensation for camera calibration errors.
• Rendering: Our rendering method for articulated billboards includes a novel way of blending the textural contributions of each billboard and features an adaptive seam correction to eliminate visible discontinuities between adjacent billboards textures. Rendering with articulated billboards does not only minimize ghosting artifacts known from conventional billboard rendering, but also alleviates restrictions to the setup and sensitivities to errors of more complex 3D representations and multiview reconstruction techniques. Our results demonstrate the flexibility and the robustness of our approach with high quality free-viewpoint video generated from broadcast footage of challenging, uncontrolled environments.

Articulated billboards representation and result images.
(Images courtesy of Teleclub and LiberoVision AG)

In future, we plan to improve our pose estimation by temporal and spacial optimization. Another focus will be the improvement of the original camera images by using the additional knowledge about the scene and our representation. With such improved super-resolution images, better close-up views of the subjects will become possible.

• M. Germann, A. Hornung, R. Keiser, R. Ziegler, S. Würmlin, M. Gross, Articulated Billboards for Video-based Rendering, Proceedings of Eurographics (Norrköping, Sweden, May 3-7, 2010), Computer Graphics Forum, vol. 29, no. 2, pp. 585-594
  [Abstract] [BibTeX] [PDF] [Video]