3D Video

Stephan Würmlin, Michael Waschbüsch, Markus Gross

As one of the many promising emerging technologies for home entertainment and spatio-temporal visual effects, 3D video acquires the dynamics and motion of a scene during recording while providing the user with the possibility to change the viewpoint at will during playback. Free navigation regarding time and space in streams of visual data directly enhances the viewing experience and interactivity.

In our work on 3D video we focus on representations and methodologies to capture and process dynamic scenes from sparsely arranged video cameras. As a fundamental primitive for FVV we employ dynamic point samples by generalizing 2D video pixels towards 3D irregular point samples.

• A special focus in our work is real-time 3D video technology which is successfully employed in the blue-c collaborative virtual reality system. blue-c combines the advantages of a CAVE™-like projection environment with simultaneous and real-time 3D video capturing and processing of the user. As a major technical achievement, users can now become part of the visualized scene while keeping visual contact.
• Furthermore, an MPEG-compliant 3D video coding framework was designed which is capable of streaming and displaying pre-recorded multi-view video data from arbitrary viewpoints. In its finest deployment, this coding framework provides multi-resolution, multi-rate and view-dependent decoding. The dynamic point sample representation is adopted by MPEG as an extension of the MPEG-4 AFX standard and combined with MPEG video codecs enable 3D video with MPEG-standardized components only.
• We are currently extending our 3D video pipeline from single objects towards complex, dynamic scenes, including background and multiple interacting foreground objects. We developed a modular scanning system based on multiple, sparsely placed 3D video bricks which acquire scene textures and depth maps using space-time stereo and structured light. This information is merged into a view-independent 3D data structure which allows for straightforward video editing and high-quality rendering using EWA volume splatting.

As future work we want to bring 3D video to a new level where not only capturing and subsequent high-quality re-rendering is cost-effective, convenient and scalable, but also editing of the spatio-temporal streams is easy to perform. We envision 3D video editing to become as convenient as 2D home video editing. Inserting novel objects to a scene or adding spatio-temporal effects should become elementary, and one has no longer to cope with the common limitations of image-based representations.

• M. Waschbüsch, S. Würmlin, M. Gross, 3D Video Billboard Clouds, Proceedings of Eurographics (Prague, Czech Republic, September 3-7, 2007), Computer Graphics Forum, vol. 26, no. 3, pp. 561-569
  [Abstract] [PDF] [Video]
• M. Waschbüsch, S. Würmlin, D. Cotting, M. Gross, Point-sampled 3D video of real-world scenes, Journal of Signal Processing: Image Communication, vol. 22, no. 2, 2007, pp. 203-216
  [Abstract] [PDF] [Video]
• M. Waschbüsch, S. Würmlin, M. Gross, Interactive 3D Video Editing, The Visual Computer, Special Issue for Pacific Graphics 2006 (Taipei, Taiwan, October 11-13, 2006), vol. 22, no. 9, pp. 631-641
  [Abstract] [PDF] [Video]
• S. Würmlin, E. Lamboray, M. Waschbüsch, P. Kaufmann, A. Smolic, M. Gross, Image-space Free-viewpoint Video, Proceedings of Vision, Modeling, Visualization '05, pp. 453-460 (Erlangen, Germany, November 16-18, 2005)
  [Abstract] [PDF]
• E. Lamboray, S. Würmlin, M. Gross, Data Streaming in Telepresence Environments, IEEE Transactions on Visualization and Computer Graphics, Special Issue on Haptics, Virtual and Augmented Reality, vol. 11, no. 6, 2005, pp. 637-648
  [Abstract] [PDF]
• S. Würmlin, E. Lamboray, M. Waschbüsch, M. Gross, Dynamic Point Samples for Free-Viewpoint Video, Proceedings of the Picture Coding Symposium 2004, San Francisco, December 15-17, 2004
  [Abstract] [PDF]
• E. Lamboray, S. Würmlin, M. Waschbüsch, M. Gross, H. Pfister, Unconstrained Free-Viewpoint Video Coding, Proceedings of the IEEE International Conference on Image Processing (ICIP) 2004 (IEEE International Conference on Image Processing (ICIP), Singapore, October 24-27, 2004)
  [Abstract] [PDF]
• S. Würmlin, E. Lamboray, M. Gross, 3D Video Fragments: Dynamic Point Samples for Real-time Free-Viewpoint Video, Computers & Graphics 28 (1), pp. 3-14 (Special Issue on Coding, Compression and Streaming Techniques for 3D and Multimedia Data, 2004)
  [Abstract] [PDF]
• E. Lamboray, S. Würmlin, M. Gross, Real-Time Streaming of Point-Based 3D Video, Proceedings of the IEEE Virtual Reality 2004 Conference, pp. 91-98 (IEEE Virtual Reality 2004 Conference, Chicago, Illinois, USA, March 27-31, 2004)
  [Abstract] [PDF]
• S. Würmlin, E. Lamboray, O. Staadt, M. Gross, 3D Video Recorder: A System for Recording and Playing Free-Viewpoint Video, Computer Graphics Forum 22 (2), David Duke and Roberto Scopigno (eds.), Blackwell Publishing Ltd, Oxford, U.K., pp. 181-193, 2003
  [Abstract] [PDF]