Mixed Reality

Rendering

• Content-Adaptive Visibility Predictor for Image Blending

  Conventional visibility models cannot reflect the dependence of the suprathreshold visibility of the blended images on the appearance of the pre-blended image content. Therefore, we have proposed a visibility model with a content-adaptive feature aggregation mechanism, which integrates the visibility for each image feature (i.e., such as spatial frequency and colors) after applying weights that are adaptively determined according to the appearance of the input image.
  
  

  • T. Fukiage and T. Oishi, "A Content-Adaptive Visibility Predictor for Perceptually Optimized Image Blending," ACM Transactions on Applied Perception, 20, 1, Article 3 (January 2023). [sup] [video]

• Visibility-based blending

  There are many situations in which virtual objects are presented half-transparently on a background in real time applications. In such cases, we often need to show the object with constant visibility. To overcome this problem, we present a framework for blending images based on a subjective metric of visibility. In our method, a blending parameter is locally and adaptively optimized so that the visibility of each location achieves the targeted level.
  
  

  • T. Fukiage, T. Oishi, K. Ikeuchi, "Visibility-Based Blending for Real-Time Applications," In Proc. 12th IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2014), pp. 63-72, Munich, Sept. 2014. [video]

• Aerial perspective rendering

  In outdoor Mixed Reality (MR), objects distant from the observer suffer from an effect called aerial perspective that fades the color of the objects and blends it to the environmental light color. We present a turbiditybased method for rendering a virtual object with aerial perspective effect in a MR application.
  
  

  
  • M. Carlos, T. Oishi, K. Ikeuchi, "Real-time rendering of aerial perspective effect based on turbidity estimation," IPSJ Transactions on Computer Vision and Applications (CVA), vol. 9, no. 1, 2017.
    
    M. Carlos, T. Oishi, K. Ikeuchi, "Turbidity-based Aerial Perspective Rendering for Mixed Reality," In Proc. 12th IEEE International Symposium on Mixed and Augmented Reality (ISMAR2014), pp. 283-284, Munich, Sept. 2014.

Image processing

• Optical flow and depth estimation

  We present an alternative method for solving the motion stereo problem for two views in a variational framework. Instead of directly solving for the depth, we simultaneously estimate the optical flow and the 3D structure by minimizing a joint energy function consisting of an optical flow constraint and a 3D constraint.
  
  

  • M. Roxas and T. Oishi, "Real-Time Simultaneous 3D Reconstruction and Optical Flow Estimation," in Proc. IEEE Winter Conf. on Applications of Computer Vision (WACV 2018), Mar. 2018. [src]

• Video completion

  We propose a novel omnidirectional video completion framework based on depth estimation. First, we recover the depth of the scene from a pixel motion model constrained by a known camera pose. The proposed approach further improves the depth map by a structure-aware refinement. We can employ the refined depth for color propagation into the holes.
  
  

  • C. Morales, S. Ono, Y. Okamoto, M. Roxas, T. Oishi and K. Ikeuchi, "Outdoor Omnidirectional Video Completion via Depth Estimation by Motion Analysis," in Proc. International Conference on Pattern Recognition (ICPR 2016), 2016.

Occlusion handling

• Occlusion handling by Semantic Segmentation and Visibility-based Blending

  We propose a novel occlusion handling method for real-time mixed-reality given a monocular image and an inaccurate depth map. We modify the intensity of the overlayed CG object based on the texture of the underlying real scene using visibility-based rendering. To determine the appropriate level of visibility, we use CNN-based semantic segmentation and assign labels to the real scene based on the complexity of object boundary and texture.
  
  

  • M. Roxas, T. Hori, T. Fukiage, Y. Okamoto, T. Oishi, "Occlusion Handling using Semantic Segmentation and Visibility-Based Rendering for Mixed Reality," ACM Symposium on Virtual Reality Software and Technology(VRST 2018). [video]

• Semi-transparent rendering and occlusion handling

  One of the challenges in mixed reality (MR) applications is handling contradictory occlusions between real and virtual objects. We proposed a solution to the occlusion problem that does not require precise foreground-background segmentation. In our method, a virtual object is blended with a real scene so that the virtual object can be perceived as being behind the foreground region.
  
  

  • T. Fukiage, T. Oishi, and K. Ikeuchi, "Reduction of contradictory partial occlusion in mixed reality by using characteristics of transparency perception," In Proc. 10th IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2012), pp. 129-139, , 2012.
  • T. Fukiage, T. Oishi, and K. Ikeuchi, "A simple photometric factor in perceived depth order of bistable transparency patterns," Journal of Vision, 14(5): 2, May 5 2014.

• Robust occlusion handling in outdoor environment

  Outdoor environment makes the occlusion handling difficult due to the unpredictable illumination changes. We proposed an outdoor illumination constraints for resolving the foreground occlusion problem in outdoor environment. In addition, we introduce an effective method to resolve the shadow occlusion problem by using shadow detection and recasting with a spherical vision camera.
  
  

  • L. B. Vinh, T. Kakuta, R. Kawakami, T. Oishi, K. Ikeuchi, "Foreground and Shadow Occlusion Handling for Outdoor Augmented Reality," In Proc. 9th IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2010), pp. 109-118, 13-16 Oct. 2010, Seoul. [video]
  • T. Kakuta, , L. B. Vinh, R. Kawakami, T. Oishi, K. Ikeuchi, "Detection of Moving Objects and Cast Shadows Using a Spherical Vision Camera for Outdoor Mixed Reality," ACM Symp. on Virtual Reality Software and Technology (VRST 2008), Oct. 2008, pp. 219-222.

Localization / tracking

• Synthesis-based localization

  A robust image-based alignment method to be used in outdoor environments is proposed. In the proposed method, the albedo of real objects is estimated using 3D shapes of these objects in advance, and the appearance is reproduced from the albedo and current light environment. The appearance of real objects and reproduced image becomes close, so a robust image-based alignment is achieved.
  
  

  • M. Inaba, A. Banno, T. Oishi, K. Ikeuchi, "Achieving Robust Alignment for Outdoor Mixed Reality using 3D Range Data," In Proc. 18th ACM Symp. on Virtual Reality Software and Technology (VRST 2012), pp. 61-68, Dec. 2012, Toronto, Canada.

Virtual Reconstruction of Cultural Heritage Assets

• Virtual Asuka-kyo Project

  We developed Mixed Reality (MR) contents that reconstructed the ancient capital of Asuka-Kyo and applied a fast shading and shadowing method that used shadowing planes. We conducted a subjective evaluation experiment with Head Mounted Display, which showed that displaying these contents increased the audience's knowledge of both Asuka-Kyo and MR technologies. We also conducted impression evaluation tests with and without shading and shadowing.
  
  

  • T. Kakuta, T. Oishi, K. Ikeuchi, "Development and Evaluation of Asuka-Kyo MR Contents with Fast Shading and Shadowing," Proc. Int. Society on Virtual Systems and MultiMedia (VSMM 2008), pp.254-260, Oct. 2008.
  • T. Kakuta, T. Oishi, K. Ikeuchi, "Virtual Kawaradera: Fast Shadow Texture for Augmented Reality," Proc. the Tenth International Conference on Virtual System and Multimedia (VSMM 2004), pp. 141-150, November 2004.