LiDAR-Camera Fusion

• LiDAR-Camera Calibration with 2DGS

  Similar to INF, this approach uses 2D Gaussian splatting generated from LiDAR data as geometric information and achieves robust LiDAR-camera calibration by optimizing view consistency across camera images.
  
  

  • S. Zhou, S. Xie, R. Ishikawa, and T. Oishi, "Robust LiDAR-Camera Calibration With 2D Gaussian Splatting," IEEE Robotics and Automation Letters, vol. 10, no. 5, pp. 4674-4681, May 2025. [code] [bib]
    
    

  INF: Implicit Neural Fusion

  Using implicit neural representations, we proposed a fusion method of data captured by LiDAR and camera. This method uses the neural density field from LiDAR data to represent the geometric information. As well as NeRF, we achieve sensor calibration and data fusion simultaneously by learning the color field that is consistent with input camera images.
  
  

  • Shuyi Zhou, Shuxiang Xie, R. Ishikawa, K. Sakurada, M. Onishi, and T. Oishi, "INF: Implicit Neural Fusion for LiDAR and Camera," 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Detroit, MI, USA, 2023, pp. 10918-10925. arXiv:2308.14414 [project]

Neural Representation

• G²fR

  This paper clarifies the basic mechanism of frequency regularisation in implicit neural representations and comprehensively discusses the expressive capabilities of NeRF with grid-based feature encoding (GFE). We also proposed a generalised frequency regularisation strategy for the problems of camera pose optimisation and few-shot reconstruction in NeRF.
  
  

  • S. Xie, S. Zhou, K. Sakurada, R. Ishikawa, M. Onishi, and T. Oishi, "G²fR: Frequency Regularization in Grid-based Feature Encoding Neural Radiance Fields," The 18th European Conference on Computer Vision (ECCV), pp. 186-203, 2024. [sup] [bib]
    
    

  Ref²-NeRF

  It is challenging to model objects through glass, such as objects in a glass case, in a 3D manner. The proposed method models the glass surface and refraction from images taken from multiple directions and separates the viewpoint-dependent reflection component and the viewpoint-independent object shape and color through neural representations.
  
  

  • W. Kim, T. Fukiage, and T. Oishi, "REF² NeRF: Reflection and Refraction-aware Neural Radiance Field," International Conference on Intelligent Robots (IROS), pp. 7196-7203, 2024. [project] [code] [bib]
    
    

  FIR NeRF

  We proposed a NeRF-based approach to model invisible components such as gases in three dimensions using image sequences from far-infrared and visible light cameras. The method learns the color and density fields of visible light in advance, and by using the same density fields as geometry information, it is possible to model invisible components from far-infrared images in three dimensions.
  
  

  • X. Li, S. Xie, K. Sakurada, R. Sagawa, and T. Oishi, "Implicit Neural Fusion of RGB and Far-Infrared 3D Imagery for Invisible Scenes," International Conference on Intelligent Robots (IROS), pp. 12501-12508, 2024. [bib]
    
    
    

Sensor System

• Ground sensor

  We developed a mobile scanning system for fastly and accurately capturing the 3D range data. Our system consists of a LiDAR and a color camera. While the laser scanner works for capturing 3D structures, the color camera captures an image sequence. The sensor motion is estimated robustly from a sensor-fused 2D/3D feature-tracking method, which helps to reconstruct the structures from the scan lines accurately.
  
  

  • R. Ishikawa, S. Zhou, Y. Sato, T. Oishi, K. Ikeuchi, "LiDAR-camera Calibration using Intensity Variance Cost," 2024 IEEE International Conference on Robotics and Automation (ICRA), Yokohama, Japan, 2024, pp. 10688-10694.
  • R. Ishikawa, T. Oishi, K. Ikeuchi, "LiDAR and Camera Calibration using Motion Estimated by Sensor Fusion Odometry," IEEE/RSJ International Conference on Intelligent Robots (IROS 2018), pp. 7342-7349, 2018. arXiv:1804.05178 [src]
  • R. Ishikawa, M. Roxas, Y. Sato, T. Oishi, T. Masuda, K. Ikeuchi, "A 3D Reconstruction with High Density and Accuracy using Laser Profiler and Camera Fusion System on a Rover," International Conference on 3D Vision (3DV), pp. 620-628, Oct 27, 2016, Palo Alto. [video]
  • B. Zheng, T. Oishi, K. Ikeuchi, "Rail Sensor: A Mobile Lidar System for 3D Archiving the Bas-reliefs in Angkor Wat," IPSJ Transactions on Computer Vision and Applications (CVA), Vol. 7, pp. 59-63, July 27, 2015.

• Aerial sensor

  We developed a flying sensor system to capture 3D data aerially. The system, consisting of a omni-directional laser scanner and a panoramic camera, can be mounted under a mobile platform to achieve the aerial scanning with high resolution and accuracy. Since the laser scanner often requires several minutes to complete an omni-directional scan, the raw data is distorted seriously due to the unknown and uncontrollable movement during the scanning period. Our approach recovers the sensor motion by utilizing the spacial and temporal features extracted both from the image sequences and point clouds.
  
  

  • B. Zheng, X. Huang, R. Ishikawa, T. Oishi, K. Ikeuchi, "A New Flying Range Sensor: Aerial Scan in Omini-directions," In Proc. International Conference on 3D Vision (3DV), pp. 623-631, Oct. 19-22, 2015, Lyon, France. [video]
  • R. Ishikawa, B. Zheng, T. Oishi, K. Ikeuchi, "Rectification of Aerial 3D Laser Scans via Line-based Registration to Ground Model," IPSJ Transactions on Computer Vision and Applications (CVA), Vol. 7, pp. 89-93, July 27, 2015. [video] [video]
  • A. Banno, T. Masuda, T. Oishi, and K. Ikeuchi, "Flying Laser Range Sensor for Large-Scale Site-Modeling and Its Applications in Bayon Digital Archival Project," International Journal of Computer Vision (IJCV), Vol. 78, No. 2-3, pp. 207-222, Jul. 2008.