Robot System and Mobility

Sensor

• INF: Implicit Neural Fusion

  Using implicit neural representations, we proposed a fusion method of data captured by LiDAR and camera. This method uses the neural density field from LiDAR data to represent the geometric information. As well as NeRF, we achieve sensor calibration and data fusion simultaneously by learning the color field that is consistent with input camera images.
  
  

  • Shuyi Zhou, Shuxiang Xie, R. Ishikawa, K. Sakurada, M. Onishi, and T. Oishi, "INF: Implicit Neural Fusion for LiDAR and Camera," International Conference on Intelligent Robots (IROS), 2023. , arXiv:2308.14414 [project]

• Agricultural support robot

  Pesticide spraying is essential for growing crops, but food safety and cost are significant issues. We have, therefore, developed a system to detect and remove worms using a quadruped robot capable of moving over uneven terrain.
  
  

  • S. Balasooriya, Y. Sato, T. Oishi, "Autonomous Robotic Platform for Proximal Data Collection Amongst Foliage Utilizing an Anisotropically Flexible Manipulator," The 2024 16th IEEE/SICE International Symposium on System Integration (SII), Ha Long, Vietnam, 2024.
    
    
  • H. Hansen, Y. Liu, R. Ishikawa, T. Oishi, Y. Sato, "Quadruped Robot Platform for Selective Pesticide Spraying," 18th International Conference on Machine Vision Applications (MVA), Hamamatsu, Japan, 2023.
    
    

  Ground sensor

  We developed a mobile scanning system for fastly and accurately capturing the 3D range data. Our system consists of a LiDAR and a color camera. While the laser scanner works for capturing 3D structures, the color camera captures an image sequence. The sensor motion is estimated robustly from a sensor-fused 2D/3D feature-tracking method, which helps to reconstruct the structures from the scan lines accurately.
  
  

  • R. Ishikawa, S. Zhou, Y. Sato, T. Oishi, K. Ikeuchi, "LiDAR-camera Calibration using Intensity Variance Cost," International Conference on Robotics and Automation (ICRA 2024), 2024. (to appear)
  • R. Ishikawa, T. Oishi, K. Ikeuchi, "LiDAR and Camera Calibration using Motion Estimated by Sensor Fusion Odometry," IEEE/RSJ International Conference on Intelligent Robots (IROS 2018), pp. 7342-7349, 2018. arXiv:1804.05178 [src]
  • R. Ishikawa, M. Roxas, Y. Sato, T. Oishi, T. Masuda, K. Ikeuchi, "A 3D Reconstruction with High Density and Accuracy using Laser Profiler and Camera Fusion System on a Rover," International Conference on 3D Vision (3DV), pp. 620-628, Oct 27, 2016, Palo Alto. [video]
  • B. Zheng, T. Oishi, K. Ikeuchi, "Rail Sensor: A Mobile Lidar System for 3D Archiving the Bas-reliefs in Angkor Wat," IPSJ Transactions on Computer Vision and Applications (CVA), Vol. 7, pp. 59-63, July 27, 2015.

• Aerial sensor

  We developed a flying sensor system to capture 3D data aerially. The system, consisting of a omni-directional laser scanner and a panoramic camera, can be mounted under a mobile platform to achieve the aerial scanning with high resolution and accuracy. Since the laser scanner often requires several minutes to complete an omni-directional scan, the raw data is distorted seriously due to the unknown and uncontrollable movement during the scanning period. Our approach recovers the sensor motion by utilizing the spacial and temporal features extracted both from the image sequences and point clouds.
  
  

  • B. Zheng, X. Huang, R. Ishikawa, T. Oishi, K. Ikeuchi, "A New Flying Range Sensor: Aerial Scan in Omini-directions," In Proc. International Conference on 3D Vision (3DV), pp. 623-631, Oct. 19-22, 2015, Lyon, France. [video]
  • R. Ishikawa, B. Zheng, T. Oishi, K. Ikeuchi, "Rectification of Aerial 3D Laser Scans via Line-based Registration to Ground Model," IPSJ Transactions on Computer Vision and Applications (CVA), Vol. 7, pp. 89-93, July 27, 2015. [video] [video]
  • A. Banno, T. Masuda, T. Oishi, and K. Ikeuchi, "Flying Laser Range Sensor for Large-Scale Site-Modeling and Its Applications in Bayon Digital Archival Project," International Journal of Computer Vision (IJCV), Vol. 78, No. 2-3, pp. 207-222, Jul. 2008.

Image processing

• Event Camera Tracking

  We proposed an event camera tracking method for object manipulation by robots. The method employs a unified camera projection model for various lenses, and an error function and optimization method that takes motion blur into account.
  
  

  • Y. Kang, G. Caron, R. Ishikawa, A. Escande, K. Chappellet, R. Sagawa, T. Oishi, "Direct 3D model-based object tracking with event camera by motion interpolation," International Conference on Robotics and Automation (ICRA), 2024. (to appear)

  RIND edge detection

  This study proposes a learning-based method to detect four different types of edges in 2D images. The network architecture is based on the SWIN Transformer and is capable of extracting fine edges by using Dice loss.
  
  

  • L. Miao, R. Ishikawa, and T. Oishi, "SWIN-RIND: Edge Detection for Reflectance, Illumination, Normal and Depth Discontinuity with Swin Transformer," British Machine Vision Conference (BMVC), 2023. [video] [code]

SLAM and Navigation

• Structure Aware Loop Closure for SLAM

  This study provides a spherical harmonics (SH) based fast structural representation (SH-FS) in visual SLAM using sparse point clouds, which extracts the structure information from sparse points into single vector. SH-FS was applied in conventional feature-based loop closing process. Furthermore, a structure-aware loop closing method in visual SLAM was proposed to improve the robustness of SLAM systems.
  
  

  • S. Xie, R. Ishikawa, K. Sakurada, M. Onishi and T. Oishi, "Fast Structural Representation and Structure-aware Loop Closing for Visual SLAM," IEEE/RSJ International Conference on Intelligent Robots (IROS), 2022. [video]

  Drone-Rover relative pose estimation

  To take advantage of the drone's wide field of view, we developed a drone pose estimation system from the ground vehicle. In this system, the relative pose is obtained by direct measurement by LiDAR and indirect measurement of the camera's vanishing directions.
  
  

  • J. Hausberg, R. Ishikawa, M. Roxas. T. Oishi, "Relative Drone - Ground Vehicle Localization using LiDAR and Fisheye Cameras through Direct and Indirect Observations," arXiv:2011.07008, 2020. [video]

  Robot navigation by SLAM-device

  We proposed a concept to attach the SLAM-device onto a robot and quickly realize the robot navigation in 3D space. The proposed method calibrates SLAM-device and robot itself by using the relative poses obtained by several robot movements and clarified the efficient ones for the calibration according to the DoF of the robot. Furthermore, the relative pose is dynamically refined so that the contact between the environment and the robot maintains the geometric consistency.
  
  

  • R. Ishikawa, T. Oishi, K. Ikeuchi, "Dynamic Calibration between a Mobile Robot and SLAM Device for Navigation," The 28th IEEE International Conference on Robot and Human Interactive Communication, 2019. [ src (HoloLensRobotNav)] [ src (HoloLens Robot ROSPackage)] [video(long)] [video(short)]

Depth Estimation

• Non-Learning Stereo-Aided Depth Completion

  We propose a non-learning depth completion method for a sparse depth map captured using a LiDAR sensor guided by a pair of stereo images. The proposed selective stereo matching (SSM) method searches the most appropriate depth value for each image pixel from its neighborly projected LiDAR points based on an energy minimization framework. This depth selection approach can handle any type of mis-projection. Moreover, SSM has an advantage in terms of long-range depth accuracy because it directly uses the LiDAR measurement rather than the depth acquired from the stereo.
  
  

  • Y. Yao, R. Ishikawa, S. Ando, K. Kurata, N. Ito, J. Shimamura, T. Oishi, "Non-Learning Stereo-Aided Depth Completion Under Mis-Projection via Selective Stereo Matching," in IEEE Access, vol. 9, pp. 136674-136686, 2021.

  Depth upsampling

  We proposed methods for estimating a dense depth map from a sparse LIDAR point cloud and images. Our unsupervised approach is a real-time dense depth completion from sparse depth maps guided by a single image. Our method generates smooth depth maps while preserving discontinuity between different objects. The key idea is a Binary Anisotropic Diffusion Tensor (B-ADT) which can eliminate smoothness constraint at intended positions and directions by applying variational regularization.
  
  

  • Y. Yao, M. Roxas, R. Ishikawa, S. Ando, J. Shimamura, and T. Oishi, "Discontinuous and Smooth Depth Completion with Binary Anisotropic Diffusion Tensor," IEEE Robotics and Automation Letters, vol. 5, no. 4, pp. 5128-5135, Oct. 2020. arXiv:2006.14374

  Semantically-guided depth upsampling

  Another approach relies on a directionally biased propagation of known depth to missing areas based on semantic segmentation. Additionally, we classify different object boundaries as either occluded or connected to limit the extent of the data propagation. At the regions with inevitably missing point cloud data, we depend on estimated depth using motion stereo.
  
  

  • A. Hirata, R. Ishikawa, M. Roxas, T. Oishi, "Real-Time Dense Depth Estimation using Semantically-Guided LIDAR Data Propagation and Motion Stereo," IEEE Robotics and Automation Letters, vol. 4, no. 4, pp. 3806-3811, Oct. 2019. [video] [ src (MATLAB for accuracy comparison)]

  Fish-eye depth estimation

  We propose a real-time dense 3D mapping method for fisheye cameras without explicit rectification and undistortion. We extend the conventional variational stereo method by constraining the correspondence search along the epipolar curve using a trajectory field induced by camera motion. We also propose a fast way of generating the trajectory field without increasing the processing time compared to conventional rectified methods.
  
  

  • M. Roxas and T. Oishi, "Variational Fisheye Stereo," IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 1303-1310, April 2020. arXiv:1909.07545 [ src] [video(result)] [video(presentation)]

Teleoperation

• Teleoperation of humanoid robot

  We proposed a method to generate robot motions for vision-based teleoperation systems. Task Model performs recognition and transmission of human motions and can simultaneously solve the issues of the structural differences between the human and the humanoid robot and time delays. We also developed a method of articulation modeling using an RGB-D sensor. The articulation parameters are estimated by a fusion of hand motion and point-cloud alignment.
  
  

  • L. Fu, R. Ishikawa, Y. Sato and T. Oishi, "CAPT: Category-level Articulation Estimation from a Single Point Cloud Using Transformer," International Conference on Robotics and Automation (ICRA 2024), 2024. (to appear)
  • R. Hartant, R. Ishikawa, M. Roxas, T. Oishi, "Hand-Motion-guided Articulation and Segmentation Estimation," The 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), Naples Italy, 2020, arXiv:2005.03691. [src] [video]
    
    

  • M. Ogawa, K. Honda, Y. Sato, T. Oishi and K. Ikeuchi, "Development of interface for teleoperation of humanoid robot using task model method," Proc. 2016 IEEE/SICE International Symposium on System Integration, Dec. 2016, Sapporo, Japan.
    
    M. Ogawa, K. Honda, Y. Sato, S. Kudoh, T. Oishi, K. Ikeuchi, "Motion Generation of the Humanoid Robot for Teleoperation by Task Model," Proc. 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pp. 71-76, Sept. 1, 2015, Kobe Japan.

City modeling

• Global positioning in urban environment

  • A. Kumar, Y. Sato, T. Oishi and K. Ikeuchi, "Identifying Reflected GPS Signals and Improving Position Estimation Using 3D Map Simultaneously Built with Laser Range Scanner," 13th ITS Asia-Pacific Forum, Auckland, New Zealand, April 2014.

• Large structure modeling

  • L. Xue, S. Ono, A. Banno, T. Oishi, Y. Sato, K. Ikeuchi, "Global 3D Modeling and its Evaluation for Large-Scale Highway Tunnel using Laser Range Sensor," In Proc. 19th ITS World Congress Vienna, Oct. 2012, Austria. (Best Paper Award)

• Object detection

  • Z. Wang, M. Kagesawa, S. Ono, A. Banno, T. Oishi, K. Ikeuchi, "Detection of Emergency Telephone Indicators in a Tunnel Environment," International Journal of Intelligent Transportation Systems Research, Jan. 2014.

• Historical Analysis of ITS

  • K. Koide, T. Oishi and K. Ikeuchi, "Historical Analysis of the ITS Progress of Japan," International Journal of Intelligent Transportation Systems Research, pp. 1-10, 2015.