IEEE Robotics & Automation Magazine - June 2017 - 66

Robotic System

Microscopic System

Work Flow

Stereo
Laparoscope

Fusion and
Visualization

PSM
dVRK
Controllers

Secondary
PC

Stereo Image

Endomicroscope
System
Scanning
Site

OCT
Probe
pCLE
Probe

Trajectory Planning
pCLE Image

OCT
System

Stereo Reconstruction

OCT Image

OCT
Depth
Estimation

Visual Servoing
on
da Vinci Robot

Endomicroscopic
Mosaic

OCT Volume Rendering

Figure 1. An overview of our experimental setup and the steps involved for autonomous optical biopsy probe scanning and multiscale
fusion. The robotic system consists of a set of dVRK controllers, and both the pCLE and OCT probes are grasped by a da Vinci PSM.
The microscopic system consists of an endomicroscope (pCLE) system, an OCT system, and a PC used to capture and process pCLE
and OCT images. The data flow streaming from the different imaging modalities is processed for visualization and servoing purposes.
From a pair of stereo images, a surface of the scene is reconstructed as a point cloud. By stitching pCLE images, a mosaic image can be
created, and a 3-D volume can be built from OCT images. These results are fused into a unified window for multiscale visualization.

I Dt
E,
T PP * = ;
0 1

(6)

where Dt = [Dt x, Dt y, Dt z] < is a displacement vector and I is
a 3 # 3 identity matrix. To obtain the displacement vector, we
use the information from either or both of the pCLE and
OCT images. The rest of the transformations in (5) are constants that can be measured or calibrated in advance. In particular, the end-effector-to-marker transformation T EM is
calibrated using a standard hand-eye calibration method
[19]. The marker-to-probe transformation T PM is determined
from the computer-aided design model of the adapter. We
also note that T EM** = (T EM) - 1 and T PM** = (T PM) - 1 . Continuous detection of the marker is not required during local
scanning with closed-loop servoing, as it is only used to determine the transformation between the robot end effector and
probe, which does not change in time with a rigid setup. Note
that the scanning surface is assumed to be a plane, as each
individual scanning region is usually small (about 2 # 2 mm).
Servoing Using pCLE Images
To allow a continuous mosaic to be generated over the desired
area of tissue, the visual servoing loop is closed on the mosaic
image. The pCLE mosaicing is performed using an approach
similar to the standard real-time technique described in
[6] and [17], using normalized cross correlation to estimate
the relative shift between each pair of consecutive frames. At
the beginning of each scanning procedure, the probe position
in the mosaic image m p (t = 0) = (0, 0) is located at the center of the image. During scanning, an estimate of the current
probe position m p (t) at time t is obtained from the mosaic
(i.e., from integrated pairwise image shifts over time). Next,
by comparing m p (t) with the kth desired probe position in
66

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2017

the trajectory m p * (k), the displacement between the current
and desired position is calculated as
cos a -sin a m p *x - m p x
Dt x
E·=
G,
= G = km ·;
sin a cos a m p *y - m p y
Dt y

(7)

where a is a 2-D rotation angle between the probe and the
mosaic image, and k m is a constant that converts the displacement from pixels to real distance. To calibrate the rotation
angle a , we drive the robot in a horizontal line scan using the
laparoscope. The angle a between the scanned line and the
desired horizontal line can be calculated.
Servoing Using OCT Images
The distance from the OCT probe to the tissue can be estimated by detecting the top surface of the tissue in the OCT
cross-sectional image, which can be seen in Figure 1. The
OCT probe is mounted slightly higher than the pCLE probe,
so that the top surface of the tissue appears at a nonzero depth
in the OCT image. Because the top surface usually maintains
the most intense signal, this can be found by simple peak
detection, taking the first peak above a user-defined threshold. To mitigate the influence of noise, a Kalman filter is
applied to assist accurate distance estimation for servoing.
Here, we consider a constant velocity model for the Kalman
filter, as the motion of the robot along the depth direction is
smooth. At the beginning of each scan, we assume that a
good initial contact has been made by the user (i.e., clear
pCLE images can be seen), and the current OCT distance
estimate d oct is recorded. This distance is then set as the
desired distance d *oct , and the robot is thus required to maintain this distance during scanning. The displacement along
the z -axis is defined as

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2017