IEEE Robotics & Automation Magazine - June 2020 - 152

types of memory for processing in the GPU. The processing
flow consists of the following steps:
1) upload data from main memory to GPU memory
2) initiate the GPU computing kernel
3) perform parallel computation in the GPU's cores
4) download the resulting data from GPU memory to main
memory.

500
Number of References

450
400
350
300
250

CUDA
OpenCV
CUDA + OpenCV

200
150
100
50

Year
Figure 1. The number of references in IEEE Xplore for CUDA and
OpenCV.

src

dst

src_gray

cv::cvtColor
CPU Memory

cv::threshold

Figure 2. The image processing flow with a CPU only.

their combined application is not so widespread. Considering
that a GPU/CUDA module for OpenCV has been available
since 2010, the number of works using both libraries published in IEEE Xplore is relatively small and has grown very
slowly. Figure 1 depicts the number of references in IEEE
Xplore citing CUDA, OpenCV, or both. In 2018 only seven
references for both are found, compared with 240 for CUDA
and 180 for OpenCV.
The aim of this article is to describe how the CUDA module for OpenCV works, with some examples of well-known
vision problems documented with source code,
in order to en courage
A significant part of
more robotics researchers to migrate their apcomputer vision is image
plications toward GPU
computation.
processing, with massive
The usefulness of
CUDA in robotics and
parallel computations.
vision has been successfully demonstrated with
significant speedups in many applications [3]-[6]. However, it introduces an overhead due to the need to transfer
data between the CPU and GPU spaces because most GPU
processors work in a dedicated memory, independent from
the system memory of the CPU. Consequently, image data
need to be moved back and forth between the different
152

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2020

The OpenCV CUDA Module
In the following example, we assume that the reader is familiar with OpenCV and C++ programming (for novices, an
introduction is provided in [7]). Unless otherwise stated, the
code snippets are based on OpenCV 3.4.0, but they can be
easily adapted to earlier (2.4) or later (4.x) versions.
In the OpenCV library, all the classes and functions are
defined in the name space cv. The main object is the class
cv::Mat, which is essentially a matrix holding pixel values
of an image. The GPU modules in OpenCV define a class
cv::cuda::GpuMat, which is a container for image data
kept in GPU memory, with a very similar interface to its
CPU counterpart.
Let's consider a quick example with a color image that is
converted into gray and binarized with a fixed threshold.
In the CPU version, the source image src is first converted to an intermediate gray image src_gray , which is
then thresholded into the resulting image dst. We need to
define the variables (line 1) and call the OpenCV functions
cv::cvtColor and cv::threshold (lines 2-3) for executing the task:
1 cv::Mat src, src_gray, dst;
2 cv::cvtColor( src, src_gray, cv::COLOR_BGR2GRAY );
3 cv::threshold( src_gray, dst, 128, 255, cv::THRESH_BINARY );

This processing flow is depicted in Figure 2, where all the data
are stored in the CPU memory, and all the operations are performed by the CPU.
In the GPU version, in addition to the variables for the initial and destination images (line 1), we need some new variables for processing the data in the GPU memory (line 2); the
intermediate image src_gray is also stored in the GPU
memory for minimizing data transfers:
1
2
3
4
5

cv::Mat src, dst;
cv::cuda::Mat gpu_src, gpu_dst, src_gray;
gpu_src.upload( src )
cv::cuda::cvtColor( gpu_src, src_gray, cv::COLOR_BGR2GRAY );
cv::cuda::threshold( src_gray, gpu_dst, 128, 255, cv::THRESH_
BINARY );
6 gpu_dst.download( dst );

The processing task is performed by the equivalent functions of the OpenCV CUDA module c v : : c u d a : :
cvtColor and cv::cuda::threshold. First, the image
is transferred from CPU to GPU memory (line 3); then, the

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020