Performance:OpenCV:BoofCV

The following is a comparison of similar algorithms in BoofCV and OpenCV for speed. Ten different algorithms were tuned to produce similar results and then run on three different architectures, desktop computer running on a Core i7, Raspberry PI 3B+, and ODROID XU4. Algorithms covered go from low level image processing (Gaussian blur) to mid-level features (SIFT).

Introduction

It’s been a while since the last runtime comparison was done between BoofCV and OpenCV in 2011. Back then I thought that neural networks (NN) were essentially worthless and C/C++ was the dominant language. Now NN dominate the field and almost everyone use Python. The main event which prompted this benchmark to be done again was concurrency (a.k.a. threads) being added to BoofCV.

The goal of this benchmark is to replicate the speed that an “average” user can expect when using either of these libraries for low level to mid-level image processing/computer vision routines. This will cover image convolution up to feature detectors. NN and machine learning are not included. It is assumed that the average user will install the library using the easiest possible method, cut and paste example code, and do some simple optimizations. For OpenCV this means “pip install opencv-python” and for BoofCV using pre-built jars on Maven Central. If memory or data structures can be easily recycled then they are.

While this approach sounds easy enough it proved to be impossible to follow 100% and exceptions were made, discussed below. Another issue is that none of the algorithms were implemented the same. In fact, only three of them have a chance of producing nearly identical results; Gaussian blur, Sobel, and Histogram. The others have known major differences. For example, BoofCV’s Canny implementation forces you to blur the image while OpenCV doesn’t. BoofCV’s SURF implementation produces significantly better features than OpenCV’s (SURF Benchmark). The default settings in each library can produce drastically different results. Thus tuning criteria are clearly stated and followed in an attempt to produce comparable output.

To replicate the results please carefully read the instructions on this page and in the source code. Especially for architectures with ARM processors, it took about 3 attempts (or 8 hrs) to get a good build of OpenCV running on Raspberry PI. Suggestions for improving the fairness of this comparison are welcomed.

Benchmark Source Code:
https://github.com/lessthanoptimal/SpeedTestCV

Library	Version
BoofCV	0.33.1
OpenCV	4.0.1

To cite this article use the following:

@misc{BoofVsOpenCV,
  title = {Performance of OpenCV vs BoofCV: March 2019},
  howpublished = {\url{https://boofcv.org/index.php?title=Performance:OpenCV:BoofCV}},
  author = {Peter Abeles},
  originalyear = {03.22.2019}
}

Algorithms, Tuning, and Exceptions

Operation	Tuning Target
Gaussian Blur	Radius = 5
Sobel Gradient	3x3 Kernel
Local Mean Thresholding	Radius = 5
Image Histogram
Canny Edge	Output edge pixel chains. ~550,000 unique pixels in chains
Binary Contour	External Only. 4-Connect Rule. Find around 1,100,000 points
Hough Line	Polar Variant. Resolutions: angle = 1 degree, range = 5 pixels. Detect 500 lines.
SIFT	Detect and Describe. 5 Octaves. 10,000 Features
SURF	Detect and Describe. 4 Octaves. 4 Scales. 10,000 Features

Two images were used in these test. The first image was 3648 x 2736 pixels of a chessboard pattern with a wood background. The second was a local mean thresholded version of the first for use in binary operators. The operation parameters were selected to be reasonable and remove potential biases. One factor that determines how fast a feature detector + descriptor run are the number of features detected. OpenCV's SIFT implementation appears to have its default settings selected for speed. While BoofCV's SIFT has defaults designed to replicate the stability found in the original paper. For SIFT the number of octaves was selected to more closely match the SIFT paper.

As previously mentioned, tuning these two libraries to produce similar results is a very difficult if not impossible problem. An attempt was made to be fair. See in code comments for specific details.

Exceptions to the Rules

SIFT and SURF are covered by patents (or were, SIFT’s just expired this month) and not included in the pip package. That means you need to build it from scratch. Major issues were found on ARM architectures where there was no version of OpenCV 4 that could be easily installed and the default JVM included lack optimizations for ARM. The build settings for OpenCV are included below. An attempt was made to find the best settings and different websites had different recommendations.

OpenCV build flags for Raspberry PI. ODROID was similar.

cmake -D CMAKE_BUILD_TYPE=RELEASE     -D CMAKE_INSTALL_PREFIX=/usr/local     -D INSTALL_PYTHON_EXAMPLES=ON     -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib-4.0.1/modules     -D ENABLE_NEON=ON     -D ENABLE_VFPV3=ON     -D WITH_FFMPEG=ON     -D WITH_GSTREAMER=ON     -D BUILD_EXAMPLES=ON -D OPENCV_ENABLE_NONFREE=ON ..

Results

Results are shown below for Intel Core i7, Odroid XU4, and Raspberry PI 3B+. Click on the arrow to change which results you are viewing.

OpenCV does very well in the Gaussian Blur test due to its hand crafted SIMD instructions being multi-threaded. For other low level SIMD friendly operations the speed difference isn't as great between Java and the C code, so it tends to come down to threading. SURF isn't a very SIMD friendly algorith and the major difference in performance come from algorithmic details. The main surprise is SIFT, which should have crushed BoofCV because the most computationally expensive part is apply Gaussian blur. OpenCV's SIFT implementation took a speed it when it was tuned to more closely approximate the original algorithm.

Results between architectures are more consistent than I thought they would be. OpenCV on desktop used the generic version contained in pypy (except for SIFT and SURF) while OpenCV for ARM architectures had been custom built for each architecture. Winners and near ties are essentially identical, except for mean-threshold on ODROID. SIFT was unable to finish computing on ARM processors. Lack of memory is the suspect. BoofCV's SIFT implementation avoids the need to save each layer at the same time.

• 
• 
• 

Conclusions

In this benchmark, BoofCV out performed OpenCV in 8 out of 10 benchmarks on desktop and 7 out of 10 on ARM processors. OpenCV does well in low level SIMD friendly operation due to superior optimization by GCC and hand written SIMD instructions. However, even with that advantage BoofCV now out performs it in many low level operations due to superior concurrent algorithms. For higher level operations performance starts to tilt even more so in BoofCV’s direction where faster algorithms play a larger role.

End Comment

The last time I published this benchmark I was a bit surprised at the lack of reading comprehension exhibited by academic authors. The results were clearly split down the middle, yet most people somehow concluded that OpenCV was the winner! The real answer to which library is faster/better is “it depends”. If you ignore programming language preference then the following would be true. Is your problem heavy in pure image convolution? OpenCV is best for you! Do you want to use a fast and stable QR code detector (see these results) then BoofCV is for you.