Performance:OpenCV:BoofCV
The following is a comparison of similar algorithms in BoofCV and OpenCV for speed. Ten different algorithms were tuned to produce similar results and then run on three different architectures, desktop computer running on a Core i7-6700, Raspberry PI 3B+, and ODROID XU4. Algorithms covered go from low level image processing (Gaussian blur) to mid-level features (SIFT).
Introduction
It’s been a while since the last runtime comparison was done between BoofCV and OpenCV in 2011. Back then I thought that neural networks (NN) were essentially worthless and C/C++ was the dominant language. Now NN dominate the field and almost everyone use Python. The main event which prompted this benchmark to be done again was concurrency (a.k.a. threads) being added to BoofCV.
The goal of this benchmark is to replicate the speed that an “average” user can expect when using either of these libraries for low level to mid-level image processing/computer vision routines. This will cover image convolution up to feature detectors. NN and machine learning are not included. It is assumed that the average user will install the library using the easiest possible method, cut and paste example code, and do some simple optimizations. For OpenCV this means “pip install opencv-python” and for BoofCV using pre-built jars on Maven Central. If memory or data structures can be easily recycled then they are.
While this approach sounds easy enough it proved to be impossible to follow 100% and exceptions were made, discussed below. Another issue is that none of the algorithms were implemented the same. In fact, only three of them have a chance of producing nearly identical results; Gaussian blur, Sobel, and Histogram. The others have known major differences. For example, BoofCV’s Canny implementation forces you to blur the image while OpenCV doesn’t. BoofCV’s SURF implementation produces significantly better features than OpenCV’s (SURF Benchmark). The default settings in each library can produce drastically different results. Thus tuning criteria are clearly stated and followed in an attempt to produce comparable output.
To replicate the results please carefully read the instructions on this page and in the source code. Especially for architectures with ARM processors, it took about 3 attempts (or 8 hrs) to get a good build of OpenCV running on Raspberry PI. Suggestions for improving the fairness of this comparison are welcomed.
https://github.com/lessthanoptimal/SpeedTestCV
| Library | Version |
|---|---|
| BoofCV | 0.33.1 |
| OpenCV | 4.0.1 |
| Device | CPU | Cores | RAM | OS |
|---|---|---|---|---|
| Desktop | Core i7-6700 | 4 | 32 GB | Ubuntu 18.04.2 @ 64bit |
| Raspberry PI 3B+ | Cortex-A53 | 4 | 1 GB | Raspbian 9.4 @ 32bit |
| ODROID XU4 | Cortex-A15 and A7 | 4+4 | 2 GB | Ubuntu 16.04.4 @ 32bit |
To cite this article use the following:
@misc{BoofVsOpenCV,
title = {Performance of OpenCV vs BoofCV: March 2019},
howpublished = {\url{https://boofcv.org/index.php?title=Performance:OpenCV:BoofCV}},
author = {Peter Abeles},
originalyear = {03.22.2019}
}
Algorithms, Tuning, and Exceptions
| Operation | Tuning Target |
|---|---|
| Gaussian Blur | Radius = 5 |
| Sobel Gradient | 3x3 Kernel |
| Local Mean Thresholding | Radius = 5 |
| Image Histogram | |
| Canny Edge | Output edge pixel chains. ~550,000 unique pixels in chains |
| Binary Contour | External Only. 4-Connect Rule. Find around 1,100,000 points |
| Good Features | Shi-Tomasi corners. Unweighted. Radius = 10 pixels. 3,000 Features |
| Hough Line | Polar Variant. Resolutions: angle = 1 degree, range = 5 pixels. Detect 500 lines. |
| SIFT | Detect and Describe. 5 Octaves. 3 Scales. No doubling of first octave. 10,000 Features |
| SURF | Detect and Describe. 4 Octaves. 4 Scales. 10,000 Features |
Two images were used in these test. The first image was 3648 x 2736 pixels of a chessboard pattern with a wood background and was processed as an 8-bit gray scale image. The second was a binary version of the just mentioned image for use by binary operators. This ensured that the binary operators had the same initial input. Tuning parameters and tuning goals mentioned above were selected based on common use cases and to remove potential biases. As an example, one factor that determines how fast a feature detector + descriptor run are the number of features detected since each detected feature must be described.
As previously mentioned, tuning these two libraries to produce similar results is a very difficult if not impossible problem. An attempt was made to be fair. See in code comments for specific details for why values were selected. The best way to ensure that two implementations are "equivalent" is to apply them to the same task and measure their performance. That approach is very labor intensive and often impossible due to difference in quality between two implementations, see the SURF Benchmark as an example, and was not done here.
Exceptions to the Rules
SIFT and SURF are covered by patents (or were, SIFT’s just expired this month) and not included in the pip package. That means you need to build OpenCV from scratch. Thus, on Desktop, those two operations were running code custom built for my architecture breaking the "average user" rule. Major issues were found on ARM architectures where there was no version of OpenCV 4 that could be easily installed and for BoofCV, the default JVM included lack optimizations for ARM making it run very slow!
The build settings for OpenCV on ARM are included below. An attempt was made to find the best settings and different websites had different recommendations. I picked one which explicitly enabled CPU specific optimizations.
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D INSTALL_PYTHON_EXAMPLES=ON -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib-4.0.1/modules -D ENABLE_NEON=ON -D ENABLE_VFPV3=ON -D WITH_FFMPEG=ON -D WITH_GSTREAMER=ON -D BUILD_EXAMPLES=ON -D OPENCV_ENABLE_NONFREE=ON ..
Instructions for installing the JVM used on ARM architectures:
jdk11.0.2-linux-arm32-vfp-hflt
While outside of the scope of this benchmark, building OpenCV on your specific architecture does provide significant performance boost for some operations. Gaussian blur ran about 2x faster on Desktop when custom built.
Results
Results are shown below for Intel Core i7, Odroid XU4, and Raspberry PI 3B+. Click on the arrow to change which results you are viewing.
OpenCV does very well in the Gaussian Blur test due to its hand crafted SIMD instructions being multi-threaded. For other low level SIMD friendly operations the speed difference isn't as great between Java and the C code (GCC does a better job optimizing for SIMD than JVM), so it tends to come down to threading performance. SURF doesn't lend itself towards SIMD optimization, meaning that the compiler is less important and algorithm more important. The main surprise is SIFT, which should have crushed BoofCV because the most computationally expensive part is applying Gaussian blur many times.
Results between architectures are more consistent than it was thought they would be. OpenCV on desktop used the generic version contained in pypy (except for SIFT and SURF) while OpenCV for ARM architectures had been custom built for each architecture. Winners and near ties are effectively the same. OpenCV's SIFT was unable to finish computing on ARM processors, threw out of memory error or just died. OpenCV's SIFT code has not been inspect to root cause this problem, but BoofCV's implementation was designed to recycle images as much as possible.
Conclusions
In this benchmark, BoofCV was the top performer in 6 out of 10, there was a tie in 2 operations, and OpenCV did best in 2 operations. This means on average BoofCV was the top overall performer. OpenCV typically does well in low level SIMD friendly operation (e.g. convolutions) due to superior optimization by GCC and in some cases, hand written SIMD instructions with concurrent code. Both OpenCV and BoofCV have spotty support for concurrency. In the case of Sobel, BoofCV now outperforms OpenCV, even though OpenCV has many advantages, due to BoofCV's implementation being concurrent. For high level algorithms, which are not dominated by SIMD friendly operations, BoofCV does very well and the likely explanation for better performance is more efficient algorithms.
