Performance:OpenCV:BoofCV:2011 - Revision history

Peter: Peter moved page Performance:OpenCV:BoofCV to Performance:OpenCV:BoofCV:2011

2019-03-23T02:42:30Z

Peter moved page Performance:OpenCV:BoofCV to Performance:OpenCV:BoofCV:2011

← Older revision	Revision as of 02:42, 23 March 2019
(No difference)

Peter at 04:01, 22 February 2013

2013-02-22T04:01:37Z

← Older revision		Revision as of 04:01, 22 February 2013
Line 48:		Line 48:
	* Ubuntu 10.10 64bit		* Ubuntu 10.10 64bit
	* Quadcore Q6600 2.4 GHz		* Quadcore Q6600 2.4 GHz
	* Memory 8194 GB		* Memory 8194 MB
	* g++ 4.4.5		* g++ 4.4.5
	* Java(TM) SE Runtime Environment (build 1.6.0_26-b03)		* Java(TM) SE Runtime Environment (build 1.6.0_26-b03)

Peter at 19:39, 8 November 2011

2011-11-08T19:39:39Z

← Older revision		Revision as of 19:39, 8 November 2011
Line 53:		Line 53:

	Compiler and JRE Configuration:		Compiler and JRE Configuration:
	* ~~All native libraries were~~ compiled with -O3		* OpenCV code was compiled without IPP (no multi-threaded code)
			* OpenCV benchmark code was compiled with -O3 optimization flag
	* Java applications were run with no special flags		* Java applications were run with no special flags

Peter at 15:46, 2 November 2011

2011-11-02T15:46:32Z

← Older revision		Revision as of 15:46, 2 November 2011
Line 13:		Line 13:
	Comparing low level algorithms was much easier than high level algorithms. Typically low level image processing algorithms don't require any parameters and have more standardized implementations. High level algorithms are open to more interpretation and it was often the case that one library would have configuration parameters not available to the other. For high level algorithms the parameters were tuned such that they produced similar output. It should be noted only speed is consider and not quality, with the exception of SURF performance. A more detailed study of SURF is available at the [[Performance:SURF\|SURF performance page]]. For example, one library might be much faster than another, but is able to achieve that speed by producing low quality output.		Comparing low level algorithms was much easier than high level algorithms. Typically low level image processing algorithms don't require any parameters and have more standardized implementations. High level algorithms are open to more interpretation and it was often the case that one library would have configuration parameters not available to the other. For high level algorithms the parameters were tuned such that they produced similar output. It should be noted only speed is consider and not quality, with the exception of SURF performance. A more detailed study of SURF is available at the [[Performance:SURF\|SURF performance page]]. For example, one library might be much faster than another, but is able to achieve that speed by producing low quality output.

	In other words, ''~~'TAKE THIS STUDY WITH A GRAIN OF SALT'''.~~		''In other words, take this study with a grain of salt.''

	= Performance Results =		= Performance Results =
Line 57:		Line 57:

	Library Version		Library Version
	* BoofCV ~~previous~~ release 10/2011		* BoofCV preview release 10/2011
	* OpenCV 2.3.1 SVN r6879		* OpenCV 2.3.1 SVN r6879

Peter at 15:44, 2 November 2011

2011-11-02T15:44:37Z

← Older revision		Revision as of 15:44, 2 November 2011
Line 41:		Line 41:
	# Kill all extraneous processes.		# Kill all extraneous processes.
	# Run benchmark application for each library		# Run benchmark application for each library
	#* For BoofCV each test was run one at a time to improve ~~stability~~. See [[http://code.google.com/p/java-matrix-benchmark/wiki/Methodology_RuntimeBenchmark\|Java Matrix benchmark]] for a discussion of this issue.		#* For BoofCV each test was run one at a time to improve performance consistency. See [[http://code.google.com/p/java-matrix-benchmark/wiki/Methodology_RuntimeBenchmark\|Java Matrix benchmark]] for a discussion of this issue.
	#* ~~As expected, no~~ improvement found for OpenCV code.		#* No improvement found for OpenCV code when running one at a time.
	# Run the whole experiment 5 times for each library and record the best performance.		# Run the whole experiment 5 times for each library and record the best performance.

Peter at 21:23, 1 November 2011

2011-11-01T21:23:18Z

← Older revision		Revision as of 21:23, 1 November 2011
Line 2:		Line 2:

	A key element in real-time computer vision is how fast the computer vision library is. Common wisdom dictates that [http://opencv.willowgarage.com/wiki/ OpenCV] should crush BoofCV as far as speed is concerned because OpenCV is written in C/C++ and has been develop/optimized since 1999, while BoofCV is written in Java and started development the summer of 2011. It is true that C/C++ is faster than Java, but just being written in C/C++ does not make an implementation fast.		A key element in real-time computer vision is how fast the computer vision library is. Common wisdom dictates that [http://opencv.willowgarage.com/wiki/ OpenCV] should crush BoofCV as far as speed is concerned because OpenCV is written in C/C++ and has been develop/optimized since 1999, while BoofCV is written in Java and started development the summer of 2011. It is true that C/C++ is faster than Java, but just being written in C/C++ does not make an implementation fast.

	To test the relative speed of each library several algorithms which have been implemented in both were tested. Algorithms were selected based upon how common their usage is. Both libraries were configured to be as similar to each other as possible. OpenCV code was primarily based upon tutorial code available at their website. Highly specialized low level optimization only a highly advanced user or developer would know was not tested.

	Comparing low level algorithms was much easier than high level algorithms. Typically low level image processing algorithms don't require any parameters and have more standardized implementations. High level algorithms are open to more interpretation and it was often the case that one library would have configuration parameters not available to the other. For high level algorithms the parameters were tuned such that they produced similar output. It should be noted only speed is consider and not quality, with the exception of SURF performance. A more detailed study of SURF is available at the [[Performance:SURF\|SURF performance page]]. For example, one library might be much faster than another, but is able to achieve that speed by producing low quality output.

	~~In other words, '''TAKE THIS STUDY WITH A GRAIN OF SALT'''.~~

	Tested Operations		Tested Operations
Line 17:		Line 11:
	# SURF detect and described.		# SURF detect and described.

	= Results =		Comparing low level algorithms was much easier than high level algorithms. Typically low level image processing algorithms don't require any parameters and have more standardized implementations. High level algorithms are open to more interpretation and it was often the case that one library would have configuration parameters not available to the other. For high level algorithms the parameters were tuned such that they produced similar output. It should be noted only speed is consider and not quality, with the exception of SURF performance. A more detailed study of SURF is available at the [[Performance:SURF\|SURF performance page]]. For example, one library might be much faster than another, but is able to achieve that speed by producing low quality output.

			In other words, '''TAKE THIS STUDY WITH A GRAIN OF SALT'''.

			= Performance Results =
	<center>		<center>
	{\| class="wikitable" style="text-align: center; width: 400px; height: 300px;"		{\| class="wikitable" style="text-align: center; width: 400px; height: 300px;"
Line 36:		Line 34:
	If both libraries had equivalent parameters then the same values would be used. Otherwise parameters would be hand tuned until the produced similar results. For feature extraction algorithms this was defined as producing the same number of features.		If both libraries had equivalent parameters then the same values would be used. Otherwise parameters would be hand tuned until the produced similar results. For feature extraction algorithms this was defined as producing the same number of features.

	Source Code:		Test Source Code:
	# [https://github.com/lessthanoptimal/BoofCV/blob/master/evaluation/benchmark/src/boofcv/benchmark/opencv/BenchmarkForOpenCV.java BoofCV]		# [https://github.com/lessthanoptimal/BoofCV/blob/master/evaluation/benchmark/src/boofcv/benchmark/opencv/BenchmarkForOpenCV.java BoofCV]
	# [https://github.com/lessthanoptimal/BoofCV/tree/master/evaluation/misc/ip/opencv OpenCV]		# [https://github.com/lessthanoptimal/BoofCV/tree/master/evaluation/misc/ip/opencv OpenCV]


	Testing Procedure:		Testing Procedure:

Peter: Created page with "= Relative Speed of BoofCV and OpenCV = A key element in real-time computer vision is how fast the computer vision library is. Common wisdom dictates that [http://opencv.willow..."

2011-11-01T20:51:53Z

Created page with "= Relative Speed of BoofCV and OpenCV = A key element in real-time computer vision is how fast the computer vision library is. Common wisdom dictates that [http://opencv.willow..."

New page

= Relative Speed of BoofCV and OpenCV =

A key element in real-time computer vision is how fast the computer vision library is. Common wisdom dictates that [http://opencv.willowgarage.com/wiki/ OpenCV] should crush BoofCV as far as speed is concerned because OpenCV is written in C/C++ and has been develop/optimized since 1999, while BoofCV is written in Java and started development the summer of 2011. It is true that C/C++ is faster than Java, but just being written in C/C++ does not make an implementation fast.

To test the relative speed of each library several algorithms which have been implemented in both were tested. Algorithms were selected based upon how common their usage is. Both libraries were configured to be as similar to each other as possible. OpenCV code was primarily based upon tutorial code available at their website. Highly specialized low level optimization only a highly advanced user or developer would know was not tested.

Comparing low level algorithms was much easier than high level algorithms. Typically low level image processing algorithms don't require any parameters and have more standardized implementations. High level algorithms are open to more interpretation and it was often the case that one library would have configuration parameters not available to the other. For high level algorithms the parameters were tuned such that they produced similar output. It should be noted only speed is consider and not quality, with the exception of SURF performance. A more detailed study of SURF is available at the [[Performance:SURF|SURF performance page]]. For example, one library might be much faster than another, but is able to achieve that speed by producing low quality output.

In other words, '''TAKE THIS STUDY WITH A GRAIN OF SALT'''.

Tested Operations
# Gaussian blur with a 5x5 kernel
# Sobel gradient with a 3x3 kernel
# Harris Corner detector with 5x5 non-max suppression
# Canny with basic contour extraction.
# Hough Line detection using polar coordinates.
# SURF detect and described.

= Results =
<center>
{| class="wikitable" style="text-align: center; width: 400px; height: 300px;"
| http://www.boofcv.org/notwiki/images/benchmark_ip/benchmark_opencv_boofcv.png
|-
| OpenCV does better on low level operations, while BoofCV does better on most of the high level operations. Higher bars are better.
|}
</center>

As expected, OpenCV performed very well for low level image processing routines. These routines can be highly optimized and take advantage of machine specific vectorization operation. Java takes a significant speed hit when traversing through arrays because for each operation it must perform a bounds check, which is not required in C/C++. Also when declaring new arrays every element is always initialized to zero, even if this is not needed. Past experience has shown that java tends to be about 3 times slower in array heavy arithmetic, which these results confirm.

For high level algorithms the language in which it was implemented seems to matter less. What is most likely the biggest determining factor is the algorithm implemented and how well it has been implemented. These implementation difference can explain BoofCV's better performance in many of the operations despite its language disadvantage.

= Test Setup =

For all operations, as many iterations were performed for at least one second and the number of operations per second (ops/sec)computed. To avoid degrading results for very fast operations, the number of operations needed to run for at least one second was automatically computed. For display purposes the ops/sec metric was converted into a relative performance metric by dividing it by the best performing library.

If both libraries had equivalent parameters then the same values would be used. Otherwise parameters would be hand tuned until the produced similar results. For feature extraction algorithms this was defined as producing the same number of features.

Source Code:
# [https://github.com/lessthanoptimal/BoofCV/blob/master/evaluation/benchmark/src/boofcv/benchmark/opencv/BenchmarkForOpenCV.java BoofCV]
# [https://github.com/lessthanoptimal/BoofCV/tree/master/evaluation/misc/ip/opencv OpenCV]

Testing Procedure:
# Kill all extraneous processes.
# Run benchmark application for each library
#* For BoofCV each test was run one at a time to improve stability. See [[http://code.google.com/p/java-matrix-benchmark/wiki/Methodology_RuntimeBenchmark|Java Matrix benchmark]] for a discussion of this issue.
#* As expected, no improvement found for OpenCV code.
# Run the whole experiment 5 times for each library and record the best performance.

Test Computer:
* Ubuntu 10.10 64bit
* Quadcore Q6600 2.4 GHz
* Memory 8194 GB
* g++ 4.4.5
* Java(TM) SE Runtime Environment (build 1.6.0_26-b03)

Compiler and JRE Configuration:
* All native libraries were compiled with -O3
* Java applications were run with no special flags

Library Version
* BoofCV previous release 10/2011
* OpenCV 2.3.1 SVN r6879