BoofCV - User contributions [en]

User:FranceDauby1095

2011-11-16T00:11:28Z

Spam: Created page with "[http://www.seatmpin.com WHATSSSSS]"

[http://www.seatmpin.com WHATSSSSS]

Performance:SURF

2011-11-01T17:32:05Z

Spam: /* Comparison of SURF implementations */

= Comparison of SURF implementations =

The [http://en.wikipedia.org/wiki/SURF SURF] descriptor is a state-of-the-art image region descriptor that is invariant with regard to scale, orientation, and illumination. By using an integral image, the descriptor can be computed efficiently across different scales. In recent years it has emerged as one of the more popular and frequently-used feature descriptors, but it is not a trivial algorithm to implement, and several different implementations exist. The following study compares several different libraries to determine relative stability and runtime performance.

Tested Implementations:

{| cellpadding="5" cellspacing="0" border="1"
! Implementation
! Version
! Language
! Threaded
! Comment
|-
| BoofCV || 10/2011 || Java || || Fast but less accurate. See FactoryDescribeRegionPoint.surf()
|-
| BoofCV-M || 10/2011 || Java || || Accurate but slower. See FactoryDescribeRegionPoint.msurf()
|-
| OpenSURF || 27/05/2010 || C++ || || http://www.chrisevansdev.com/computer-vision-opensurf.html
|-
| Reference || 1.0.9 || C++ || || http://www.vision.ee.ethz.ch/~surf/
|-
| JOpenSURF || SVN r24 || Java || || http://code.google.com/p/jopensurf/
|-
| JavaSURF || SVN r4 || Java || || http://code.google.com/p/javasurf/
|-
| OpenCV || 2.3.1 SVN r6879 || C++ || Yes || http://opencv.willowgarage.com/wiki/
|}

Benchmark Source Code:
* [https://github.com/lessthanoptimal/BoofCV/tree/master/evaluation/benchmark BoofCV benchmark]
* [https://github.com/lessthanoptimal/BoofCV/tree/master/evaluation/misc External libraries]

= Conclusions =

{|
| http://www.boofcv.org/notwiki/images/benchmark_surf/overall_describe_speed.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/overall_detect_speed.gif
|-
! Lower is better. !! Lower is better.
|-
|colspan="2"| <center>http://www.boofcv.org/notwiki/images/benchmark_surf/overall_stability.gif </center>
|-
!colspan="2"| Higher is better.
|}

For the sake of those with short attention spans, the summary results are posted first and a discussion of testing methodology follows. The top two plots show feature detection and feature describe speed. Feature detection is the detection of the location and scale of interest points inside the image using each library's implementation of the Fast Hessian detector. A feature is described by estimating its orientation and computing the SURF-64 descriptor. The bottommost plot shows a summary of the descriptors' relative stabilities across a standard set of test images.

One reason for JavaSURF's poor stability is that it only implements an upright version of SURF, so rotated images defeat the descriptor. Not computing orientation helps JavaSURF on the description runtime benchmark, because it has fewer computations to perform. JOpenSURF is a straightforward port of the OpenSURF library to Java and shows comparable stability with the expected hit on runtime performance. JOpenSURF, OpenSURF and BoofCV-M all compute an enhanced version of the SURF descriptor, while the BoofCV descriptor is closer to the SURF paper with some improvements. I suspect that the descriptor computed by the reference library is also an improvement over what was presented in the SURF paper, but the source code is closed, so this theory cannot be directly verified.

OpenCV is a bit of an oddball library as far as SURF is concerned. It did not provide an interface that would allow it to be tested in the same manner as the other libraries, and comments in the code indicated that parts of it are multi-threaded. Every other library tested is single-threaded. Because of these issues, OpenCV's own interest points were used instead of the precomputed ones. Speed-wise, a special test was done for OpenCV where features were detected and described at the same time, which took 1940 (ms) for 6485 features. This was approximately 20% slower than OpenSURF's combined detect and describe time.

= Descriptor Stability =

<center>
<gallery caption="Images from evaluation data set" heights=150 widths=200 >
Image:Performance-descriptor-Graffiti.jpg|Graffiti
Image:Performance-descriptor-Boat.jpg|Boat
Image:Performance-descriptor-Trees.jpg|Trees
Image:Performance-descriptor-Bricks.jpg|Bricks
</gallery>
</center>

The stability benchmark was performed using standardized test images from [http://www.robots.ox.ac.uk/~vgg/research/affine/], which have known transformations. Stability was measured based on the number of correct associations between two images in the data set. The testing procedure for each library is summarized below:

# For each image, detect features (scale and location) using the fast Hessian detector in BoofCV.
#* Save results to a file and use the same file for all libraries.
# For each image, compute a feature description (including orientation) for all features found.
# In each image sequence, associate features in the first image to the Nth image, where N > 1.
#* Association is done by minimizing Euclidean error.
#* Validation is done using reverse association, i.e. the association must be the optimal association going from frame 1 to N and N to 1.
# Compute the number of correct associations.
#* An association is correct if it is within 3 pixels of the true location.

Since the transformation is known between images, the true location could have been used. However, in reality features will not lie at the exact point, and a descriptor needs to be tolerant of this type of error. Thus, this is a more accurate measure of the descriptor's strength.

The relative stability metric described above is computed by summing up the total percent of correctly associated features across the whole test data set, and then choosing the library with the best performance. The relative stability is computed by dividing each library's score by the best performer's score.

'''Configuration:''' All libraries were configured to describe oriented SURF-64 features as defined in the original SURF paper. JavaSURF does not support orientation estimation. OpenCV forces orientation to be estimated inside the feature detector; therefore it was decided that the lesser evil would be to let OpenCV detect its own features. OpenCV's threshold was adjusted so that it detected about the same number of features.

== Stability Results ==

{|
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_bike.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_boat.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_graf.gif] || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_leuven.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_ubc.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_trees.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_wall.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_bark.gif
|}

= Runtime Speed =
Each library's speed in describing and detecting features was also benchmarked. Each test was performed several times, but only the best time is shown. Java libraries tended to exhibit more variation than native libraries, although all libraries showed a significant amount of variation from trial to trial.

Only image processing time essential to SURF was measured, not image loading time. This would include the time to convert an image to integral image format, but not the time to convert the image to grayscale, assuming it was possible to not include the grayscale conversion. Elapsed time was measured in the actual application using System.currentTimeMillis() in Java and clock() in C++.

Testing Procedure:
# Kill all extraneous processes.
# Load feature location and size from file.
# Compute descriptors (including orientation) for each feature while recording elapsed time.
# Compute elapsed time 10 times and output best result.
# Run the whole experiment 4 times for each library and record the best time.

Test Computer:
* Ubuntu 10.10 64bit
* Quadcore Q6600 2.4 GHz
* Memory 8194 GB
* g++ 4.4.5
* Java(TM) SE Runtime Environment (build 1.6.0_26-b03)

Compiler and JRE Configuration:
* All native libraries were compiled with -O3
* Java applications were run with no special flags

Describe Specific Setup:
* input image was boat/img1
* Fast Hessian features from BoofCV
** 6415 Total

Detect Specific Setup:
* Impossible to configure libraries to detect exact same features
** Adjusted detection threshold to top out at around 2000 features
* Octaves: 4
* Scales: 4
* Base Size: 9
* Initial Pixel Skip: 1

Results can be found at the top of the page. OpenCV was omitted from runtime results because it could not be configured identically to the other libraries. A special test was performed just for OpenCV and is discussed above. It is not known what pixel skip was used inside of OpenCV.

Performance:SURF

2011-11-01T17:31:54Z

Spam: /* Runtime Speed */

= Comparison of SURF implementations =

The [http://en.wikipedia.org/wiki/SURF SURF] descriptor is a state-of-the-art image region descriptor that is invariant with regard to scale, orientation, and illumination. By using an integral image, the descriptor can be computed efficiently across different scales. In recent years it has emerged as one of the more popular and frequently-used feature descriptors, but it is not a trivial algorithm to implement, and several different implementations exist. The following study compares several different libraries to determine relative stability and run-time performance.

Tested Implementations:

{| cellpadding="5" cellspacing="0" border="1"
! Implementation
! Version
! Language
! Threaded
! Comment
|-
| BoofCV || 10/2011 || Java || || Fast but less accurate. See FactoryDescribeRegionPoint.surf()
|-
| BoofCV-M || 10/2011 || Java || || Accurate but slower. See FactoryDescribeRegionPoint.msurf()
|-
| OpenSURF || 27/05/2010 || C++ || || http://www.chrisevansdev.com/computer-vision-opensurf.html
|-
| Reference || 1.0.9 || C++ || || http://www.vision.ee.ethz.ch/~surf/
|-
| JOpenSURF || SVN r24 || Java || || http://code.google.com/p/jopensurf/
|-
| JavaSURF || SVN r4 || Java || || http://code.google.com/p/javasurf/
|-
| OpenCV || 2.3.1 SVN r6879 || C++ || Yes || http://opencv.willowgarage.com/wiki/
|}

Benchmark Source Code:
* [https://github.com/lessthanoptimal/BoofCV/tree/master/evaluation/benchmark BoofCV benchmark]
* [https://github.com/lessthanoptimal/BoofCV/tree/master/evaluation/misc External libraries]

= Conclusions =

{|
| http://www.boofcv.org/notwiki/images/benchmark_surf/overall_describe_speed.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/overall_detect_speed.gif
|-
! Lower is better. !! Lower is better.
|-
|colspan="2"| <center>http://www.boofcv.org/notwiki/images/benchmark_surf/overall_stability.gif </center>
|-
!colspan="2"| Higher is better.
|}

For the sake of those with short attention spans, the summary results are posted first and a discussion of testing methodology follows. The top two plots show feature detection and feature describe speed. Feature detection is the detection of the location and scale of interest points inside the image using each library's implementation of the Fast Hessian detector. A feature is described by estimating its orientation and computing the SURF-64 descriptor. The bottommost plot shows a summary of the descriptors' relative stabilities across a standard set of test images.

One reason for JavaSURF's poor stability is that it only implements an upright version of SURF, so rotated images defeat the descriptor. Not computing orientation helps JavaSURF on the description runtime benchmark, because it has fewer computations to perform. JOpenSURF is a straightforward port of the OpenSURF library to Java and shows comparable stability with the expected hit on runtime performance. JOpenSURF, OpenSURF and BoofCV-M all compute an enhanced version of the SURF descriptor, while the BoofCV descriptor is closer to the SURF paper with some improvements. I suspect that the descriptor computed by the reference library is also an improvement over what was presented in the SURF paper, but the source code is closed, so this theory cannot be directly verified.

OpenCV is a bit of an oddball library as far as SURF is concerned. It did not provide an interface that would allow it to be tested in the same manner as the other libraries, and comments in the code indicated that parts of it are multi-threaded. Every other library tested is single-threaded. Because of these issues, OpenCV's own interest points were used instead of the precomputed ones. Speed-wise, a special test was done for OpenCV where features were detected and described at the same time, which took 1940 (ms) for 6485 features. This was approximately 20% slower than OpenSURF's combined detect and describe time.

= Descriptor Stability =

<center>
<gallery caption="Images from evaluation data set" heights=150 widths=200 >
Image:Performance-descriptor-Graffiti.jpg|Graffiti
Image:Performance-descriptor-Boat.jpg|Boat
Image:Performance-descriptor-Trees.jpg|Trees
Image:Performance-descriptor-Bricks.jpg|Bricks
</gallery>
</center>

The stability benchmark was performed using standardized test images from [http://www.robots.ox.ac.uk/~vgg/research/affine/], which have known transformations. Stability was measured based on the number of correct associations between two images in the data set. The testing procedure for each library is summarized below:

# For each image, detect features (scale and location) using the fast Hessian detector in BoofCV.
#* Save results to a file and use the same file for all libraries.
# For each image, compute a feature description (including orientation) for all features found.
# In each image sequence, associate features in the first image to the Nth image, where N > 1.
#* Association is done by minimizing Euclidean error.
#* Validation is done using reverse association, i.e. the association must be the optimal association going from frame 1 to N and N to 1.
# Compute the number of correct associations.
#* An association is correct if it is within 3 pixels of the true location.

Since the transformation is known between images, the true location could have been used. However, in reality features will not lie at the exact point, and a descriptor needs to be tolerant of this type of error. Thus, this is a more accurate measure of the descriptor's strength.

The relative stability metric described above is computed by summing up the total percent of correctly associated features across the whole test data set, and then choosing the library with the best performance. The relative stability is computed by dividing each library's score by the best performer's score.

'''Configuration:''' All libraries were configured to describe oriented SURF-64 features as defined in the original SURF paper. JavaSURF does not support orientation estimation. OpenCV forces orientation to be estimated inside the feature detector; therefore it was decided that the lesser evil would be to let OpenCV detect its own features. OpenCV's threshold was adjusted so that it detected about the same number of features.

== Stability Results ==

{|
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_bike.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_boat.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_graf.gif] || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_leuven.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_ubc.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_trees.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_wall.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_bark.gif
|}

= Runtime Speed =
Each library's speed in describing and detecting features was also benchmarked. Each test was performed several times, but only the best time is shown. Java libraries tended to exhibit more variation than native libraries, although all libraries showed a significant amount of variation from trial to trial.

Only image processing time essential to SURF was measured, not image loading time. This would include the time to convert an image to integral image format, but not the time to convert the image to grayscale, assuming it was possible to not include the grayscale conversion. Elapsed time was measured in the actual application using System.currentTimeMillis() in Java and clock() in C++.

Testing Procedure:
# Kill all extraneous processes.
# Load feature location and size from file.
# Compute descriptors (including orientation) for each feature while recording elapsed time.
# Compute elapsed time 10 times and output best result.
# Run the whole experiment 4 times for each library and record the best time.

Test Computer:
* Ubuntu 10.10 64bit
* Quadcore Q6600 2.4 GHz
* Memory 8194 GB
* g++ 4.4.5
* Java(TM) SE Runtime Environment (build 1.6.0_26-b03)

Compiler and JRE Configuration:
* All native libraries were compiled with -O3
* Java applications were run with no special flags

Describe Specific Setup:
* input image was boat/img1
* Fast Hessian features from BoofCV
** 6415 Total

Detect Specific Setup:
* Impossible to configure libraries to detect exact same features
** Adjusted detection threshold to top out at around 2000 features
* Octaves: 4
* Scales: 4
* Base Size: 9
* Initial Pixel Skip: 1

Results can be found at the top of the page. OpenCV was omitted from runtime results because it could not be configured identically to the other libraries. A special test was performed just for OpenCV and is discussed above. It is not known what pixel skip was used inside of OpenCV.

Performance:SURF

2011-11-01T17:20:28Z

Spam: /* Descriptor Stability */

= Comparison of SURF implementations =

The [http://en.wikipedia.org/wiki/SURF SURF] descriptor is a state-of-the-art image region descriptor that is invariant with regard to scale, orientation, and illumination. By using an integral image, the descriptor can be computed efficiently across different scales. In recent years it has emerged as one of the more popular and frequently-used feature descriptors, but it is not a trivial algorithm to implement, and several different implementations exist. The following study compares several different libraries to determine relative stability and run-time performance.

Tested Implementations:

{| cellpadding="5" cellspacing="0" border="1"
! Implementation
! Version
! Language
! Threaded
! Comment
|-
| BoofCV || 10/2011 || Java || || Fast but less accurate. See FactoryDescribeRegionPoint.surf()
|-
| BoofCV-M || 10/2011 || Java || || Accurate but slower. See FactoryDescribeRegionPoint.msurf()
|-
| OpenSURF || 27/05/2010 || C++ || || http://www.chrisevansdev.com/computer-vision-opensurf.html
|-
| Reference || 1.0.9 || C++ || || http://www.vision.ee.ethz.ch/~surf/
|-
| JOpenSURF || SVN r24 || Java || || http://code.google.com/p/jopensurf/
|-
| JavaSURF || SVN r4 || Java || || http://code.google.com/p/javasurf/
|-
| OpenCV || 2.3.1 SVN r6879 || C++ || Yes || http://opencv.willowgarage.com/wiki/
|}

Benchmark Source Code:
* [https://github.com/lessthanoptimal/BoofCV/tree/master/evaluation/benchmark BoofCV benchmark]
* [https://github.com/lessthanoptimal/BoofCV/tree/master/evaluation/misc External libraries]

= Conclusions =

{|
| http://www.boofcv.org/notwiki/images/benchmark_surf/overall_describe_speed.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/overall_detect_speed.gif
|-
! Lower is better. !! Lower is better.
|-
|colspan="2"| <center>http://www.boofcv.org/notwiki/images/benchmark_surf/overall_stability.gif </center>
|-
!colspan="2"| Higher is better.
|}

For the sake of those with short attention spans, the summary results are posted first and a discussion of testing methodology follows. The top two plots show feature detection and feature describe speed. Feature detection is the detection of the location and scale of interest points inside the image using each library's implementation of the Fast Hessian detector. A feature is described by estimating its orientation and computing the SURF-64 descriptor. The bottommost plot shows a summary of the descriptors' relative stabilities across a standard set of test images.

One reason for JavaSURF's poor stability is that it only implements an upright version of SURF, so rotated images defeat the descriptor. Not computing orientation helps JavaSURF on the description runtime benchmark, because it has fewer computations to perform. JOpenSURF is a straightforward port of the OpenSURF library to Java and shows comparable stability with the expected hit on runtime performance. JOpenSURF, OpenSURF and BoofCV-M all compute an enhanced version of the SURF descriptor, while the BoofCV descriptor is closer to the SURF paper with some improvements. I suspect that the descriptor computed by the reference library is also an improvement over what was presented in the SURF paper, but the source code is closed, so this theory cannot be directly verified.

OpenCV is a bit of an oddball library as far as SURF is concerned. It did not provide an interface that would allow it to be tested in the same manner as the other libraries, and comments in the code indicated that parts of it are multi-threaded. Every other library tested is single-threaded. Because of these issues, OpenCV's own interest points were used instead of the precomputed ones. Speed-wise, a special test was done for OpenCV where features were detected and described at the same time, which took 1940 (ms) for 6485 features. This was approximately 20% slower than OpenSURF's combined detect and describe time.

= Descriptor Stability =

<center>
<gallery caption="Images from evaluation data set" heights=150 widths=200 >
Image:Performance-descriptor-Graffiti.jpg|Graffiti
Image:Performance-descriptor-Boat.jpg|Boat
Image:Performance-descriptor-Trees.jpg|Trees
Image:Performance-descriptor-Bricks.jpg|Bricks
</gallery>
</center>

The stability benchmark was performed using standardized test images from [http://www.robots.ox.ac.uk/~vgg/research/affine/], which have known transformations. Stability was measured based on the number of correct associations between two images in the data set. The testing procedure for each library is summarized below:

# For each image, detect features (scale and location) using the fast Hessian detector in BoofCV.
#* Save results to a file and use the same file for all libraries.
# For each image, compute a feature description (including orientation) for all features found.
# In each image sequence, associate features in the first image to the Nth image, where N > 1.
#* Association is done by minimizing Euclidean error.
#* Validation is done using reverse association, i.e. the association must be the optimal association going from frame 1 to N and N to 1.
# Compute the number of correct associations.
#* An association is correct if it is within 3 pixels of the true location.

Since the transformation is known between images, the true location could have been used. However, in reality features will not lie at the exact point, and a descriptor needs to be tolerant of this type of error. Thus, this is a more accurate measure of the descriptor's strength.

The relative stability metric described above is computed by summing up the total percent of correctly associated features across the whole test data set, and then choosing the library with the best performance. The relative stability is computed by dividing each library's score by the best performer's score.

'''Configuration:''' All libraries were configured to describe oriented SURF-64 features as defined in the original SURF paper. JavaSURF does not support orientation estimation. OpenCV forces orientation to be estimated inside the feature detector; therefore it was decided that the lesser evil would be to let OpenCV detect its own features. OpenCV's threshold was adjusted so that it detected about the same number of features.

== Stability Results ==

{|
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_bike.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_boat.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_graf.gif] || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_leuven.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_ubc.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_trees.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_wall.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_bark.gif
|}

= Runtime Speed =
How fast enough library can compute the description and detect features was also benchmarked. Each test was performed several times with only the best time being shown. Java libraries tended to exhibit more variability than native libraries, while all libraries showed a significant amount of variability from trial to trial.

Only image processing time essential to SURF was measured and not loading in images. This would include converting an image to integral image format, but not converting the image to gray scale. Assuming that it was possible to not include the gray scale conversion. Elapsed time was measured in the actual application using System.currentTimeMillis() in Java and clock() in C++.

Testing Procedure:
# Kill all extraneous processes.
# Load feature location and size from file.
# Compute descriptors (including orientation) for each feature while recording elapsed time.
# Compute elapsed time 10 times and output best result.
# Run the whole experiment 4 times for each library and record the best time.

Test Computer:
* Ubuntu 10.10 64bit
* Quadcore Q6600 2.4 GHz
* Memory 8194 GB
* g++ 4.4.5
* Java(TM) SE Runtime Environment (build 1.6.0_26-b03)

Compiler and JRE Configuration
* All native libraries were compiled with -O3
* Java applications were run with no special flags

Describe Specific Setup:
* input image was boat/img1
* Fast Hessian features from BoofCV
** 6415 Total

Detect Specific Setup:
* Impossible to configure libraries to detect exact same features.
** Adjusted detection threshold to top out at around 2000 features
* Octaves: 4
* Scales: 4
* Base Size: 9
* Initial Pixel Skip: 1

Results can be found at the top of the page. OpenCV was ommited from runtime results because it could not be configured the same way as the other libraries. A special test was performed just for OpenCV and is discussed above. It should also be noted that it is not known what pixel skip was used inside of OpenCV.

Performance:SURF

2011-11-01T17:08:52Z

Spam: /* Conclusions */

= Comparison of SURF implementations =

The [http://en.wikipedia.org/wiki/SURF SURF] descriptor is a state-of-the-art image region descriptor that is invariant with regard to scale, orientation, and illumination. By using an integral image, the descriptor can be computed efficiently across different scales. In recent years it has emerged as one of the more popular and frequently-used feature descriptors, but it is not a trivial algorithm to implement, and several different implementations exist. The following study compares several different libraries to determine relative stability and run-time performance.

Tested Implementations:

{| cellpadding="5" cellspacing="0" border="1"
! Implementation
! Version
! Language
! Threaded
! Comment
|-
| BoofCV || 10/2011 || Java || || Fast but less accurate. See FactoryDescribeRegionPoint.surf()
|-
| BoofCV-M || 10/2011 || Java || || Accurate but slower. See FactoryDescribeRegionPoint.msurf()
|-
| OpenSURF || 27/05/2010 || C++ || || http://www.chrisevansdev.com/computer-vision-opensurf.html
|-
| Reference || 1.0.9 || C++ || || http://www.vision.ee.ethz.ch/~surf/
|-
| JOpenSURF || SVN r24 || Java || || http://code.google.com/p/jopensurf/
|-
| JavaSURF || SVN r4 || Java || || http://code.google.com/p/javasurf/
|-
| OpenCV || 2.3.1 SVN r6879 || C++ || Yes || http://opencv.willowgarage.com/wiki/
|}

Benchmark Source Code:
* [https://github.com/lessthanoptimal/BoofCV/tree/master/evaluation/benchmark BoofCV benchmark]
* [https://github.com/lessthanoptimal/BoofCV/tree/master/evaluation/misc External libraries]

= Conclusions =

{|
| http://www.boofcv.org/notwiki/images/benchmark_surf/overall_describe_speed.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/overall_detect_speed.gif
|-
! Lower is better. !! Lower is better.
|-
|colspan="2"| <center>http://www.boofcv.org/notwiki/images/benchmark_surf/overall_stability.gif </center>
|-
!colspan="2"| Higher is better.
|}

For the sake of those with short attention spans, the summary results are posted first and a discussion of testing methodology follows. The top two plots show feature detection and feature describe speed. Feature detection is the detection of the location and scale of interest points inside the image using each library's implementation of the Fast Hessian detector. A feature is described by estimating its orientation and computing the SURF-64 descriptor. The bottommost plot shows a summary of the descriptors' relative stabilities across a standard set of test images.

One reason for JavaSURF's poor stability is that it only implements an upright version of SURF, so rotated images defeat the descriptor. Not computing orientation helps JavaSURF on the description runtime benchmark, because it has fewer computations to perform. JOpenSURF is a straightforward port of the OpenSURF library to Java and shows comparable stability with the expected hit on runtime performance. JOpenSURF, OpenSURF and BoofCV-M all compute an enhanced version of the SURF descriptor, while the BoofCV descriptor is closer to the SURF paper with some improvements. I suspect that the descriptor computed by the reference library is also an improvement over what was presented in the SURF paper, but the source code is closed, so this theory cannot be directly verified.

OpenCV is a bit of an oddball library as far as SURF is concerned. It did not provide an interface that would allow it to be tested in the same manner as the other libraries, and comments in the code indicated that parts of it are multi-threaded. Every other library tested is single-threaded. Because of these issues, OpenCV's own interest points were used instead of the precomputed ones. Speed-wise, a special test was done for OpenCV where features were detected and described at the same time, which took 1940 (ms) for 6485 features. This was approximately 20% slower than OpenSURF's combined detect and describe time.

= Descriptor Stability =

<center>
<gallery caption="Images from evaluation data set" heights=150 widths=200 >
Image:Performance-descriptor-Graffiti.jpg|Graffiti
Image:Performance-descriptor-Boat.jpg|Boat
Image:Performance-descriptor-Trees.jpg|Trees
Image:Performance-descriptor-Bricks.jpg|Bricks
</gallery>
</center>

The stability benchmark was performed using standardized test images from [http://www.robots.ox.ac.uk/~vgg/research/affine/], which have known transformations. Stability was measured based upon the number of correct associations between two images in the data set. The testing procedure for each library is summarized below:

# For each image, detect features (scale and location) using the fast Hessian detector in BoofCV.
#* Save results to a file and use the same file for all libraries.
# For each image, compute a feature description (including orientation) for all found features.
# In each image sequence, associate features in the first image to the Nth image, where N > 1.
#* Association is done by minimizing Euclidean error
#* Validation is done using reverse association. E.g. This association must be the optimal association going from frame 1 to N and N to 1.
# Compute the number of correct associations.
#* An association is correct if it is within 3 pixels of the true location.

Since the transformation is known between images the true location could have been used. However, in reality features will not lie at the exact point and a descriptor needs to be tolerant to this type of errors. Thus this is a more accurate measure of the description's strength.

The relative stability metric shown in the conclusions section above is computed by first summing up the total percent of correctly associated features across the whole test data set. Then the library with the best performance is found and the relative stability is computed by dividing each library's score by the best performer's score.

'''Configuration:''' All libraries were configured to describe oriented SURF-64 features as defined in the original SURF paper. JavaSURF does not support orientation estimation. OpenCV forces orientation to be estimated inside the feature detector. Thus it was decided that the lesser evil for OpenCV was to let it detect its own features. OpenCV's threshold was adjusted so that it detected about the same number of features.

== Stability Results ==

{|
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_bike.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_boat.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_graf.gif] || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_leuven.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_ubc.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_trees.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_wall.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_bark.gif
|}

= Runtime Speed =
How fast enough library can compute the description and detect features was also benchmarked. Each test was performed several times with only the best time being shown. Java libraries tended to exhibit more variability than native libraries, while all libraries showed a significant amount of variability from trial to trial.

Only image processing time essential to SURF was measured and not loading in images. This would include converting an image to integral image format, but not converting the image to gray scale. Assuming that it was possible to not include the gray scale conversion. Elapsed time was measured in the actual application using System.currentTimeMillis() in Java and clock() in C++.

Testing Procedure:
# Kill all extraneous processes.
# Load feature location and size from file.
# Compute descriptors (including orientation) for each feature while recording elapsed time.
# Compute elapsed time 10 times and output best result.
# Run the whole experiment 4 times for each library and record the best time.

Test Computer:
* Ubuntu 10.10 64bit
* Quadcore Q6600 2.4 GHz
* Memory 8194 GB
* g++ 4.4.5
* Java(TM) SE Runtime Environment (build 1.6.0_26-b03)

Compiler and JRE Configuration
* All native libraries were compiled with -O3
* Java applications were run with no special flags

Describe Specific Setup:
* input image was boat/img1
* Fast Hessian features from BoofCV
** 6415 Total

Detect Specific Setup:
* Impossible to configure libraries to detect exact same features.
** Adjusted detection threshold to top out at around 2000 features
* Octaves: 4
* Scales: 4
* Base Size: 9
* Initial Pixel Skip: 1

Results can be found at the top of the page. OpenCV was ommited from runtime results because it could not be configured the same way as the other libraries. A special test was performed just for OpenCV and is discussed above. It should also be noted that it is not known what pixel skip was used inside of OpenCV.

Performance:SURF

2011-11-01T16:39:51Z

Spam: /* Conclusions */

= Comparison of SURF implementations =

The [http://en.wikipedia.org/wiki/SURF SURF] descriptor is a state-of-the-art image region descriptor that is invariant with regard to scale, orientation, and illumination. By using an integral image, the descriptor can be computed efficiently across different scales. In recent years it has emerged as one of the more popular and frequently-used feature descriptors, but it is not a trivial algorithm to implement, and several different implementations exist. The following study compares several different libraries to determine relative stability and run-time performance.

Tested Implementations:

{| cellpadding="5" cellspacing="0" border="1"
! Implementation
! Version
! Language
! Threaded
! Comment
|-
| BoofCV || 10/2011 || Java || || Fast but less accurate. See FactoryDescribeRegionPoint.surf()
|-
| BoofCV-M || 10/2011 || Java || || Accurate but slower. See FactoryDescribeRegionPoint.msurf()
|-
| OpenSURF || 27/05/2010 || C++ || || http://www.chrisevansdev.com/computer-vision-opensurf.html
|-
| Reference || 1.0.9 || C++ || || http://www.vision.ee.ethz.ch/~surf/
|-
| JOpenSURF || SVN r24 || Java || || http://code.google.com/p/jopensurf/
|-
| JavaSURF || SVN r4 || Java || || http://code.google.com/p/javasurf/
|-
| OpenCV || 2.3.1 SVN r6879 || C++ || Yes || http://opencv.willowgarage.com/wiki/
|}

Benchmark Source Code:
* [https://github.com/lessthanoptimal/BoofCV/tree/master/evaluation/benchmark BoofCV benchmark]
* [https://github.com/lessthanoptimal/BoofCV/tree/master/evaluation/misc External libraries]

= Conclusions =

{|
| http://www.boofcv.org/notwiki/images/benchmark_surf/overall_describe_speed.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/overall_detect_speed.gif
|-
! Lower is better. !! Lower is better.
|-
|colspan="2"| <center>http://www.boofcv.org/notwiki/images/benchmark_surf/overall_stability.gif </center>
|-
!colspan="2"| Higher is better.
|}

For the sake of those with short attention spans, the summary results are posted first and a discussion of testing methodology follows. The top two plots show feature detection and feature describe speed. Detection is when the location and scale of interest points are detected inside the image using each library's implementation of the Fast Hessian detector. A feature is described by estimating its orientation and computing the SURF-64 descriptor. The bottommost plot shows a summary of the descriptors' relative stabilities across a standard set of test images.

One reason for JavaSURF's poor stability performance is that it only implements an upright version of SURF, so images with any rotation cause it to fail. Not computing the orientation also helps JavaSURF on the description runtime benchmark because it has fewer computations to perform. JOpenSURF is a straight forward port of the OpenSURF library to Java and shows comparable stability with the expected hit on runtime performance. JOpenSURF, OpenSURF and BoofCV-M all compute an enhanced version of the SURF descriptor, while the BoofCV descriptor is closer to the SURF paper with some improvements. I suspect that the descriptor computed by the reference library is also an improvement over what was presented in the SURF paper, but source code is closed so this theory cannot be directly verified.

OpenCV is a bit of an odd ball library as far as SURF is concerned. It did not provide an interface which allowed it to be tested in the same manor as the other libraries and comments in the code mention that parts of it are multi-threaded. Every other library tested is single threaded. Because of these issues, its own interest points were used instead of the precomputed ones. Speed wise a special test was done for OpenCV where both features were detected and described at the same time, which took 1940 (ms) for 6485 features. Making it about 20% slower than OpenSURF's combined detect and describe time.

= Descriptor Stability =

<center>
<gallery caption="Images from evaluation data set" heights=150 widths=200 >
Image:Performance-descriptor-Graffiti.jpg|Graffiti
Image:Performance-descriptor-Boat.jpg|Boat
Image:Performance-descriptor-Trees.jpg|Trees
Image:Performance-descriptor-Bricks.jpg|Bricks
</gallery>
</center>

The stability benchmark was performed using standardized test images from [http://www.robots.ox.ac.uk/~vgg/research/affine/], which have known transformations. Stability was measured based upon the number of correct associations between two images in the data set. The testing procedure for each library is summarized below:

# For each image, detect features (scale and location) using the fast Hessian detector in BoofCV.
#* Save results to a file and use the same file for all libraries.
# For each image, compute a feature description (including orientation) for all found features.
# In each image sequence, associate features in the first image to the Nth image, where N > 1.
#* Association is done by minimizing Euclidean error
#* Validation is done using reverse association. E.g. This association must be the optimal association going from frame 1 to N and N to 1.
# Compute the number of correct associations.
#* An association is correct if it is within 3 pixels of the true location.

Since the transformation is known between images the true location could have been used. However, in reality features will not lie at the exact point and a descriptor needs to be tolerant to this type of errors. Thus this is a more accurate measure of the description's strength.

The relative stability metric shown in the conclusions section above is computed by first summing up the total percent of correctly associated features across the whole test data set. Then the library with the best performance is found and the relative stability is computed by dividing each library's score by the best performer's score.

'''Configuration:''' All libraries were configured to describe oriented SURF-64 features as defined in the original SURF paper. JavaSURF does not support orientation estimation. OpenCV forces orientation to be estimated inside the feature detector. Thus it was decided that the lesser evil for OpenCV was to let it detect its own features. OpenCV's threshold was adjusted so that it detected about the same number of features.

== Stability Results ==

{|
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_bike.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_boat.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_graf.gif] || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_leuven.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_ubc.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_trees.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_wall.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_bark.gif
|}

= Runtime Speed =
How fast enough library can compute the description and detect features was also benchmarked. Each test was performed several times with only the best time being shown. Java libraries tended to exhibit more variability than native libraries, while all libraries showed a significant amount of variability from trial to trial.

Only image processing time essential to SURF was measured and not loading in images. This would include converting an image to integral image format, but not converting the image to gray scale. Assuming that it was possible to not include the gray scale conversion. Elapsed time was measured in the actual application using System.currentTimeMillis() in Java and clock() in C++.

Testing Procedure:
# Kill all extraneous processes.
# Load feature location and size from file.
# Compute descriptors (including orientation) for each feature while recording elapsed time.
# Compute elapsed time 10 times and output best result.
# Run the whole experiment 4 times for each library and record the best time.

Test Computer:
* Ubuntu 10.10 64bit
* Quadcore Q6600 2.4 GHz
* Memory 8194 GB
* g++ 4.4.5
* Java(TM) SE Runtime Environment (build 1.6.0_26-b03)

Compiler and JRE Configuration
* All native libraries were compiled with -O3
* Java applications were run with no special flags

Describe Specific Setup:
* input image was boat/img1
* Fast Hessian features from BoofCV
** 6415 Total

Detect Specific Setup:
* Impossible to configure libraries to detect exact same features.
** Adjusted detection threshold to top out at around 2000 features
* Octaves: 4
* Scales: 4
* Base Size: 9
* Initial Pixel Skip: 1

Results can be found at the top of the page. OpenCV was ommited from runtime results because it could not be configured the same way as the other libraries. A special test was performed just for OpenCV and is discussed above. It should also be noted that it is not known what pixel skip was used inside of OpenCV.

Performance:SURF

2011-11-01T16:26:33Z

Spam: /* Comparison of SURF implementations */

= Comparison of SURF implementations =

The [http://en.wikipedia.org/wiki/SURF SURF] descriptor is a state-of-the-art image region descriptor that is invariant with regard to scale, orientation, and illumination. By using an integral image, the descriptor can be computed efficiently across different scales. In recent years it has emerged as one of the more popular and frequently-used feature descriptors, but it is not a trivial algorithm to implement, and several different implementations exist. The following study compares several different libraries to determine relative stability and run-time performance.

Tested Implementations:

{| cellpadding="5" cellspacing="0" border="1"
! Implementation
! Version
! Language
! Threaded
! Comment
|-
| BoofCV || 10/2011 || Java || || Fast but less accurate. See FactoryDescribeRegionPoint.surf()
|-
| BoofCV-M || 10/2011 || Java || || Accurate but slower. See FactoryDescribeRegionPoint.msurf()
|-
| OpenSURF || 27/05/2010 || C++ || || http://www.chrisevansdev.com/computer-vision-opensurf.html
|-
| Reference || 1.0.9 || C++ || || http://www.vision.ee.ethz.ch/~surf/
|-
| JOpenSURF || SVN r24 || Java || || http://code.google.com/p/jopensurf/
|-
| JavaSURF || SVN r4 || Java || || http://code.google.com/p/javasurf/
|-
| OpenCV || 2.3.1 SVN r6879 || C++ || Yes || http://opencv.willowgarage.com/wiki/
|}

Benchmark Source Code:
* [https://github.com/lessthanoptimal/BoofCV/tree/master/evaluation/benchmark BoofCV benchmark]
* [https://github.com/lessthanoptimal/BoofCV/tree/master/evaluation/misc External libraries]

= Conclusions =

{|
| http://www.boofcv.org/notwiki/images/benchmark_surf/overall_describe_speed.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/overall_detect_speed.gif
|-
! Lower is better. !! Lower is better.
|-
|colspan="2"| <center>http://www.boofcv.org/notwiki/images/benchmark_surf/overall_stability.gif </center>
|-
!colspan="2"| Higher is better.
|}

For sake of those with no attention span, the summary results are posted first and a discussion of testing methodology follows below. The top two plots show feature detection and feature describe speed. Detection is when the location and scale of interest points are detected inside the image using each library's implementation of the Fast Hessian detector. A feature is described by estimating its orientation and computing the SURF-64 descriptor. The bottom most plot shows a summary of the descriptors relative stability across a standard set of test images.

One reason for JavaSURF's poor stability performance is that is only implements an upright version of SURF, so images with any rotation cause it to fail. Not computing the orientation also helps JavaSURF on the description runtime benchmark because it has fewer computations to perform. JOpenSURF is a straight forward port of the OpenSURF library to Java and shows comparable stability with the expected hit on runtime performance. JOpenSURF, OpenSURF and BoofCV-M all compute an enhanced version of the SURF descriptor, while the BoofCV descriptor is closer to the SURF paper with some improvements. I suspect that the descriptor computed by the reference library is also an improvement over what was presented in the SURF paper, but source code is closed so this theory cannot be directly verified.

OpenCV is a bit of an odd ball library as far as SURF is concerned. It did not provide an interface which allowed it to be tested in the same manor as the other libraries and comments in the code mention that parts of it are multi-threaded. Every other library tested is single threaded. Because of these issues, its own interest points were used instead of the precomputed ones. Speed wise a special test was done for OpenCV where both features were detected and described at the same time, which took 1940 (ms) for 6485 features. Making it about 20% slower than OpenSURF's combined detect and describe time.

= Descriptor Stability =

<center>
<gallery caption="Images from evaluation data set" heights=150 widths=200 >
Image:Performance-descriptor-Graffiti.jpg|Graffiti
Image:Performance-descriptor-Boat.jpg|Boat
Image:Performance-descriptor-Trees.jpg|Trees
Image:Performance-descriptor-Bricks.jpg|Bricks
</gallery>
</center>

The stability benchmark was performed using standardized test images from [http://www.robots.ox.ac.uk/~vgg/research/affine/], which have known transformations. Stability was measured based upon the number of correct associations between two images in the data set. The testing procedure for each library is summarized below:

# For each image, detect features (scale and location) using the fast Hessian detector in BoofCV.
#* Save results to a file and use the same file for all libraries.
# For each image, compute a feature description (including orientation) for all found features.
# In each image sequence, associate features in the first image to the Nth image, where N > 1.
#* Association is done by minimizing Euclidean error
#* Validation is done using reverse association. E.g. This association must be the optimal association going from frame 1 to N and N to 1.
# Compute the number of correct associations.
#* An association is correct if it is within 3 pixels of the true location.

Since the transformation is known between images the true location could have been used. However, in reality features will not lie at the exact point and a descriptor needs to be tolerant to this type of errors. Thus this is a more accurate measure of the description's strength.

The relative stability metric shown in the conclusions section above is computed by first summing up the total percent of correctly associated features across the whole test data set. Then the library with the best performance is found and the relative stability is computed by dividing each library's score by the best performer's score.

'''Configuration:''' All libraries were configured to describe oriented SURF-64 features as defined in the original SURF paper. JavaSURF does not support orientation estimation. OpenCV forces orientation to be estimated inside the feature detector. Thus it was decided that the lesser evil for OpenCV was to let it detect its own features. OpenCV's threshold was adjusted so that it detected about the same number of features.

== Stability Results ==

{|
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_bike.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_boat.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_graf.gif] || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_leuven.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_ubc.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_trees.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_wall.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_bark.gif
|}

= Runtime Speed =
How fast enough library can compute the description and detect features was also benchmarked. Each test was performed several times with only the best time being shown. Java libraries tended to exhibit more variability than native libraries, while all libraries showed a significant amount of variability from trial to trial.

Only image processing time essential to SURF was measured and not loading in images. This would include converting an image to integral image format, but not converting the image to gray scale. Assuming that it was possible to not include the gray scale conversion. Elapsed time was measured in the actual application using System.currentTimeMillis() in Java and clock() in C++.

Testing Procedure:
# Kill all extraneous processes.
# Load feature location and size from file.
# Compute descriptors (including orientation) for each feature while recording elapsed time.
# Compute elapsed time 10 times and output best result.
# Run the whole experiment 4 times for each library and record the best time.

Test Computer:
* Ubuntu 10.10 64bit
* Quadcore Q6600 2.4 GHz
* Memory 8194 GB
* g++ 4.4.5
* Java(TM) SE Runtime Environment (build 1.6.0_26-b03)

Compiler and JRE Configuration
* All native libraries were compiled with -O3
* Java applications were run with no special flags

Describe Specific Setup:
* input image was boat/img1
* Fast Hessian features from BoofCV
** 6415 Total

Detect Specific Setup:
* Impossible to configure libraries to detect exact same features.
** Adjusted detection threshold to top out at around 2000 features
* Octaves: 4
* Scales: 4
* Base Size: 9
* Initial Pixel Skip: 1

Results can be found at the top of the page. OpenCV was ommited from runtime results because it could not be configured the same way as the other libraries. A special test was performed just for OpenCV and is discussed above. It should also be noted that it is not known what pixel skip was used inside of OpenCV.

Performance:SURF

2011-11-01T16:19:46Z

Spam: /* Comparison of SURF implementations */

= Comparison of SURF implementations =

The [http://en.wikipedia.org/wiki/SURF SURF] descriptor is a state-of-the-art image region descriptor that is invariant with regard to scale, orientation, and illumination. By using an integral image, it can be computed efficiently across different scales. In recent years it has emerged as one of the more popular and frequently-used feature descriptors, but it is not a trivial algorithm to implement, and several different implementations exist. The following study compares several different libraries to determine relative stability and run-time performance.

Tested Implementations:

{| cellpadding="5" cellspacing="0" border="1"
! Implementation
! Version
! Language
! Threaded
! Comment
|-
| BoofCV || 10/2011 || Java || || Fast but less accurate. See FactoryDescribeRegionPoint.surf()
|-
| BoofCV-M || 10/2011 || Java || || Accurate but slower. See FactoryDescribeRegionPoint.msurf()
|-
| OpenSURF || 27/05/2010 || C++ || || http://www.chrisevansdev.com/computer-vision-opensurf.html
|-
| Reference || 1.0.9 || C++ || || http://www.vision.ee.ethz.ch/~surf/
|-
| JOpenSURF || SVN r24 || Java || || http://code.google.com/p/jopensurf/
|-
| JavaSURF || SVN r4 || Java || || http://code.google.com/p/javasurf/
|-
| OpenCV || 2.3.1 SVN r6879 || C++ || Yes || http://opencv.willowgarage.com/wiki/
|}

Benchmark Source Code:
* [https://github.com/lessthanoptimal/BoofCV/tree/master/evaluation/benchmark BoofCV benchmark]
* [https://github.com/lessthanoptimal/BoofCV/tree/master/evaluation/misc External libraries]

= Conclusions =

{|
| http://www.boofcv.org/notwiki/images/benchmark_surf/overall_describe_speed.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/overall_detect_speed.gif
|-
! Lower is better. !! Lower is better.
|-
|colspan="2"| <center>http://www.boofcv.org/notwiki/images/benchmark_surf/overall_stability.gif </center>
|-
!colspan="2"| Higher is better.
|}

For sake of those with no attention span, the summary results are posted first and a discussion of testing methodology follows below. The top two plots show feature detection and feature describe speed. Detection is when the location and scale of interest points are detected inside the image using each library's implementation of the Fast Hessian detector. A feature is described by estimating its orientation and computing the SURF-64 descriptor. The bottom most plot shows a summary of the descriptors relative stability across a standard set of test images.

One reason for JavaSURF's poor stability performance is that is only implements an upright version of SURF, so images with any rotation cause it to fail. Not computing the orientation also helps JavaSURF on the description runtime benchmark because it has fewer computations to perform. JOpenSURF is a straight forward port of the OpenSURF library to Java and shows comparable stability with the expected hit on runtime performance. JOpenSURF, OpenSURF and BoofCV-M all compute an enhanced version of the SURF descriptor, while the BoofCV descriptor is closer to the SURF paper with some improvements. I suspect that the descriptor computed by the reference library is also an improvement over what was presented in the SURF paper, but source code is closed so this theory cannot be directly verified.

OpenCV is a bit of an odd ball library as far as SURF is concerned. It did not provide an interface which allowed it to be tested in the same manor as the other libraries and comments in the code mention that parts of it are multi-threaded. Every other library tested is single threaded. Because of these issues, its own interest points were used instead of the precomputed ones. Speed wise a special test was done for OpenCV where both features were detected and described at the same time, which took 1940 (ms) for 6485 features. Making it about 20% slower than OpenSURF's combined detect and describe time.

= Descriptor Stability =

<center>
<gallery caption="Images from evaluation data set" heights=150 widths=200 >
Image:Performance-descriptor-Graffiti.jpg|Graffiti
Image:Performance-descriptor-Boat.jpg|Boat
Image:Performance-descriptor-Trees.jpg|Trees
Image:Performance-descriptor-Bricks.jpg|Bricks
</gallery>
</center>

The stability benchmark was performed using standardized test images from [http://www.robots.ox.ac.uk/~vgg/research/affine/], which have known transformations. Stability was measured based upon the number of correct associations between two images in the data set. The testing procedure for each library is summarized below:

# For each image, detect features (scale and location) using the fast Hessian detector in BoofCV.
#* Save results to a file and use the same file for all libraries.
# For each image, compute a feature description (including orientation) for all found features.
# In each image sequence, associate features in the first image to the Nth image, where N > 1.
#* Association is done by minimizing Euclidean error
#* Validation is done using reverse association. E.g. This association must be the optimal association going from frame 1 to N and N to 1.
# Compute the number of correct associations.
#* An association is correct if it is within 3 pixels of the true location.

Since the transformation is known between images the true location could have been used. However, in reality features will not lie at the exact point and a descriptor needs to be tolerant to this type of errors. Thus this is a more accurate measure of the description's strength.

The relative stability metric shown in the conclusions section above is computed by first summing up the total percent of correctly associated features across the whole test data set. Then the library with the best performance is found and the relative stability is computed by dividing each library's score by the best performer's score.

'''Configuration:''' All libraries were configured to describe oriented SURF-64 features as defined in the original SURF paper. JavaSURF does not support orientation estimation. OpenCV forces orientation to be estimated inside the feature detector. Thus it was decided that the lesser evil for OpenCV was to let it detect its own features. OpenCV's threshold was adjusted so that it detected about the same number of features.

== Stability Results ==

{|
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_bike.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_boat.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_graf.gif] || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_leuven.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_ubc.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_trees.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_wall.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_bark.gif
|}

= Runtime Speed =
How fast enough library can compute the description and detect features was also benchmarked. Each test was performed several times with only the best time being shown. Java libraries tended to exhibit more variability than native libraries, while all libraries showed a significant amount of variability from trial to trial.

Only image processing time essential to SURF was measured and not loading in images. This would include converting an image to integral image format, but not converting the image to gray scale. Assuming that it was possible to not include the gray scale conversion. Elapsed time was measured in the actual application using System.currentTimeMillis() in Java and clock() in C++.

Testing Procedure:
# Kill all extraneous processes.
# Load feature location and size from file.
# Compute descriptors (including orientation) for each feature while recording elapsed time.
# Compute elapsed time 10 times and output best result.
# Run the whole experiment 4 times for each library and record the best time.

Test Computer:
* Ubuntu 10.10 64bit
* Quadcore Q6600 2.4 GHz
* Memory 8194 GB
* g++ 4.4.5
* Java(TM) SE Runtime Environment (build 1.6.0_26-b03)

Compiler and JRE Configuration
* All native libraries were compiled with -O3
* Java applications were run with no special flags

Describe Specific Setup:
* input image was boat/img1
* Fast Hessian features from BoofCV
** 6415 Total

Detect Specific Setup:
* Impossible to configure libraries to detect exact same features.
** Adjusted detection threshold to top out at around 2000 features
* Octaves: 4
* Scales: 4
* Base Size: 9
* Initial Pixel Skip: 1

Results can be found at the top of the page. OpenCV was ommited from runtime results because it could not be configured the same way as the other libraries. A special test was performed just for OpenCV and is discussed above. It should also be noted that it is not known what pixel skip was used inside of OpenCV.

Performance:SURF

2011-11-01T16:15:16Z

Spam:

= Comparison of SURF implementations =

The [http://en.wikipedia.org/wiki/SURF SURF] descriptor is a state-of-the-art image region descriptor that is scale-, orientation-, and illumination-invariant. By using an integral image, it can be computed efficiently across different scales. In recent years it has emerged as one of the more popular and frequently-used feature descriptors, but it is not a trivial algorithm to implement, and several different implementations exist. The following study compares several different libraries to determine relative stability and run-time performance.

Tested Implementations:

{| cellpadding="5" cellspacing="0" border="1"
! Implementation
! Version
! Language
! Threaded
! Comment
|-
| BoofCV || 10/2011 || Java || || Fast but less accurate. See FactoryDescribeRegionPoint.surf()
|-
| BoofCV-M || 10/2011 || Java || || Accurate but slower. See FactoryDescribeRegionPoint.msurf()
|-
| OpenSURF || 27/05/2010 || C++ || || http://www.chrisevansdev.com/computer-vision-opensurf.html
|-
| Reference || 1.0.9 || C++ || || http://www.vision.ee.ethz.ch/~surf/
|-
| JOpenSURF || SVN r24 || Java || || http://code.google.com/p/jopensurf/
|-
| JavaSURF || SVN r4 || Java || || http://code.google.com/p/javasurf/
|-
| OpenCV || 2.3.1 SVN r6879 || C++ || Yes || http://opencv.willowgarage.com/wiki/
|}

Benchmark Source Code:
* [https://github.com/lessthanoptimal/BoofCV/tree/master/evaluation/benchmark BoofCV benchmark]
* [https://github.com/lessthanoptimal/BoofCV/tree/master/evaluation/misc External libraries]

= Conclusions =

{|
| http://www.boofcv.org/notwiki/images/benchmark_surf/overall_describe_speed.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/overall_detect_speed.gif
|-
! Lower is better. !! Lower is better.
|-
|colspan="2"| <center>http://www.boofcv.org/notwiki/images/benchmark_surf/overall_stability.gif </center>
|-
!colspan="2"| Higher is better.
|}

For sake of those with no attention span, the summary results are posted first and a discussion of testing methodology follows below. The top two plots show feature detection and feature describe speed. Detection is when the location and scale of interest points are detected inside the image using each library's implementation of the Fast Hessian detector. A feature is described by estimating its orientation and computing the SURF-64 descriptor. The bottom most plot shows a summary of the descriptors relative stability across a standard set of test images.

One reason for JavaSURF's poor stability performance is that is only implements an upright version of SURF, so images with any rotation cause it to fail. Not computing the orientation also helps JavaSURF on the description runtime benchmark because it has fewer computations to perform. JOpenSURF is a straight forward port of the OpenSURF library to Java and shows comparable stability with the expected hit on runtime performance. JOpenSURF, OpenSURF and BoofCV-M all compute an enhanced version of the SURF descriptor, while the BoofCV descriptor is closer to the SURF paper with some improvements. I suspect that the descriptor computed by the reference library is also an improvement over what was presented in the SURF paper, but source code is closed so this theory cannot be directly verified.

OpenCV is a bit of an odd ball library as far as SURF is concerned. It did not provide an interface which allowed it to be tested in the same manor as the other libraries and comments in the code mention that parts of it are multi-threaded. Every other library tested is single threaded. Because of these issues, its own interest points were used instead of the precomputed ones. Speed wise a special test was done for OpenCV where both features were detected and described at the same time, which took 1940 (ms) for 6485 features. Making it about 20% slower than OpenSURF's combined detect and describe time.

= Descriptor Stability =

<center>
<gallery caption="Images from evaluation data set" heights=150 widths=200 >
Image:Performance-descriptor-Graffiti.jpg|Graffiti
Image:Performance-descriptor-Boat.jpg|Boat
Image:Performance-descriptor-Trees.jpg|Trees
Image:Performance-descriptor-Bricks.jpg|Bricks
</gallery>
</center>

The stability benchmark was performed using standardized test images from [http://www.robots.ox.ac.uk/~vgg/research/affine/], which have known transformations. Stability was measured based upon the number of correct associations between two images in the data set. The testing procedure for each library is summarized below:

# For each image, detect features (scale and location) using the fast Hessian detector in BoofCV.
#* Save results to a file and use the same file for all libraries.
# For each image, compute a feature description (including orientation) for all found features.
# In each image sequence, associate features in the first image to the Nth image, where N > 1.
#* Association is done by minimizing Euclidean error
#* Validation is done using reverse association. E.g. This association must be the optimal association going from frame 1 to N and N to 1.
# Compute the number of correct associations.
#* An association is correct if it is within 3 pixels of the true location.

Since the transformation is known between images the true location could have been used. However, in reality features will not lie at the exact point and a descriptor needs to be tolerant to this type of errors. Thus this is a more accurate measure of the description's strength.

The relative stability metric shown in the conclusions section above is computed by first summing up the total percent of correctly associated features across the whole test data set. Then the library with the best performance is found and the relative stability is computed by dividing each library's score by the best performer's score.

'''Configuration:''' All libraries were configured to describe oriented SURF-64 features as defined in the original SURF paper. JavaSURF does not support orientation estimation. OpenCV forces orientation to be estimated inside the feature detector. Thus it was decided that the lesser evil for OpenCV was to let it detect its own features. OpenCV's threshold was adjusted so that it detected about the same number of features.

== Stability Results ==

{|
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_bike.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_boat.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_graf.gif] || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_leuven.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_ubc.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_trees.gif
|-
| http://www.boofcv.org/notwiki/images/benchmark_surf/stability_wall.gif || http://www.boofcv.org/notwiki/images/benchmark_surf/stability_bark.gif
|}

= Runtime Speed =
How fast enough library can compute the description and detect features was also benchmarked. Each test was performed several times with only the best time being shown. Java libraries tended to exhibit more variability than native libraries, while all libraries showed a significant amount of variability from trial to trial.

Only image processing time essential to SURF was measured and not loading in images. This would include converting an image to integral image format, but not converting the image to gray scale. Assuming that it was possible to not include the gray scale conversion. Elapsed time was measured in the actual application using System.currentTimeMillis() in Java and clock() in C++.

Testing Procedure:
# Kill all extraneous processes.
# Load feature location and size from file.
# Compute descriptors (including orientation) for each feature while recording elapsed time.
# Compute elapsed time 10 times and output best result.
# Run the whole experiment 4 times for each library and record the best time.

Test Computer:
* Ubuntu 10.10 64bit
* Quadcore Q6600 2.4 GHz
* Memory 8194 GB
* g++ 4.4.5
* Java(TM) SE Runtime Environment (build 1.6.0_26-b03)

Compiler and JRE Configuration
* All native libraries were compiled with -O3
* Java applications were run with no special flags

Describe Specific Setup:
* input image was boat/img1
* Fast Hessian features from BoofCV
** 6415 Total

Detect Specific Setup:
* Impossible to configure libraries to detect exact same features.
** Adjusted detection threshold to top out at around 2000 features
* Octaves: 4
* Scales: 4
* Base Size: 9
* Initial Pixel Skip: 1

Results can be found at the top of the page. OpenCV was ommited from runtime results because it could not be configured the same way as the other libraries. A special test was performed just for OpenCV and is discussed above. It should also be noted that it is not known what pixel skip was used inside of OpenCV.

Manual

2011-10-06T02:56:04Z

Spam:

Coding Standards

2011-10-06T02:45:22Z

Spam: Created page with "== Coding Standards == One of the primary reasons Java was selected as a programming language for this library are the set of development tools available for Java. Less restric..."

== Coding Standards ==

One of the primary reasons Java was selected as a programming language for this library are the set of development tools available for Java. Less restrictive and structured languages such as C/C++ essentially allow the developer to redefine much of the language on the fly. While at first this sounds like a positive characteristic, it has hindered the ability to create development tools which allow [http://en.wikipedia.org/wiki/Code_refactoring refactoring] and other modern techniques. BoofCV's API has been designed to take full advantage of integrated development environments (IDE) such as [http://www.jetbrains.com/idea/ IntelliJ] and [http://www.eclipse.org/ Eclipse].

Unfortunately Java does not provide templates. For example, if two classes are identical in every respect except that they use different primitive data structures internally in Java you need to create to classes. In languages such as C++ you can create one template class. The later is easier to maintain and requires less code. To get around this issue auto code generation is extensively used throughout the library.

A common pitfall which developers fall into is over abstraction. This leads to code which is difficult to use or develop because of its complexity and layers of abstraction. Computer vision has many complex algorithms and competing ideologies on how to catalog these algorithms together. To mitigate this issue the code in BoofCV has been divided into algorithms and abstractions. Code inside of algorithms has minimal abstraction and is implemented in whichever way makes the most sense to that particular algorithm. Abstraction containers generalized interfaced and wrappers around the algorithms so that they fit that particular formalism. This also greatly reduces unnecessary dependencies between the code allowing pieces of BoofCV to be cut off and used independently.

PROCEDURAL VS OOP.

<h3>Packages</h3>

Each project module (e.g. image processing) has a standardized package structure.

{| style="width: 640px;" border="1"
! Java Package
! Description
|-
| boofcv.abst
| Contains abstracted interfaces or wrappers for algorithms in the same family or to enforce different formalisms. Sometimes to conform to these algorithms performance or unique capabilities are lost but added flexibility is gained by making it easier to switch algorithms.
|-
| boofcv.alg
| Bare algorithms which minimal abstraction. Using algorithms directly allows for greater performance and or for a developer to wrap inside their own interface/frame work.
|-
| boofcv.factory
| Contains factories for creating raw algorithms or abstracted implementations of algorithms easily. Using a factory makes it easier to get started but the simplicity comes at the cost of flexibility. Flexibility is lost because factories hard code some parameters.
|}
<h3>Directory Structure</h3>

Each project module also has a standardized directory structure.

{| style="width: 640px;" border="1"
! Directory
! Description
|-
| src/
| Contains library source code.
|-
| test/
| Contains unit tests for source code.
|-
| generate/
| Source code for auto-generating source code in "src/". The package of the generator corresponds to the package the output should be placed inside.
|-
| benchmark/
| Contains simple codes for testing runtime performance of different algorithms. Used during development to ensure efficiency.
|}

<h3>Class Name Standards</h3>

The following standard prefixes and suffices are used throughout the code base.

{| style="width: 640px;" border="1"
! name
! Description
|-
| Impl* || Is a low level (often auto-generated image type specific) implementation of an algorithm. Directly accessing these class is rarely required and discouraged.
|-
| Wrap* || Wrapper around an algorithm for a specific interface. These classes are found inside the boofcv.abst package and its children.
|-
| Factory* || Factory for creating abstracted algorithms.
|-
| Factory*Alg || Factory for creating a raw algorithms.
|-
| *_F32, *_F64 || Implementation for 32-bit/64-bit floating point images or data structures, respectively.
|-
| *_S32 || Implementation for signed 32-bit integer images or data structures.
|-
| *_U16, *_S16 || Implementation for unsigned/signed 16-bit integer images or data structures, respectively.
|-
| *_U8, *_S8 || Implementation for unsigned/signed 8-bit integer images or data structures, respectively.
|}

Manual

2011-10-06T02:44:21Z

Spam:

= Development with BoofCV =

The following manual provides an introduction to development with BoofCV. It is assumed that the reader is familiar with development in the Java programming language and the basics of computer vision. This manual primarily takes the form of tutorials. However, before going through the tutorials one should be familiar with coding standards use in BoofCV. Once these are understood one will be able to efficiently search the library or even guess the names of classes which are needed.

== Coding Standards ==

One of the primary reasons Java was selected as a programming language for this library are the set of development tools available for Java. Less restrictive and structured languages such as C/C++ essentially allow the developer to redefine much of the language on the fly. While at first this sounds like a positive characteristic, it has hindered the ability to create development tools which allow [http://en.wikipedia.org/wiki/Code_refactoring refactoring] and other modern techniques. BoofCV's API has been designed to take full advantage of integrated development environments (IDE) such as [http://www.jetbrains.com/idea/ IntelliJ] and [http://www.eclipse.org/ Eclipse].

Unfortunately Java does not provide templates. For example, if two classes are identical in every respect except that they use different primitive data structures internally in Java you need to create to classes. In languages such as C++ you can create one template class. The later is easier to maintain and requires less code. To get around this issue auto code generation is extensively used throughout the library.

A common pitfall which developers fall into is over abstraction. This leads to code which is difficult to use or develop because of its complexity and layers of abstraction. Computer vision has many complex algorithms and competing ideologies on how to catalog these algorithms together. To mitigate this issue the code in BoofCV has been divided into algorithms and abstractions. Code inside of algorithms has minimal abstraction and is implemented in whichever way makes the most sense to that particular algorithm. Abstraction containers generalized interfaced and wrappers around the algorithms so that they fit that particular formalism. This also greatly reduces unnecessary dependencies between the code allowing pieces of BoofCV to be cut off and used independently.

PROCEDURAL VS OOP.

<h3>Packages</h3>

Each project module (e.g. image processing) has a standardized package structure.

{| style="width: 640px;" border="1"
! Java Package
! Description
|-
| boofcv.abst
| Contains abstracted interfaces or wrappers for algorithms in the same family or to enforce different formalisms. Sometimes to conform to these algorithms performance or unique capabilities are lost but added flexibility is gained by making it easier to switch algorithms.
|-
| boofcv.alg
| Bare algorithms which minimal abstraction. Using algorithms directly allows for greater performance and or for a developer to wrap inside their own interface/frame work.
|-
| boofcv.factory
| Contains factories for creating raw algorithms or abstracted implementations of algorithms easily. Using a factory makes it easier to get started but the simplicity comes at the cost of flexibility. Flexibility is lost because factories hard code some parameters.
|}
<h3>Directory Structure</h3>

Each project module also has a standardized directory structure.

{| style="width: 640px;" border="1"
! Directory
! Description
|-
| src/
| Contains library source code.
|-
| test/
| Contains unit tests for source code.
|-
| generate/
| Source code for auto-generating source code in "src/". The package of the generator corresponds to the package the output should be placed inside.
|-
| benchmark/
| Contains simple codes for testing runtime performance of different algorithms. Used during development to ensure efficiency.
|}

<h3>Class Name Standards</h3>

The following standard prefixes and suffices are used throughout the code base.

{| style="width: 640px;" border="1"
! name
! Description
|-
| Impl* || Is a low level (often auto-generated image type specific) implementation of an algorithm. Directly accessing these class is rarely required and discouraged.
|-
| Wrap* || Wrapper around an algorithm for a specific interface. These classes are found inside the boofcv.abst package and its children.
|-
| Factory* || Factory for creating abstracted algorithms.
|-
| Factory*Alg || Factory for creating a raw algorithms.
|-
| *_F32, *_F64 || Implementation for 32-bit/64-bit floating point images or data structures, respectively.
|-
| *_S32 || Implementation for signed 32-bit integer images or data structures.
|-
| *_U16, *_S16 || Implementation for unsigned/signed 16-bit integer images or data structures, respectively.
|-
| *_U8, *_S8 || Implementation for unsigned/signed 8-bit integer images or data structures, respectively.
|}

= List of Tutorial =

= Building BoofCV =

The easiest way to build the library from source is using the provided ant scripts.

# Download source code.
# Run the ant script inside the boofcv/main directory.
# Copy the BoofCV.jar from boofcv/lib.

In Linux this looks something like this:
<pre>
$ cd boofcv/main/
$ ant
---- lots of text -----
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 15 warnings

jar:
[jar] Building jar: /home/pja/projects/boofcv/main/jar/BoofCV_IO.jar

main:
[jar] Building jar: /home/pja/projects/boofcv/lib/BoofCV.jar

main:

BUILD SUCCESSFUL
Total time: 7 seconds
</pre>

Main Page

2011-09-23T21:48:07Z

Spam:

__NOTOC__

{| style="width:600pt;"
| style="background-color:#ffffdd;" align="center" |
= BoofCV =
''' A Computer Vision Library with Style'''
|-
|
BoofCV is a Java library for real-time computer vision and robotics applications. Written from scratch for ease of use and high performance, it provides both basic and advanced features needed for creating a modern computer vision system. Functionality include highly optimized low level image processing routines (e.g. convolution, interpolation, gradient) to high level functionality such as image stabilization.

It is divided up into four package, 1) image processing, 2) features, 3) geometric vision, and 4) input output. Image processing contains core functionality which are most commonly used and typically operate directly on pixels. Features contains feature (e.g. points, lines, and regions) extraction algorithms for use in higher level operations. Geometric vision contains routines for processing extracted image features using 2D and 3D geometry. IO stands for input/output and contains common routines for reading in and displaying compute images.

BoofCV is still in an early development phase, but is still highly capable. To demonstrate this capability several example applets have been provided which can be run in your own browser, provided you have
version 1.6 or later of Java installed.
|}

{| style="width:600pt;"
| colspan=2 style="background-color:#ffffdd;" align="center" |
== Table of Contents ==
|-
| Introduction
| asdfaf
|-
| Examples
| asfasdf
|-
| JavaDOC
| asdfasdf
|-
| Applets
| List of example applets [[List of Applets]]
|}

Main Page

2011-09-23T21:39:18Z

Spam: /* BoofCV */

Main Page

2011-09-14T02:17:47Z

Spam:

'''MediaWiki has been successfully installed.'''

Consult the [http://meta.wikimedia.org/wiki/Help:Contents User's Guide] for information on using the wiki software.

== Getting started ==
* [http://www.mediawiki.org/wiki/Manual:Configuration_settings Configuration settings list]
* [http://www.mediawiki.org/wiki/Manual:FAQ MediaWiki FAQ]
* [https://lists.wikimedia.org/mailman/listinfo/mediawiki-announce MediaWiki release mailing list]