Performance:QrCode - Revision history

Peter: /* Categories */

2019-09-17T14:38:49Z

Peter at 00:32, 20 March 2019

2019-03-20T00:32:57Z

← Older revision		Revision as of 00:32, 20 March 2019
Line 6:		Line 6:

	Evaluated Libraries:		Evaluated Libraries:
	* [http://boofcv.org BoofCV]		* [http://boofcv.org BoofCV] (Ver. 0.33)
	* [http://opencv.org OpenCV]		* [http://opencv.org OpenCV] (Ver. 4.0.1)
	* [https://github.com/dlbeer/quirc Quirc]		* [https://github.com/dlbeer/quirc Quirc] (Feb 1, 2018. SHA 307473db)
	* [http://zbar.sourceforge.net/ ZBar]		* [http://zbar.sourceforge.net/ ZBar] (Ver. 0.10)
	* [https://github.com/zxing/zxing ZXing]		* [https://github.com/zxing/zxing ZXing] (Ver. 3.3.3)

	To cite this study please use the following:		To cite this study please use the following:
Line 99:		Line 99:
	'''BoofCV'''		'''BoofCV'''

	Returns four precise corners (sub-pixel accuracy) for each position pattern and an approximate location of each alignment pattern. A bounding box is also returned. Since there is no feature at the fourth corner it is found by intersecting lines from finder patterns. ~~Version 0.33~~		Returns four precise corners (sub-pixel accuracy) for each position pattern and an approximate location of each alignment pattern. A bounding box is also returned. Since there is no feature at the fourth corner it is found by intersecting lines from finder patterns.

	'''OpenCV'''		'''OpenCV'''

	Can only detect a single QR Code in an image. Returns a single bounding box at sub-pixel accuracy. New since OpenCV 4		Can only detect a single QR Code in an image. Returns a single bounding box at sub-pixel accuracy. New since OpenCV 4.

	'''Quirc'''		'''Quirc'''

	Returns bounding box as represented by four corners. It's a bit unusual in that it doesn't fully process the image but delays some processing until a request is made to decode a specific marker. ~~Date: Feb 1, 2018. SHA 307473db~~		Returns bounding box as represented by four corners. It's a bit unusual in that it doesn't fully process the image but delays some processing until a request is made to decode a specific marker.

	'''ZBar'''		'''ZBar'''

	Because ZBar is designed to detect multiple types of fiducials the source code can be a bit vague. By inspection it was determined that it returned four corners bounding a QR code. ~~Versio:n 0.10~~		Because ZBar is designed to detect multiple types of fiducials the source code can be a bit vague. By inspection it was determined that it returned four corners bounding a QR code.

	'''ZXing'''		'''ZXing'''
Line 125:		Line 125:
	</syntaxhighlight>		</syntaxhighlight>

	ZXing was intentionally designed for speed over 100% detection rate (citation needed). The idea being that when processing a video stream eventually one image will be readable. ~~Version 3.3.3~~		ZXing was intentionally designed for speed over 100% detection rate (citation needed). The idea being that when processing a video stream eventually one image will be readable.

	== Evaluation Procedure ==		== Evaluation Procedure ==
Line 191:		Line 191:
	To give you an idea of the size of this test set and the diversity of images in each category, where are the number of images in each category.		To give you an idea of the size of this test set and the diversity of images in each category, where are the number of images in each category.

	[[file:Qrcode_category_counts.png\|center\|Number of Images in Each Category]]		[[file:Qrcode_category_counts.png\|600px\|center\|Number of Images in Each Category]]

	== Conclusions ==		== Conclusions ==
Line 202:		Line 202:

	* Added two new categories		* Added two new categories
	* High Version and Perspective		** High Version and Perspective
	* A few additional images to blur and others		* A few additional images to blur and others
	* Updated BoofCV to 0.33 and ZXing to 3.3.3		* Updated BoofCV to 0.33 and ZXing to 3.3.3

Peter at 00:28, 20 March 2019

2019-03-20T00:28:12Z

← Older revision		Revision as of 00:28, 20 March 2019
Line 3:		Line 3:
	QR Codes (Quick Response Code) are fiducial markers used to embed digital information visually on 2D surfaces. QR Codes are commonly used in consumer and industrial applications was a way to embed information, such as website addresses or part numbers. A QR Code reader or scanner can visually detect a QR code and decode the message inside. There are numerous free open source and commercial proprietary software libraries available. This study compares all known open source libraries in their ability to correctly detect QR codes and efficiently process them. Results are presented where performance is broken down based on different categories. While there is no library which dominates every category, on average BoofCV's detector is the fastest and most reliable detector.		QR Codes (Quick Response Code) are fiducial markers used to embed digital information visually on 2D surfaces. QR Codes are commonly used in consumer and industrial applications was a way to embed information, such as website addresses or part numbers. A QR Code reader or scanner can visually detect a QR code and decode the message inside. There are numerous free open source and commercial proprietary software libraries available. This study compares all known open source libraries in their ability to correctly detect QR codes and efficiently process them. Results are presented where performance is broken down based on different categories. While there is no library which dominates every category, on average BoofCV's detector is the fastest and most reliable detector.

	[[#Changes\|Updated: ~~JANUARY 16~~, 2019]]		[[#Changes\|Updated: March 19, 2019]]

	Evaluated Libraries:		Evaluated Libraries:
Line 15:		Line 15:
	<pre>		<pre>
	@misc{QRBenchmark,		@misc{QRBenchmark,
	title = {Study of QR Code Scanning Performance in Different Environments},		title = {Study of QR Code Scanning Performance in Different Environments. V3},
	howpublished = {\url{https://boofcv.org/index.php?title=Performance:QrCode}},		howpublished = {\url{https://boofcv.org/index.php?title=Performance:QrCode}},
	author = {Peter Abeles},		author = {Peter Abeles},
	originalyear = {01.16.2019}		originalyear = {03.19.2019}
	}		}
	</pre>		</pre>
Line 99:		Line 99:
	'''BoofCV'''		'''BoofCV'''

	Returns four precise corners (sub-pixel accuracy) for each position pattern and an approximate location of each alignment pattern. A bounding box is also returned. Since there is no feature at the fourth corner it is found by intersecting lines from finder patterns.		Returns four precise corners (sub-pixel accuracy) for each position pattern and an approximate location of each alignment pattern. A bounding box is also returned. Since there is no feature at the fourth corner it is found by intersecting lines from finder patterns. Version 0.33

	'''OpenCV'''		'''OpenCV'''
Line 107:		Line 107:
	'''Quirc'''		'''Quirc'''

	Returns bounding box as represented by four corners. It's a bit unusual in that it doesn't fully process the image but delays some processing until a request is made to decode a specific marker.		Returns bounding box as represented by four corners. It's a bit unusual in that it doesn't fully process the image but delays some processing until a request is made to decode a specific marker. Date: Feb 1, 2018. SHA 307473db

	'''ZBar'''		'''ZBar'''

	Because ZBar is designed to detect multiple types of fiducials the source code can be a bit vague. By inspection it was determined that it returned four corners bounding a QR code.		Because ZBar is designed to detect multiple types of fiducials the source code can be a bit vague. By inspection it was determined that it returned four corners bounding a QR code. Versio:n 0.10

	'''ZXing'''		'''ZXing'''
Line 125:		Line 125:
	</syntaxhighlight>		</syntaxhighlight>

	ZXing was intentionally designed for speed over 100% detection rate (citation needed). The idea being that when processing a video stream eventually one image will be readable.		ZXing was intentionally designed for speed over 100% detection rate (citation needed). The idea being that when processing a video stream eventually one image will be readable. Version 3.3.3

	== Evaluation Procedure ==		== Evaluation Procedure ==
Line 198:		Line 198:

	== Changes ==		== Changes ==

			=== March 19, 2019 ===

			* Added two new categories
			* High Version and Perspective
			* A few additional images to blur and others
			* Updated BoofCV to 0.33 and ZXing to 3.3.3

	=== January 16, 2019 ===		=== January 16, 2019 ===
Line 203:		Line 210:
	* Added OpenCV 4's new QR Code detector		* Added OpenCV 4's new QR Code detector
	* Using latest 33-SNAPSHOT of BoofCV (subject to change)		* Using latest 33-SNAPSHOT of BoofCV (subject to change)

	=== October 2018 ===		=== October 2018 ===
	* Initial Publication		* Initial Publication

Peter: /* Categories */

2019-03-20T00:23:19Z

Peter: /* Categories */

2019-03-20T00:18:15Z

Peter: /* Detection Results */

2019-01-17T03:44:24Z

Detection Results

← Older revision		Revision as of 03:44, 17 January 2019
Line 165:		Line 165:

	No library dominates all categories. BoofCV is the best or a close second best in every categories, except in the non-compliant and pathological categories. It requires stricter adherence to the specification. ZBar tends to do better than ZXing in most categories but loses out in the summary plot due to the categories which ZXing does best in. While Quirc does well in most common situations it performs poorly in categories which all other libraries do well in. OpenCV's scanner needs a lot of work in general when it comes to detection. OpenCV being limited to detecting one marker in an image didn't help.		No library dominates all categories. BoofCV is the best or a close second best in every categories, except in the non-compliant and pathological categories. It requires stricter adherence to the specification. ZBar tends to do better than ZXing in most categories but loses out in the summary plot due to the categories which ZXing does best in. While Quirc does well in most common situations it performs poorly in categories which all other libraries do well in. OpenCV's scanner needs a lot of work in general when it comes to detection. OpenCV being limited to detecting one marker in an image didn't help.

			<div style="text-align: center;">'''Higher is Better'''</div>


	[[file:Qrcode_detection_categories.png\|800px\|center\|Detection Performance by Categories]]		[[file:Qrcode_detection_categories.png\|800px\|center\|Detection Performance by Categories]]

Peter at 03:31, 17 January 2019

2019-01-17T03:31:44Z

← Older revision		Revision as of 03:31, 17 January 2019
Line 159:		Line 159:
	It was not obvious always what should be measured and what should not be measured. For example, after the image has been loaded should the conversion from color to gray be included? OpenCV would load an image as gray even if it was color so it would be unfair to penalize libraries which didn't use OpenCV. Most libraries did all the heavy processing inside their process()/scan() functions. However some libraries delayed processing until the QR code message/location was requested. In that case the timing would be extended around the qr code message accessors.		It was not obvious always what should be measured and what should not be measured. For example, after the image has been loaded should the conversion from color to gray be included? OpenCV would load an image as gray even if it was color so it would be unfair to penalize libraries which didn't use OpenCV. Most libraries did all the heavy processing inside their process()/scan() functions. However some libraries delayed processing until the QR code message/location was requested. In that case the timing would be extended around the qr code message accessors.

	Relative runtime is shown for each category. This was done because there are no standard image sizes so some categories might tend to have larger or smaller images confusing the results. The summary performance was found by averaging the median runtime across all categories, where each category had equal weight. Median performance was used here in an attempt to limit outliers skewing results. A few library had large spikes in processing time under specific conditions.		Relative runtime is shown for each category. This was done because there are no standard image sizes and some categories might tend to have larger or smaller images confusing the results. The summary performance was found by averaging the median runtime across all categories, where each category had equal weight. Median performance was used here in an attempt to limit outliers skewing results. A few library had large spikes in processing time under specific conditions.

	~~== Category Counts ==~~

	~~The number of images in each category is shown as a histogram below.~~

	~~[[file:Qrcode_category_counts.png\|center\|Number of Images in Each Category]]~~

	== Detection Results ==		== Detection Results ==
Line 182:		Line 177:
	[[file:Qrcode_runtime_category_relative_mean.png\|800px\|center\|Relative Runtime by Categories]]		[[file:Qrcode_runtime_category_relative_mean.png\|800px\|center\|Relative Runtime by Categories]]
	[[file:Qrcode_runtime_summary.png\|400px\|center\|Runtime Performance Summary]]		[[file:Qrcode_runtime_summary.png\|400px\|center\|Runtime Performance Summary]]

			== Category Counts ==

			To give you an idea of the size of this test set and the diversity of images in each category, where are the number of images in each category.

			[[file:Qrcode_category_counts.png\|center\|Number of Images in Each Category]]

	== Conclusions ==		== Conclusions ==

Peter: /* Evaluation Procedure */

2019-01-17T03:29:38Z

Evaluation Procedure

← Older revision		Revision as of 03:29, 17 January 2019
Line 157:		Line 157:
	</syntaxhighlight>		</syntaxhighlight>

	It was not obvious always what should be measured and what should not be measured. For example, after the image has been loaded should the conversion from color to gray be included? OpenCV would load an image as gray even if it was color so it would be unfair to penalize libraries which didn't use OpenCV. Most libraries did all the heavy processing inside their process()/scan() functions. However some libraries delayed processing until the QR code was ~~quested~~. In that case the timing would be extended around the qr code message accessors.		It was not obvious always what should be measured and what should not be measured. For example, after the image has been loaded should the conversion from color to gray be included? OpenCV would load an image as gray even if it was color so it would be unfair to penalize libraries which didn't use OpenCV. Most libraries did all the heavy processing inside their process()/scan() functions. However some libraries delayed processing until the QR code message/location was requested. In that case the timing would be extended around the qr code message accessors.

	Relative runtime is shown for each category. This was done because there are no standard image sizes so some categories might tend to have larger or smaller images confusing the results. The summary performance was found by averaging the median runtime across all categories, where each category had equal weight. Median performance was used here in an attempt to limit outliers skewing results. A few library had large spikes in processing time under specific conditions.		Relative runtime is shown for each category. This was done because there are no standard image sizes so some categories might tend to have larger or smaller images confusing the results. The summary performance was found by averaging the median runtime across all categories, where each category had equal weight. Median performance was used here in an attempt to limit outliers skewing results. A few library had large spikes in processing time under specific conditions.

Peter: /* Evaluated Libraries */

2019-01-17T03:28:14Z

Evaluated Libraries

← Older revision		Revision as of 03:28, 17 January 2019
Line 94:		Line 94:

	Returns four precise corners (sub-pixel accuracy) for each position pattern and an approximate location of each alignment pattern. A bounding box is also returned. Since there is no feature at the fourth corner it is found by intersecting lines from finder patterns.		Returns four precise corners (sub-pixel accuracy) for each position pattern and an approximate location of each alignment pattern. A bounding box is also returned. Since there is no feature at the fourth corner it is found by intersecting lines from finder patterns.

			'''OpenCV'''

			Can only detect a single QR Code in an image. Returns a single bounding box at sub-pixel accuracy. New since OpenCV 4

	'''Quirc'''		'''Quirc'''
Line 116:		Line 120:

	ZXing was intentionally designed for speed over 100% detection rate (citation needed). The idea being that when processing a video stream eventually one image will be readable.		ZXing was intentionally designed for speed over 100% detection rate (citation needed). The idea being that when processing a video stream eventually one image will be readable.


	== Evaluation Procedure ==		== Evaluation Procedure ==

Peter: /* Introduction */

2019-01-17T03:26:23Z

Introduction

← Older revision		Revision as of 03:26, 17 January 2019
Line 29:		Line 29:
	[[File:QR_Code_Structure.png\|400px\|center\|Structure of a QR Code. From Wikipedia]]		[[File:QR_Code_Structure.png\|400px\|center\|Structure of a QR Code. From Wikipedia]]

	This benchmark evaluates the performance of different open source QR Code scanners under different real world environmental conditions. Only open source libraries are considered ~~because~~ they are ~~free and can be quickly integrated into your hardware~~. One thing clear from this study is that the behavior of scanners is difficult to succinctly summarize.		This benchmark evaluates the performance of different open source QR Code scanners under different real world environmental conditions. Only open source libraries are considered since they are easily accessible. One thing clear from this study is that the behavior of scanners is difficult to succinctly summarize.

	Each library's ability to detect and decode a QR code varies widely depending on the environmental conditions. It's not uncommon for a library to get near 100% in one category then fail completely in another. How fast each library runs high dependent not only on image size but also on the scenes structure and the number of QR codes visible.		Each library's ability to detect and decode a QR code varies widely depending on the environmental conditions. It's not uncommon for a library to get near 100% in one category then fail completely in another. How fast each library runs high dependent not only on image size but also on the scenes structure and the number of QR codes visible.

	All test images and evaluation source code is freely ~~available3~~		All test images and evaluation source code is freely available
	* [https://github.com/lessthanoptimal/ValidationBoof/tree/master/thirdparty/qrcode Benchmark Code v0.33-SNAPSHOT]		* [https://github.com/lessthanoptimal/ValidationBoof/tree/master/thirdparty/qrcode Benchmark Code v0.33-SNAPSHOT]
	* [https://boofcv.org/notwiki/regression/fiducial/qrcodes_v2.zip QR Images V2]		* [https://boofcv.org/notwiki/regression/fiducial/qrcodes_v2.zip QR Images V2]


	== Categories ==		== Categories ==

← Older revision		Revision as of 14:38, 17 September 2019
Line 40:		Line 40:

	Images where broken up into several categories, each of which would stress a detector in a different way. Representative images for each category are shown below. Categories were selected based on known failure conditions and common use cases.		Images where broken up into several categories, each of which would stress a detector in a different way. Representative images for each category are shown below. Categories were selected based on known failure conditions and common use cases.

			<center>[https://boofcv.org/notwiki/regression/fiducial/qrcodes_v3.zip Image Dataset]</center>

	; Blurred		; Blurred

← Older revision		Revision as of 00:23, 20 March 2019
Line 55:		Line 55:
	; Glare		; Glare
	: Glare obscuring or degrading part of the QR Code		: Glare obscuring or degrading part of the QR Code
			; High Version
			: Unusually high version numbers, e.g. 25 and 40
	; Monitor		; Monitor
	: Pictures taken of QR Codes on a monitor. If close the pixels are visible.		: Pictures taken of QR Codes on a monitor. If close the pixels are visible.
Line 79:		Line 81:
	File:qrcode_damaged.jpg\| damaged		File:qrcode_damaged.jpg\| damaged
	File:qrcode_glare.jpg\| Glare		File:qrcode_glare.jpg\| Glare
			File:Qrcode_highversion.jpg\| High Version
	File:qrcode_lots.jpg\| Lots		File:qrcode_lots.jpg\| Lots
	File:qrcode_monitor.jpg\| Monitor		File:qrcode_monitor.jpg\| Monitor

← Older revision		Revision as of 00:18, 20 March 2019
Line 63:		Line 63:
	; Pathological		; Pathological
	: Intentionally corrupted markers designed to break detectors		: Intentionally corrupted markers designed to break detectors
			; Perspective
			: Marker viewed at increasingly skewed angles
	; Rotated		; Rotated
	: Same image but rotated 360 degrees		: Same image but rotated 360 degrees
Line 82:		Line 84:
	File:qrcode_noncompliant.jpg\| Non-Compliant		File:qrcode_noncompliant.jpg\| Non-Compliant
	File:qrcode_pathological.png\| Pathological		File:qrcode_pathological.png\| Pathological
			File:Qrcode_perspective.jpg\| Perspective
	File:qrcode_rotations.jpg\| Rotations		File:qrcode_rotations.jpg\| Rotations
	File:qrcode_shadows.jpg\| Shadows		File:qrcode_shadows.jpg\| Shadows