Performance:QrCode
Abstract
QR Codes (Quick Response Code) are fiducial markers used to embed digital information visually on 2D surfaces. QR Codes are commonly used in consumer and industrial applications was a way to embed information, such as website addresses or part numbers. A QR Code reader or scanner can visually detect a QR code and decode the message inside. There are numerous free open source and commercial proprietary software libraries available. This study compares all known open source libraries in their ability to correctly detect QR codes and efficiently process them. Results are presented where performance is broken down based on different categories. While there is no library which dominates every category, on average BoofCV's detector is the fastest and most reliable detector.
To cite this study please use the following:
@misc{QRBenchmark,
title = {Study of QR Code Scanning Performance in Different Situations},
howpublished = {\url{https://boofcv.org/index.php?title=Performance:QrCode}},
author = {Peter Abeles},
originalyear = {07.10.2018}
}
Introduction
Originally designed in 1994 by the Japanese company Denso Wave for the automotive industry QR Codes are widely used in consumer and industrial applications today. Each digital bit is encoded as a square in a 2D array. Different parts of the marker are used to locate the QR code, aid in decoding, and error correction. The amount of information which can be stored is depends on the version and configuration. The QR Code marker is fully specified in the ISO/IEC 18004 document. For more about the history see wikipedia.
This benchmark evaluates the performance of different open source QR Code scanners under different real world environmental conditions. Only open source libraries are considered because they are free and can be quickly integrated into your hardware. One thing clear from this study is that the behavior of scanners is difficult to succinctly summarize.
Each library's ability to detect and decode a QR code varies widely depending on the environmental conditions. It's not uncommon for a library to get near 100% in one category then fail completely in another. How fast each library runs high dependent not only on image size but also on the scenes structure and the number of QR codes visible.
All test images and evaluation source code is freely available.
Categories
Images where broken up into several categories, each of which would stress a detector in a different way. Representative images for each category are shown below. Categories were selected based on known failure conditions and common use cases.
- Blurred
- Images which are blurred due to focus or motion
- Brightness
- Same scene but with the exposure manually adjusted from under to over exposed
- Bright Spots
- Abrupt changes in intensity can throw off adaptive thresholding techniques
- Close
- Images taken so close to the image that black regions are no longer uniformly black
- Curved
- QR Codes on curved or uneven surfaces. Specification says it needs to be on a flat surface.
- Damaged
- Printing errors, scratches, pens, ... etc
- Glare
- Glare obscuring or degrading part of the QR Code
- Monitor
- Pictures taken of QR Codes on a monitor. If close the pixels are visible.
- Nominal
- Intended to be representative of normal use cases. All of these should be readable
- Non-Compliant
- Specification was intentionally broken. Often for artistic reasons
- Pathological
- Intentionally corrupted markers designed to break detectors
- Rotated
- Same image but rotated 360 degrees
- Shadows
- Non uniform lighting caused by shadows
Blurred
Brightness
Bright Spots
Close
Curved
damaged
Glare
Lots
Monitor
Nominal
Non-Compliant
Pathological
Rotations
Shadows
Evaluation Procedure
For all QR Codes in every image in the test set four corners were selected and saved. These four corners define the bounding box and are used to decide of a library correctly detected a QR Code or not. If a detected marker was found to overlap labeled markers in the image by a minimum of 10% then it was considered a match. 10% might seem like a low threshold but not all libraries return precise location information. The exact threshold for determining a match was found to not be important.
Detection performance was evaluated using F-Measure. To compute F-Measure we need to precision and recall. To find precision and recall we need to count the number of true positives, false positives, true negatives, and false negatives. Detecting the same QR code more than once was also considered but none of the libraries were found to do that.
- True Positives
- When a labeled marker is matched to a detected marker
- False Positive
- When no labeled marker is matched to a detected marker
- True Negative
- Always 100% in this scenario.
- False Negative
- When a labeled marker is not matched to any detected marker
Summary detection results were found lumping all results into a single bin. Categories will more images/markers will have greater weight. For this reason you should inspect the category results and use knowledge of your application to decide which is the best match.
As mentioned runtime performance (how fast a library can process an image) is dependent on scene structure, number of qr codes, and image size. The results presented here focus on scene structure dependency. A more fine grained analysis related to image size and number of visible markers might be added in the future. Processing time for an image was measured using the system clock and measured actual time as compared to CPU time. Some argue that CPU time is more accurate but would be very difficult to measure here because we did not want to measure how long each application took to decode the input JPEG image just processing time.
Detection Results
Runtime Results
Conclusions
