Difference between revisions of "Android support"

From BoofCV
Jump to navigationJump to search
m
m
(8 intermediate revisions by the same user not shown)
Line 1: Line 1:
BoofCV works without modification on Android systems, however to build in image types are not directly compatible with BoofCV.  To get around this issue BoofCV provides code in its integration package for converting to and from several Android image types.  Support is currently provided for converting Bitmap and NV21 images. These functions are many times faster than trying to convert images using Bitmap's RGB values or through some more convoluted process like NV21->jpeg->Bitmap->BoofCV.  Functions have been tested on Android 2.3.3 API and will most likely work on even earlier versions.
BoofCV's Android library provides support for converting between different image types and managing your android activity. If you use BoofCV to manager your activity it will select the camera, open the camera, create a thread pool, synchronizes data structures, handle the android life cycle (open/close the camera properly, stop threads), convert the YUV420 image, correctly align input pixels to screen pixels, and change camera settings upon request.


* [http://boofcv.org/javadoc/integration/boofcv/android/ConvertBitmap.html Convert Bitmap]
= Dependencies =
* [http://boofcv.org/javadoc/integration/boofcv/android/ConvertNV21.html Convert NV21]


To use these additional functions you must either use the precompiled BoofCVAndroid.jar or compile it yourself from source code.  Source code for Android integration can be found in the 'boofcv/integration/android' directory.  An ant script is provided for compiling the jar file, but needs to be modified so that it points towards your Android API installation.
To use BoofCV in your project all you need to do is add the following line to dependencies in app/build.gradle


= Having Trouble in Android Studio? =
<syntaxhighlight lang="java">
api group: 'org.boofcv', name: 'boofcv-android', version: '0.31'
</syntaxhighlight>


Here are a few common issues with using BoofCV in Android.
<syntaxhighlight lang="java">
 
// conflicts with Android dependencies
* Don't compile code that's byte compatible with JDK 8
* Remove conflicting libraries.
* Look at this working [https://github.com/lessthanoptimal/BoofCV/tree/SNAPSHOT/integration/android/examples/video example].
 
Make sure your 'app/build.gradle' file has the following in it:
 
<pre>
// Remove libraries that conflict with libraries already included with Android
configurations {
configurations {
     all*.exclude group: "xmlpull", module: "xmlpull"
     all*.exclude group: "xmlpull", module: "xmlpull"
     all*.exclude group: "org.apache.commons", module: "commons-compress"
     all*.exclude group: "org.apache.commons", module: "commons-compress"
    all*.exclude group: "com.thoughtworks.xstream", module: "commons-compress"
}
}
</syntaxhighlight>
= API Summary =
Support for Camera2 API was added May of 2018. The Camera 1 API is still available but not currently being updated. Static functions are provided for converting different image formats. If you wish to have BoofCV manage the Android activity you need to extend SimpleCamera2Activity or VisualizeCamera2Activity. The former will
only provide you access to control of the camera and access to its images. The latter handles converting
image data and assists in visualization. To use these activities you need to grant permission to the camera in your AndroidManifest.xml:
<syntaxhighlight lang="xml">
<uses-permission android:name="android.permission.CAMERA" />
<uses-feature android:name="android.hardware.camera2.full" />
</syntaxhighlight>
== SimpleCamera2Activity ==
To start the camera invoke ''startCamera} inside your Activity's onCreate() function.
<ul>
    <li>''selectResolution''
    <li>''onCameraResolutionChange''
    <li>''configureCamera''
    <li>''selectCamera''
    <li>''processFrame''
    <li>''onCameraOpened''
    <li>''onCameraDisconnected''
    <li>''changeCameraConfiguratio''
</ul>
To turn on and off verbose printing to Log you can toggle the SimpleCamera2Activity.verbose variable.
=== Selecting Camera / Resolution ===
By default BoofCV will select the back facing camera and the resolution which post closely matches the texture's width. NOTE: This behavior is changed if you extend VisualizeCamera2Activity. You can modify
this behavior by overriding one of the functions list above. For example, to change the resolution selection
behavior you will override ''selectResolution''.


dependencies {
=== Camera Settings ===
    ['android', 'core'].each { String a -> compile group: 'org.boofcv', name: a, version: 'ENTER YOUR BOOFCV VERSION HERE' }
To customize the camera settings you need to override ''configureCamera''. If after opening the camera
    ... other dependencies ...
you want to change the camera settings you need to then first call ''isCameraReadyReconfiguration''
}
and make sure the camera is in a state that it can be reconfigured. If that returns truen then you're
</pre>
free to call ''changeCameraConfiguration'' which will eventually result in ''configureCamera''
being called again.
 
== VisualizeCamera2Activity ==
 
Extension of ''SimpleCamera2Activity'' which adds visualization and hooks for image
processing. Video frames are automatically converted into a format which can be processed by
BoofCV routines Optionally multiple threads can be generated so that video frames are
processed concurrently. The input image is automatically converted into a Bitmap image if
requested. If multiple threads are being used the user can toggle if they want visualization
to be shown if an old image finished being processed after a newer one.
 
Inside of onCreate() you need to call ''setImageType'' so that it knows what type of image to convert camera's YUV420_888 into. ImageType.single(GrayU8.class) for gray scale and ImageType.il(Gray8.class,3) for interleaved RGB. After that is done you will call ''startCamera(ViewGroup, TextureView)'' The textureview is optional and only needed if you want to display the camera preview directly.
 
<ul>
    <li><b>imageToView</b>: Matrix that converts a pixel in video frame to view frame</li>
    <li><b>displayView</b>: Where visualizations are rendered in.</li>
    <li><b>targetResolution</b>: Specifies how many pixels you want in the video frame</li>
    <li><b>stretchToFill</b>: true to stretch the video frame to fill the entire view</li>
    <li><b>bitmapMode</b>: Species how and if the bitmap should be drawn to the screen</li>
    <li><b>visualizeOnlyMostRecent</b>: If more than one thread is enabled should it show old results</li>
</ul>
 
=== Processing Images ===
 
You must implement ''processImage()'' and typecast the image it provides into the format you specified earlier
 
=== Visualization ===
 
Override ''onDrawFrame()'' and invoke the super.
 
== Converting Image Types ==


* [http://boofcv.org/javadoc/integration/boofcv/android/ConvertBitmap.html Convert Bitmap]
* [http://boofcv.org/javadoc/integration/boofcv/android/ConvertCameraImage.html Convert YUV_420_888]
* [http://boofcv.org/javadoc/integration/boofcv/android/ConvertNV21.html Convert NV21]


= Usage Examples =
= Usage Examples =
Line 34: Line 96:
A more detailed and working example for processing video images in Android can be found at the link below.  Short code snippets are included in this section.
A more detailed and working example for processing video images in Android can be found at the link below.  Short code snippets are included in this section.


Full Examples:
== Full Working Examples ==
* [[Example_Android_Video|Android Video]]
* [[Example_Android_Gradient|Visualize Gradient]]
* [https://github.com/lessthanoptimal/BoofCV/blob/SNAPSHOT/integration/boofcv-android/examples/video/app/src/main/java/org/boofcv/video/ExposureActivity.java Exposure Control]
* [https://github.com/lessthanoptimal/BoofCV/blob/SNAPSHOT/integration/boofcv-android/examples/video/app/src/main/java/org/boofcv/video/QrCodeActivity.java QR Code Detect and Visualize]


== Converting Bitmap images ==
== Converting Bitmap images ==
<syntaxhighlight lang="java">
<syntaxhighlight lang="java">
    // Easiest way to convert a Bitmap into a BoofCV type
// Easiest way to convert a Bitmap into a BoofCV type
    GrayU8 image = ConvertBitmap.bitmapToGray(bitmap, (GrayU8)null, null);
GrayU8 image = ConvertBitmap.bitmapToGray(bitmap, (GrayU8)null, null);
    // If you are converting a sequence of images it is faster reuse a
// If you are converting a sequence of images it is faster reuse a
    // previously declare image and buffer
// previously declare image and buffer
    byte[] workBuffer = ConvertBitmap.declareStorage(bitmap, null);
byte[] workBuffer = ConvertBitmap.declareStorage(bitmap, null);
    ConvertBitmap.bitmapToGray(bitmap, image, workBuffer);
ConvertBitmap.bitmapToGray(bitmap, image, workBuffer);
    
    
    // Convert back into a Bitmap
// Convert back into a Bitmap
    ConvertBitmap.grayToBitmap(image, bitmap, workBuffer);
ConvertBitmap.grayToBitmap(image, bitmap, workBuffer);


    // another less efficient way
// another less efficient way
    bitmap = ConvertBitmap.grayToBitmap(image, Bitmap.Config.ARGB_8888);
bitmap = ConvertBitmap.grayToBitmap(image, Bitmap.Config.ARGB_8888);
    
    
    // Functions are also provided for multi-spectral images
// Functions are also provided for multi-spectral images
    Planar<GrayF32> color = ConvertBitmap.bitmapToMS(bitmap, null, GrayF32.class, null);
Planar<GrayF32> color = ConvertBitmap.bitmapToMS(bitmap, null, GrayF32.class, null);
</syntaxhighlight>
 
== Converting YUV420 Images ==
 
<syntaxhighlight lang="java">
Image image; // From camera in YUV 420 format
byte work[] = ConvertCameraImage.declareWork(image,null);
ConvertCameraImage.imageToBoof(image, ColorFormat.RGB, grayU8, work);
</syntaxhighlight>
</syntaxhighlight>


== Converting NV21 Images ==
== Converting NV21 Images ==
Data from camera previews is made available in the NV21 image format.
Data from camera previews in Camera 1 API is made available in the NV21 image format.


<syntaxhighlight lang="java">
<syntaxhighlight lang="java">
    // from NV21 to gray scale
// from NV21 to gray scale
    ConvertNV21.nv21ToGray(bytes,width,height,gray)
ConvertNV21.nv21ToGray(bytes,width,height,gray)


  // from NV21 to YUV Multi-Spectral
// from NV21 to YUV Planar
    ConvertNV21.nv21ToMsYuv(bytes,width,height,colorMS,GrayU8.class);
ConvertNV21.nv21ToPlanarYuv(bytes,width,height,colorPL,GrayU8.class);
</syntaxhighlight>
</syntaxhighlight>


Besides converting images, VisualizeImageData is provided for visualizing  
Besides converting images, VisualizeImageData is provided for visualizing


= Android Apps =  
= Android Apps =  
Line 74: Line 146:
Android demonstration and benchmark applications are available for download from the google play store.
Android demonstration and benchmark applications are available for download from the google play store.
* [https://play.google.com/store/apps/details?id=org.boofcv.android Demonstration]
* [https://play.google.com/store/apps/details?id=org.boofcv.android Demonstration]
* [https://play.google.com/store/apps/details?id=boofcv.benchmark.android Benchmark]


The demonstration visualizes several computer vision algorithms and provides a good estimate on what sort of runtime performance can be expected when using BoofCV on Android.  The benchmark application is primarily a diagnostic app and if you submit results helps the BoofCV developers understand how will it performs on different devices.  Additional information on each of these apps can be accessed using the links below.  Full source is available for both applications.
The demonstration visualizes several computer vision algorithms and provides a good estimate on what sort of runtime performance can be expected when using BoofCV on Android.


* [http://peterabeles.com/blog/?p=204 Demonstration Instructions]
* [http://peterabeles.com/blog/?p=204 Demonstration Instructions]
* [[Android_benchmark_app|Benchmark Wiki]]


= Problems? =
= Trouble Shooting =


* If you try to compile BoofCV in an Android you will get errors due to the swing code.  The solution to this problem is to not compile it inside the Android app and instead just link to its jars.
Does your code run much slower than the demo? Try switching the build to release. Click on "Build Variants" on the bottom left, then in the panel that pops up select "release" from the build variant.
* Compiler can't find ConvertBitmap?  Make sure BoofCVAndroid.jar is included in your project.

Revision as of 07:37, 14 November 2018

BoofCV's Android library provides support for converting between different image types and managing your android activity. If you use BoofCV to manager your activity it will select the camera, open the camera, create a thread pool, synchronizes data structures, handle the android life cycle (open/close the camera properly, stop threads), convert the YUV420 image, correctly align input pixels to screen pixels, and change camera settings upon request.

Dependencies

To use BoofCV in your project all you need to do is add the following line to dependencies in app/build.gradle

api group: 'org.boofcv', name: 'boofcv-android', version: '0.31'
// conflicts with Android dependencies
configurations {
    all*.exclude group: "xmlpull", module: "xmlpull"
    all*.exclude group: "org.apache.commons", module: "commons-compress"
    all*.exclude group: "com.thoughtworks.xstream", module: "commons-compress"
}


API Summary

Support for Camera2 API was added May of 2018. The Camera 1 API is still available but not currently being updated. Static functions are provided for converting different image formats. If you wish to have BoofCV manage the Android activity you need to extend SimpleCamera2Activity or VisualizeCamera2Activity. The former will only provide you access to control of the camera and access to its images. The latter handles converting image data and assists in visualization. To use these activities you need to grant permission to the camera in your AndroidManifest.xml:

 <uses-permission android:name="android.permission.CAMERA" />
 <uses-feature android:name="android.hardware.camera2.full" />

SimpleCamera2Activity

To start the camera invoke startCamera} inside your Activity's onCreate() function.

  • selectResolution
  • onCameraResolutionChange
  • configureCamera
  • selectCamera
  • processFrame
  • onCameraOpened
  • onCameraDisconnected
  • changeCameraConfiguratio

To turn on and off verbose printing to Log you can toggle the SimpleCamera2Activity.verbose variable.

Selecting Camera / Resolution

By default BoofCV will select the back facing camera and the resolution which post closely matches the texture's width. NOTE: This behavior is changed if you extend VisualizeCamera2Activity. You can modify this behavior by overriding one of the functions list above. For example, to change the resolution selection behavior you will override selectResolution.

Camera Settings

To customize the camera settings you need to override configureCamera. If after opening the camera you want to change the camera settings you need to then first call isCameraReadyReconfiguration and make sure the camera is in a state that it can be reconfigured. If that returns truen then you're free to call changeCameraConfiguration which will eventually result in configureCamera being called again.

VisualizeCamera2Activity

Extension of SimpleCamera2Activity which adds visualization and hooks for image processing. Video frames are automatically converted into a format which can be processed by BoofCV routines Optionally multiple threads can be generated so that video frames are processed concurrently. The input image is automatically converted into a Bitmap image if requested. If multiple threads are being used the user can toggle if they want visualization to be shown if an old image finished being processed after a newer one.

Inside of onCreate() you need to call setImageType so that it knows what type of image to convert camera's YUV420_888 into. ImageType.single(GrayU8.class) for gray scale and ImageType.il(Gray8.class,3) for interleaved RGB. After that is done you will call startCamera(ViewGroup, TextureView) The textureview is optional and only needed if you want to display the camera preview directly.

  • imageToView: Matrix that converts a pixel in video frame to view frame
  • displayView: Where visualizations are rendered in.
  • targetResolution: Specifies how many pixels you want in the video frame
  • stretchToFill: true to stretch the video frame to fill the entire view
  • bitmapMode: Species how and if the bitmap should be drawn to the screen
  • visualizeOnlyMostRecent: If more than one thread is enabled should it show old results

Processing Images

You must implement processImage() and typecast the image it provides into the format you specified earlier

Visualization

Override onDrawFrame() and invoke the super.

Converting Image Types

Usage Examples

A more detailed and working example for processing video images in Android can be found at the link below. Short code snippets are included in this section.

Full Working Examples

Converting Bitmap images

// Easiest way to convert a Bitmap into a BoofCV type
GrayU8 image = ConvertBitmap.bitmapToGray(bitmap, (GrayU8)null, null);
	
// If you are converting a sequence of images it is faster reuse a
// previously declare image and buffer
byte[] workBuffer = ConvertBitmap.declareStorage(bitmap, null);
ConvertBitmap.bitmapToGray(bitmap, image, workBuffer);
    	
// Convert back into a Bitmap
ConvertBitmap.grayToBitmap(image, bitmap, workBuffer);

// another less efficient way
bitmap = ConvertBitmap.grayToBitmap(image, Bitmap.Config.ARGB_8888);
    	
// Functions are also provided for multi-spectral images
Planar<GrayF32> color = ConvertBitmap.bitmapToMS(bitmap, null, GrayF32.class, null);

Converting YUV420 Images

Image image; // From camera in YUV 420 format
byte work[] = ConvertCameraImage.declareWork(image,null);
ConvertCameraImage.imageToBoof(image, ColorFormat.RGB, grayU8, work);

Converting NV21 Images

Data from camera previews in Camera 1 API is made available in the NV21 image format.

// from NV21 to gray scale
ConvertNV21.nv21ToGray(bytes,width,height,gray)

// from NV21 to YUV Planar
ConvertNV21.nv21ToPlanarYuv(bytes,width,height,colorPL,GrayU8.class);

Besides converting images, VisualizeImageData is provided for visualizing

Android Apps

Android demonstration and benchmark applications are available for download from the google play store.

The demonstration visualizes several computer vision algorithms and provides a good estimate on what sort of runtime performance can be expected when using BoofCV on Android.

Trouble Shooting

Does your code run much slower than the demo? Try switching the build to release. Click on "Build Variants" on the bottom left, then in the panel that pops up select "release" from the build variant.