Difference between revisions of "Android support"

From BoofCV
Jump to navigationJump to search
(Clarified several points about android support)
m
(19 intermediate revisions by the same user not shown)
Line 1: Line 1:
= Android Support =
BoofCV's Android library provides support for converting between different image types and managing your android activity. If you use BoofCV to manager your activity it will select the camera, open the camera, create a thread pool, synchronizes data structures, handle the android life cycle (open/close the camera properly, stop threads), convert the YUV420 image, correctly align input pixels to screen pixels, and change camera settings upon request.


'''BoofCV and required jars will all run on Android as is without any modifications.'''  However, since Android does not support swing be sure not to call any GUI functions provided in the library or else it will crash.  All the core functions in BoofCV are compatible with Android.  It is not necessary to remove the classes from the jars which invoke swing, just don't call them.
= Dependencies =


The one key piece of functionality missing for running on Android is a function that converts Android Bitmaps into BoofCV images.  [[User:Maarten_Van_Lier|Maarten]] has provided unoptimized source code for converting bitmaps into BoofCV images at his thesis website [http://dl.dropbox.com/u/16413201/BoofCV%20v0.2%20Partial%20Android%20Port.zip].
To use BoofCV in your project all you need to do is add the following line to dependencies in app/build.gradle


Better and more complete Android support will be added in the future. Contributions of welcome.
<syntaxhighlight lang="java">
api group: 'org.boofcv', name: 'boofcv-android', version: '0.31'
</syntaxhighlight>


== Partial Android Port of BoofCV v0.2 ==
<syntaxhighlight lang="java">
// conflicts with Android dependencies
configurations {
    all*.exclude group: "xmlpull", module: "xmlpull"
    all*.exclude group: "org.apache.commons", module: "commons-compress"
    all*.exclude group: "com.thoughtworks.xstream", module: "commons-compress"
}
</syntaxhighlight>


For his [[Projects_Using_BoofCV|master thesis]], [[User:Maarten_Van_Lier|Maarten]] made a partial port of BoofCV v0.2 to Android.  Note that the changes listed below are not neccisary to use BoofCV on Android.  The original jar than you can download from the website will work just fine.  However, additional work is need to compile the source code with out Swing.


In this port, the following was done to get it working:
= API Summary =
* Just use the EJML, GeoRegression and libpja libraries that BoofCV uses, they work on Android as is.
* Modify the core.image.ConvertBufferedImage class to the core.image.ConvertBitmap class. This new class uses BufferedImages and Android API calls to do the same as the ConvertBufferedImage class. The implementation is not as efficient as the original one, though.
* Leave out the gui package because of the java swing windows that were used. It was not needed for the thesis.
* Leave out problematic methods in io.image.UtilIO and remove other classes in io.image, io.video and io.wrapper.images, because they use java swing window calls. They were not needed for the thesis.
* Leave out the methods in testing.BoofTesting that used the BufferedImage class.


Basically, code for loading images was ported, while code to show windows was left out.
Support for Camera2 API was added May of 2018. The Camera 1 API is still available but not currently being updated. Static functions are provided for converting different image formats. If you wish to have BoofCV manage the Android activity you need to extend SimpleCamera2Activity or VisualizeCamera2Activity. The former will
The code is available here[http://dl.dropbox.com/u/16413201/BoofCV%20v0.2%20Partial%20Android%20Port.zip]. Note that this is an incomplete port that may not have what you need, or may not work as expected.
only provide you access to control of the camera and access to its images. The latter handles converting
image data and assists in visualization. To use these activities you need to grant permission to the camera in your AndroidManifest.xml:
 
<syntaxhighlight lang="xml">
<uses-permission android:name="android.permission.CAMERA" />
<uses-feature android:name="android.hardware.camera2.full" />
</syntaxhighlight>
 
== SimpleCamera2Activity ==
 
To start the camera invoke ''startCamera} inside your Activity's onCreate() function.
<ul>
    <li>''selectResolution''
    <li>''onCameraResolutionChange''
    <li>''configureCamera''
    <li>''selectCamera''
    <li>''processFrame''
    <li>''onCameraOpened''
    <li>''onCameraDisconnected''
    <li>''changeCameraConfiguratio''
</ul>
To turn on and off verbose printing to Log you can toggle the SimpleCamera2Activity.verbose variable.
 
=== Selecting Camera / Resolution ===
 
By default BoofCV will select the back facing camera and the resolution which post closely matches the texture's width. NOTE: This behavior is changed if you extend VisualizeCamera2Activity. You can modify
this behavior by overriding one of the functions list above. For example, to change the resolution selection
behavior you will override ''selectResolution''.
 
=== Camera Settings ===
To customize the camera settings you need to override ''configureCamera''. If after opening the camera
you want to change the camera settings you need to then first call ''isCameraReadyReconfiguration''
and make sure the camera is in a state that it can be reconfigured. If that returns truen then you're
free to call ''changeCameraConfiguration'' which will eventually result in ''configureCamera''
being called again.
 
== VisualizeCamera2Activity ==
 
Extension of ''SimpleCamera2Activity'' which adds visualization and hooks for image
processing. Video frames are automatically converted into a format which can be processed by
BoofCV routines Optionally multiple threads can be generated so that video frames are
processed concurrently. The input image is automatically converted into a Bitmap image if
requested. If multiple threads are being used the user can toggle if they want visualization
to be shown if an old image finished being processed after a newer one.
 
Inside of onCreate() you need to call ''setImageType'' so that it knows what type of image to convert camera's YUV420_888 into. ImageType.single(GrayU8.class) for gray scale and ImageType.il(Gray8.class,3) for interleaved RGB. After that is done you will call ''startCamera(ViewGroup, TextureView)'' The textureview is optional and only needed if you want to display the camera preview directly.
 
<ul>
    <li><b>imageToView</b>: Matrix that converts a pixel in video frame to view frame</li>
    <li><b>displayView</b>: Where visualizations are rendered in.</li>
    <li><b>targetResolution</b>: Specifies how many pixels you want in the video frame</li>
    <li><b>stretchToFill</b>: true to stretch the video frame to fill the entire view</li>
    <li><b>bitmapMode</b>: Species how and if the bitmap should be drawn to the screen</li>
    <li><b>visualizeOnlyMostRecent</b>: If more than one thread is enabled should it show old results</li>
</ul>
 
=== Processing Images ===
 
You must implement ''processImage()'' and typecast the image it provides into the format you specified earlier
 
=== Visualization ===
 
Override ''onDrawFrame()'' and invoke the super.
 
== Converting Image Types ==
 
* [http://boofcv.org/javadoc/integration/boofcv/android/ConvertBitmap.html Convert Bitmap]
* [http://boofcv.org/javadoc/integration/boofcv/android/ConvertCameraImage.html Convert YUV_420_888]
* [http://boofcv.org/javadoc/integration/boofcv/android/ConvertNV21.html Convert NV21]
 
= Usage Examples =
 
A more detailed and working example for processing video images in Android can be found at the link below.  Short code snippets are included in this section.
 
== Full Working Examples ==
* [[Example_Android_Gradient|Visualize Gradient]]
* [https://github.com/lessthanoptimal/BoofCV/blob/SNAPSHOT/integration/boofcv-android/examples/video/app/src/main/java/org/boofcv/video/ExposureActivity.java Exposure Control]
* [https://github.com/lessthanoptimal/BoofCV/blob/SNAPSHOT/integration/boofcv-android/examples/video/app/src/main/java/org/boofcv/video/QrCodeActivity.java QR Code Detect and Visualize]
 
== Converting Bitmap images ==
<syntaxhighlight lang="java">
// Easiest way to convert a Bitmap into a BoofCV type
GrayU8 image = ConvertBitmap.bitmapToGray(bitmap, (GrayU8)null, null);
// If you are converting a sequence of images it is faster reuse a
// previously declare image and buffer
byte[] workBuffer = ConvertBitmap.declareStorage(bitmap, null);
ConvertBitmap.bitmapToGray(bitmap, image, workBuffer);
   
// Convert back into a Bitmap
ConvertBitmap.grayToBitmap(image, bitmap, workBuffer);
 
// another less efficient way
bitmap = ConvertBitmap.grayToBitmap(image, Bitmap.Config.ARGB_8888);
   
// Functions are also provided for multi-spectral images
Planar<GrayF32> color = ConvertBitmap.bitmapToMS(bitmap, null, GrayF32.class, null);
</syntaxhighlight>
 
== Converting YUV420 Images ==
 
<syntaxhighlight lang="java">
Image image; // From camera in YUV 420 format
byte work[] = ConvertCameraImage.declareWork(image,null);
ConvertCameraImage.imageToBoof(image, ColorFormat.RGB, grayU8, work);
</syntaxhighlight>
 
== Converting NV21 Images ==
Data from camera previews in Camera 1 API is made available in the NV21 image format.
 
<syntaxhighlight lang="java">
// from NV21 to gray scale
ConvertNV21.nv21ToGray(bytes,width,height,gray)
 
// from NV21 to YUV Planar
ConvertNV21.nv21ToPlanarYuv(bytes,width,height,colorPL,GrayU8.class);
</syntaxhighlight>
 
Besides converting images, VisualizeImageData is provided for visualizing
 
= Android Apps =
 
Android demonstration and benchmark applications are available for download from the google play store.
* [https://play.google.com/store/apps/details?id=org.boofcv.android Demonstration]
 
The demonstration visualizes several computer vision algorithms and provides a good estimate on what sort of runtime performance can be expected when using BoofCV on Android.
 
* [http://peterabeles.com/blog/?p=204 Demonstration Instructions]
 
= Trouble Shooting =
 
Does your code run much slower than the demo? Try switching the build to release. Click on "Build Variants" on the bottom left, then in the panel that pops up select "release" from the build variant.

Revision as of 07:37, 14 November 2018

BoofCV's Android library provides support for converting between different image types and managing your android activity. If you use BoofCV to manager your activity it will select the camera, open the camera, create a thread pool, synchronizes data structures, handle the android life cycle (open/close the camera properly, stop threads), convert the YUV420 image, correctly align input pixels to screen pixels, and change camera settings upon request.

Dependencies

To use BoofCV in your project all you need to do is add the following line to dependencies in app/build.gradle

api group: 'org.boofcv', name: 'boofcv-android', version: '0.31'
// conflicts with Android dependencies
configurations {
    all*.exclude group: "xmlpull", module: "xmlpull"
    all*.exclude group: "org.apache.commons", module: "commons-compress"
    all*.exclude group: "com.thoughtworks.xstream", module: "commons-compress"
}


API Summary

Support for Camera2 API was added May of 2018. The Camera 1 API is still available but not currently being updated. Static functions are provided for converting different image formats. If you wish to have BoofCV manage the Android activity you need to extend SimpleCamera2Activity or VisualizeCamera2Activity. The former will only provide you access to control of the camera and access to its images. The latter handles converting image data and assists in visualization. To use these activities you need to grant permission to the camera in your AndroidManifest.xml:

 <uses-permission android:name="android.permission.CAMERA" />
 <uses-feature android:name="android.hardware.camera2.full" />

SimpleCamera2Activity

To start the camera invoke startCamera} inside your Activity's onCreate() function.

  • selectResolution
  • onCameraResolutionChange
  • configureCamera
  • selectCamera
  • processFrame
  • onCameraOpened
  • onCameraDisconnected
  • changeCameraConfiguratio

To turn on and off verbose printing to Log you can toggle the SimpleCamera2Activity.verbose variable.

Selecting Camera / Resolution

By default BoofCV will select the back facing camera and the resolution which post closely matches the texture's width. NOTE: This behavior is changed if you extend VisualizeCamera2Activity. You can modify this behavior by overriding one of the functions list above. For example, to change the resolution selection behavior you will override selectResolution.

Camera Settings

To customize the camera settings you need to override configureCamera. If after opening the camera you want to change the camera settings you need to then first call isCameraReadyReconfiguration and make sure the camera is in a state that it can be reconfigured. If that returns truen then you're free to call changeCameraConfiguration which will eventually result in configureCamera being called again.

VisualizeCamera2Activity

Extension of SimpleCamera2Activity which adds visualization and hooks for image processing. Video frames are automatically converted into a format which can be processed by BoofCV routines Optionally multiple threads can be generated so that video frames are processed concurrently. The input image is automatically converted into a Bitmap image if requested. If multiple threads are being used the user can toggle if they want visualization to be shown if an old image finished being processed after a newer one.

Inside of onCreate() you need to call setImageType so that it knows what type of image to convert camera's YUV420_888 into. ImageType.single(GrayU8.class) for gray scale and ImageType.il(Gray8.class,3) for interleaved RGB. After that is done you will call startCamera(ViewGroup, TextureView) The textureview is optional and only needed if you want to display the camera preview directly.

  • imageToView: Matrix that converts a pixel in video frame to view frame
  • displayView: Where visualizations are rendered in.
  • targetResolution: Specifies how many pixels you want in the video frame
  • stretchToFill: true to stretch the video frame to fill the entire view
  • bitmapMode: Species how and if the bitmap should be drawn to the screen
  • visualizeOnlyMostRecent: If more than one thread is enabled should it show old results

Processing Images

You must implement processImage() and typecast the image it provides into the format you specified earlier

Visualization

Override onDrawFrame() and invoke the super.

Converting Image Types

Usage Examples

A more detailed and working example for processing video images in Android can be found at the link below. Short code snippets are included in this section.

Full Working Examples

Converting Bitmap images

// Easiest way to convert a Bitmap into a BoofCV type
GrayU8 image = ConvertBitmap.bitmapToGray(bitmap, (GrayU8)null, null);
	
// If you are converting a sequence of images it is faster reuse a
// previously declare image and buffer
byte[] workBuffer = ConvertBitmap.declareStorage(bitmap, null);
ConvertBitmap.bitmapToGray(bitmap, image, workBuffer);
    	
// Convert back into a Bitmap
ConvertBitmap.grayToBitmap(image, bitmap, workBuffer);

// another less efficient way
bitmap = ConvertBitmap.grayToBitmap(image, Bitmap.Config.ARGB_8888);
    	
// Functions are also provided for multi-spectral images
Planar<GrayF32> color = ConvertBitmap.bitmapToMS(bitmap, null, GrayF32.class, null);

Converting YUV420 Images

Image image; // From camera in YUV 420 format
byte work[] = ConvertCameraImage.declareWork(image,null);
ConvertCameraImage.imageToBoof(image, ColorFormat.RGB, grayU8, work);

Converting NV21 Images

Data from camera previews in Camera 1 API is made available in the NV21 image format.

// from NV21 to gray scale
ConvertNV21.nv21ToGray(bytes,width,height,gray)

// from NV21 to YUV Planar
ConvertNV21.nv21ToPlanarYuv(bytes,width,height,colorPL,GrayU8.class);

Besides converting images, VisualizeImageData is provided for visualizing

Android Apps

Android demonstration and benchmark applications are available for download from the google play store.

The demonstration visualizes several computer vision algorithms and provides a good estimate on what sort of runtime performance can be expected when using BoofCV on Android.

Trouble Shooting

Does your code run much slower than the demo? Try switching the build to release. Click on "Build Variants" on the bottom left, then in the panel that pops up select "release" from the build variant.