android - PreviewCallback onPreviewFrame does not change data

Question

I want to do some image processing with images from camera and display it on a SurfaceView but I don't know how to modify the camera frame. I tried to use setPreviewCallbackWithBuffer and onPreviewFrame but they do not work as expected, the frame is not modified.

/** A basic Camera preview class */
public class CameraPreview extends SurfaceView implements
        SurfaceHolder.Callback, Camera.PreviewCallback {
    private SurfaceHolder mHolder;
    private Camera mCamera;
    private byte[] mData;
    private long prevFrameTick = System.currentTimeMillis();
    Canvas mCanvas;

    public CameraPreview(Context context, Camera camera) {
        super(context);
        mCamera = camera;

        // Install a SurfaceHolder.Callback so we get notified when the
        // underlying surface is created and destroyed.
        mHolder = getHolder();
        mHolder.addCallback(this);
        Size previewSize = mCamera.getParameters().getPreviewSize();
        mData = new byte[(int) (previewSize.height * previewSize.width * 1.5)];
        initBuffer();
        // deprecated setting, but required on Android versions prior to 3.0
        mHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
    }

    private void initBuffer() {
        mCamera.addCallbackBuffer(mData);
        mCamera.addCallbackBuffer(mData);
        mCamera.addCallbackBuffer(mData);
        mCamera.setPreviewCallbackWithBuffer(this);
    }

    public void setCamera(Camera cam) {
        mCamera = cam;
        initBuffer();
    }

    public void surfaceCreated(SurfaceHolder holder) {
        // The Surface has been created, now tell the camera where to draw the
        // preview.
        try {
            mCamera.setPreviewDisplay(holder);
            initBuffer();
            mCamera.startPreview();
        } catch (IOException e) {
            Log.d("APP",
                    "Error setting camera preview: " + e.getMessage());
        }
    }

    public void surfaceDestroyed(SurfaceHolder holder) {
        // empty. Take care of releasing the Camera preview in your activity.
    }

    public void surfaceChanged(SurfaceHolder holder, int format, int w, int h) {
        // If your preview can change or rotate, take care of those events here.
        // Make sure to stop the preview before resizing or reformatting it.

        if (mHolder.getSurface() == null) {
            // preview surface does not exist
            return;
        }

        // stop preview before making changes
        try {
            mCamera.stopPreview();
        } catch (Exception e) {
            // ignore: tried to stop a non-existent preview
        }

        // set preview size and make any resize, rotate or
        // reformatting changes here

        // start preview with new settings
        try {
            mCamera.setPreviewDisplay(mHolder);
            initBuffer();
            mCamera.startPreview();
        } catch (Exception e) {
            Log.d("APP",
                    "Error starting camera preview: " + e.getMessage());
        }
    }

    public void onPreviewFrame(byte[] data, Camera camera) {
        // System.arraycopy(data, 0, mData, 0, data.length);
        Log.e("onPreviewFrame", data.length + " "
                + (System.currentTimeMillis() - prevFrameTick));
        prevFrameTick = System.currentTimeMillis();
        mData = new byte[data.length];
        mCamera.addCallbackBuffer(mData);
    }
}

Answer 1

You cannot modify the preview data sent to a SurfaceView, if you're using the setPreviewDisplay() call. The preview video stream is managed entirely outside of your application and isn't accessible to it.

There are a few options you can take:

1. You can place a second view on top of the SurfaceView, such as an ImageView or another SurfaceView, and draw the data received by the onPreviewFrame callback into this view. You'll have to do some color/pixel format conversion from the preview callback format (usually NV21) for display, and obviously you have to run your image processing on that data first as well. This isn't very efficient, unless you're willing to write some JNI code.

2. On Android 3.0 or newer, you can use the Camera.setPreviewTexture() method, and pipe the camera preview stream into an OpenGL texture by using a SurfaceTexture object, which you can then manipulate in OpenGL before displaying. Then you don't need the preview callbacks at all. This is more efficient if GPU processing is sufficient. You can also use the OpenGL readPixels call to get the processed preview data back to your application, if you want to display it/process it some other way.