使用javacv录像，同时进行讯飞声纹认证

由于最近的demo中需要在活体检测的同时进行音视频录制 , 尝试使用MediaRecord和camera来录制视频 , 然而Camera.onPreviewFrame 不能与 MediaRecord同时调用。活体检测的原理其实是把camera的预览回调onPreviewFrame(byte[] data, Camera camera) 中的图片数据data作为参数传递到活体检测引擎中去拿返回的检测结果码,由于种种原因 , 不能使用Camera2实现 , 于是通过谷歌了解到javacv这个库可以录制视频 , 下了几个demo , 感觉不仅满足需求 , 录制的视频质量也还可以。使用javacv中的FrameRecorder进行录像，录像的时候，调用record方法写帧数据和音频数据，这时候我们有一个需求，录像的同时，要把声音实时拿过来进行声纹认证。由此产生了2个问题：

问题1：

语音识别用的是讯飞的SDK，要求声音采样率8k或16k。而设置FrameRecorder.setSampleRate(8000)后，再FrameRecorder.start()会报错，报错如下：

avcodec_encode_audio2() error 2: Could not encode audio packet.

问题2：

javacv官方录制demo中，从AudioRecord中read到的是ShortBuffer，而讯飞SDK方法要求传入byte，他的方法如下：

public void writeAudio(byte[] data, int start, int length)

百度谷歌无果，只好自己研究。

使用javacv进行录像

下面是使用javacv进行录像的示例代码：

1. 初始化 ffmpeg_recorder

public void initRecorder() {

String ffmpeg_link = parentPath + "/" + "video.mp4";

Log.w(LOG_TAG, "init recorder");

if (yuvIplimage == null) {

yuvIplimage = IplImage.create(cameraManager.getDefaultSize().width,

cameraManager.getDefaultSize().height, IPL_DEPTH_8U, 2);

Log.i(LOG_TAG, "create yuvIplimage");

}

Log.i(LOG_TAG, "ffmpeg_url: " + ffmpeg_link);

recorder = new FFmpegFrameRecorder(ffmpeg_link,

cameraManager.getDefaultSize().width,

cameraManager.getDefaultSize().height, 1);

recorder.setFormat("mp4");

recorder.setSampleRate(sampleAudioRateInHz);

// Set in the surface changed method

recorder.setFrameRate(frameRate);

Log.i(LOG_TAG, "recorder initialize success");

audioRecordRunnable = new AudioRecordRunnable();

audioThread = new Thread(audioRecordRunnable);

try {

recorder.start();

} catch (Exception e) {

// TODO Auto-generated catch block

e.printStackTrace();

}

audioThread.start();

}

2. 捕捉摄像头视频数据：

public void onPreviewFrame(byte[] data, Camera camera) {

int during = checkIfMax(new Date().getTime());

/* get video data */

if (yuvIplimage != null && isStart) {

yuvIplimage.getByteBuffer().put(data);

//yuvIplimage = rotateImage(yuvIplimage.asCvMat(), 90).asIplImage();

Log.v(LOG_TAG, "Writing Frame");

try {

System.out.println(System.currentTimeMillis() - videoStartTime);

if (during < 6000) {

recorder.setTimestamp(1000 * during);

recorder.record(yuvIplimage);

}

} catch (FFmpegFrameRecorder.Exception e) {

Log.v(LOG_TAG, e.getMessage());

e.printStackTrace();

}

}

}

3. 捕捉声音数据：

class AudioRecordRunnable implements Runnable {

@Override

public void run() {

android.os.Process

.setThreadPriority(android.os.Process.THREAD_PRIORITY_URGENT_AUDIO);

// Audio

int bufferSize;

short[] audioData;

int bufferReadResult;

bufferSize = AudioRecord.getMinBufferSize(sampleAudioRateInHz,

AudioFormat.CHANNEL_CONFIGURATION_MONO,

AudioFormat.ENCODING_PCM_16BIT);

audioRecord = new AudioRecord(MediaRecorder.AudioSource.MIC,

sampleAudioRateInHz,

AudioFormat.CHANNEL_CONFIGURATION_MONO,

AudioFormat.ENCODING_PCM_16BIT, bufferSize);

audioData = new short[bufferSize];

Log.d(LOG_TAG, "audioRecord.startRecording()");

audioRecord.startRecording();

/* ffmpeg_audio encoding loop */

while (!isFinished) {

// Log.v(LOG_TAG,"recording? " + recording);

bufferReadResult = audioRecord.read(audioData, 0,

audioData.length);

if (bufferReadResult > 0) {

// Log.v(LOG_TAG, "bufferReadResult: " + bufferReadResult);

// If "recording" isn't true when start this thread, it

// never get's set according to this if statement...!!!

// Why? Good question...

if (isStart) {

try {

Buffer[] barray = new Buffer[1];

barray[0] = ShortBuffer.wrap(audioData, 0,

bufferReadResult);

recorder.record(barray);

// Log.v(LOG_TAG,"recording " + 1024*i + " to " +

// 1024*i+1024);

} catch (FFmpegFrameRecorder.Exception e) {

Log.v(LOG_TAG, e.getMessage());

e.printStackTrace();

}

}

}

}

Log.v(LOG_TAG, "AudioThread Finished, release audioRecord");

/* encoding finish, release recorder */

if (audioRecord != null) {

audioRecord.stop();

audioRecord.release();

audioRecord = null;

Log.v(LOG_TAG, "audioRecord released");

}

}

}

解决问题1：

demo中默认设置FrameRecorder.setSampleRate(44100)没问题，想到一个办法，这个地方设置44100，在语音采集的地方设置8000，最后成功了。不过这个计算时间的方法要修改：

public static int getTimeStampInNsFromSampleCounted(int paramInt) {

// return (int) (paramInt / 0.0441D);

return (int) (paramInt / 0.0080D);

}

解决问题2：

short数组转byte数组，注意数组长度变为原来的2倍

public static byte[] short2byte(short[] sData) {

int shortArrsize = sData.length;

byte[] bytes = new byte[shortArrsize * 2];

for (int i = 0; i < shortArrsize; i++) {

bytes[i * 2] = (byte) (sData[i] & 0x00FF);

bytes[(i * 2) + 1] = (byte) (sData[i] >> 8);

sData[i] = 0;

}

return bytes;

}

录制音频源码：

 /**

     * 录制音频的线程

     */

    class AudioRecordRunnable implements Runnable {

        short[] audioData;

        private final AudioRecord audioRecord;

        private int mCount = 0;

        int sampleRate = Constants.AUDIO_SAMPLING_RATE;

        private AudioRecordRunnable() {

            int bufferSize = AudioRecord.getMinBufferSize(sampleRate,

                    AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT);

            audioRecord = new AudioRecord(MediaRecorder.AudioSource.MIC, sampleRate,

                    AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT, bufferSize);

            audioData = new short[bufferSize];

        }

        /**

         * 包含了音频的数据和起始位置

         *

         * @param buffer

         */

        private void record(Buffer buffer) {

            synchronized (mAudioRecordLock) {

                this.mCount += buffer.limit();

                if (!mIsPause) {

                    try {

                        if (mRecorder != null) {

                            mRecorder.record(sampleRate, new Buffer[]{buffer});

                        }

                    } catch (FrameRecorder.Exception e) {

                        e.printStackTrace();

                    }

                }

            }

        }

        /**

         * 更新音频的时间戳

         */

        private void updateTimestamp() {

            int i = Util.getTimeStampInNsFromSampleCounted(this.mCount);

            if (mAudioTimestamp != i) {

                mAudioTimestamp = i;

                mAudioTimeRecorded = System.nanoTime();

            }

        }

        public void run() {

            android.os.Process.setThreadPriority(android.os.Process.THREAD_PRIORITY_URGENT_AUDIO);

            if (audioRecord != null) {

                //判断音频录制是否被初始化

                while (this.audioRecord.getState() == 0) {

                    try {

                        Thread.sleep(100L);

                    } catch (InterruptedException localInterruptedException) {

                    }

                }

                this.audioRecord.startRecording();

                while ((runAudioThread)) {

                    updateTimestamp();

                    int bufferReadResult = this.audioRecord.read(audioData, 0, audioData.length);

                    if (bufferReadResult > 0) {

                        if (recording || (mVideoTimestamp > mAudioTimestamp)) {

                            record(ShortBuffer.wrap(audioData, 0, bufferReadResult));

                        }

                        if (SpeechManager.getInstance().isListening()) {

                            SpeechManager.getInstance().writeAudio(Util.short2byte(audioData), 0, bufferReadResult * 2);

                        }

                    }

                }

                SpeechManager.getInstance().stopListener();

                this.audioRecord.stop();

                this.audioRecord.release();

            }

        }

    }

巴特西

使用javacv录像，同时进行讯飞声纹认证

使用javacv进行录像

解决问题1：

解决问题2：

最新文章

热门文章