一般來講在安卓中常使用AudioRecord藐石、MediaRecorder對音頻進(jìn)行采集,使用SoundPool、MediaPlayer定拟、AudioTrack進(jìn)行音頻播放于微。
但是這些接口都是java層的,而NDK其實(shí)也提供了一個叫做OpenSL的C語言引擎用于聲音的處理。
OpenSL入門難度比較大,而且網(wǎng)上也沒有什么特別好的教程,我這里把自己了解到的一些知識記錄下來,希望以后忘記的時候可以快速回憶起來,也希望對大家有用青自。
這篇筆記的很多內(nèi)容都參考了OpenSL的官方文檔OpenSL_ES_Specification_1.0.1.pdf,它是全英文的,可以在NDK的安裝目錄下找到,大家可以大概瀏覽一下,具體路徑為:
$NDK_ROOT/docs/Additional_library_docs/opensles
為什么要學(xué)OpenSL呢?除了C/C++的性能優(yōu)勢(不過其實(shí)java的效率也不低)之外,最主要是因?yàn)樽罱肟覨Fmpeg,如果使用java層的接口,還需要通過一層JNI,比較復(fù)雜,性能消耗也大株依。如果用OpenSL的話就能直接在C/C++里面把事情都處理了。
基本概念
Object和Interface
在OpenSL里面,Object和Interface是兩個很重要的概念,基本上所有的操作都是通過它們兩個去執(zhí)行的延窜。
Object和Interface是包含關(guān)系,一個Object里面包含了多個Interface:
Object
Object是一個資源的抽象集合,可以通過它獲取各種資源勺三。
例如我們可以通過Object的GetInterface方法獲取Interface。
所有的Object在OpenSL里面我們拿到的都是一個SLObjectItf:
struct SLObjectItf_ {
SLresult (*Realize) (SLObjectItf self,SLboolean async);
SLresult (*Resume) (SLObjectItf self,SLboolean async);
SLresult (*GetState) (SLObjectItf self,SLuint32 * pState);
SLresult (*GetInterface) (SLObjectItf self, const SLInterfaceID iid, void * pInterface);
SLresult (*RegisterCallback) (SLObjectItf self, slObjectCallback callback, void * pContext);
void (*AbortAsyncOperation) (SLObjectItf self);
void (*Destroy) (SLObjectItf self);
SLresult (*SetPriority) (SLObjectItf self, SLint32 priority, SLboolean preemptable);
SLresult (*GetPriority) (SLObjectItf self, SLint32 *pPriority, SLboolean *pPreemptable);
SLresult (*SetLossOfControlInterfaces) (SLObjectItf self, SLint16 numInterfaces, SLInterfaceID * pInterfaceIDs, SLboolean enabled);
};
typedef const struct SLObjectItf_ * const * SLObjectItf;
在創(chuàng)建出來之后必須先調(diào)用Realize方法做初始化需曾。在不需要使用的時候調(diào)用Destroy方法釋放資源。
GetInterface
GetInterface可以說是OpenSL里使用頻率最高的方法,通過它我們可以獲取Object里面的Interface祈远。
由于一個Object里面可能包含了多個Interface,所以GetInterface方法有個SLInterfaceID參數(shù)來指定到的需要獲取Object里面的那個Interface呆万。
例如下面代碼我們通過EngineObject去獲取SL_IID_ENGINE這個id的Interface,而這個id對應(yīng)的Interface就是SLEngineItf:
//create EngineObject
SLObjectItf engineObject;
slCreateEngine(&engineObject, 0, NULL, 0, NULL, NULL);
(*engineObject)->Realize(engineObject, SL_BOOLEAN_FALSE);
//get SLEngineItf
SLEngineItf engineInterface;
(*engineObject)->GetInterface(engineObject, SL_IID_ENGINE, &engineInterface);
Interface
Interface則是方法的集合,例如SLRecordItf里面包含了和錄音相關(guān)的方法,SLPlayItf包含了和播放相關(guān)的方法。我們功能都是通過調(diào)用Interfaces的方法去實(shí)現(xiàn)的车份。
SLEngineItf
SLEngineItf是OpenSL里面最重要的一個Interface,我們可以通過它去創(chuàng)建各種Object,例如播放器谋减、錄音器、混音器的Object,然后在用這些Object去獲取各種Interface去實(shí)現(xiàn)各種功能扫沼。
struct SLEngineItf_ {
SLresult (*CreateAudioPlayer) (SLEngineItf self, SLObjectItf * pPlayer, SLDataSource *pAudioSrc, SLDataSink *pAudioSnk, SLuint32 numInterfaces, const SLInterfaceID * pInterfaceIds, const SLboolean * pInterfaceRequired);
SLresult (*CreateAudioRecorder) (SLEngineItf self, SLObjectItf * pRecorder, SLDataSource *pAudioSrc, SLDataSink *pAudioSnk, SLuint32 numInterfaces, const SLInterfaceID * pInterfaceIds, const SLboolean * pInterfaceRequired);
SLresult (*CreateOutputMix) (SLEngineItf self, SLObjectItf * pMix, SLuint32 numInterfaces, const SLInterfaceID * pInterfaceIds, const SLboolean * pInterfaceRequired);
...
};
錄音
OpenSL的錄音功能是通過AudioRecorder來實(shí)現(xiàn)的,而AudioRecorder是通過SLEngineItf.CreateAudioRecorder方法創(chuàng)建的:
SLresult (*CreateAudioRecorder) (
SLEngineItf self,
SLObjectItf * pRecorder,
SLDataSource * pAudioSrc,
SLDataSink * pAudioSnk,
SLuint32 numInterfaces,
const SLInterfaceID * pInterfaceIds,
const SLboolean * pInterfaceRequired
);
各個參數(shù)的意義如下:
- SLEngineItf C語言不像c++,沒有this指針,只能每次調(diào)用SLEngineItf的方法的時候手動傳入
- SLObjectItf 用于保存創(chuàng)建出來的AudioRecorderObject
- SLDataSource 數(shù)據(jù)的來源
- SLDataSink 數(shù)據(jù)的去處
- numInterfaces 與下面的SLInterfaceID和SLboolean配合使用,用于標(biāo)記SLInterfaceID數(shù)組和SLboolean的大小
- SLInterfaceID 這里需要傳入一個數(shù)組,指定創(chuàng)建的AudioRecorderObject會包含哪些Interface
- SLboolean 這里也是一個數(shù)組,用來標(biāo)記每個需要包含的Interface,如果AudioRecorderObject不支持,是不是需要直接創(chuàng)建AudioRecorderObject失敗出爹。
最后的三個參數(shù)用于指定AudioRecorderObject需要包含哪些Interface,如果不包含,是不是要直接創(chuàng)建失敗。如果成功的話我們就能使用AudioRecorderObject的GetInterface方法獲取到這些Interface了缎除。
SLDataSource和SLDataSink可能比較難理解严就。我們可以看下OpenSL錄音的原理:
簡而言之, AudioRecorder會從SLDataSource指定的數(shù)據(jù)源獲取數(shù)據(jù),然后將數(shù)據(jù)保存到SLDataSink指定的接收器。
SLDataSource很明顯就是錄音設(shè)備(SL_IODEVICE_AUDIOINPUT):
SLDataLocator_IODevice device;
device.locatorType = SL_DATALOCATOR_IODEVICE;
device.deviceType = SL_IODEVICE_AUDIOINPUT;
device.deviceID = SL_DEFAULTDEVICEID_AUDIOINPUT;
device.device = NULL; //Must be NULL if deviceID parameter is to be used.
SLDataSource source;
source.pLocator = &device;
source.pFormat = NULL; //This parameter is ignored if pLocator is SLDataLocator_IODevice.
而SLDataSink就可以任由我們指定了,它官方支持下面的類型:
SLDataLocator_Address
SLDataLocator_IODevice
SLDataLocator_OutputMix
SLDataLocator_URI
SLDataLocator_BufferQueue
SLDataLocator_MIDIBufferQueue
Android又拓展了下面幾種類型:
SLDataLocator_AndroidFD
SLDataLocator_AndroidBufferQueue
SLDataLocator_AndroidSimpleBufferQueue
我這邊把它設(shè)置成SLDataLocator_AndroidSimpleBufferQueue,它比較通用, AudioRecorder把數(shù)據(jù)放到這個隊(duì)列中,我們再可以從這個隊(duì)列中拿出來使用:
SLDataLocator_AndroidSimpleBufferQueue queue;
queue.locatorType = SL_DATALOCATOR_ANDROIDSIMPLEBUFFERQUEUE;
queue.numBuffers = 2;
SLDataFormat_PCM format;
format.formatType = SL_DATAFORMAT_PCM;
format.numChannels = numChannels;
format.samplesPerSec = samplingRate;
format.bitsPerSample = SL_PCMSAMPLEFORMAT_FIXED_16;
format.containerSize = SL_PCMSAMPLEFORMAT_FIXED_16;
format.channelMask = getChannelMask(numChannels);
format.endianness = SL_BYTEORDER_LITTLEENDIAN;
SLDataSink sink;
sink.pLocator = &queue;
sink.pFormat = &format;
同時在創(chuàng)建的時候需要檢測下SL_DATALOCATOR_ANDROIDSIMPLEBUFFERQUEUE是不是支持:
SLInterfaceID id[] = {SL_IID_ANDROIDSIMPLEBUFFERQUEUE};
SLboolean required[] = {SL_BOOLEAN_TRUE};
SLObjectItf recorderObject;
(engineInterface)->CreateAudioRecorder(
engineInterface,
&(recorderObject),
&source,
&sink,
1,
id,
required
);
(*recorderObject)->Realize(recorderObject, SL_BOOLEAN_FALSE);
所以我們可以通過GetInterface獲取SLAndroidSimpleBufferQueueItf,然后注冊個隊(duì)列滿的監(jiān)聽回調(diào):
SLAndroidSimpleBufferQueueItf queueInterface;
(*recorderObject)->GetInterface(
recorderObject,
SL_IID_ANDROIDSIMPLEBUFFERQUEUE,
&(queueInterface)
);
(*queueInterface)->RegisterCallback(
queueInterface,
bufferQueueCallback,
NULL
);
回調(diào)函數(shù)如下,我們可以在這個時候從隊(duì)列里面讀取下來的音頻數(shù)據(jù):
static void bufferQueueCallback(SLAndroidSimpleBufferQueueItf queue, void *pContext) {
...
}
最后需要打開錄音設(shè)備開始錄音:
SLRecordItf recorderInterface;
(*recorderObject)->GetInterface(
recorderObject,
SL_IID_RECORD,
&(recorderInterface)
);
(*recorderInterface)->SetRecordState(
recorderInterface,
SL_RECORDSTATE_RECORDING
);
這里需要注意的是我們必須在隊(duì)列滿的時候?qū)?shù)據(jù)取出來,如果不取,那隊(duì)列里面就沒有空間可以繼續(xù)存儲音頻數(shù)據(jù)了:
(*queueInterface)->Enqueue(queueInterface, buffer, BUFFER_SIZE*sizeof(short));
播放
播放的代碼和錄音很類似器罐。我們需要先創(chuàng)建AudioPlayer:
SLresult (*CreateAudioPlayer) (
SLEngineItf self,
SLObjectItf * pPlayer,
SLDataSource *pAudioSrc,
SLDataSink *pAudioSnk,
SLuint32 numInterfaces,
const SLInterfaceID * pInterfaceIds,
const SLboolean * pInterfaceRequired
);
它的參數(shù)和CreateAudioRecorder一樣,我就不再一個個去解釋了,可以看看播放的過程:
SLDataSource我也用SLDataLocator_AndroidSimpleBufferQueue,這樣我們可以往隊(duì)列中不斷寫入音頻數(shù)據(jù),AudioRecorder會從隊(duì)列中不斷獲取數(shù)據(jù)傳遞到混音器中:
SLDataLocator_AndroidSimpleBufferQueue queue;
queue.locatorType = SL_DATALOCATOR_ANDROIDSIMPLEBUFFERQUEUE;
queue.numBuffers = 2;
SLDataFormat_PCM format;
format.formatType = SL_DATAFORMAT_PCM;
format.numChannels = numChannels;
format.samplesPerSec = samplingRate;
format.bitsPerSample = SL_PCMSAMPLEFORMAT_FIXED_16;
format.containerSize = SL_PCMSAMPLEFORMAT_FIXED_16;
format.channelMask = getChannelMask(numChannels);
format.endianness = SL_BYTEORDER_LITTLEENDIAN;
SLDataSource source;
source.pLocator = &queue;
source.pFormat = &format;
而SLDataSink需要配置成混音器梢为。混音器用于將多個音頻混合并且輸出到喇叭:
SLObjectItf outputMixObject;
(*engineInterface)->CreateOutputMix(
engineInterface,
&(outputMixObject),
0,
NULL,
NULL
);
(*outputMixObject)->Realize(
outputMixObject,
SL_BOOLEAN_FALSE
);
SLDataLocator_OutputMix outputMix;
outputMix.locatorType = SL_DATALOCATOR_OUTPUTMIX;
outputMix.outputMix = outputMixObject;
SLDataSink sink;
sink.pLocator = &outputMix;
sink.pFormat = NULL; //This parameter is ignored if pLocator is SLDataLocator_IODevice or SLDataLocator_OutputMix.
同樣的我們在創(chuàng)建AudioPlayer的時候會檢查是不是支持SL_IID_ANDROIDSIMPLEBUFFERQUEUE:
SLObjectItf playerObject;
SLInterfaceID id[] = {SL_IID_ANDROIDSIMPLEBUFFERQUEUE};
SLboolean required[] = {SL_BOOLEAN_TRUE};
(*engineInterface)->CreateAudioPlayer(
engineInterface,
&(playerObject),
&source,
&sink,
1,
id,
required
);
(*playerObject)->Realize(playerObject, SL_BOOLEAN_FALSE);
最后我們需要注冊隊(duì)列空的監(jiān)聽和打開播放器開始播放:
SLAndroidSimpleBufferQueueItf queueInterface;
(*playerObject)->GetInterface(
playerObject,
SL_IID_ANDROIDSIMPLEBUFFERQUEUE,
&(queueInterface)
);
(*queueInterface)->RegisterCallback(
queueInterface,
bufferQueueCallback,
NULL
);
//////Begin Playing//////
SLPlayItf playInterface;
(*playerObject)->GetInterface(
playerObject,
SL_IID_PLAY,
&(playInterface)
);
(*playInterface)->SetPlayState(
playInterface,
SL_PLAYSTATE_PLAYING
);
Demo
這里有個簡單的錄音和播放的demo,按興趣的同學(xué)可以參考一下。
