我們知道Metal可以完成圖片渲染秒裕，我們也知道視頻是由一幀一幀的圖片組成的几蜻。今天我們將使用Metal來完成攝像頭的實時顯示体斩。
實現(xiàn)原理
1 利用AVFoundation做圖像采集
2 將AVFoundation的采集回調(diào)中的CMSampleBufferRef對象轉(zhuǎn)換成紋理對象
3 我們將拿到的紋理對象通過Metal實時渲染到屏幕上(MTKView)
Metal實時渲染攝像頭畫面圖解.png
配置Metal
-
配置Metal圖解
Metal配置圖解.png - 實現(xiàn)代碼
//1.創(chuàng)建MTKView
self.mtkView = [[MTKView alloc] initWithFrame:self.view.bounds];
self.mtkView.device = MTLCreateSystemDefaultDevice();
[self.view insertSubview:self.mtkView atIndex:0];
self.mtkView.delegate = self;
//2.創(chuàng)建命令隊列.
self.commandQueue = [self.mtkView.device newCommandQueue];
//3.注意: 在初始化MTKView 的基本操作以外. 還需要多下面2行代碼.
/*
1. 設置MTKView 的drawable 紋理是可讀寫的(默認是只讀);
2. 創(chuàng)建CVMetalTextureCacheRef _textureCache; 這是Core Video的Metal紋理緩存
*/
//允許讀寫操作
self.mtkView.framebufferOnly = NO;
/*
CVMetalTextureCacheCreate(CFAllocatorRef allocator,
CFDictionaryRef cacheAttributes,
id <MTLDevice> metalDevice,
CFDictionaryRef textureAttributes,
CVMetalTextureCacheRef * CV_NONNULL cacheOut )
功能: 創(chuàng)建紋理緩存區(qū)
參數(shù)1: allocator 內(nèi)存分配器.默認即可.NULL
參數(shù)2: cacheAttributes 緩存區(qū)行為字典.默認為NULL
參數(shù)3: metalDevice
參數(shù)4: textureAttributes 緩存創(chuàng)建紋理選項的字典. 使用默認選項NULL
參數(shù)5: cacheOut 返回時弧烤，包含新創(chuàng)建的紋理緩存暇昂。
*/
CVMetalTextureCacheCreate(NULL, NULL, self.mtkView.device, NULL, &_textureCache);
配置視頻采集
-
配置步驟圖解
配置視頻采集圖解.png - 具體代碼如下：
//創(chuàng)建mCaptureSession
self.mCaptureSession = [[AVCaptureSession alloc] init];
//設置視頻采集的分辨率
self.mCaptureSession.sessionPreset = AVCaptureSessionPreset1920x1080;
//創(chuàng)建串行隊列
self.mProcessQueue = dispatch_queue_create("mProcessQueue", DISPATCH_QUEUE_SERIAL);
//獲取攝像頭設備(前置/后置攝像頭設備)
NSArray *devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo];
AVCaptureDevice *inputCamera = nil;
//循環(huán)設備數(shù)組,找到后置攝像頭.設置為當前inputCamera
for (AVCaptureDevice *device in devices) {
if ([device position] == AVCaptureDevicePositionBack) {
inputCamera = device;
}
}
//將AVCaptureDevice 轉(zhuǎn)換為AVCaptureDeviceInput
self.mCaptureDeviceInput = [[AVCaptureDeviceInput alloc] initWithDevice:inputCamera error:nil];
//將設備添加到mCaptureSession中
if ([self.mCaptureSession canAddInput:self.mCaptureDeviceInput]) {
[self.mCaptureSession addInput:self.mCaptureDeviceInput];
}
//6.創(chuàng)建AVCaptureVideoDataOutput 對象
self.mCaptureDeviceOutput = [[AVCaptureVideoDataOutput alloc] init];
/*設置視頻幀延遲到底時是否丟棄數(shù)據(jù).
YES: 處理現(xiàn)有幀的調(diào)度隊列在captureOutput:didOutputSampleBuffer:FromConnection:Delegate方法中被阻止時从铲，對象會立即丟棄捕獲的幀。
NO: 在丟棄新幀之前吉嫩，允許委托有更多的時間處理舊幀自娩，但這樣可能會內(nèi)存增加.
*/
[self.mCaptureDeviceOutput setAlwaysDiscardsLateVideoFrames:NO];
//這里設置格式為BGRA忙迁，而不用YUV的顏色空間碎乃，避免使用Shader轉(zhuǎn)換
//注意:這里必須和后面CVMetalTextureCacheCreateTextureFromImage 保存圖像像素存儲格式保持一致.否則視頻會出現(xiàn)異常現(xiàn)象.
[self.mCaptureDeviceOutput setVideoSettings:[NSDictionary dictionaryWithObject:[NSNumber numberWithInt:kCVPixelFormatType_32BGRA] forKey:(id)kCVPixelBufferPixelFormatTypeKey]];
//設置視頻捕捉輸出的代理方法
[self.mCaptureDeviceOutput setSampleBufferDelegate:self queue:self.mProcessQueue];
//添加輸出
if ([self.mCaptureSession canAddOutput:self.mCaptureDeviceOutput]) {
[self.mCaptureSession addOutput:self.mCaptureDeviceOutput];
}
//輸入與輸出鏈接
AVCaptureConnection *connection = [self.mCaptureDeviceOutput connectionWithMediaType:AVMediaTypeVideo];
//設置視頻方向
//注意: 一定要設置視頻方向.否則視頻會是朝向異常的.
[connection setVideoOrientation:AVCaptureVideoOrientationPortrait];
//開始捕捉
[self.mCaptureSession startRunning];
AVFundation采集回調(diào)方法
拿到CMSampleBufferRef之后的處理.png
具體代碼：
//1.從sampleBuffer 獲取視頻像素緩存區(qū)對象
CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
//2.獲取捕捉視頻的寬和高
size_t width = CVPixelBufferGetWidth(pixelBuffer);
size_t height = CVPixelBufferGetHeight(pixelBuffer);
/*3. 根據(jù)視頻像素緩存區(qū) 創(chuàng)建 Metal 紋理緩存區(qū)
CVReturn CVMetalTextureCacheCreateTextureFromImage(CFAllocatorRef allocator, CVMetalTextureCacheRef textureCache,
CVImageBufferRef sourceImage,
CFDictionaryRef textureAttributes,
MTLPixelFormat pixelFormat,
size_t width,
size_t height,
size_t planeIndex,
CVMetalTextureRef *textureOut);
功能: 從現(xiàn)有圖像緩沖區(qū)創(chuàng)建核心視頻Metal紋理緩沖區(qū)梅誓。
參數(shù)1: allocator 內(nèi)存分配器,默認kCFAllocatorDefault
參數(shù)2: textureCache 紋理緩存區(qū)對象
參數(shù)3: sourceImage 視頻圖像緩沖區(qū)
參數(shù)4: textureAttributes 紋理參數(shù)字典.默認為NULL
參數(shù)5: pixelFormat 圖像緩存區(qū)數(shù)據(jù)的Metal 像素格式常量.注意如果MTLPixelFormatBGRA8Unorm和攝像頭采集時設置的顏色格式不一致，則會出現(xiàn)圖像異常的情況嵌言；
參數(shù)6: width,紋理圖像的寬度（像素）
參數(shù)7: height,紋理圖像的高度（像素）
參數(shù)8: planeIndex.如果圖像緩沖區(qū)是平面的，則為映射紋理數(shù)據(jù)的平面索引及穗。對于非平面圖像緩沖區(qū)忽略。
參數(shù)9: textureOut,返回時埂陆，返回創(chuàng)建的Metal紋理緩沖區(qū)苛白。
// Mapping a BGRA buffer:
CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault, textureCache, pixelBuffer, NULL, MTLPixelFormatBGRA8Unorm, width, height, 0, &outTexture);
// Mapping the luma plane of a 420v buffer:
CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault, textureCache, pixelBuffer, NULL, MTLPixelFormatR8Unorm, width, height, 0, &outTexture);
// Mapping the chroma plane of a 420v buffer as a source texture:
CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault, textureCache, pixelBuffer, NULL, MTLPixelFormatRG8Unorm width/2, height/2, 1, &outTexture);
// Mapping a yuvs buffer as a source texture (note: yuvs/f and 2vuy are unpacked and resampled -- not colorspace converted)
CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault, textureCache, pixelBuffer, NULL, MTLPixelFormatGBGR422, width, height, 1, &outTexture);
*/
CVMetalTextureRef tmpTexture = NULL;
CVReturn status = CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault, self.textureCache, pixelBuffer, NULL, MTLPixelFormatBGRA8Unorm, width, height, 0, &tmpTexture);
//4.判斷tmpTexture 是否創(chuàng)建成功
if(status == kCVReturnSuccess)
{
//5.設置可繪制紋理的當前大小焚虱。
self.mtkView.drawableSize = CGSizeMake(width, height);
//6.返回紋理緩沖區(qū)的Metal紋理對象购裙。
self.texture = CVMetalTextureGetTexture(tmpTexture);
//7.使用完畢,則釋放tmpTexture
CFRelease(tmpTexture);
}
MTKView的代理方法
-
步驟圖解
drawInMTKView方法中的處理.png - 具體代碼
//1.判斷是否獲取了AVFoundation 采集的紋理數(shù)據(jù)
if (self.texture) {
//2.創(chuàng)建指令緩沖
id<MTLCommandBuffer> commandBuffer = [self.commandQueue commandBuffer];
//3.將MTKView 作為目標渲染紋理
id<MTLTexture> drawingTexture = view.currentDrawable.texture;
//4.設置濾鏡
/*
MetalPerformanceShaders是Metal的一個集成庫私股，有一些濾鏡處理的Metal實現(xiàn);
MPSImageGaussianBlur 高斯模糊處理;
*/
//創(chuàng)建高斯濾鏡處理filter
//注意:sigma值可以修改黄娘，sigma值越高圖像越模糊;
MPSImageGaussianBlur *filter = [[MPSImageGaussianBlur alloc] initWithDevice:self.mtkView.device sigma:1];
//5.MPSImageGaussianBlur以一個Metal紋理作為輸入誓焦，以一個Metal紋理作為輸出杂伟；
//輸入:攝像頭采集的圖像 self.texture
//輸出:創(chuàng)建的紋理 drawingTexture(其實就是view.currentDrawable.texture)
[filter encodeToCommandBuffer:commandBuffer sourceTexture:self.texture destinationTexture:drawingTexture];
//6.展示顯示的內(nèi)容
[commandBuffer presentDrawable:view.currentDrawable];
//7.提交命令
[commandBuffer commit];
//8.清空當前紋理,準備下一次的紋理數(shù)據(jù)讀取.
self.texture = NULL;
}
