Metal 渲染視頻

Metal實時處理攝像頭采集內(nèi)容仅胞，添加高斯模糊效果。demo

未命名.gif

1、AVFoundation視頻采集

視頻采集

創(chuàng)建負(fù)責(zé)管理輸入、輸出設(shè)備數(shù)據(jù)傳遞對象AVCaptureSession

   //1.創(chuàng)建mCaptureSession
    self.mCaptureSession = [[AVCaptureSession alloc] init];
    //設(shè)置視頻采集的分辨率
    self.mCaptureSession.sessionPreset = AVCaptureSessionPreset1920x1080;

輸入設(shè)備對象AVCaptureDeviceInput
1初厚、獲取攝相頭設(shè)備
2、轉(zhuǎn)化為輸入對象
3、輸入對象添加到負(fù)責(zé)管理輸入产禾、輸出的對象中

   //3.獲取攝像頭設(shè)備(前置/后置攝像頭設(shè)備)
    NSArray *devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo];
    AVCaptureDevice *inputCamera = nil;
    //循環(huán)設(shè)備數(shù)組,找到后置攝像頭.設(shè)置為當(dāng)前inputCamera
    for (AVCaptureDevice *device in devices) {
        if ([device position] == AVCaptureDevicePositionBack) {
            inputCamera = device;
        }
    }
    
    //4.將AVCaptureDevice 轉(zhuǎn)換為AVCaptureDeviceInput
    self.mCaptureDeviceInput = [[AVCaptureDeviceInput alloc] initWithDevice:inputCamera error:nil];
    
    //5. 將設(shè)備添加到mCaptureSession中
    if ([self.mCaptureSession canAddInput:self.mCaptureDeviceInput]) {
        [self.mCaptureSession addInput:self.mCaptureDeviceInput];
    }

輸出設(shè)備AVCaptureVideoDataOutput
1排作、創(chuàng)建輸出對象
2、設(shè)置視頻幀延遲時是否掉幀
3亚情、設(shè)置視頻像素顏色格式
4妄痪、設(shè)置視頻捕捉輸出的代理方法
5、將輸出對象添加至負(fù)責(zé)管理輸入楞件、輸出的對象中

    //創(chuàng)建串行隊列, 處理captureSession交互時衫生，不影響主隊列
    self.mProcessQueue = dispatch_queue_create("mProcessQueue", DISPATCH_QUEUE_SERIAL);

    //6.創(chuàng)建AVCaptureVideoDataOutput 對象
    self.mCaptureDeviceOutput = [[AVCaptureVideoDataOutput alloc] init];
    
    /*設(shè)置視頻幀延遲到底時是否丟棄數(shù)據(jù).
     YES: 處理現(xiàn)有幀的調(diào)度隊列在captureOutput:didOutputSampleBuffer:FromConnection:Delegate方法中被阻止時，對象會立即丟棄捕獲的幀土浸。
     NO: 在丟棄新幀之前罪针，允許委托有更多的時間處理舊幀，但這樣可能會內(nèi)存增加.
     */
    [self.mCaptureDeviceOutput setAlwaysDiscardsLateVideoFrames:NO];
    
    //這里設(shè)置格式為BGRA栅迄，而不用YUV的顏色空間站故，避免使用Shader轉(zhuǎn)換
    //注意:這里必須和后面CVMetalTextureCacheCreateTextureFromImage 保存圖像像素存儲格式保持一致.否則視頻會出現(xiàn)異常現(xiàn)象.
    [self.mCaptureDeviceOutput setVideoSettings:[NSDictionary dictionaryWithObject:[NSNumber numberWithInt:kCVPixelFormatType_32BGRA] forKey:(id)kCVPixelBufferPixelFormatTypeKey]];
    
    //設(shè)置視頻捕捉輸出的代理方法
    [self.mCaptureDeviceOutput setSampleBufferDelegate:self queue:self.mProcessQueue];
    
    //7.添加輸出
    if ([self.mCaptureSession canAddOutput:self.mCaptureDeviceOutput]) {
        [self.mCaptureSession addOutput:self.mCaptureDeviceOutput];
    }

輸入與輸出鏈接

//8.輸入與輸出鏈接
    AVCaptureConnection *connection = [self.mCaptureDeviceOutput connectionWithMediaType:AVMediaTypeVideo];
    
    //9.設(shè)置視頻方向
    //注意: 一定要設(shè)置視頻方向.否則視頻會是朝向異常的.
    [connection setVideoOrientation:AVCaptureVideoOrientationPortrait];
    
    //10.開始捕捉
    [self.mCaptureSession startRunning];

視頻采集回調(diào)
1毅舆、獲取視頻像素緩沖區(qū)對象(位圖)
2西篓、獲取視頻幀寬高
3、根據(jù)視頻像素緩存區(qū) 創(chuàng)建 Metal 紋理緩存區(qū)(位圖轉(zhuǎn)化Metal紋理)
4憋活、設(shè)置可繪制紋理的當(dāng)前大小
5岂津、返回紋理緩沖區(qū)的Metal紋理對象
6、使用完畢悦即，釋放紋理緩存區(qū)

//AVFoundation 視頻采集回調(diào)方法
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
    
    //1.從sampleBuffer 獲取視頻像素緩存區(qū)對象
    CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
   
    //2.獲取捕捉視頻的寬和高
    size_t width = CVPixelBufferGetWidth(pixelBuffer);
    size_t height = CVPixelBufferGetHeight(pixelBuffer);
    
    /*3. 根據(jù)視頻像素緩存區(qū) 創(chuàng)建 Metal 紋理緩存區(qū)
     CVReturn CVMetalTextureCacheCreateTextureFromImage(CFAllocatorRef allocator,                         CVMetalTextureCacheRef textureCache,
     CVImageBufferRef sourceImage,
     CFDictionaryRef textureAttributes,
     MTLPixelFormat pixelFormat,
     size_t width,
     size_t height,
     size_t planeIndex,
     CVMetalTextureRef  *textureOut);
     
     功能: 從現(xiàn)有圖像緩沖區(qū)創(chuàng)建核心視頻Metal紋理緩沖區(qū)吮成。
     參數(shù)1: allocator 內(nèi)存分配器,默認(rèn)kCFAllocatorDefault
     參數(shù)2: textureCache 紋理緩存區(qū)對象
     參數(shù)3: sourceImage 視頻圖像緩沖區(qū)
     參數(shù)4: textureAttributes 紋理參數(shù)字典.默認(rèn)為NULL
     參數(shù)5: pixelFormat 圖像緩存區(qū)數(shù)據(jù)的Metal 像素格式常量.注意如果MTLPixelFormatBGRA8Unorm和攝像頭采集時設(shè)置的顏色格式不一致，則會出現(xiàn)圖像異常的情況辜梳；
     參數(shù)6: width,紋理圖像的寬度（像素）
     參數(shù)7: height,紋理圖像的高度（像素）
     參數(shù)8: planeIndex.如果圖像緩沖區(qū)是平面的粱甫，則為映射紋理數(shù)據(jù)的平面索引。對于非平面圖像緩沖區(qū)忽略作瞄。
     參數(shù)9: textureOut,返回時茶宵，返回創(chuàng)建的Metal紋理緩沖區(qū)。
     
     // Mapping a BGRA buffer:
     CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault, textureCache, pixelBuffer, NULL, MTLPixelFormatBGRA8Unorm, width, height, 0, &outTexture);
     
     // Mapping the luma plane of a 420v buffer:
     CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault, textureCache, pixelBuffer, NULL, MTLPixelFormatR8Unorm, width, height, 0, &outTexture);
     
     // Mapping the chroma plane of a 420v buffer as a source texture:
     CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault, textureCache, pixelBuffer, NULL, MTLPixelFormatRG8Unorm width/2, height/2, 1, &outTexture);
     
     // Mapping a yuvs buffer as a source texture (note: yuvs/f and 2vuy are unpacked and resampled -- not colorspace converted)
     CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault, textureCache, pixelBuffer, NULL, MTLPixelFormatGBGR422, width, height, 1, &outTexture);
     
     */
    CVMetalTextureRef tmpTexture = NULL;
    CVReturn status = CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault, self.textureCache, pixelBuffer, NULL, MTLPixelFormatBGRA8Unorm, width, height, 0, &tmpTexture);
    
    //4.判斷tmpTexture 是否創(chuàng)建成功
    if(status == kCVReturnSuccess)
    {
        //5.設(shè)置可繪制紋理的當(dāng)前大小宗挥。
        self.mtkView.drawableSize = CGSizeMake(width, height);
        //6.返回紋理緩沖區(qū)的Metal紋理對象乌庶。
        self.texture = CVMetalTextureGetTexture(tmpTexture);
        //7.使用完畢,則釋放tmpTexture
        CFRelease(tmpTexture);
    }
}

2、MTKView渲染

MTKView渲染

初始化MTKView對象
1契耿、指定當(dāng)前設(shè)備device(GPU)
2瞒大、創(chuàng)建命令隊列
3、設(shè)置MTKView對象允許讀寫
4搪桂、創(chuàng)建紋理緩存區(qū)

   //1.獲取MTKView
    self.mtkView = [[MTKView alloc] initWithFrame:self.view.bounds];
    self.mtkView.device = MTLCreateSystemDefaultDevice();
    [self.view insertSubview:self.mtkView atIndex:0];
    self.mtkView.delegate = self;

    //2.創(chuàng)建命令隊列.
    self.commandQueue = [self.mtkView.device newCommandQueue];

    //3.注意: 在初始化MTKView 的基本操作以外. 還需要多下面2行代碼.
    /*
     1. 設(shè)置MTKView 的drawable 紋理是可讀寫的(默認(rèn)是只讀);
     2. 創(chuàng)建CVMetalTextureCacheRef _textureCache; 這是Core Video的Metal紋理緩存
     */
    //允許讀寫操作
    self.mtkView.framebufferOnly = NO;
    /*
     CVMetalTextureCacheCreate(CFAllocatorRef  allocator,
     CFDictionaryRef cacheAttributes,
     id <MTLDevice>  metalDevice,
     CFDictionaryRef  textureAttributes,
     CVMetalTextureCacheRef * CV_NONNULL cacheOut )

     功能: 創(chuàng)建紋理緩存區(qū)
     參數(shù)1: allocator 內(nèi)存分配器.默認(rèn)即可.NULL
     參數(shù)2: cacheAttributes 緩存區(qū)行為字典.默認(rèn)為NULL
     參數(shù)3: metalDevice
     參數(shù)4: textureAttributes 緩存創(chuàng)建紋理選項的字典. 使用默認(rèn)選項NULL
     參數(shù)5: cacheOut 返回時透敌，包含新創(chuàng)建的紋理緩存。

     */
    CVMetalTextureCacheCreate(NULL, NULL, self.mtkView.device, NULL, &_textureCache);

MTKView視圖渲染代理方法
1、創(chuàng)建命令緩沖區(qū)
2酗电、將MTKView作為目標(biāo)渲染紋理
3淌山、設(shè)置濾鏡
4、將濾鏡以Metal紋理作為輸入顾瞻、輸出
5、展示顯示內(nèi)容
6德绿、提交命令緩沖區(qū)
7荷荤、釋放當(dāng)前紋理、準(zhǔn)備下一次紋理數(shù)據(jù)讀取

//視圖渲染則會調(diào)用此方法
- (void)drawInMTKView:(MTKView *)view {
  
    //1.判斷是否獲取了AVFoundation 采集的紋理數(shù)據(jù)
    if (self.texture) {
        
        //2.創(chuàng)建命令緩沖區(qū)
        id<MTLCommandBuffer> commandBuffer = [self.commandQueue commandBuffer];
        
        //3.將MTKView 作為目標(biāo)渲染紋理
        id<MTLTexture> drawingTexture = view.currentDrawable.texture;
        
        //4.設(shè)置濾鏡
        /*
         MetalPerformanceShaders是Metal的一個集成庫移稳，有一些濾鏡處理的Metal實現(xiàn);
         MPSImageGaussianBlur 高斯模糊處理;
         */
       
        //創(chuàng)建高斯濾鏡處理filter
        //注意:sigma值可以修改蕴纳，sigma值越高圖像越模糊;
        MPSImageGaussianBlur *filter = [[MPSImageGaussianBlur alloc] initWithDevice:self.mtkView.device sigma:10];
        
        //5.MPSImageGaussianBlur以一個Metal紋理作為輸入，以一個Metal紋理作為輸出个粱；
        //輸入:攝像頭采集的圖像 self.texture
        //輸出:創(chuàng)建的紋理 drawingTexture(其實就是view.currentDrawable.texture)
        [filter encodeToCommandBuffer:commandBuffer sourceTexture:self.texture destinationTexture:drawingTexture];
        
        //6.展示顯示的內(nèi)容
        [commandBuffer presentDrawable:view.currentDrawable];
        
        //7.提交命令
        [commandBuffer commit];
        
        //8.清空當(dāng)前紋理,準(zhǔn)備下一次的紋理數(shù)據(jù)讀取.
        self.texture = NULL;
    }
}