目前主要的編碼方式為h264,h265雖然更好舞蔽，但是ios11以上才支持，并且cpu負(fù)荷比較大

• 硬編碼：基于GPU

  • 視頻：VideoToolBox
  • 音頻：AudioToolBox

• 軟編碼：基于CPU

  • 視頻壓縮：視頻編碼MPEG码撰，H264
    X264把視頻原數(shù)據(jù)YUV/RGB編碼H264
  • 音頻：AudioToolBox
    fdk_aac將音頻數(shù)據(jù)PCM轉(zhuǎn)AAC

H264基本概念.

I幀: 關(guān)鍵幀,采用幀內(nèi)壓縮技術(shù).

• 舉個(gè)例子,如果攝像頭對(duì)著你拍攝,1秒之內(nèi),實(shí)際你發(fā)生的變化是非常少的.1秒鐘之內(nèi)實(shí)際少很少有大幅度的變化.攝像機(jī)一般一秒鐘會(huì)抓取幾十幀的數(shù)據(jù).比如像動(dòng)畫,就是25幀/s,一般視頻文件都是在30幀/s左右.對(duì)于一些要求比較高的,對(duì)動(dòng)作的精細(xì)度有要求,想要捕捉到完整的動(dòng)作的,高級(jí)的攝像機(jī)一般是60幀/s.那些對(duì)于一組幀的它的變化很小.為了便于壓縮數(shù)據(jù),那怎么辦了?將第一幀完整的保存下來.如果沒有這個(gè)關(guān)鍵幀后面解碼數(shù)據(jù),是完成不了的.所以I幀特別關(guān)鍵.

P幀: 向前參考幀.壓縮時(shí)只參考前一個(gè)幀.屬于幀間壓縮技術(shù).

• 視頻的第一幀會(huì)被作為關(guān)鍵幀完整保存下來.而后面的幀會(huì)向前依賴.也就是第二幀依賴于第一個(gè)幀.后面所有的幀只存儲(chǔ)于前一幀的差異.這樣就能將數(shù)據(jù)大大的減少.從而達(dá)到一個(gè)高壓縮率的效果.

B幀: 雙向參考幀,壓縮時(shí)即參考前一幀也參考后一幀.幀間壓縮技術(shù).

• B幀,即參考前一幀,也參考后一幀.這樣就使得它的壓縮率更高.存儲(chǔ)的數(shù)據(jù)量更小.如果B幀的數(shù)量越多,你的壓縮率就越高.這是B幀的優(yōu)點(diǎn),但是B幀最大的缺點(diǎn)是,如果是實(shí)時(shí)互動(dòng)的直播,那時(shí)與B幀就要參考后面的幀才能解碼,那在網(wǎng)絡(luò)中就要等待后面的幀傳輸過來.這就與網(wǎng)絡(luò)有關(guān)了.如果網(wǎng)絡(luò)狀態(tài)很好的話,解碼會(huì)比較快,如果網(wǎng)絡(luò)不好時(shí)解碼會(huì)稍微慢一些.丟包時(shí)還需要重傳.對(duì)實(shí)時(shí)互動(dòng)的直播,一般不會(huì)使用B幀.
• 如果在泛娛樂的直播中,可以接受一定度的延時(shí),需要比較高的壓縮比就可以使用B幀.
• 如果我們?cè)趯?shí)時(shí)互動(dòng)的直播,我們需要提高時(shí)效性,這時(shí)就不能使用B幀了.

二. GOF(Group of Frame)一組幀

兩個(gè)I幀之間形成的一組圖片,就是GOP(Group of Picture).
通常在編碼器設(shè)置參數(shù)時(shí)渗柿，必須會(huì)設(shè)置gop_ size 的值其實(shí)就是代表2個(gè)|幀之間的幀數(shù)目.在一個(gè)GOP組中容量最大的就是I幀.所以相對(duì)而言, gop_ size 設(shè)置的越大，整個(gè)視頻畫面質(zhì)量就會(huì)越好.但是解碼端必須從接收的第一個(gè)|幀開始才可以正確解碼出原始圖像.否則無法正確解碼.

image

SPS/PPS

SPS/PPS實(shí)際上就是存儲(chǔ)GOP的參數(shù).

SPS: (Sequence Parameter Set,序列參數(shù)集)存放幀數(shù),參考幀數(shù)目,解碼圖像尺寸,幀場(chǎng)編碼模式選擇標(biāo)識(shí)等.

• 一組幀的參數(shù)集.

PPS:(Picture Parameter Set,圖像參數(shù)集).存放熵編碼模式選擇標(biāo)識(shí),片組數(shù)目,初始量化參數(shù)和去方塊濾波系數(shù)調(diào)整標(biāo)識(shí)等.(與圖像相關(guān)的信息)

在一組幀之前我們首先收到的是SPS/PPS數(shù)據(jù).如果沒有這組參數(shù)的話,我們是無法解碼.

如果我們?cè)诮獯a時(shí)發(fā)生錯(cuò)誤,首先要檢查是否有SPS/PPS.如果沒有,是因?yàn)閷?duì)端沒有發(fā)送過來還是因?yàn)閷?duì)端在發(fā)送過程中丟失了.

SPS/PPS數(shù)據(jù),我們也把其歸類到I幀.這2組數(shù)據(jù)是絕對(duì)不能丟的.

那么下面我們來看一下實(shí)際開發(fā)中遇到的問題.

視頻花屏/卡頓原因

我們?cè)谟^看視頻時(shí),會(huì)遇到花屏或者卡頓現(xiàn)象.那這個(gè)與我們剛剛所講的GOF就息息相關(guān)了.

• 如果GOP分組中的P幀丟失就會(huì)造成解碼端的圖像發(fā)生錯(cuò)誤.
• 為了避免花屏問題的發(fā)生,一般如果發(fā)現(xiàn)P幀或者I幀丟失.就不顯示本GOP內(nèi)的所有幀.只到下一個(gè)I幀來后重新刷新圖像.
• 當(dāng)這時(shí)因?yàn)闆]有刷新屏幕.丟包的這一組幀全部扔掉了.圖像就會(huì)卡在哪里不動(dòng).這就是卡頓的原因.

所以總結(jié)起來,花屏是因?yàn)槟銇G了P幀或者I幀.導(dǎo)致解碼錯(cuò)誤. 而卡頓是因?yàn)闉榱伺禄ㄆ?將整組錯(cuò)誤的GOP數(shù)據(jù)扔掉了.直達(dá)下一組正確的GOP再重新刷屏.而這中間的時(shí)間差,就是我們所感受的卡頓.

VideoToolBox

在iOS4.0脖岛，蘋果就已經(jīng)支持硬編解碼但是硬編解碼在當(dāng)時(shí)屬于私有API.不提供給開發(fā)者使用在2014年的WWDC大會(huì)上,iOS 8.0之后朵栖，蘋果開放了硬編解碼的APl。就是VideoToolbox. framework的API柴梆。VideoToolbox 是一套純C語言API陨溅。其中包含了很多C語言函數(shù). VideoToolbox . framework是基于Core Foundation庫函數(shù),基于C語言

VideoToolBox框架的流程

• 創(chuàng)建session
• 設(shè)置編碼相關(guān)參數(shù)
• 開始編碼
• 循環(huán)獲取采集數(shù)據(jù)
• 獲取編碼后數(shù)據(jù).
• 將數(shù)據(jù)寫入H264文件

編碼的輸入與輸出.png

h264編碼采集

#import <AVFoundation/AVFoundation.h>
#import <VideoToolbox/VideoToolbox.h>


@interface ViewController ()<AVCaptureVideoDataOutputSampleBufferDelegate>

@property(nonatomic,strong)UILabel *cLabel;
@property(nonatomic,strong)AVCaptureSession *cCapturesession;//捕捉會(huì)話，用于輸入輸出設(shè)備之間的數(shù)據(jù)傳遞
@property(nonatomic,strong)AVCaptureDeviceInput *cCaptureDeviceInput;//捕捉輸入
@property(nonatomic,strong)AVCaptureVideoDataOutput *cCaptureDataOutput;//捕捉輸出
@property(nonatomic,strong)AVCaptureVideoPreviewLayer *cPreviewLayer;//預(yù)覽圖層

@end

@implementation ViewController
{
    int  frameID; //幀ID
    dispatch_queue_t cCaptureQueue; //捕獲隊(duì)列
    dispatch_queue_t cEncodeQueue;  //編碼隊(duì)列
    VTCompressionSessionRef cEncodeingSession;//編碼session
    CMFormatDescriptionRef format; //編碼格式
    NSFileHandle *fileHandele;
    
    
}

- (void)viewDidLoad {
    [super viewDidLoad];
    // Do any additional setup after loading the view, typically from a nib.
    
    //基礎(chǔ)UI實(shí)現(xiàn)
    _cLabel = [[UILabel alloc]initWithFrame:CGRectMake(20, 20, 200, 100)];
    _cLabel.text = @"cc課堂之H.264硬編碼";
    _cLabel.textColor = [UIColor redColor];
    [self.view addSubview:_cLabel];
    
    UIButton *cButton = [[UIButton alloc]initWithFrame:CGRectMake(200, 20, 100, 100)];
    [cButton setTitle:@"play" forState:UIControlStateNormal];
    [cButton setTitleColor:[UIColor whiteColor] forState:UIControlStateNormal];
    [cButton setBackgroundColor:[UIColor orangeColor]];
    [cButton addTarget:self action:@selector(buttonClick:) forControlEvents:UIControlEventTouchUpInside];
    [self.view addSubview:cButton];
    

    
}


-(void)buttonClick:(UIButton *)button
{
    
    //判斷_cCapturesession 和 _cCapturesession是否正在捕捉
    if (!_cCapturesession || !_cCapturesession.isRunning ) {
        
        //修改按鈕狀態(tài)
        [button setTitle:@"Stop" forState:UIControlStateNormal];
        
        //開始捕捉
        [self startCapture];
        
        
    }else
    {
        [button setTitle:@"Play" forState:UIControlStateNormal];
        
        //停止捕捉
        [self stopCapture];
    }
    
}

//開始捕捉
- (void)startCapture
{
    self.cCapturesession = [[AVCaptureSession alloc]init];
    
    //設(shè)置捕捉分辨率
    self.cCapturesession.sessionPreset = AVCaptureSessionPreset640x480;
    
    //使用函數(shù)dispath_get_global_queue去得到隊(duì)列
    cCaptureQueue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
    cEncodeQueue  = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
    
    AVCaptureDevice *inputCamera = nil;
    //獲取iPhone視頻捕捉的設(shè)備轩性，例如前置攝像頭声登、后置攝像頭......
    NSArray *devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo];
    for (AVCaptureDevice *device in devices) {
        
        //拿到后置攝像頭
        if ([device position] == AVCaptureDevicePositionBack) {
            
            inputCamera = device;
        }
    }
    
    //將捕捉設(shè)備 封裝成 AVCaptureDeviceInput 對(duì)象
    self.cCaptureDeviceInput = [[AVCaptureDeviceInput alloc]initWithDevice:inputCamera error:nil];
    
    //判斷是否能加入后置攝像頭作為輸入設(shè)備
    if ([self.cCapturesession canAddInput:self.cCaptureDeviceInput]) {
        
        //將設(shè)備添加到會(huì)話中
        [self.cCapturesession addInput:self.cCaptureDeviceInput];
        
        
    }
    
    
    //配置輸出
    self.cCaptureDataOutput = [[AVCaptureVideoDataOutput alloc]init];
    
    //設(shè)置丟棄最后的video frame 為NO
    [self.cCaptureDataOutput setAlwaysDiscardsLateVideoFrames:NO];
    
    //設(shè)置video的視頻捕捉的像素點(diǎn)壓縮方式為 420
    [self.cCaptureDataOutput setVideoSettings:[NSDictionary dictionaryWithObject:[NSNumber numberWithInt:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange] forKey:(id)kCVPixelBufferPixelFormatTypeKey]];
    
    //設(shè)置捕捉代理 和 捕捉隊(duì)列
    [self.cCaptureDataOutput setSampleBufferDelegate:self queue:cCaptureQueue];
    
    //判斷是否能添加輸出
    if ([self.cCapturesession canAddOutput:self.cCaptureDataOutput]) {
        
        //添加輸出
        [self.cCapturesession addOutput:self.cCaptureDataOutput];
    }
    
    //創(chuàng)建連接
    AVCaptureConnection *connection = [self.cCaptureDataOutput connectionWithMediaType:AVMediaTypeVideo];
    
    //設(shè)置連接的方向
    [connection setVideoOrientation:AVCaptureVideoOrientationPortrait];
    
    //初始化圖層
    self.cPreviewLayer = [[AVCaptureVideoPreviewLayer alloc]initWithSession:self.cCapturesession];
    
    //設(shè)置視頻重力
    [self.cPreviewLayer setVideoGravity:AVLayerVideoGravityResizeAspect];
    
    //設(shè)置圖層的frame
    [self.cPreviewLayer setFrame:self.view.bounds];
    
    //添加圖層
    [self.view.layer addSublayer:self.cPreviewLayer];
    
    
    //文件寫入沙盒
    NSString *filePath = [[NSSearchPathForDirectoriesInDomains(NSDocumentDirectory,NSUserDomainMask,YES)lastObject]stringByAppendingPathComponent:@"cc_video.h264"];
   // NSString *filePath = [NSHomeDirectory()stringByAppendingPathComponent:@"/Documents/cc_video.h264"];
    
    
    //先移除已存在的文件
    [[NSFileManager defaultManager] removeItemAtPath:filePath error:nil];
    
    //新建文件
    BOOL createFile = [[NSFileManager defaultManager] createFileAtPath:filePath contents:nil attributes:nil];
    if (!createFile) {
        
        NSLog(@"create file failed");
    }else
    {
        NSLog(@"create file success");

    }
    
    NSLog(@"filePaht = %@",filePath);
    fileHandele = [NSFileHandle fileHandleForWritingAtPath:filePath];
    
    
    //初始化videoToolbBox
    [self initVideoToolBox];
    
    //開始捕捉
    [self.cCapturesession startRunning];
    
    
    
    
}


//停止捕捉
- (void)stopCapture
{
    //停止捕捉
    [self.cCapturesession stopRunning];
    
    //移除預(yù)覽圖層
    [self.cPreviewLayer removeFromSuperlayer];
    
    //結(jié)束videoToolbBox
    [self endVideoToolBox];
    
    //關(guān)閉文件
    [fileHandele closeFile];
    
    fileHandele = NULL;
    
}

#pragma mark - AVCaptureVideoDataOutputSampleBufferDelegate
//AV Foundation 獲取到視頻流
-(void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
    //開始視頻錄制狠鸳，獲取到攝像頭的視頻幀，傳入encode 方法中
    dispatch_sync(cEncodeQueue, ^{
        [self encode:sampleBuffer];
    });
    
}



//初始化videoToolBox
-(void)initVideoToolBox
{
    dispatch_sync(cEncodeQueue, ^{
        
        frameID = 0;
        
        int width = 480,height = 640;
        
        //1.調(diào)用VTCompressionSessionCreate創(chuàng)建編碼session
        //參數(shù)1：NULL 分配器,設(shè)置NULL為默認(rèn)分配
        //參數(shù)2：width悯嗓，像素為單位件舵，如果此數(shù)據(jù)非法，編碼會(huì)改為合理的值
        //參數(shù)3：height
        //參數(shù)4：編碼類型,如kCMVideoCodecType_H264
        //參數(shù)5：NULL encoderSpecification: 編碼規(guī)范脯厨。設(shè)置NULL由videoToolbox自己選擇
        //參數(shù)6：NULL sourceImageBufferAttributes: 源像素緩沖區(qū)屬性.設(shè)置NULL不讓videToolbox創(chuàng)建,而自己創(chuàng)建
        //參數(shù)7：NULL compressedDataAllocator: 壓縮數(shù)據(jù)分配器.設(shè)置NULL,默認(rèn)的分配
        //參數(shù)8：回調(diào)  當(dāng)VTCompressionSessionEncodeFrame被調(diào)用壓縮一次后會(huì)被異步調(diào)用.注:當(dāng)你設(shè)置NULL的時(shí)候,你需要調(diào)用VTCompressionSessionEncodeFrameWithOutputHandler方法進(jìn)行壓縮幀處理,支持iOS9.0以上
        //參數(shù)9：outputCallbackRefCon: 回調(diào)客戶定義的參考值
        //參數(shù)10：compressionSessionOut: 編碼會(huì)話變量
        OSStatus status = VTCompressionSessionCreate(NULL, width, height, kCMVideoCodecType_H264, NULL, NULL, NULL, didCompressH264, (__bridge void *)(self), &cEncodeingSession);
        
        NSLog(@"H264:VTCompressionSessionCreate:%d",(int)status);
        
        if (status != 0) {
            
            NSLog(@"H264:Unable to create a H264 session");
            return ;
        }
        /*
        VTSessionSetProperty(VTSessionRef  _Nonnull session, CFStringRef  _Nonnull propertyKey, CFTypeRef  _Nullable propertyValue)
         * 參數(shù)設(shè)置對(duì)象 cEncodeingSession
         
         */
        //設(shè)置實(shí)時(shí)編碼輸出（避免延遲）
        VTSessionSetProperty(cEncodeingSession, kVTCompressionPropertyKey_RealTime, kCFBooleanTrue);
        
        //舍棄B幀
        VTSessionSetProperty(cEncodeingSession, kVTCompressionPropertyKey_ProfileLevel,kVTProfileLevel_H264_Baseline_AutoLevel);
        
        //是否產(chǎn)生B幀(因?yàn)锽幀在解碼時(shí)并不是必要的,是可以拋棄B幀的)
        VTSessionSetProperty(cEncodeingSession, kVTCompressionPropertyKey_AllowFrameReordering, kCFBooleanFalse);
        
        //設(shè)置關(guān)鍵幀（GOPsize）間隔铅祸，GOP太小的話圖像會(huì)模糊，太大視頻體積增大
        int frameInterval = 10;
        //VTSessionSetProperty 不能直接設(shè)置int/float 作為屬性值
        /*
        CFNumberCreate(CFAllocatorRef allocator, CFNumberType theType, const void *valuePtr)
         * allocator : 分配器合武，一般默認(rèn)kCFAllocatorDefault
         * theType : 數(shù)據(jù)類型
         * *valuePtr : 地址
         */
        CFNumberRef frameIntervalRaf = CFNumberCreate(kCFAllocatorDefault, kCFNumberIntType, &frameInterval);
        VTSessionSetProperty(cEncodeingSession, kVTCompressionPropertyKey_MaxKeyFrameInterval, frameIntervalRaf);
        
        //設(shè)置期望幀率临梗，不是實(shí)際幀率
        int fps = 10;
        CFNumberRef fpsRef = CFNumberCreate(kCFAllocatorDefault, kCFNumberIntType, &fps);
        VTSessionSetProperty(cEncodeingSession, kVTCompressionPropertyKey_ExpectedFrameRate, fpsRef);
        
        //碼率的理解：碼率大了話就會(huì)非常清晰，但同時(shí)文件也會(huì)比較大稼跳。碼率小的話盟庞，圖像有時(shí)會(huì)模糊，但也勉強(qiáng)能看
        //碼率計(jì)算公式汤善，參考印象筆記
        //設(shè)置碼率什猖、上限、單位是bps
        int bitRate = width * height * 3 * 4 * 8;
        CFNumberRef bitRateRef = CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, &bitRate);
        VTSessionSetProperty(cEncodeingSession, kVTCompressionPropertyKey_AverageBitRate, bitRateRef);
        
        //設(shè)置碼率红淡，均值不狮，單位是byte
        int bigRateLimit = width * height * 3 * 4;
        CFNumberRef bitRateLimitRef = CFNumberCreate(kCFAllocatorDefault, kCFNumberSInt32Type, &bigRateLimit);
        VTSessionSetProperty(cEncodeingSession, kVTCompressionPropertyKey_DataRateLimits, bitRateLimitRef);
        
        //開始編碼
        VTCompressionSessionPrepareToEncodeFrames(cEncodeingSession);
        
        
    });
    
}



- (void) encode:(CMSampleBufferRef )sampleBuffer
{
    //拿到每一幀未編碼數(shù)據(jù)
    CVImageBufferRef imageBuffer = (CVImageBufferRef)CMSampleBufferGetImageBuffer(sampleBuffer);
    
    //設(shè)置幀時(shí)間，如果不設(shè)置會(huì)導(dǎo)致時(shí)間軸過長在旱。
    CMTime presentationTimeStamp = CMTimeMake(frameID++, 1000);
    //同步摇零，異步
    VTEncodeInfoFlags flags;
    
    //參數(shù)1：編碼會(huì)話變量
    //參數(shù)2：未編碼數(shù)據(jù)
    //參數(shù)3：獲取到的這個(gè)sample buffer數(shù)據(jù)的展示時(shí)間戳。每一個(gè)傳給這個(gè)session的時(shí)間戳都要大于前一個(gè)展示時(shí)間戳.
    //參數(shù)4：對(duì)于獲取到sample buffer數(shù)據(jù),這個(gè)幀的展示時(shí)間.如果沒有時(shí)間信息,可設(shè)置kCMTimeInvalid.
    //參數(shù)5：frameProperties: 包含這個(gè)幀的屬性.幀的改變會(huì)影響后邊的編碼幀.一般為null
    //參數(shù)6：ourceFrameRefCon: 回調(diào)函數(shù)會(huì)引用你設(shè)置的這個(gè)幀的參考值. null
    //參數(shù)7：infoFlagsOut: 指向一個(gè)VTEncodeInfoFlags來接受一個(gè)編碼操作.如果使用異步運(yùn)行,kVTEncodeInfo_Asynchronous被設(shè)置桶蝎；同步運(yùn)行,kVTEncodeInfo_FrameDropped被設(shè)置驻仅；設(shè)置NULL為不想接受這個(gè)信息.
    
    OSStatus statusCode = VTCompressionSessionEncodeFrame(cEncodeingSession, imageBuffer, presentationTimeStamp, kCMTimeInvalid, NULL, NULL, &flags);
    
    if (statusCode != noErr) {
        
        NSLog(@"H.264:VTCompressionSessionEncodeFrame faild with %d",(int)statusCode);
        
        VTCompressionSessionInvalidate(cEncodeingSession);
        CFRelease(cEncodeingSession);
        cEncodeingSession = NULL;
        return;
    }
    
    NSLog(@"H264:VTCompressionSessionEncodeFrame Success");
    
}


//編碼完成回調(diào)
/*
    1.H264硬編碼完成后，回調(diào)VTCompressionOutputCallback
    2.將硬編碼成功的CMSampleBuffer轉(zhuǎn)換成H264碼流俊嗽，通過網(wǎng)絡(luò)傳播
    3.解析出參數(shù)集SPS & PPS雾家，加上開始碼組裝成 NALU铃彰。提現(xiàn)出視頻數(shù)據(jù)绍豁，將長度碼轉(zhuǎn)換為開始碼，組成NALU牙捉，將NALU發(fā)送出去竹揍。
 */
void didCompressH264(void *outputCallbackRefCon, void *sourceFrameRefCon, OSStatus status, VTEncodeInfoFlags infoFlags, CMSampleBufferRef sampleBuffer)
{
    NSLog(@"didCompressH264 called with status %d infoFlags %d",(int)status,(int)infoFlags);
    //狀態(tài)錯(cuò)誤
    if (status != 0) {
        
        return;
    }
    
    //沒準(zhǔn)備好
    if (!CMSampleBufferDataIsReady(sampleBuffer)) {
        
        NSLog(@"didCompressH264 data is not ready");
        return;
        
    }
    //C轉(zhuǎn)OC
    ViewController *encoder = (__bridge ViewController *)outputCallbackRefCon;
    
    //判斷當(dāng)前幀是否為關(guān)鍵幀
    /* 分步驟判斷
    CFArrayRef array = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, true);
    CFDictionaryRef dic = CFArrayGetValueAtIndex(array, 0);
    bool isKeyFrame = !CFDictionaryContainsKey(dic, kCMSampleAttachmentKey_NotSync);
    */
    bool keyFrame = !CFDictionaryContainsKey((CFArrayGetValueAtIndex(CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, true), 0)), kCMSampleAttachmentKey_NotSync);
    
    //判斷當(dāng)前幀是否為關(guān)鍵幀
    //獲取sps & pps 數(shù)據(jù) 只獲取1次，保存在h264文件開頭的第一幀中
    //sps(sample per second 采樣次數(shù)/s),是衡量模數(shù)轉(zhuǎn)換（ADC）時(shí)采樣速率的單位邪铲，幀的參數(shù)信息
    //pps()芬位，單個(gè)圖像的參數(shù)信息
    if (keyFrame) {
        
        //圖像存儲(chǔ)方式，編碼器等格式描述
        CMFormatDescriptionRef format = CMSampleBufferGetFormatDescription(sampleBuffer);
        
        //sps
        size_t sparameterSetSize,sparameterSetCount;
        const uint8_t *sparameterSet;
        
        OSStatus statusCode = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(format, 0, &sparameterSet, &sparameterSetSize, &sparameterSetCount, 0);
        
        if (statusCode == noErr) {
            
            //獲取pps
            size_t pparameterSetSize,pparameterSetCount;
            const uint8_t *pparameterSet;
            
            //從第一個(gè)關(guān)鍵幀獲取sps & pps
            OSStatus statusCode = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(format, 1, &pparameterSet, &pparameterSetSize, &pparameterSetCount, 0);
            
            //獲取H264參數(shù)集合中的SPS和PPS
            if (statusCode == noErr)
            {
                //Found pps & sps
                NSData *sps = [NSData dataWithBytes:sparameterSet length:sparameterSetSize];
                NSData *pps = [NSData dataWithBytes:pparameterSet length:pparameterSetSize];
                
                if(encoder)
                {
                    [encoder gotSpsPps:sps pps:pps];
                }
            }
        }
        
    }
    
    CMBlockBufferRef dataBuffer = CMSampleBufferGetDataBuffer(sampleBuffer);
    size_t length,totalLength;
    char *dataPointer;
    /*
     CMBlockBufferGetDataPointer(CMBlockBufferRef  _Nonnull theBuffer, size_t offset, size_t * _Nullable lengthAtOffsetOut, size_t * _Nullable totalLengthOut, char * _Nullable * _Nullable dataPointerOut)
     * theBuffer: 數(shù)據(jù)源
     * offset : 偏移量
     * lengthAtOffsetOut : 單個(gè)數(shù)據(jù)長度
     * totalLengthOut : 總數(shù)據(jù)長度
     * dataPointerOut : 數(shù)據(jù)塊首地址
     */
    OSStatus statusCodeRet = CMBlockBufferGetDataPointer(dataBuffer, 0, &length, &totalLength, &dataPointer);
    if (statusCodeRet == noErr) {
        /*
         大端： 01 23 45 67
         小端： 67 45 23 01
         */
        size_t bufferOffset = 0;
        static const int AVCCHeaderLength = 4;//返回的nalu數(shù)據(jù)前4個(gè)字節(jié)不是001的startcode,而是大端模式的幀長度length
        
        //循環(huán)獲取nalu數(shù)據(jù)
        while (bufferOffset < totalLength - AVCCHeaderLength) {
            
            uint32_t NALUnitLength = 0;
            
            //讀取 一單元長度的 nalu
            memcpy(&NALUnitLength, dataPointer + bufferOffset, AVCCHeaderLength);
            
            //從大端模式轉(zhuǎn)換為系統(tǒng)端模式(小端)
            NALUnitLength = CFSwapInt32BigToHost(NALUnitLength);
            
            //獲取nalu數(shù)據(jù)
            NSData *data = [[NSData alloc]initWithBytes:(dataPointer + bufferOffset + AVCCHeaderLength) length:NALUnitLength];
            
            //將nalu數(shù)據(jù)寫入到文件
            [encoder gotEncodedData:data isKeyFrame:keyFrame];
            
            //move to the next NAL unit in the block buffer
            //讀取下一個(gè)nalu 一次回調(diào)可能包含多個(gè)nalu數(shù)據(jù)
            bufferOffset += AVCCHeaderLength + NALUnitLength;
            
            
        }
       
    }
    
    
}

//第一幀寫入 sps & pps
- (void)gotSpsPps:(NSData*)sps pps:(NSData*)pps
{
    NSLog(@"gotSpsPp %d %d",(int)[sps length],(int)[pps length]);
    //寫入之前(起始位)
    const char bytes[] = "\x00\x00\x00\x01";
    //去除末尾的/0
    size_t length = (sizeof bytes) - 1;
    
    NSData *ByteHeader = [NSData dataWithBytes:bytes length:length];
    
    [fileHandele writeData:ByteHeader];
    [fileHandele writeData:sps];
    [fileHandele writeData:ByteHeader];
    [fileHandele writeData:pps];
    
    
    
}


- (void)gotEncodedData:(NSData*)data isKeyFrame:(BOOL)isKeyFrame
{
    NSLog(@"gotEncodeData %d",(int)[data length]);
    
    if (fileHandele != NULL) {
        
        //添加4個(gè)字節(jié)的H264 協(xié)議 start code 分割符
        //一般來說編碼器編出的首幀數(shù)據(jù)為PPS & SPS
        //H264編碼時(shí)带到，在每個(gè)NAL前添加起始碼 0x000001,解碼器在碼流中檢測(cè)起始碼昧碉，當(dāng)前NAL結(jié)束。
        /*
         為了防止NAL內(nèi)部出現(xiàn)0x000001的數(shù)據(jù)，h.264又提出'防止競(jìng)爭(zhēng) emulation prevention"機(jī)制被饿，在編碼完一個(gè)NAL時(shí)四康，如果檢測(cè)出有連續(xù)兩個(gè)0x00字節(jié)，就在后面插入一個(gè)0x03狭握。當(dāng)解碼器在NAL內(nèi)部檢測(cè)到0x000003的數(shù)據(jù)闪金，就把0x03拋棄，恢復(fù)原始數(shù)據(jù)论颅。
         
         總的來說H264的碼流的打包方式有兩種,一種為annex-b byte stream format 的格式哎垦，這個(gè)是絕大部分編碼器的默認(rèn)輸出格式，就是每個(gè)幀的開頭的3~4個(gè)字節(jié)是H264的start_code,0x00000001或者0x000001恃疯。
         另一種是原始的NAL打包格式漏设，就是開始的若干字節(jié)（1，2今妄，4字節(jié)）是NAL的長度愿题，而不是start_code,此時(shí)必須借助某個(gè)全局的數(shù)據(jù)來獲得編 碼器的profile,level,PPS,SPS等信息才可以解碼。
         
         */
        const char bytes[] ="\x00\x00\x00\x01";
        
        //長度
        size_t length = (sizeof bytes) - 1;
        
        //頭字節(jié)
        NSData *ByteHeader = [NSData dataWithBytes:bytes length:length];
        
        //寫入頭字節(jié)
        [fileHandele writeData:ByteHeader];
        
        //寫入H264數(shù)據(jù)
        [fileHandele writeData:data];
        
    }
    
}

//結(jié)束VideoToolBox
-(void)endVideoToolBox
{
    //完成
    VTCompressionSessionCompleteFrames(cEncodeingSession, kCMTimeInvalid);
    //釋放
    VTCompressionSessionInvalidate(cEncodeingSession);
    CFRelease(cEncodeingSession);
    cEncodeingSession = NULL;
}

結(jié)構(gòu).png

h264解碼

一.解碼的思路:

1. 解析數(shù)據(jù)(NALU Unit) I/P/B...
2. 初始化解碼器
3. 將解析后的H264 NALU Unit輸入解碼器
4. 解碼完成回調(diào),輸出解碼數(shù)據(jù)
5. 解碼數(shù)據(jù)顯示(OpenGL ES)

二.解碼三個(gè)核心函數(shù):

1. 創(chuàng)建session, VTDecompressionSessionCreate
2. 解碼一個(gè)frame, VTDecompressionSessionDecodeFrame
3. 銷毀解碼session, VTDecompressionSessionInvalidate

三.原理分析:

• H264原始碼流-->NALU.
  • I幀:保留了一張完整視頻幀.解碼關(guān)鍵!
  • P幀:先前參考幀.差異數(shù)據(jù).解碼需要依賴于I幀
  • B幀:雙向參考幀,解碼時(shí)既需要|幀蛙奖，也需要P幀!

如果H264碼流中I幀錯(cuò)誤/丟失,就會(huì)導(dǎo)致錯(cuò)誤傳遞,P/B幀單獨(dú)是完成不了解碼工作!花屏的現(xiàn)象產(chǎn)生. VideoToolBox硬編碼編碼H264幀.I幀!手動(dòng)加入SPS/PPS.
解碼時(shí):需要使用SPS/PPS數(shù)據(jù)來對(duì)解碼器進(jìn)行初始化!

#import <AVFoundation/AVFoundation.h>
#import <VideoToolbox/VideoToolbox.h>

@interface CCVideoDecoder ()
@property (nonatomic, strong) dispatch_queue_t decodeQueue;
@property (nonatomic, strong) dispatch_queue_t callbackQueue;
/**解碼會(huì)話*/
@property (nonatomic) VTDecompressionSessionRef decodeSesion;

@end

@implementation CCVideoDecoder{
    uint8_t *_sps;
    NSUInteger _spsSize;
    uint8_t *_pps;
    NSUInteger _ppsSize;
    CMVideoFormatDescriptionRef _decodeDesc;
}

/**解碼回調(diào)函數(shù)*/
/*
 參數(shù)1: 回調(diào)引用
 參數(shù)2: 幀引用
 參數(shù)3: 狀態(tài)標(biāo)識(shí)
 參數(shù)4: 同步/異步解碼
 參數(shù)5: 實(shí)際圖像緩存
 參數(shù)6: 出現(xiàn)時(shí)間戳
 參數(shù)7: 出現(xiàn)持續(xù)時(shí)間
 */
void videoDecompressionOutputCallback(void * CM_NULLABLE decompressionOutputRefCon,
                                      void * CM_NULLABLE sourceFrameRefCon,
                                      OSStatus status,
                                      VTDecodeInfoFlags infoFlags,
                                      CM_NULLABLE CVImageBufferRef imageBuffer,
                                      CMTime presentationTimeStamp,
                                      CMTime presentationDuration ) {
    if (status != noErr) {
        NSLog(@"Video hard decode callback error status=%d", (int)status);
        return;
    }
    //解碼后的數(shù)據(jù)sourceFrameRefCon -> CVPixelBufferRef
    CVPixelBufferRef *outputPixelBuffer = (CVPixelBufferRef *)sourceFrameRefCon;
    *outputPixelBuffer = CVPixelBufferRetain(imageBuffer);
    
    //獲取self
    CCVideoDecoder *decoder = (__bridge CCVideoDecoder *)(decompressionOutputRefCon);
    
    //調(diào)用回調(diào)隊(duì)列
    dispatch_async(decoder.callbackQueue, ^{
        
        //將解碼后的數(shù)據(jù)給decoder代理.viewController
        [decoder.delegate videoDecodeCallback:imageBuffer];
        //釋放數(shù)據(jù)
        CVPixelBufferRelease(imageBuffer);
    });
}


- (instancetype)initWithConfig:(CCVideoConfig *)config
{
    self = [super init];
    if (self) {
        //初始化VideoConfig 信息
        _config = config;
        //創(chuàng)建解碼隊(duì)列與回調(diào)隊(duì)列
        _decodeQueue = dispatch_queue_create("h264 hard decode queue", DISPATCH_QUEUE_SERIAL);
        _callbackQueue = dispatch_queue_create("h264 hard decode callback queue", DISPATCH_QUEUE_SERIAL);
    }
    return self;
}

/*初始化解碼器**/
- (BOOL)initDecoder {
    
    if (_decodeSesion) return true;
    const uint8_t * const parameterSetPointers[2] = {_sps, _pps};
    const size_t parameterSetSizes[2] = {_spsSize, _ppsSize};
    int naluHeaderLen = 4;
    
    /**
     根據(jù)sps pps設(shè)置解碼參數(shù)
     param kCFAllocatorDefault 分配器
     param 2 參數(shù)個(gè)數(shù)
     param parameterSetPointers 參數(shù)集指針
     param parameterSetSizes 參數(shù)集大小
     param naluHeaderLen nalu nalu start code 的長度 4
     param _decodeDesc 解碼器描述
     return 狀態(tài)
     */
    OSStatus status = CMVideoFormatDescriptionCreateFromH264ParameterSets(kCFAllocatorDefault, 2, parameterSetPointers, parameterSetSizes, naluHeaderLen, &_decodeDesc);
    if (status != noErr) {
        NSLog(@"Video hard DecodeSession create H264ParameterSets(sps, pps) failed status= %d", (int)status);
        return false;
    }
    
    /*
     解碼參數(shù):
    * kCVPixelBufferPixelFormatTypeKey:攝像頭的輸出數(shù)據(jù)格式
     kCVPixelBufferPixelFormatTypeKey潘酗，已測(cè)可用值為
        kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange，即420v
        kCVPixelFormatType_420YpCbCr8BiPlanarFullRange雁仲，即420f
        kCVPixelFormatType_32BGRA仔夺，iOS在內(nèi)部進(jìn)行YUV至BGRA格式轉(zhuǎn)換
     YUV420一般用于標(biāo)清視頻，YUV422用于高清視頻攒砖，這里的限制讓人感到意外缸兔。但是，在相同條件下吹艇，YUV420計(jì)算耗時(shí)和傳輸壓力比YUV422都小惰蜜。
     
    * kCVPixelBufferWidthKey/kCVPixelBufferHeightKey: 視頻源的分辨率 width*height
     * kCVPixelBufferOpenGLCompatibilityKey : 它允許在 OpenGL 的上下文中直接繪制解碼后的圖像，而不是從總線和 CPU 之間復(fù)制數(shù)據(jù)受神。這有時(shí)候被稱為零拷貝通道抛猖，因?yàn)樵诶L制過程中沒有解碼的圖像被拷貝.
     
     */
    NSDictionary *destinationPixBufferAttrs =
    @{
      (id)kCVPixelBufferPixelFormatTypeKey: [NSNumber numberWithInt:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange], //iOS上 nv12(uvuv排布) 而不是nv21（vuvu排布）
      (id)kCVPixelBufferWidthKey: [NSNumber numberWithInteger:_config.width],
      (id)kCVPixelBufferHeightKey: [NSNumber numberWithInteger:_config.height],
      (id)kCVPixelBufferOpenGLCompatibilityKey: [NSNumber numberWithBool:true]
      };
    
    //解碼回調(diào)設(shè)置
    /*
     VTDecompressionOutputCallbackRecord 是一個(gè)簡(jiǎn)單的結(jié)構(gòu)體，它帶有一個(gè)指針 (decompressionOutputCallback)鼻听，指向幀解壓完成后的回調(diào)方法财著。你需要提供可以找到這個(gè)回調(diào)方法的實(shí)例 (decompressionOutputRefCon)。VTDecompressionOutputCallback 回調(diào)方法包括七個(gè)參數(shù)：
            參數(shù)1: 回調(diào)的引用
            參數(shù)2: 幀的引用
            參數(shù)3: 一個(gè)狀態(tài)標(biāo)識(shí) (包含未定義的代碼)
            參數(shù)4: 指示同步/異步解碼撑碴，或者解碼器是否打算丟幀的標(biāo)識(shí)
            參數(shù)5: 實(shí)際圖像的緩沖
            參數(shù)6: 出現(xiàn)的時(shí)間戳
            參數(shù)7: 出現(xiàn)的持續(xù)時(shí)間
     */
    VTDecompressionOutputCallbackRecord callbackRecord;
    callbackRecord.decompressionOutputCallback = videoDecompressionOutputCallback;
    callbackRecord.decompressionOutputRefCon = (__bridge void * _Nullable)(self);
    
    //創(chuàng)建session
    
    /*!
     @function    VTDecompressionSessionCreate
     @abstract    創(chuàng)建用于解壓縮視頻幀的會(huì)話撑教。
     @discussion  解壓后的幀將通過調(diào)用OutputCallback發(fā)出
     @param    allocator  內(nèi)存的會(huì)話。通過使用默認(rèn)的kCFAllocatorDefault的分配器醉拓。
     @param    videoFormatDescription 描述源視頻幀
     @param    videoDecoderSpecification 指定必須使用的特定視頻解碼器.NULL
     @param    destinationImageBufferAttributes 描述源像素緩沖區(qū)的要求 NULL
     @param    outputCallback 使用已解壓縮的幀調(diào)用的回調(diào)
     @param    decompressionSessionOut 指向一個(gè)變量以接收新的解壓會(huì)話
     */
    status = VTDecompressionSessionCreate(kCFAllocatorDefault, _decodeDesc, NULL, (__bridge CFDictionaryRef _Nullable)(destinationPixBufferAttrs), &callbackRecord, &_decodeSesion);
    
    //判斷一下status
    if (status != noErr) {
        NSLog(@"Video hard DecodeSession create failed status= %d", (int)status);
        return false;
    }
    
    //設(shè)置解碼會(huì)話屬性(實(shí)時(shí)編碼)
    status = VTSessionSetProperty(_decodeSesion, kVTDecompressionPropertyKey_RealTime,kCFBooleanTrue);
    
    NSLog(@"Vidoe hard decodeSession set property RealTime status = %d", (int)status);
    
    return true;
}

/**解碼函數(shù)（private）*/
- (CVPixelBufferRef)decode:(uint8_t *)frame withSize:(uint32_t)frameSize {
    // CVPixelBufferRef 解碼后的數(shù)據(jù)伟姐，編碼之前源視頻數(shù)據(jù)
    // CMBlockBufferRef 編碼之后的數(shù)據(jù)
    CVPixelBufferRef outputPixelBuffer = NULL;
    CMBlockBufferRef blockBuffer = NULL;
    CMBlockBufferFlags flag0 = 0;
    
    //創(chuàng)建blockBuffer
    /*!
     參數(shù)1: structureAllocator kCFAllocatorDefault  默認(rèn)內(nèi)存分配
     參數(shù)2: memoryBlock  frame    內(nèi)容
     參數(shù)3: frame size
     參數(shù)4: blockAllocator: Pass NULL
     參數(shù)5: customBlockSource Pass NULL
     參數(shù)6: offsetToData  數(shù)據(jù)偏移
     參數(shù)7: dataLength 數(shù)據(jù)長度
     參數(shù)8: flags 功能和控制標(biāo)志
     參數(shù)9: newBBufOut blockBuffer地址,不能為空
     */
    OSStatus status = CMBlockBufferCreateWithMemoryBlock(kCFAllocatorDefault, frame, frameSize, kCFAllocatorNull, NULL, 0, frameSize, flag0, &blockBuffer);
    
    if (status != kCMBlockBufferNoErr) {
        NSLog(@"Video hard decode create blockBuffer error code=%d", (int)status);
        return outputPixelBuffer;
    }
    
    CMSampleBufferRef sampleBuffer = NULL;
    const size_t sampleSizeArray[] = {frameSize};
    
    //創(chuàng)建sampleBuffer
    /*
     參數(shù)1: allocator 分配器,使用默認(rèn)內(nèi)存分配, kCFAllocatorDefault
     參數(shù)2: blockBuffer.需要編碼的數(shù)據(jù)blockBuffer.不能為NULL
     參數(shù)3: formatDescription,視頻輸出格式
     參數(shù)4: numSamples.CMSampleBuffer 個(gè)數(shù).
     參數(shù)5: numSampleTimingEntries 必須為0,1,numSamples 
     參數(shù)6: sampleTimingArray.  數(shù)組.為空
     參數(shù)7: numSampleSizeEntries 默認(rèn)為1
     參數(shù)8: sampleSizeArray
     參數(shù)9: sampleBuffer對(duì)象
     */
    status = CMSampleBufferCreateReady(kCFAllocatorDefault, blockBuffer, _decodeDesc, 1, 0, NULL, 1, sampleSizeArray, &sampleBuffer);
    
    if (status != noErr || !sampleBuffer) {
        NSLog(@"Video hard decode create sampleBuffer failed status=%d", (int)status);
        CFRelease(blockBuffer);
        return outputPixelBuffer;
    }
    
    //解碼
    //向視頻解碼器提示使用低功耗模式是可以的
    VTDecodeFrameFlags flag1 = kVTDecodeFrame_1xRealTimePlayback;
    //異步解碼
    VTDecodeInfoFlags  infoFlag = kVTDecodeInfo_Asynchronous;
    //解碼數(shù)據(jù)
    /*
     參數(shù)1: 解碼session
     參數(shù)2: 源數(shù)據(jù) 包含一個(gè)或多個(gè)視頻幀的CMsampleBuffer
     參數(shù)3: 解碼標(biāo)志
     參數(shù)4: 解碼后數(shù)據(jù)outputPixelBuffer
     參數(shù)5: 同步/異步解碼標(biāo)識(shí)
     */
    status = VTDecompressionSessionDecodeFrame(_decodeSesion, sampleBuffer, flag1, &outputPixelBuffer, &infoFlag);
    
    if (status == kVTInvalidSessionErr) {
        NSLog(@"Video hard decode  InvalidSessionErr status =%d", (int)status);
    } else if (status == kVTVideoDecoderBadDataErr) {
        NSLog(@"Video hard decode  BadData status =%d", (int)status);
    } else if (status != noErr) {
        NSLog(@"Video hard decode failed status =%d", (int)status);
    }
    CFRelease(sampleBuffer);
    CFRelease(blockBuffer);
    
    
    return outputPixelBuffer;
}

// private
- (void)decodeNaluData:(uint8_t *)frame size:(uint32_t)size {
    //數(shù)據(jù)類型:frame的前4個(gè)字節(jié)是NALU數(shù)據(jù)的開始碼收苏，也就是00 00 00 01，
    // 第5個(gè)字節(jié)是表示數(shù)據(jù)類型愤兵，轉(zhuǎn)為10進(jìn)制后倒戏，7是sps, 8是pps, 5是IDR（I幀）信息
    int type = (frame[4] & 0x1F);
    
    // 將NALU的開始碼轉(zhuǎn)為4字節(jié)大端NALU的長度信息
    uint32_t naluSize = size - 4;
    uint8_t *pNaluSize = (uint8_t *)(&naluSize);
    CVPixelBufferRef pixelBuffer = NULL;
    frame[0] = *(pNaluSize + 3);
    frame[1] = *(pNaluSize + 2);
    frame[2] = *(pNaluSize + 1);
    frame[3] = *(pNaluSize);
    
    //第一次解析時(shí): 初始化解碼器initDecoder
    /*
     關(guān)鍵幀/其他幀數(shù)據(jù): 調(diào)用[self decode:frame withSize:size] 方法
     sps/pps數(shù)據(jù):則將sps/pps數(shù)據(jù)賦值到_sps/_pps中.
     */
    switch (type) {
        case 0x05: //關(guān)鍵幀
            if ([self initDecoder]) {
                pixelBuffer= [self decode:frame withSize:size];
            }
            break;
        case 0x06:
            //NSLog(@"SEI");//增強(qiáng)信息
            break;
        case 0x07: //sps
            _spsSize = naluSize;
            _sps = malloc(_spsSize);
            memcpy(_sps, &frame[4], _spsSize);
            break;
        case 0x08: //pps
            _ppsSize = naluSize;
            _pps = malloc(_ppsSize);
            memcpy(_pps, &frame[4], _ppsSize);
            break;
        default: //其他幀（1-5）
            if ([self initDecoder]) {
                pixelBuffer = [self decode:frame withSize:size];
            }
            break;
    }
}

// public
- (void)decodeNaluData:(NSData *)frame {
    //將解碼放在異步隊(duì)列.
    dispatch_async(_decodeQueue, ^{
        //獲取frame 二進(jìn)制數(shù)據(jù)
        uint8_t *nalu = (uint8_t *)frame.bytes;
        //調(diào)用解碼Nalu數(shù)據(jù)方法,參數(shù)1:數(shù)據(jù) 參數(shù)2:數(shù)據(jù)長度
        [self decodeNaluData:nalu size:(uint32_t)frame.length];
    });
}

//銷毀
- (void)dealloc
{
    if (_decodeSesion) {
        VTDecompressionSessionInvalidate(_decodeSesion);
        CFRelease(_decodeSesion);
        _decodeSesion = NULL;
    }
    
}

/**
 nal_unit_type  NAL類型                         C
 0              未使用
 1              非IDR圖像中不采用數(shù)據(jù)劃分的片段     2,3,4
 2              非IDR圖像中A類數(shù)據(jù)劃分片段         2
 3              非IDR圖像中B類數(shù)據(jù)劃分片段         3
 4              非IDR圖像中C類數(shù)據(jù)劃分片段         4
 5              IDR圖像的片                     2,3
 6              補(bǔ)充增強(qiáng)信息單元（SEI）           5
 7              序列參數(shù)集                       0
 8              圖像參數(shù)集                       1
 9              分界符                          6
 10             序列結(jié)束                         7
 11             碼流結(jié)束                        8
 12             填充                            9
 13..23         保留
 
 24..31        不保留（RTP打包時(shí)會(huì)用到）
 
 
 NSString * const naluTypesStrings[] =
 {
 @"0: Unspecified (non-VCL)",
 @"1: Coded slice of a non-IDR picture (VCL)",    // P frame
 @"2: Coded slice data partition A (VCL)",
 @"3: Coded slice data partition B (VCL)",
 @"4: Coded slice data partition C (VCL)",
 @"5: Coded slice of an IDR picture (VCL)",      // I frame
 @"6: Supplemental enhancement information (SEI) (non-VCL)",
 @"7: Sequence parameter set (non-VCL)",         // SPS parameter
 @"8: Picture parameter set (non-VCL)",          // PPS parameter
 @"9: Access unit delimiter (non-VCL)",
 @"10: End of sequence (non-VCL)",
 @"11: End of stream (non-VCL)",
 @"12: Filler data (non-VCL)",
 @"13: Sequence parameter set extension (non-VCL)",
 @"14: Prefix NAL unit (non-VCL)",
 @"15: Subset sequence parameter set (non-VCL)",
 @"16: Reserved (non-VCL)",
 @"17: Reserved (non-VCL)",
 @"18: Reserved (non-VCL)",
 @"19: Coded slice of an auxiliary coded picture without partitioning (non-VCL)",
 @"20: Coded slice extension (non-VCL)",
 @"21: Coded slice extension for depth view components (non-VCL)",
 @"22: Reserved (non-VCL)",
 @"23: Reserved (non-VCL)",
 @"24: STAP-A Single-time aggregation packet (non-VCL)",
 @"25: STAP-B Single-time aggregation packet (non-VCL)",
 @"26: MTAP16 Multi-time aggregation packet (non-VCL)",
 @"27: MTAP24 Multi-time aggregation packet (non-VCL)",
 @"28: FU-A Fragmentation unit (non-VCL)",
 @"29: FU-B Fragmentation unit (non-VCL)",
 @"30: Unspecified (non-VCL)",
 @"31: Unspecified (non-VCL)",
 };
 */


@end

渲染

通過OpenGL渲染

#import <AVFoundation/AVUtilities.h>
#import <mach/mach_time.h>
#include <AVFoundation/AVFoundation.h>
#import <UIKit/UIScreen.h>
#include <OpenGLES/EAGL.h>
#include <OpenGLES/ES2/gl.h>
#include <OpenGLES/ES2/glext.h>

// Uniform index.
enum
{
    UNIFORM_Y,
    UNIFORM_UV,
    UNIFORM_ROTATION_ANGLE,
    UNIFORM_COLOR_CONVERSION_MATRIX,
    NUM_UNIFORMS
};
GLint uniforms[NUM_UNIFORMS];

// Attribute index.
enum
{
    ATTRIB_VERTEX,
    ATTRIB_TEXCOORD,
    NUM_ATTRIBUTES
};

// Color Conversion Constants (YUV to RGB) including adjustment from 16-235/16-240 (video range)

// BT.601, which is the standard for SDTV.
static const GLfloat kColorConversion601[] = {
    1.164,  1.164, 1.164,
    0.0, -0.392, 2.017,
    1.596, -0.813,   0.0,
};

// BT.709, which is the standard for HDTV.
static const GLfloat kColorConversion709[] = {
    1.164,  1.164, 1.164,
    0.0, -0.213, 2.112,
    1.793, -0.533,   0.0,
};



@interface AAPLEAGLLayer ()
{
    // The pixel dimensions of the CAEAGLLayer.
    GLint _backingWidth;
    GLint _backingHeight;
    
    EAGLContext *_context;
    CVOpenGLESTextureRef _lumaTexture;
    CVOpenGLESTextureRef _chromaTexture;
    
    GLuint _frameBufferHandle;
    GLuint _colorBufferHandle;
    
    const GLfloat *_preferredConversion;
}
@property GLuint program;

@end
@implementation AAPLEAGLLayer
@synthesize pixelBuffer = _pixelBuffer;

-(CVPixelBufferRef) pixelBuffer
{
    return _pixelBuffer;
}

- (void)setPixelBuffer:(CVPixelBufferRef)pb
{
    if(_pixelBuffer) {
        CVPixelBufferRelease(_pixelBuffer);
    }
    _pixelBuffer = CVPixelBufferRetain(pb);
    
    int frameWidth = (int)CVPixelBufferGetWidth(_pixelBuffer);
    int frameHeight = (int)CVPixelBufferGetHeight(_pixelBuffer);
    [self displayPixelBuffer:_pixelBuffer width:frameWidth height:frameHeight];
}

- (instancetype)initWithFrame:(CGRect)frame
{
    self = [super init];
    if (self) {
        CGFloat scale = [[UIScreen mainScreen] scale];
        self.contentsScale = scale;
        
        self.opaque = TRUE;
        self.drawableProperties = @{ kEAGLDrawablePropertyRetainedBacking :[NSNumber numberWithBool:YES]};
        
        [self setFrame:frame];
        
        // Set the context into which the frames will be drawn.
        _context = [[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES2];
        
        if (!_context) {
            return nil;
        }
        
        // Set the default conversion to BT.709, which is the standard for HDTV.
        _preferredConversion = kColorConversion709;
        
        [self setupGL];
    }
    
    return self;
}

- (void)displayPixelBuffer:(CVPixelBufferRef)pixelBuffer width:(uint32_t)frameWidth height:(uint32_t)frameHeight
{
    if (!_context || ![EAGLContext setCurrentContext:_context]) {
        return;
    }
    
    if(pixelBuffer == NULL) {
        NSLog(@"Pixel buffer is null");
        return;
    }
    
    CVReturn err;
    
    size_t planeCount = CVPixelBufferGetPlaneCount(pixelBuffer);
    
    /*
     Use the color attachment of the pixel buffer to determine the appropriate color conversion matrix.
     */
    CFTypeRef colorAttachments = CVBufferGetAttachment(pixelBuffer, kCVImageBufferYCbCrMatrixKey, NULL);
    
    if (CFStringCompare(colorAttachments, kCVImageBufferYCbCrMatrix_ITU_R_601_4, 0) == kCFCompareEqualTo) {
        _preferredConversion = kColorConversion601;
    }
    else {
        _preferredConversion = kColorConversion709;
    }
    
    /*
     CVOpenGLESTextureCacheCreateTextureFromImage will create GLES texture optimally from CVPixelBufferRef.
     */
    
    /*
     Create Y and UV textures from the pixel buffer. These textures will be drawn on the frame buffer Y-plane.
     */
    
    CVOpenGLESTextureCacheRef _videoTextureCache;
    
    // Create CVOpenGLESTextureCacheRef for optimal CVPixelBufferRef to GLES texture conversion.
    err = CVOpenGLESTextureCacheCreate(kCFAllocatorDefault, NULL, _context, NULL, &_videoTextureCache);
    if (err != noErr) {
        NSLog(@"Error at CVOpenGLESTextureCacheCreate %d", err);
        return;
    }
    
    glActiveTexture(GL_TEXTURE0);
    
    err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
                                                       _videoTextureCache,
                                                       pixelBuffer,
                                                       NULL,
                                                       GL_TEXTURE_2D,
                                                       GL_RED_EXT,
                                                       frameWidth,
                                                       frameHeight,
                                                       GL_RED_EXT,
                                                       GL_UNSIGNED_BYTE,
                                                       0,
                                                       &_lumaTexture);
    if (err) {
        NSLog(@"Error at CVOpenGLESTextureCacheCreateTextureFromImage %d", err);
    }
    
    glBindTexture(CVOpenGLESTextureGetTarget(_lumaTexture), CVOpenGLESTextureGetName(_lumaTexture));
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
    
    if(planeCount == 2) {
        // UV-plane.
        glActiveTexture(GL_TEXTURE1);
        err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
                                                           _videoTextureCache,
                                                           pixelBuffer,
                                                           NULL,
                                                           GL_TEXTURE_2D,
                                                           GL_RG_EXT,
                                                           frameWidth / 2,
                                                           frameHeight / 2,
                                                           GL_RG_EXT,
                                                           GL_UNSIGNED_BYTE,
                                                           1,
                                                           &_chromaTexture);
        if (err) {
            NSLog(@"Error at CVOpenGLESTextureCacheCreateTextureFromImage %d", err);
        }
        
        glBindTexture(CVOpenGLESTextureGetTarget(_chromaTexture), CVOpenGLESTextureGetName(_chromaTexture));
        glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
        glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
    }
    
    glBindFramebuffer(GL_FRAMEBUFFER, _frameBufferHandle);
    
    // Set the view port to the entire view.
    glViewport(0, 0, _backingWidth, _backingHeight);
    
    glClearColor(0.0f, 0.0f, 0.0f, 1.0f);
    glClear(GL_COLOR_BUFFER_BIT);
    
    // Use shader program.
    glUseProgram(self.program);
    //    glUniform1f(uniforms[UNIFORM_LUMA_THRESHOLD], 1);
    //    glUniform1f(uniforms[UNIFORM_CHROMA_THRESHOLD], 1);
    glUniform1f(uniforms[UNIFORM_ROTATION_ANGLE], 0);
    glUniformMatrix3fv(uniforms[UNIFORM_COLOR_CONVERSION_MATRIX], 1, GL_FALSE, _preferredConversion);
    
    // Set up the quad vertices with respect to the orientation and aspect ratio of the video.
    CGRect viewBounds = self.bounds;
    CGSize contentSize = CGSizeMake(frameWidth, frameHeight);
    CGRect vertexSamplingRect = AVMakeRectWithAspectRatioInsideRect(contentSize, viewBounds);
    
    // Compute normalized quad coordinates to draw the frame into.
    CGSize normalizedSamplingSize = CGSizeMake(0.0, 0.0);
    CGSize cropScaleAmount = CGSizeMake(vertexSamplingRect.size.width/viewBounds.size.width,
                                        vertexSamplingRect.size.height/viewBounds.size.height);
    
    // Normalize the quad vertices.
    if (cropScaleAmount.width > cropScaleAmount.height) {
        normalizedSamplingSize.width = 1.0;
        normalizedSamplingSize.height = cropScaleAmount.height/cropScaleAmount.width;
    }
    else {
        normalizedSamplingSize.width = cropScaleAmount.width/cropScaleAmount.height;
        normalizedSamplingSize.height = 1.0;;
    }
    
    /*
     The quad vertex data defines the region of 2D plane onto which we draw our pixel buffers.
     Vertex data formed using (-1,-1) and (1,1) as the bottom left and top right coordinates respectively, covers the entire screen.
     */
    GLfloat quadVertexData [] = {
        -1 * normalizedSamplingSize.width, -1 * normalizedSamplingSize.height,
        normalizedSamplingSize.width, -1 * normalizedSamplingSize.height,
        -1 * normalizedSamplingSize.width, normalizedSamplingSize.height,
        normalizedSamplingSize.width, normalizedSamplingSize.height,
    };
    
    // Update attribute values.
    glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, 0, 0, quadVertexData);
    glEnableVertexAttribArray(ATTRIB_VERTEX);
    
    /*
     The texture vertices are set up such that we flip the texture vertically. This is so that our top left origin buffers match OpenGL's bottom left texture coordinate system.
     */
    CGRect textureSamplingRect = CGRectMake(0, 0, 1, 1);
    GLfloat quadTextureData[] =  {
        CGRectGetMinX(textureSamplingRect), CGRectGetMaxY(textureSamplingRect),
        CGRectGetMaxX(textureSamplingRect), CGRectGetMaxY(textureSamplingRect),
        CGRectGetMinX(textureSamplingRect), CGRectGetMinY(textureSamplingRect),
        CGRectGetMaxX(textureSamplingRect), CGRectGetMinY(textureSamplingRect)
    };
    
    glVertexAttribPointer(ATTRIB_TEXCOORD, 2, GL_FLOAT, 0, 0, quadTextureData);
    glEnableVertexAttribArray(ATTRIB_TEXCOORD);
    
    glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
    
    glBindRenderbuffer(GL_RENDERBUFFER, _colorBufferHandle);
    [_context presentRenderbuffer:GL_RENDERBUFFER];
    
    [self cleanUpTextures];
    // Periodic texture cache flush every frame
    CVOpenGLESTextureCacheFlush(_videoTextureCache, 0);
    
    if(_videoTextureCache) {
        CFRelease(_videoTextureCache);
    }
}

# pragma mark - OpenGL setup

- (void)setupGL
{
    if (!_context || ![EAGLContext setCurrentContext:_context]) {
        return;
    }
    
    [self setupBuffers];
    [self loadShaders];
    
    glUseProgram(self.program);
    
    // 0 and 1 are the texture IDs of _lumaTexture and _chromaTexture respectively.
    glUniform1i(uniforms[UNIFORM_Y], 0);
    glUniform1i(uniforms[UNIFORM_UV], 1);
    glUniform1f(uniforms[UNIFORM_ROTATION_ANGLE], 0);
    glUniformMatrix3fv(uniforms[UNIFORM_COLOR_CONVERSION_MATRIX], 1, GL_FALSE, _preferredConversion);
}

#pragma mark - Utilities

- (void)setupBuffers
{
    glDisable(GL_DEPTH_TEST);
    
    glEnableVertexAttribArray(ATTRIB_VERTEX);
    glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, GL_FALSE, 2 * sizeof(GLfloat), 0);
    
    glEnableVertexAttribArray(ATTRIB_TEXCOORD);
    glVertexAttribPointer(ATTRIB_TEXCOORD, 2, GL_FLOAT, GL_FALSE, 2 * sizeof(GLfloat), 0);
    
    [self createBuffers];
}

- (void) createBuffers
{
    glGenFramebuffers(1, &_frameBufferHandle);
    glBindFramebuffer(GL_FRAMEBUFFER, _frameBufferHandle);
    
    glGenRenderbuffers(1, &_colorBufferHandle);
    glBindRenderbuffer(GL_RENDERBUFFER, _colorBufferHandle);
    
    [_context renderbufferStorage:GL_RENDERBUFFER fromDrawable:self];
    glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_WIDTH, &_backingWidth);
    glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_HEIGHT, &_backingHeight);
    
    glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, _colorBufferHandle);
    if (glCheckFramebufferStatus(GL_FRAMEBUFFER) != GL_FRAMEBUFFER_COMPLETE) {
        NSLog(@"Failed to make complete framebuffer object %x", glCheckFramebufferStatus(GL_FRAMEBUFFER));
    }
}

- (void) releaseBuffers
{
    if(_frameBufferHandle) {
        glDeleteFramebuffers(1, &_frameBufferHandle);
        _frameBufferHandle = 0;
    }
    
    if(_colorBufferHandle) {
        glDeleteRenderbuffers(1, &_colorBufferHandle);
        _colorBufferHandle = 0;
    }
}

- (void) resetRenderBuffer
{
    if (!_context || ![EAGLContext setCurrentContext:_context]) {
        return;
    }
    
    [self releaseBuffers];
    [self createBuffers];
}

- (void) cleanUpTextures
{
    if (_lumaTexture) {
        CFRelease(_lumaTexture);
        _lumaTexture = NULL;
    }
    
    if (_chromaTexture) {
        CFRelease(_chromaTexture);
        _chromaTexture = NULL;
    }
}

#pragma mark -  OpenGL ES 2 shader compilation

const GLchar *shader_fsh = (const GLchar*)"varying highp vec2 texCoordVarying;"
"precision mediump float;"
"uniform sampler2D SamplerY;"
"uniform sampler2D SamplerUV;"
"uniform mat3 colorConversionMatrix;"
"void main()"
"{"
"    mediump vec3 yuv;"
"    lowp vec3 rgb;"
//   Subtract constants to map the video range start at 0
"    yuv.x = (texture2D(SamplerY, texCoordVarying).r - (16.0/255.0));"
"    yuv.yz = (texture2D(SamplerUV, texCoordVarying).rg - vec2(0.5, 0.5));"
"    rgb = colorConversionMatrix * yuv;"
"    gl_FragColor = vec4(rgb, 1);"
"}";

const GLchar *shader_vsh = (const GLchar*)"attribute vec4 position;"
"attribute vec2 texCoord;"
"uniform float preferredRotation;"
"varying vec2 texCoordVarying;"
"void main()"
"{"
"    mat4 rotationMatrix = mat4(cos(preferredRotation), -sin(preferredRotation), 0.0, 0.0,"
"                               sin(preferredRotation),  cos(preferredRotation), 0.0, 0.0,"
"                               0.0,                        0.0, 1.0, 0.0,"
"                               0.0,                        0.0, 0.0, 1.0);"
"    gl_Position = position * rotationMatrix;"
"    texCoordVarying = texCoord;"
"}";

- (BOOL)loadShaders
{
    GLuint vertShader = 0, fragShader = 0;
    
    // Create the shader program.
    self.program = glCreateProgram();
    
    if(![self compileShaderString:&vertShader type:GL_VERTEX_SHADER shaderString:shader_vsh]) {
        NSLog(@"Failed to compile vertex shader");
        return NO;
    }
    
    if(![self compileShaderString:&fragShader type:GL_FRAGMENT_SHADER shaderString:shader_fsh]) {
        NSLog(@"Failed to compile fragment shader");
        return NO;
    }
    
    // Attach vertex shader to program.
    glAttachShader(self.program, vertShader);
    
    // Attach fragment shader to program.
    glAttachShader(self.program, fragShader);
    
    // Bind attribute locations. This needs to be done prior to linking.
    glBindAttribLocation(self.program, ATTRIB_VERTEX, "position");
    glBindAttribLocation(self.program, ATTRIB_TEXCOORD, "texCoord");
    
    // Link the program.
    if (![self linkProgram:self.program]) {
        NSLog(@"Failed to link program: %d", self.program);
        
        if (vertShader) {
            glDeleteShader(vertShader);
            vertShader = 0;
        }
        if (fragShader) {
            glDeleteShader(fragShader);
            fragShader = 0;
        }
        if (self.program) {
            glDeleteProgram(self.program);
            self.program = 0;
        }
        
        return NO;
    }
    
    // Get uniform locations.
    uniforms[UNIFORM_Y] = glGetUniformLocation(self.program, "SamplerY");
    uniforms[UNIFORM_UV] = glGetUniformLocation(self.program, "SamplerUV");
    //    uniforms[UNIFORM_LUMA_THRESHOLD] = glGetUniformLocation(self.program, "lumaThreshold");
    //    uniforms[UNIFORM_CHROMA_THRESHOLD] = glGetUniformLocation(self.program, "chromaThreshold");
    uniforms[UNIFORM_ROTATION_ANGLE] = glGetUniformLocation(self.program, "preferredRotation");
    uniforms[UNIFORM_COLOR_CONVERSION_MATRIX] = glGetUniformLocation(self.program, "colorConversionMatrix");
    
    // Release vertex and fragment shaders.
    if (vertShader) {
        glDetachShader(self.program, vertShader);
        glDeleteShader(vertShader);
    }
    if (fragShader) {
        glDetachShader(self.program, fragShader);
        glDeleteShader(fragShader);
    }
    
    return YES;
}

- (BOOL)compileShaderString:(GLuint *)shader type:(GLenum)type shaderString:(const GLchar*)shaderString
{
    *shader = glCreateShader(type);
    glShaderSource(*shader, 1, &shaderString, NULL);
    glCompileShader(*shader);
    
#if defined(DEBUG)
    GLint logLength;
    glGetShaderiv(*shader, GL_INFO_LOG_LENGTH, &logLength);
    if (logLength > 0) {
        GLchar *log = (GLchar *)malloc(logLength);
        glGetShaderInfoLog(*shader, logLength, &logLength, log);
        NSLog(@"Shader compile log:\n%s", log);
        free(log);
    }
#endif
    
    GLint status = 0;
    glGetShaderiv(*shader, GL_COMPILE_STATUS, &status);
    if (status == 0) {
        glDeleteShader(*shader);
        return NO;
    }
    
    return YES;
}

- (BOOL)compileShader:(GLuint *)shader type:(GLenum)type URL:(NSURL *)URL
{
    NSError *error;
    NSString *sourceString = [[NSString alloc] initWithContentsOfURL:URL encoding:NSUTF8StringEncoding error:&error];
    if (sourceString == nil) {
        NSLog(@"Failed to load vertex shader: %@", [error localizedDescription]);
        return NO;
    }
    
    const GLchar *source = (GLchar *)[sourceString UTF8String];
    
    return [self compileShaderString:shader type:type shaderString:source];
}

- (BOOL)linkProgram:(GLuint)prog
{
    GLint status;
    glLinkProgram(prog);
    
#if defined(DEBUG)
    GLint logLength;
    glGetProgramiv(prog, GL_INFO_LOG_LENGTH, &logLength);
    if (logLength > 0) {
        GLchar *log = (GLchar *)malloc(logLength);
        glGetProgramInfoLog(prog, logLength, &logLength, log);
        NSLog(@"Program link log:\n%s", log);
        free(log);
    }
#endif
    
    glGetProgramiv(prog, GL_LINK_STATUS, &status);
    if (status == 0) {
        return NO;
    }
    
    return YES;
}

- (BOOL)validateProgram:(GLuint)prog
{
    GLint logLength, status;
    
    glValidateProgram(prog);
    glGetProgramiv(prog, GL_INFO_LOG_LENGTH, &logLength);
    if (logLength > 0) {
        GLchar *log = (GLchar *)malloc(logLength);
        glGetProgramInfoLog(prog, logLength, &logLength, log);
        NSLog(@"Program validate log:\n%s", log);
        free(log);
    }
    
    glGetProgramiv(prog, GL_VALIDATE_STATUS, &status);
    if (status == 0) {
        return NO;
    }
    
    return YES;
}

- (void)dealloc
{
    if (!_context || ![EAGLContext setCurrentContext:_context]) {
        return;
    }
    
    [self cleanUpTextures];
    
    if(_pixelBuffer) {
        CVPixelBufferRelease(_pixelBuffer);
    }
    
    if (self.program) {
        glDeleteProgram(self.program);
        self.program = 0;
    }
    if(_context) {
        //[_context release];
        _context = nil;
    }
    //[super dealloc];
}
@end