大道如青天纹磺，我獨不得出
先來個圖
效果.png
前言
在上一篇iOS Core ML與Vision初識中，初步了解到了vision的作用，并在文章最后留了個疑問硼莽，就是類似下面的一些函數(shù)有什么用
- (instancetype)initWithCIImage:(CIImage *)image options:(NSDictionary<VNImageOption, id> *)options;
- (instancetype)initWithCVPixelBuffer:(CVPixelBufferRef)pixelBuffer options:(NSDictionary<VNImageOption, id> *)options;
在查閱一些資料后，最終通過這些函數(shù)得到了如下的效果
face.gif
對煮纵，沒錯懂鸵，這就是通過
initWithCVPixelBuffer函數(shù)來實現(xiàn)的。當(dāng)然vision的作用遠(yuǎn)不于此行疏，還有如下的效果1匆光、圖像匹配（上篇文章中的效果）
2、矩形檢測
3酿联、二維碼殴穴、條碼檢測
4、目標(biāo)跟蹤
5、文字檢測
6采幌、人臉檢測
7劲够、人臉面部特征檢測
由于對人臉識別比較感興趣，所以這里就主要簡單了解了下人臉部分休傍，下面就針對人臉檢測和面部檢測寫寫
Vision支持的圖片類型
通過查看VNRequestHandler.h文件征绎，我們可以看到里面的所有初始化函數(shù)，通過這些初始化函數(shù)磨取，我們可以了解到支持的類型有：
1人柿、CVPixelBufferRef
2、CGImageRef
3忙厌、CIImage
4凫岖、NSURL
5、NSData
Vision使用
在使用vision的時候逢净，我們首先需要明確自己需要什么效果哥放，然后根據(jù)想要的效果來選擇不同的類，最后實現(xiàn)自己的效果
1爹土、需要一個RequestHandler甥雕，在創(chuàng)建RequestHandler的時候，需要一個合適的輸入源胀茵，及圖片類型
2社露、需要一個Request，在創(chuàng)建Request的時候琼娘，也需要根據(jù)實際情況來選擇峭弟，Request大概有如下這么些
request.jpeg
3、通過
requestHandler將request聯(lián)系起來脱拼，然后得到結(jié)果
[handler performRequests:@[requset] error:&error];
4瞒瘸、處理結(jié)果VNObservation，在VNRequest的results數(shù)組中挪拟，包含了VNObservation結(jié)果挨务，VNObservation也分很多種，這和你Request的類型是相關(guān)聯(lián)的
Vision結(jié)果繼承關(guān)系.png
在完成上述步驟后玉组，我們就可以根據(jù)結(jié)果來實現(xiàn)一些我們想要的效果
人臉矩形檢測
這里我們需要用到VNDetectFaceRectanglesRequest
requset = [[VNDetectFaceRectanglesRequest alloc] initWithCompletionHandler:completionHandler];
在得到結(jié)果后谎柄，我們需要處理下坐標(biāo)
for (VNFaceObservation *faceObservation in observations) {
//boundingBox
CGRect transFrame = [self convertRect:faceObservation.boundingBox imageSize:image.size];
[rects addObject:[NSValue valueWithCGRect:transFrame]];
}
// 轉(zhuǎn)換Rect
- (CGRect)convertRect:(CGRect)boundingBox imageSize:(CGSize)imageSize{
CGFloat w = boundingBox.size.width * imageSize.width;
CGFloat h = boundingBox.size.height * imageSize.height;
CGFloat x = boundingBox.origin.x * imageSize.width;
CGFloat y = imageSize.height * (1 - boundingBox.origin.y - boundingBox.size.height);//- (boundingBox.origin.y * imageSize.height) - h;
return CGRectMake(x, y, w, h);
}
在返回結(jié)果中的boundingBox中的坐標(biāo)，我們并不能立即使用惯雳，而是需要進(jìn)行轉(zhuǎn)換朝巫，因為這里是相對于image的一個比例，這里需要注意的是y坐標(biāo)的轉(zhuǎn)換石景，因為坐標(biāo)系的y軸和UIView的y軸是相反的劈猿。
最后就是通過返回的坐標(biāo)進(jìn)行矩形的繪制
+ (UIImage *)gl_drawImage:(UIImage *)image withRects:(NSArray *)rects
{
UIImage *newImage = nil;
UIGraphicsBeginImageContextWithOptions(image.size, NO, [UIScreen mainScreen].scale);
CGContextRef context = UIGraphicsGetCurrentContext();
CGContextSetLineCap(context,kCGLineCapRound); //邊緣樣式
CGContextSetLineJoin(context, kCGLineJoinRound);
CGContextSetLineWidth(context,2); //線寬
CGContextSetAllowsAntialiasing(context,YES); //打開抗鋸齒
CGContextSetStrokeColorWithColor(context, [UIColor redColor].CGColor);
CGContextSetFillColorWithColor(context, [UIColor clearColor].CGColor);
//繪制圖片
[image drawInRect:CGRectMake(0, 0,image.size.width, image.size.height)];
CGContextBeginPath(context);
for (int i = 0; i < rects.count; i ++) {
CGRect rect = [rects[i] CGRectValue];
CGPoint sPoints[4];//坐標(biāo)點
sPoints[0] = CGPointMake(rect.origin.x, rect.origin.y);//坐標(biāo)1
sPoints[1] = CGPointMake(rect.origin.x + rect.size.width, rect.origin.y);//坐標(biāo)2
sPoints[2] = CGPointMake(rect.origin.x + rect.size.width, rect.origin.y + rect.size.height);//坐標(biāo)3
sPoints[3] = CGPointMake(rect.origin.x , rect.origin.y + rect.size.height);
CGContextAddLines(context, sPoints, 4);//添加線
CGContextClosePath(context); //封閉
}
CGContextDrawPath(context, kCGPathFillStroke); //根據(jù)坐標(biāo)繪制路徑
newImage = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();
return newImage;
}
效果如下
faceRect.jpg
人臉特征識別
這里我們需要用到VNDetectFaceLandmarksRequest
requset = [[VNDetectFaceLandmarksRequest alloc] initWithCompletionHandler:completionHandler];
處理結(jié)果
for (VNFaceObservation *faceObservation in observations) {
//boundingBox
CGRect transFrame = [self convertRect:faceObservation.boundingBox imageSize:image.size];
[rects addObject:[NSValue valueWithCGRect:transFrame]];
}
pointModel.faceRectPoints = rects;
return pointModel;
}
- (GLDiscernPointModel *)handlerFaceLandMark:(NSArray *)observations image:(UIImage *)image
{
GLDiscernPointModel *pointModel = [[GLDiscernPointModel alloc] init];
NSMutableArray *rects = @[].mutableCopy;
for (VNFaceObservation *faceObservation in observations) {
VNFaceLandmarks2D *faceLandMarks2D = faceObservation.landmarks;
[self getKeysWithClass:[VNFaceLandmarks2D class] block:^(NSString *key) {
if ([key isEqualToString:@"allPoints"]) {
return ;
}
VNFaceLandmarkRegion2D *faceLandMarkRegion2D = [faceLandMarks2D valueForKey:key];
NSMutableArray *sPoints = [[NSMutableArray alloc] initWithCapacity:faceLandMarkRegion2D.pointCount];
for (int i = 0; i < faceLandMarkRegion2D.pointCount; i ++) {
CGPoint point = faceLandMarkRegion2D.normalizedPoints[i];
CGFloat rectWidth = image.size.width * faceObservation.boundingBox.size.width;
CGFloat rectHeight = image.size.height * faceObservation.boundingBox.size.height;
CGPoint p = CGPointMake(point.x * rectWidth + faceObservation.boundingBox.origin.x * image.size.width, faceObservation.boundingBox.origin.y * image.size.height + point.y * rectHeight);
[sPoints addObject:[NSValue valueWithCGPoint:p]];
}
[rects addObject:sPoints];
}];
}
在這里拙吉，我們需要注意到landmarks這個屬性，這是一個VNFaceLandmarks2D類型的對象揪荣，里面包含著許多面部特征的VNFaceLandmarkRegion2D對象筷黔，如：faceContour，leftEye仗颈，nose....分別表示面部輪廓佛舱、左眼、鼻子挨决。這些對象中请祖，又包含下面這么一個屬性
@property (readonly, assign, nullable) const CGPoint* normalizedPoints
這是一個包含該面部特征的的數(shù)組，所以我們可以通過下面的方式取出里面的坐標(biāo)
CGPoint point = faceLandMarkRegion2D.normalizedPoints[i];
當(dāng)然這里面也存在坐標(biāo)的轉(zhuǎn)換脖祈，見上面代碼
最后也是畫線肆捕，代碼如下
+ (UIImage *)gl_drawImage:(UIImage *)image faceLandMarkPoints:(NSArray *)landMarkPoints
{
UIImage * newImage = image;
for (NSMutableArray *points in landMarkPoints) {
CGPoint sPoints [points.count];
for (int i = 0;i <points.count;i++) {
NSValue *pointValue = points[i];
CGPoint point = pointValue.CGPointValue;
sPoints[i] = point;
}
//畫線
UIGraphicsBeginImageContextWithOptions(newImage.size, NO, [UIScreen mainScreen].scale);
CGContextRef context = UIGraphicsGetCurrentContext();
CGContextSetLineCap(context,kCGLineCapRound); //邊緣樣式
CGContextSetLineJoin(context, kCGLineJoinRound);
CGContextSetLineWidth(context,2); //線寬
CGContextSetAllowsAntialiasing(context,YES); //打開抗鋸齒
// 設(shè)置翻轉(zhuǎn)
CGContextTranslateCTM(context, 0, newImage.size.height);
CGContextScaleCTM(context, 1.0, -1.0);
CGContextSetStrokeColorWithColor(context, [UIColor redColor].CGColor);
CGContextSetFillColorWithColor(context, [UIColor clearColor].CGColor);
CGContextDrawImage(context, CGRectMake(0, 0,newImage.size.width,newImage.size.height), newImage.CGImage);
CGContextBeginPath(context);
CGContextAddLines(context, sPoints,points.count);//添加線
CGContextClosePath(context); //封閉
CGContextDrawPath(context, kCGPathFillStroke); //根據(jù)坐標(biāo)繪制路徑
newImage = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();
}
return newImage;
}
效果如下
faceLandmark.png
動態(tài)人臉矩形檢測
要動態(tài)來檢測，那么我們肯定需要通過相機(jī)來實時取出資源盖高，然后再實現(xiàn)慎陵，所以我們這里選擇了AVCapture，關(guān)于相機(jī)的初始化及使用方法這里就不在累贅了或舞，我們直接上代碼
在AVCaptureVideoDataOutputSampleBufferDelegate中荆姆，通過下面的方法
- (void)captureOutput:(AVCaptureOutput *)output didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
我們可以進(jìn)行這么一個轉(zhuǎn)換
CVPixelBufferRef cvpixeBufferRef = CMSampleBufferGetImageBuffer(sampleBuffer);
然后通過
VNImageRequestHandler *handler = [[VNImageRequestHandler alloc] initWithCVPixelBuffer:cvpixeBufferRef options:@{}];
將相機(jī)返回的圖片與request進(jìn)行關(guān)聯(lián)了蒙幻。
后續(xù)操作如下
request = [[VNDetectFaceRectanglesRequest alloc] initWithCompletionHandler:^(VNRequest * _Nonnull request, NSError * _Nullable error) {
NSLog(@" 打印信息:%lu",request.results.count);
NSArray *vnobservations = request.results;
dispatch_async(dispatch_get_main_queue(), ^{
//先移除之前的矩形框
[self.rectLayers makeObjectsPerformSelector:@selector(removeFromSuperlayer)];
AVCaptureDevicePosition position = [[self.avInput device] position];
for (VNFaceObservation *faceObservation in vnobservations) {
//boundingBox
CGRect transFrame = [[GLTools sharedInstance] convertRect:faceObservation.boundingBox imageSize:self.view.frame.size];
//前置攝像頭的時候 記得轉(zhuǎn)換
if (position == AVCaptureDevicePositionFront){
transFrame.origin.x = self.view.frame.size.width - transFrame.origin.x - transFrame.size.width;
}
CALayer *rectLayer = [CALayer layer];
rectLayer.frame = transFrame;
rectLayer.borderColor = [UIColor purpleColor].CGColor;
rectLayer.borderWidth = 2;
[self.view.layer addSublayer:rectLayer];
[self.rectLayers addObject:rectLayer];
}
});
}];
在這里存在一個問題映凳，就是攝像頭分為前后攝像頭，所以在前置攝像頭和后置攝像頭切換的時候邮破，需要重新配置下
//需要重新進(jìn)行配置輸出 特別是下面的輸出方向
AVCaptureConnection *captureConnection = [self.avOutput connectionWithMediaType:AVMediaTypeVideo];
if ([captureConnection isVideoOrientationSupported]) {
[captureConnection setVideoOrientation:AVCaptureVideoOrientationPortrait];
}
// 視頻穩(wěn)定設(shè)置
if ([captureConnection isVideoStabilizationSupported]) {
captureConnection.preferredVideoStabilizationMode = AVCaptureVideoStabilizationModeAuto;
}
// 設(shè)置輸出圖片方向
captureConnection.videoOrientation = AVCaptureVideoOrientationPortrait;
還有個問題就是在坐標(biāo)轉(zhuǎn)化的時候诈豌，前置攝像頭的x軸和UIView的x軸也是相反的，所以這里也需要在進(jìn)行一次轉(zhuǎn)化
transFrame.origin.x = self.view.frame.size.width - transFrame.origin.x - transFrame.size.width;
效果如下
動態(tài)1.gif
動態(tài)添加場景
關(guān)于動態(tài)添加場景抒和，其實就像我們平時用的美顏相機(jī)那樣矫渔，在適當(dāng)?shù)奈恢锰砑有┟弊印⒀坨R等各種搞笑的圖片摧莽。這里我們還是需要用到AVCapture庙洼，并且和動態(tài)添加矩形的方法類似，只是在request上和處理方式上不一樣
下面我們先看代碼
request = [[VNDetectFaceLandmarksRequest alloc] initWithCompletionHandler:^(VNRequest * _Nonnull request, NSError * _Nullable error) {
NSArray *vnobservations = request.results;
for (VNFaceObservation *faceObservation in vnobservations) {
VNFaceLandmarks2D *faceLandMarks2D = faceObservation.landmarks;
VNFaceLandmarkRegion2D *leftEyefaceLandMarkRegion2D = faceLandMarks2D.leftEye;
VNFaceLandmarkRegion2D *rightEyefaceLandMarkRegion2D = faceLandMarks2D.rightEye;
dispatch_async(dispatch_get_main_queue(), ^{
// //先移除之前的矩形框
// [self.rectLayers makeObjectsPerformSelector:@selector(removeFromSuperlayer)];
//
// AVCaptureDevicePosition position = [[self.avInput device] position];
//
// CGRect transFrame = [[GLTools sharedInstance] convertRect:faceObservation.boundingBox imageSize:self.view.frame.size];
// //前置攝像頭的時候 記得轉(zhuǎn)換
// if (position == AVCaptureDevicePositionFront){
// transFrame.origin.x = self.view.frame.size.width - transFrame.origin.x - transFrame.size.width;
// }
//
// CALayer *rectLayer = [CALayer layer];
// rectLayer.frame = transFrame;
// rectLayer.borderColor = [UIColor purpleColor].CGColor;
// rectLayer.borderWidth = 2;
// [self.view.layer addSublayer:rectLayer];
//
// [self.rectLayers addObject:rectLayer];
AVCaptureDevicePosition position = [[self.avInput device] position];
CGPoint sPoints[leftEyefaceLandMarkRegion2D.pointCount + rightEyefaceLandMarkRegion2D.pointCount];
NSMutableArray *pointXs = [[NSMutableArray alloc] init];
NSMutableArray *pointYs = [[NSMutableArray alloc] init];
for (int i = 0; i < leftEyefaceLandMarkRegion2D.pointCount; i ++) {
CGPoint point = leftEyefaceLandMarkRegion2D.normalizedPoints[i];
CGFloat rectWidth = self.view.bounds.size.width * faceObservation.boundingBox.size.width;
CGFloat rectHeight = self.view.bounds.size.height * faceObservation.boundingBox.size.height;
CGFloat boundingBoxY = self.view.bounds.size.height * (1 - faceObservation.boundingBox.origin.y - faceObservation.boundingBox.size.height);
CGPoint p = CGPointZero;
if (position == AVCaptureDevicePositionFront){
CGFloat boundingX = self.view.frame.size.width - faceObservation.boundingBox.origin.x * self.view.bounds.size.width - rectWidth;
p = CGPointMake(point.x * rectWidth + boundingX, boundingBoxY + (1-point.y) * rectHeight);
}else{
p = CGPointMake(point.x * rectWidth + faceObservation.boundingBox.origin.x * self.view.bounds.size.width, boundingBoxY + (1-point.y) * rectHeight);
}
sPoints[i] = p;
[pointXs addObject:[NSNumber numberWithFloat:p.x]];
[pointYs addObject:[NSNumber numberWithFloat:p.y]];
}
for (int j = 0; j < rightEyefaceLandMarkRegion2D.pointCount; j ++) {
CGPoint point = rightEyefaceLandMarkRegion2D.normalizedPoints[j];
CGFloat rectWidth = self.view.bounds.size.width * faceObservation.boundingBox.size.width;
CGFloat rectHeight = self.view.bounds.size.height * faceObservation.boundingBox.size.height;
CGFloat boundingBoxY = self.view.bounds.size.height * (1 - faceObservation.boundingBox.origin.y - faceObservation.boundingBox.size.height);
CGPoint p = CGPointZero;
if (position == AVCaptureDevicePositionFront){
CGFloat boundingX = self.view.frame.size.width - faceObservation.boundingBox.origin.x * self.view.bounds.size.width - rectWidth;
p = CGPointMake(point.x * rectWidth + boundingX, boundingBoxY + (1-point.y) * rectHeight);
}else{
p = CGPointMake(point.x * rectWidth + faceObservation.boundingBox.origin.x * self.view.bounds.size.width, boundingBoxY + (1-point.y) * rectHeight);
}
sPoints[leftEyefaceLandMarkRegion2D.pointCount + j] = p;
[pointXs addObject:[NSNumber numberWithFloat:p.x]];
[pointYs addObject:[NSNumber numberWithFloat:p.y]];
}
// for (UIView *view in self.view.subviews) {
// if ([view isKindOfClass:[UIImageView class]]) {
// [view removeFromSuperview];
// }
// }
//
// for (int i = 0; i < rightEyefaceLandMarkRegion2D.pointCount + leftEyefaceLandMarkRegion2D.pointCount; i++) {
// CGFloat x = sPoints[i].x;
// CGFloat y = sPoints[i].y;
// UIImageView *view = [[UIImageView alloc] initWithFrame:CGRectMake(x, y, 2, 2)];
// view.backgroundColor = [UIColor redColor];
// [self.view addSubview:view];
// }
//排序 得到最小的x和最大的x
NSArray *sortPointXs = [pointXs sortedArrayWithOptions:NSSortStable usingComparator:
^NSComparisonResult(id _Nonnull obj1, id _Nonnull obj2) {
int value1 = [obj1 floatValue];
int value2 = [obj2 floatValue];
if (value1 > value2) {
return NSOrderedDescending;
}else if (value1 == value2){
return NSOrderedSame;
}else{
return NSOrderedAscending;
}
}];
NSArray *sortPointYs = [pointYs sortedArrayWithOptions:NSSortStable usingComparator:
^NSComparisonResult(id _Nonnull obj1, id _Nonnull obj2) {
int value1 = [obj1 floatValue];
int value2 = [obj2 floatValue];
if (value1 > value2) {
return NSOrderedDescending;
}else if (value1 == value2){
return NSOrderedSame;
}else{
return NSOrderedAscending;
}
}];
UIImage *image =[UIImage imageNamed:@"eyes"];
CGFloat imageWidth = [sortPointXs.lastObject floatValue] - [sortPointXs.firstObject floatValue] + 40;
CGFloat imageHeight = (imageWidth * image.size.height)/image.size.width;
self.glassesImageView.frame = CGRectMake([sortPointXs.firstObject floatValue]-20, [sortPointYs.firstObject floatValue]-5, imageWidth, imageHeight);
});
}
}];
由于時間關(guān)系镊辕，代碼有點亂油够，將就將就
先說說思路，我是想動態(tài)添加一個眼鏡的征懈，所以我必須先得到兩個眼睛的位置石咬，然后在計算出兩個眼睛的寬高，最后適當(dāng)?shù)恼{(diào)整眼鏡的大小卖哎，再動態(tài)的添加上去
這里必須要說的一個問題鬼悠，就是我在實現(xiàn)過程中遇到的---坐標(biāo)
首先是y坐標(biāo)删性，如果還是按照靜態(tài)圖片的那種獲取方式，那么得到的結(jié)果將會是完全相反的焕窝。
faceObservation.boundingBox.origin.y * image.size.height + point.y * rectHeight
這里我做了 一個假設(shè)蹬挺，估計是由于攝像機(jī)成像的原因造成的，所以必須反其道而行它掂，于是我如下改造了下
CGFloat boundingBoxY = self.view.bounds.size.height * (1 - faceObservation.boundingBox.origin.y - faceObservation.boundingBox.size.height);
p = CGPointMake(point.x * rectWidth + faceObservation.boundingBox.origin.x * self.view.bounds.size.width, boundingBoxY + (1-point.y) * rectHeight);
從中可以看到汗侵，所有的point.y都用1減去了，這個試驗的過程有點惱火群发，我還沒怎么相通晰韵，若有知道的，希望可以告訴我下熟妓，當(dāng)然我也會再研究研究雪猪。
再說完y坐標(biāo)后，就是x坐標(biāo)了起愈，x坐標(biāo)在前置攝像頭的時候一切正常只恨，然而在切換成后置攝像頭的時候，又反了抬虽。??官觅！心累啊，所以沒辦法阐污，我就只要加判斷休涤，然后進(jìn)行測試，有了如下代碼
CGFloat boundingX = self.view.frame.size.width - faceObservation.boundingBox.origin.x * self.view.bounds.size.width - rectWidth;
最后終于大功告成笛辟！
效果就是文章最頂?shù)哪莻€效果
注意
1功氨、在使用過程中，我發(fā)現(xiàn)當(dāng)檢測圖片的時候內(nèi)存和cpu的消耗還是很高的手幢，比如我的5s就成功的崩潰過.....
2捷凄、圖片方向是有要求的....
- (instancetype)initWithCVPixelBuffer:(CVPixelBufferRef)pixelBuffer options:(NSDictionary<VNImageOption, id> *)options;
/*!
@brief initWithCVPixelBuffer:options creates a VNImageRequestHandler to be used for performing requests against the image passed in as buffer.
@param pixelBuffer A CVPixelBuffer containing the image to be used for performing the requests. The content of the buffer cannot be modified for the lifetime of the VNImageRequestHandler.
@param orientation The orientation of the image/buffer based on the EXIF specification. For details see kCGImagePropertyOrientation. The value has to be an integer from 1 to 8. This superceeds every other orientation information.
@param options A dictionary with options specifying auxilary information for the buffer/image like VNImageOptionCameraIntrinsics
*/
- (instancetype)initWithCVPixelBuffer:(CVPixelBufferRef)pixelBuffer orientation:(CGImagePropertyOrientation)orientation options:(NSDictionary<VNImageOption, id> *)options;
通過對比上面兩個函數(shù)，我們可以發(fā)現(xiàn)围来，多了一個CGImagePropertyOrientation類型的參數(shù)跺涤，沒錯，這就是指定傳入圖片的方向监透，如果指定了方向桶错，而圖片方向卻不一致，那么恭喜你才漆，檢測不出來....這里我用的都是第一個方法牛曹，及沒有參數(shù)，好像默認(rèn)是up的醇滥。
最后
還是附上Demo黎比，如果覺得還行的話超营，歡迎大家給個star！有什么問題阅虫，可以多多溝通
