未經(jīng)同意蚊惯，不得轉(zhuǎn)載

人臉遮擋檢測（Face occlusion detect）有助于構(gòu)建高質(zhì)量的人臉識別底庫愿卸。本文定義人臉的遮擋位置為5個區(qū)域：左眼，右眼截型，鼻子趴荸，嘴和下巴，基于Tensorflow + Keras訓(xùn)練一個簡單的CNN人臉遮擋檢測模型宦焦，并結(jié)合Dlib人臉檢測器實現(xiàn)測試Demo的示例发钝。

項目地址：https://github.com/Oreobird/Face-Occlusion-Detect

一、樣本準(zhǔn)備

1波闹、Cofw數(shù)據(jù)庫

數(shù)據(jù)是 .mat 格式酝豪，其中包括了圖像像素值，29個點的landmarks坐標(biāo)以及每個點的遮擋情況精堕，可以通過以下的代碼轉(zhuǎn)換為jpg圖像以及相應(yīng)的ground_true數(shù)據(jù)：

#mat_file: COFW_train.mat, COFW_test.mat
#img_token: 'IsTr', 'IsT'
#bbox_token: 'bboxesTr', 'bboxesT'
#phis_token: 'phisTr', 'phisT'
def mat_to_files(mat_file, img_token, bbox_token, phis_token, img_dir, gt_txt_file):
    train_mat = h5py.File(mat_file, 'r')
    tr_imgs_obj = train_mat[img_token][:]
    total_num = tr_imgs_obj.shape[1]
    # print(total_num)

    with open(gt_txt_file, "w+") as trf:
        for i in range(total_num):
            img = train_mat[tr_imgs_obj[0][i]][:]
            bbox = train_mat[bbox_token][:]
            bbox = np.transpose(bbox)[i]

            img = np.transpose(img)
            if not os.path.exists(img_dir):
                os.mkdir(img_dir)

            cv2.imwrite(img_dir + "/{}.jpg".format(i), img)
            gt = train_mat[phis_token][:]
            gt = np.transpose(gt)[i]

            content = img_dir + "/{}.jpg,".format(i)
            for k in range(bbox.shape[0]):
                content = content + bbox[k].astype(str) + ' '
            content += ','
            for k in range(gt.shape[0]):
                content = content + gt[k].astype(str) + ' '
            content += '\n'
            trf.write(content)

if not os.path.exists(data_root + "train_ground_true.txt"):
    mat_to_files(data_root + "COFW_train.mat",
                     'IsTr', 'bboxesTr', 'phisTr',
                     data_root + "train",
                     data_root + "train_ground_true.txt")

if not os.path.exists(data_root + "test_ground_true.txt"):
    mat_to_files(data_root + "COFW_test.mat",
                     'IsT', 'bboxesT', 'phisT',
                     data_root + "test",
                     data_root + "test_ground_true.txt")

打開生成的train_ground_true.txt文件后孵淘，以逗號隔開的內(nèi)容為：

cofw數(shù)據(jù)信息

landmarks信息

[normal载萌，left-eye，right-eye巡扇，nose扭仁，mouth，chin]

對應(yīng)的landmarks遮擋標(biāo)記下標(biāo)索引為：

left-eye：8,10,12,13,16
right-eye：9,11,14,15,17
nose：18,19,20,21
mouth：22,23,24,25,26,27
chin：28

當(dāng)其中的這些點有遮擋時厅翔，則認為此區(qū)域被遮擋乖坠。可以通過以下代碼對圖像生成相應(yīng)的標(biāo)注信息：

# gt_txt：ground_true文件
# face_img_dir: 圖像目錄
# face_txt：要生成的標(biāo)注信息文件
# show：在人臉圖像上顯示29個landmarks點

def face_label(gt_txt, face_img_dir, face_txt, show=False):
    img_num = 1
    with open(face_txt, "w+") as face_txt_fp:
        with open(gt_txt, 'r') as gt_fp:
            line = gt_fp.readline()
            while line:
                img_path, bbox, phis = line.split(',')

                phis = phis.strip('\n').strip(' ').split(' ')
                phis = [int(float(x)) for x in phis]
                # print(phis)

                if show:
                    for i in range(29):
                        cv2.circle(img, (phis[i], phis[i + 29]), 2, (0, 255, 255))
                        cv2.putText(img, str(i), (phis[i], phis[i + 29]),
 cv2.FONT_HERSHEY_COMPLEX,0.3,(0,0,255),1)
                        cv2.imshow("img", img)
                        cv2.waitKey(0)

                slot = phis[58:]
                label = [1, 0, 0, 0, 0, 0]
                # if slot[0] and slot[2] and slot[4] and slot[5]:
                # label[1] = 1 # left eyebrow
                # label[0] = 0
                if slot[16]: # slot[10] or slot[12] or slot[13] or slot[16] or slot[8]:
                    label[1] = 1 # left eye
                    label[0] = 0
                # if slot[1] and slot[3] and slot[6] and slot[7]:
                # label[3] = 1 # right eyebrow
                # label[0] = 0
                if slot[17]: # slot[11] or slot[14] or slot[15] or slot[17] or slot[9]:
                    label[2] = 1 # right eye
                    label[0] = 0
                if slot[20]: # slot[18] or slot[19] or slot[20] or slot[21]:
                    label[3] = 1 # nose
                    label[0] = 0
                if slot[22] or slot[23] or slot[25] or slot[26] or slot[27]: # or slot[24]
                    label[4] = 1 # mouth
                    label[0] = 0
                if slot[28]:
                    label[5] = 1 # chin
                    label[0] = 0

                lab_str = ''
                for x in label:
                    lab_str += str(x) + ' '

                content = face_img_dir + "{}.jpg".format(img_num) + ',' + lab_str.rstrip(' ')
                content += '\n'
                # print(content)
                face_txt_fp.write(content)

                line = gt_fp.readline()
                img_num += 1

將show參數(shù)置為True后刀闷，可以得到landmarks的29個點位置如下：

cofw人臉landmarks點

3熊泵、數(shù)據(jù)預(yù)處理
由于Cofw的樣本數(shù)據(jù)只有1000多個仰迁，并且有遮擋的人臉很少，為了避免訓(xùn)練時過擬合顽分，需要對原始樣本做一個處理徐许。本文通過對5個區(qū)域隨機疊加隨機灰度值的遮擋塊來擴展遮擋樣本：

遮擋塊構(gòu)造

二梯皿、模型構(gòu)建

1、模型結(jié)構(gòu)

由于可以是人臉多區(qū)域的遮擋县恕，所以可以建模為多標(biāo)簽的分類問題东羹。模型結(jié)構(gòu)如下圖所示：

模型結(jié)構(gòu)

多標(biāo)簽分類的輸出層激活函數(shù)使用Sigmoid，loss類型為binary_crossentropy忠烛，即對每個標(biāo)簽做二分類属提。模型類的代碼如下，其中實現(xiàn)了自定義的簡單模型和基于VGG16來finetune的兩種構(gòu)建方式美尸。

import tensorflow as tf
import os
import numpy as np
import cv2
import scipy.io as sio
import heapq
# import tensorflow.contrib.eager as tfe
# tfe.enable_eager_execution()
# np.set_printoptions(threshold=np.nan)
EPOCHS = 25
class FodNet:
def __init__(self, dataset, class_num, batch_size, input_size, fine_tune=True, fine_tune_model_file='imagenet'):
    self.class_num = class_num
    self.batch_size = batch_size
    self.input_size = input_size
    self.dataset = dataset
    self.fine_tune_model_file = fine_tune_model_file
    if fine_tune:
        self.model = self.fine_tune_model()
    else:
        self.model = self.__create_model()

def __base_model(self, inputs):
    feature = tf.keras.layers.Conv2D(filters=64, kernel_size=(5, 5), strides=(1, 1), padding='same')(inputs)
    feature = tf.keras.layers.BatchNormalization()(feature)
    feature = tf.keras.layers.Activation(activation=tf.nn.relu)(feature)
    feature = tf.keras.layers.Conv2D(filters=64, kernel_size=(5, 5), strides=(1, 1), padding='same')(feature)
    feature = tf.keras.layers.BatchNormalization()(feature)
    feature = tf.keras.layers.Activation(activation=tf.nn.relu)(feature)
    feature = tf.keras.layers.MaxPool2D(pool_size=(2, 2), strides=(2, 2))(feature)

    feature = tf.keras.layers.Conv2D(filters=64, kernel_size=(3, 3), strides=(1, 1), padding='same')(feature)
    feature = tf.keras.layers.BatchNormalization()(feature)
    feature = tf.keras.layers.Activation(activation=tf.nn.relu)(feature)
    feature = tf.keras.layers.Conv2D(filters=64, kernel_size=(3, 3), strides=(1, 1), padding='same')(feature)
    feature = tf.keras.layers.BatchNormalization()(feature)
    feature = tf.keras.layers.Activation(activation=tf.nn.relu)(feature)
    feature = tf.keras.layers.MaxPool2D(pool_size=(2, 2), strides=(2, 2))(feature)
    return feature

 def __dense(self, feature):
    feature = tf.keras.layers.Flatten()(feature)
    feature = tf.keras.layers.Dense(units=128)(feature)
    feature = tf.keras.layers.BatchNormalization()(feature)
    feature = tf.keras.layers.Activation(activation=tf.nn.relu)(feature)
    feature = tf.keras.layers.Dropout(0.5)(feature)
    feature = tf.keras.layers.Dense(units=256)(feature)
    feature = tf.keras.layers.BatchNormalization()(feature)
    feature = tf.keras.layers.Activation(activation=tf.nn.relu)(feature)
    feature = tf.keras.layers.Dropout(0.5)(feature)
    return feature

def __create_model(self):
    input_fod = tf.keras.layers.Input(name='fod_input', shape=(self.input_size,   self.input_size, 1))
    feature_fod = self.__base_model(input_fod)
    feature_fod = self.__dense(feature_fod)
    output_fod = tf.keras.layers.Dense(name='fod_output', units=self.class_num, activation=tf.nn.sigmoid)(feature_fod)
    model = tf.keras.Model(inputs=[input_fod], outputs=[output_fod])
    losses = {
      'fod_output': 'binary_crossentropy',
    }
    model.compile(optimizer=tf.train.AdamOptimizer(),
                  loss=losses,
                  metrics=['accuracy'])
    return model
def __extract_output(self, model, name, input):
    model._name = name
    for layer in model.layers:
        layer.trainable = True
    return model(input)

def fine_tune_model(self):
    input_fod = tf.keras.layers.Input(name='fod_input', shape=(self.input_size, self.input_size, 3))
    # resnet_fod = tf.keras.applications.ResNet50(weights='imagenet', include_top=False)
    # feature_fod = self.__extract_output(resnet_fod, 'resnet_fod', input_fod)

    vgg16_fod = tf.keras.applications.VGG16(weights=self.fine_tune_model_file, include_top=False)
    feature_fod = self.__extract_output(vgg16_fod, 'vgg16_fod', input_fod)

    feature_fod = self.__dense(feature_fod)
    output_fod = tf.keras.layers.Dense(name='fod_output', units=self.class_num, activation=tf.nn.sigmoid)(feature_fod)

    model = tf.keras.Model(inputs=[input_fod], outputs=[output_fod])
    losses = {
        'fod_output': 'binary_crossentropy',
    }

    model.compile(optimizer=tf.train.AdamOptimizer(),
    loss=losses,
    metrics=['accuracy'])
    return model

def fit(self, model_file, checkpoint_dir, log_dir, max_epoches=EPOCHS, train=True):
    self.model.summary()
    if not train:
        self.model.load_weights(model_file)
    else:
        cp_callback = tf.keras.callbacks.ModelCheckpoint(checkpoint_dir,
                                                         save_weights_only=True,
                                                          save_best_only=True,
                                                          period=2,
                                                          verbose=1)
        earlystop_cb = tf.keras.callbacks.EarlyStopping(monitor='val_loss',
                                                        mode='min',
                                                        min_delta=0.001,
                                                        patience=3,
                                                        verbose=1)
        tb_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir)
        input_name_list = ['fod_input']
        output_name_list = ['fod_output']           
        self.model.fit_generator(generator=self.dataset.data_generator(input_name_list, output_name_list, 'train.txt'),
                                epochs=max_epoches,
                                steps_per_epoch=self.dataset.train_num() // self.batch_size,
                                validation_data=self.dataset.data_generator(input_name_list, output_name_list, 'val.txt'),
                                validation_steps=self.dataset.val_num() // self.batch_size,
                                callbacks=[cp_callback, earlystop_cb, tb_callback],
                                max_queue_size=10,
                                workers=1,
                                verbose=1)
        self.model.save(model_file)

def predict(self):
    input_name_list = ['fod_input']
    output_name_list = ['fod_output']
    predictions = self.model.predict_generator(generator=self.dataset.data_generator(input_name_list, output_name_list, 'test.txt', shuffle=False),
 steps=self.dataset.test_num() // self.batch_size,
 verbose=1)
    if len(predictions) > 0:
        fod_preds = predictions
        # print(fod_preds)
        test_data = self.dataset.data_generator(input_name_list, output_name_list, 'test.txt', shuffle=False)
        correct = 0
        steps = self.dataset.test_num() // self.batch_size
        total = steps * self.batch_size

        for step in range(steps):
            _, test_batch_y = next(test_data)
            fod_real_batch = test_batch_y['fod_output']
            for i, fod_real in enumerate(fod_real_batch):
                fod_real = fod_real.tolist()
                one_num = fod_real.count(1)
                fod_pred_idxs = sorted(list(map(fod_preds[self.batch_size * step + i].tolist().index,
                                            heapq.nlargest(one_num, fod_preds[self.batch_size * step + i]))))
                fod_real_idxs = [i for i,x in enumerate(fod_real) if x == 1]
                # print(fod_pred_idxs)
                # print(fod_real_idxs)
                if fod_real_idxs == fod_pred_idxs:
                    correct += 1
        print("fod==> correct:{}, total:{}, correct_rate:{}".format(correct, total, 1.0 * correct / total))
    return predictions

def test_online(self, face_imgs):
    batch_x = np.array(face_imgs[0]['fod_input'], dtype=np.float32)
    batch_x = np.expand_dims(batch_x, 0)

    predictions = self.model.predict({'fod_input': batch_x}, batch_size=1)
    # predictions = np.asarray(predictions)
    return predictions

2冤议、訓(xùn)練曲線

acc曲線

loss曲線

三、測試

通過Dlib實時檢測人臉實現(xiàn)师坎，需要先下載好Dlib的人臉檢測模型文件：

shape_predictor_68_face_landmarks.dat

dlib實現(xiàn)了68個人臉landmarks點的檢測恕酸，這里根據(jù)這些點來crop出人臉區(qū)，在有遮擋時在 相應(yīng)的區(qū)域畫圈圈胯陋，代碼如下：

import os
import dlib
from imutils import face_utils
import cv2
import numpy as np
class CameraTester():
    def __init__(self, net=None, input_size=96, fine_tune=False, face_landmark_path='./model/shape_predictor_68_face_landmarks.dat'):
        self.cap = cv2.VideoCapture(0)
        if not self.cap.isOpened():
            raise Exception("Unable to connect to camera.")
        self.detector = dlib.get_frontal_face_detector()
        self.predictor = dlib.shape_predictor(face_landmark_path)
        self.net = net
        self.input_size = input_size
        self.fine_tune = fine_tune

    def crop_face(self, shape, img, input_size):
        x = []
        y = []
        for (_x, _y) in shape:
            x.append(_x)
            y.append(_y)

        max_x = min(max(x), img.shape[1])
        min_x = max(min(x), 0)
        max_y = min(max(y), img.shape[0])
        min_y = max(min(y), 0)

        Lx = max_x - min_x
        Ly = max_y - min_y
        Lmax = int(max(Lx, Ly))
        delta = Lmax // 2

        center_x = (max(x) + min(x)) // 2
        center_y = (max(y) + min(y)) // 2
        start_x = int(center_x - delta)
        start_y = int(center_y - 0.99 * delta)
        end_x = int(center_x + delta)
        end_y = int(center_y + 1.01 * delta)
        start_y = 0 if start_y < 0 else start_y
        start_x = 0 if start_x < 0 else start_x
        end_x = img.shape[1] if end_x > img.shape[1] else end_x
        end_y = img.shape[0] if end_y > img.shape[0] else end_y

        crop_face = img[start_y:end_y, start_x:end_x]
        print(crop_face.shape)
        crop_face = cv2.cvtColor(crop_face, cv2.COLOR_BGR2GRAY)

        crop_face = cv2.resize(crop_face, (input_size, input_size)) / 255
        channel = 3 if self.fine_tune else 1
        crop_face = np.resize(crop_face, (self.input_size, self.input_size, channel))
        return crop_face, start_y, end_y, start_x, end_x

    def get_area(self, shape, idx):
        #[[x, y], radius]
        left_eye = [(shape[42] + shape[45]) // 2, abs(shape[45][0] - shape[42][0])]
        right_eye = [(shape[36] + shape[39]) // 2, abs(shape[39][0] - shape[36][0])]
        nose = [shape[30], int(abs(shape[31][0] - shape[35][0]) / 1.5)]
        mouth = [(shape[48] + shape[54]) // 2, abs(shape[48][0] - shape[54][0]) // 2]
        chin = [shape[8], nose[1]]
        area = [None, left_eye, right_eye, nose, mouth, chin]
        block_area = [x for i, x in enumerate(area) if i in idx]
        return block_area

    def draw_occlusion_area(self, img, shape, idx):
        area = self.get_area(shape, idx)
        for k, v in enumerate(area):
            if v:
                cv2.circle(img, tuple(v[0]), v[1], (0, 255, 0))

    def run(self):
        frames = []
        while self.cap.isOpened():
            ret, frame = self.cap.read()
            if ret:
                frame = cv2.resize(frame, (640, 480))
                face_rects = self.detector(frame, 0)
                if len(face_rects) > 0:
                    shape = self.predictor(frame, face_rects[0])
                    shape = face_utils.shape_to_np(shape)
                    input_img, start_y, end_y, start_x, end_x = self.crop_face(shape, frame, self.input_size)
                    cv2.rectangle(frame, (start_x, start_y), (end_x, end_y), (0, 255, 0), thickness=2)
                    frames.append({'fod_input': input_img})
                    if len(frames) == 1:
                        pred = self.net.test_online(frames)
                        print(pred)
                        idx = [i for i, x in enumerate(pred[0]) if x > 0.5]
                        frames = []
                        print(idx)
                        if len(idx):
                            self.draw_occlusion_area(frame, shape, idx)
                        else:
                            print("No face detect")
            cv2.imshow("frame", frame)
            if cv2.waitKey(1) & 0xFF == ord('q'):
                break

四蕊温、總結(jié)

本文基于Tensorflow+keras+Dlib實現(xiàn)了一個人臉遮擋實時檢測的Demo。由于訓(xùn)練樣本的比較單一遏乔，模型簡單义矛，實現(xiàn)的效果準(zhǔn)確率還有待提高。