inception_model

題圖來自: github
本文主要介紹了PrettyTensor呆盖，用來快速構(gòu)建神經(jīng)網(wǎng)絡(luò)株扛。當(dāng)然盆繁，原文寫于16年冕碟，現(xiàn)在有更方便的API厕妖，后續(xù)會(huì)介紹迎捺。本文有大段前篇教程的文字及代碼范舀，如果看過上一篇的朋友可以快速翻到下文PrettyTensor實(shí)現(xiàn)的那一部分去。

by Magnus Erik Hvass Pedersen / GitHub / Videos on YouTube中文翻譯 thrillerist / Github

介紹

之前的教程演示了如何在TensorFlow中實(shí)現(xiàn)一個(gè)卷積神經(jīng)網(wǎng)絡(luò)玫锋，這需要了解一些TensorFlow工作的底層原理。它有點(diǎn)復(fù)雜础爬，實(shí)現(xiàn)起來還容易犯錯(cuò)供炎。

這篇教程為我們說明了如何使用TensorFlow的一個(gè)附加包PrettyTensor，它也是Google開發(fā)的。PrettyTensor提供了在TensorFlow中創(chuàng)建神經(jīng)網(wǎng)絡(luò)的更簡(jiǎn)單的方法幽勒，讓我們可以關(guān)注自己想要實(shí)現(xiàn)的想法咪惠，而不用過多擔(dān)心底層的實(shí)現(xiàn)細(xì)節(jié)永脓。這也讓代碼更短排宰、更容易閱讀和修改寞奸。

除了用PrettyTensor構(gòu)造圖之外，這篇教程的大部分代碼和教程 #02 中的一樣彼哼，當(dāng)然還有一些細(xì)微的變化孝常。

這篇教程是基于教程 #02 之上的，如果你是TensorFlow新手的話稿茉，推薦先學(xué)完上一份教程。你需要熟悉基本的線性代數(shù)少态、Python和Jupyter Notebook編輯器为肮。

流程圖

下面的圖表直接展示了之后實(shí)現(xiàn)的卷積神經(jīng)網(wǎng)絡(luò)中數(shù)據(jù)的傳遞重斑。關(guān)于卷積的詳細(xì)描述請(qǐng)看上一篇教程

from IPython.display import Image
Image('images/02_network_flowchart.png')

image.png

輸入圖像在第一層卷基層中用權(quán)重過濾器處理。結(jié)果在16張新圖里笛丙，每個(gè)代表了卷積層里一個(gè)過濾器（的處理結(jié)果）漾脂。圖像也經(jīng)過降采樣，因此圖像分辨率從28x28減少到14x14胚鸯。

這16張小圖在第二個(gè)卷積層中處理骨稿。這16個(gè)通道都需要一個(gè)權(quán)重過濾，這層的輸出的每個(gè)通道也各需要一個(gè)權(quán)重過濾√构冢總共有36個(gè)輸出形耗，所以在第二個(gè)卷積層有16 x 36 = 576個(gè)濾波器。輸出圖再一次降采樣到7x7個(gè)像素辙浑。

第二個(gè)卷積層的輸出是36張7x7像素的圖像激涤。它們被壓到一個(gè)長(zhǎng)為7 x 7 x 36 = 1764的向量中去，它作為一個(gè)有128個(gè)神經(jīng)元（或元素）的全連接網(wǎng)絡(luò)的輸入例衍。這些又輸入到另一個(gè)有10個(gè)神經(jīng)元的全連接層中昔期，每個(gè)神經(jīng)元代表一個(gè)類別砚偶，用來確定圖像的類別遣蚀，也即圖像上的數(shù)字机打。

卷積濾波一開始是隨機(jī)挑選的敛瓷，因此分類也是隨機(jī)完成的膳犹。根據(jù)交叉熵（cross-entropy）來測(cè)量輸入圖預(yù)測(cè)值和真實(shí)類別間的錯(cuò)誤案糙。然后優(yōu)化器用鏈?zhǔn)椒▌t自動(dòng)地將這個(gè)誤差傳在卷積網(wǎng)絡(luò)中傳遞瘸恼，更新濾波權(quán)重來提升分類質(zhì)量骑科。這個(gè)過程迭代了幾千次奥吩，直到分類誤差足夠低哼蛆。

這些特定的濾波權(quán)重和中間圖像是一個(gè)優(yōu)化的結(jié)果，和你執(zhí)行這些代碼所看到的可能會(huì)有所不同霞赫。

注意腮介，這些在TensorFlow上的計(jì)算是在一部分圖像上執(zhí)行，而非單獨(dú)的一張圖端衰，這使得計(jì)算更有效叠洗。也意味著在TensorFlow上實(shí)現(xiàn)時(shí)，這個(gè)流程圖實(shí)際上會(huì)有更多的數(shù)據(jù)維度旅东。

導(dǎo)入

%matplotlib inline
import matplotlib.pyplot as plt
import tensorflow as tf
import numpy as np
from sklearn.metrics import confusion_matrix
import time
from datetime import timedelta
import math

# We also need PrettyTensor.
import prettytensor as pt

使用Python3.5（Anaconda）開發(fā)灭抑，TensorFlow版本是

tf.__version__

'1.0.0-rc0'

載入數(shù)據(jù)

MNIST數(shù)據(jù)集大約12MB，如果沒在文件夾中找到就會(huì)自動(dòng)下載抵代。

from tensorflow.examples.tutorials.mnist import input_data
data = input_data.read_data_sets('data/MNIST/', one_hot=True)

Extracting data/MNIST/train-images-idx3-ubyte.gz
Extracting data/MNIST/train-labels-idx1-ubyte.gz
Extracting data/MNIST/t10k-images-idx3-ubyte.gz
Extracting data/MNIST/t10k-labels-idx1-ubyte.gz

現(xiàn)在已經(jīng)載入了MNIST數(shù)據(jù)集腾节，它由70,000張圖像和對(duì)應(yīng)的標(biāo)簽（比如圖像的類別）組成。數(shù)據(jù)集分成三份互相獨(dú)立的子集荤牍。我們?cè)诮坛讨兄挥糜?xùn)練集和測(cè)試集案腺。

print("Size of:")
print("- Training-set:\t\t{}".format(len(data.train.labels)))
print("- Test-set:\t\t{}".format(len(data.test.labels)))
print("- Validation-set:\t{}".format(len(data.validation.labels)))

Size of:
-Training-set: 55000
-Test-set: 10000
-Validation-set: 5000

類型標(biāo)簽使用One-Hot編碼，這意外每個(gè)標(biāo)簽是長(zhǎng)為10的向量康吵，除了一個(gè)元素之外救湖，其他的都為零。這個(gè)元素的索引就是類別的數(shù)字涎才，即相應(yīng)圖片中畫的數(shù)字鞋既。我們也需要測(cè)試數(shù)據(jù)集類別數(shù)字的整型值力九，用下面的方法來計(jì)算。

data.test.cls = np.argmax(data.test.labels, axis=1)

數(shù)據(jù)維度

在下面的源碼中邑闺，有很多地方用到了數(shù)據(jù)維度跌前。它們只在一個(gè)地方定義，因此我們可以在代碼中使用這些數(shù)字而不是直接寫數(shù)字陡舅。

# We know that MNIST images are 28 pixels in each dimension.
img_size = 28

# Images are stored in one-dimensional arrays of this length.
img_size_flat = img_size * img_size

# Tuple with height and width of images used to reshape arrays.
img_shape = (img_size, img_size)

# Number of colour channels for the images: 1 channel for gray-scale.
num_channels = 1

# Number of classes, one class for each of 10 digits.
num_classes = 10

用來繪制圖片的幫助函數(shù)

這個(gè)函數(shù)用來在3x3的柵格中畫9張圖像抵乓，然后在每張圖像下面寫出真實(shí)類別和預(yù)測(cè)類別。

def plot_images(images, cls_true, cls_pred=None):
    assert len(images) == len(cls_true) == 9

    # Create figure with 3x3 sub-plots.
    fig, axes = plt.subplots(3, 3)
    fig.subplots_adjust(hspace=0.3, wspace=0.3)

    for i, ax in enumerate(axes.flat):
        # Plot image.
        ax.imshow(images[i].reshape(img_shape), cmap='binary')

        # Show true and predicted classes.
        if cls_pred is None:
            xlabel = "True: {0}".format(cls_true[i])
        else:
            xlabel = "True: {0}, Pred: {1}".format(cls_true[i], cls_pred[i])

        # Show the classes as the label on the x-axis.
        ax.set_xlabel(xlabel)

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()

繪制幾張圖像來看看數(shù)據(jù)是否正確

# Get the first images from the test-set.
images = data.test.images[0:9]

# Get the true classes for those images.
cls_true = data.test.cls[0:9]

# Plot the images and labels using our helper-function above.
plot_images(images=images, cls_true=cls_true)

image.png

TensorFlow圖

TensorFlow的全部目的就是使用一個(gè)稱之為計(jì)算圖（computational graph）的東西靶衍，它會(huì)比直接在Python中進(jìn)行相同計(jì)算量要高效得多灾炭。TensorFlow比Numpy更高效，因?yàn)門ensorFlow了解整個(gè)需要運(yùn)行的計(jì)算圖颅眶，然而Numpy只知道某個(gè)時(shí)間點(diǎn)上唯一的數(shù)學(xué)運(yùn)算蜈出。

TensorFlow也能夠自動(dòng)地計(jì)算需要優(yōu)化的變量的梯度，使得模型有更好的表現(xiàn)涛酗。這是由于圖是簡(jiǎn)單數(shù)學(xué)表達(dá)式的結(jié)合铡原，因此整個(gè)圖的梯度可以用鏈?zhǔn)椒▌t推導(dǎo)出來。

TensorFlow還能利用多核CPU和GPU商叹，Google也為TensorFlow制造了稱為TPUs（Tensor Processing Units）的特殊芯片燕刻，它比GPU更快。

一個(gè)TensorFlow圖由下面幾個(gè)部分組成剖笙，后面會(huì)詳細(xì)描述：

• 占位符變量（Placeholder）用來改變圖的輸入卵洗。
• 模型變量（Model）將會(huì)被優(yōu)化，使得模型表現(xiàn)得更好弥咪。
• 模型本質(zhì)上就是一些數(shù)學(xué)函數(shù)过蹂，它根據(jù)Placeholder和模型的輸入變量來計(jì)算一些輸出。
• 一個(gè)cost度量用來指導(dǎo)變量的優(yōu)化酪夷。
• 一個(gè)優(yōu)化策略會(huì)更新模型的變量。

另外孽惰，TensorFlow圖也包含了一些調(diào)試狀態(tài)晚岭，比如用TensorBoard打印log數(shù)據(jù)，本教程不涉及這些勋功。

占位符 （Placeholder）變量

Placeholder是作為圖的輸入坦报，每次我們運(yùn)行圖的時(shí)候都可能會(huì)改變它們。將這個(gè)過程稱為feeding placeholder變量狂鞋，后面將會(huì)描述它片择。

首先我們?yōu)檩斎雸D像定義placeholder變量。這讓我們可以改變輸入到TensorFlow圖中的圖像骚揍。這也是一個(gè)張量（tensor）字管，代表一個(gè)多維向量或矩陣啰挪。數(shù)據(jù)類型設(shè)置為float32，形狀設(shè)為[None, img_size_flat]嘲叔，None代表tensor可能保存著任意數(shù)量的圖像亡呵，每張圖象是一個(gè)長(zhǎng)度為img_size_flat的向量。

x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='x')

卷積層希望x被編碼為4維張量硫戈，因此我們需要將它的形狀轉(zhuǎn)換至[num_images, img_height, img_width, num_channels]锰什。注意img_height == img_width == img_size，如果第一維的大小設(shè)為-1丁逝， num_images的大小也會(huì)被自動(dòng)推導(dǎo)出來汁胆。轉(zhuǎn)換運(yùn)算如下：

x_image = tf.reshape(x, [-1, img_size, img_size, num_channels])

接下來我們?yōu)檩斎胱兞縳中的圖像所對(duì)應(yīng)的真實(shí)標(biāo)簽定義placeholder變量。變量的形狀是[None, num_classes]霜幼，這代表著它保存了任意數(shù)量的標(biāo)簽嫩码，每個(gè)標(biāo)簽是長(zhǎng)度為num_classes的向量，本例中長(zhǎng)度為10辛掠。

y_true = tf.placeholder(tf.float32, shape=[None, 10], name='y_true')

我們也可以為class-number提供一個(gè)placeholder谢谦，但這里用argmax來計(jì)算它。這里只是TensorFlow中的一些操作萝衩，沒有執(zhí)行什么運(yùn)算回挽。

y_true_cls = tf.argmax(y_true, dimension=1)

TensorFlow 實(shí)現(xiàn)

這一節(jié)顯示了教程 #02 中直接用TensorFlow實(shí)現(xiàn)卷積神經(jīng)網(wǎng)絡(luò)的源代碼。這份Notebook中并沒有直接用到這些代碼猩谊，只是為了方便和下面PrettyTensor的實(shí)現(xiàn)進(jìn)行比較千劈。

這里要注意的是有多少代碼量以及TensorFlow保存數(shù)據(jù)、進(jìn)行運(yùn)算的底層細(xì)節(jié)牌捷。即使在很小的神經(jīng)網(wǎng)絡(luò)中也容易犯錯(cuò)墙牌。

幫助函數(shù)

在直接用TensorFlow實(shí)現(xiàn)時(shí)，我們創(chuàng)建一些在構(gòu)造圖時(shí)常用到的幫助函數(shù)暗甥。

這兩個(gè)函數(shù)在TensorFlow圖中創(chuàng)建新的變量并用隨機(jī)值初始化喜滨。

def new_weights(shape):
    return tf.Variable(tf.truncated_normal(shape, stddev=0.05))

def new_biases(length):
    return tf.Variable(tf.constant(0.05, shape=[length]))

下面的幫助函數(shù)創(chuàng)建一個(gè)新的卷積網(wǎng)絡(luò)。輸入和輸出是4維的張量（4-rank tensors）撤防。注意TensorFlow API的底層細(xì)節(jié)虽风，比如權(quán)重變量的大小。這里很容易犯錯(cuò)寄月，可能會(huì)導(dǎo)致奇怪的錯(cuò)誤信息辜膝，并且很難調(diào)試。

def new_conv_layer(input,              # The previous layer.
                   num_input_channels, # Num. channels in prev. layer.
                   filter_size,        # Width and height of filters.
                   num_filters,        # Number of filters.
                   use_pooling=True):  # Use 2x2 max-pooling.

    # Shape of the filter-weights for the convolution.
    # This format is determined by the TensorFlow API.
    shape = [filter_size, filter_size, num_input_channels, num_filters]

    # Create new weights aka. filters with the given shape.
    weights = new_weights(shape=shape)

    # Create new biases, one for each filter.
    biases = new_biases(length=num_filters)

    # Create the TensorFlow operation for convolution.
    # Note the strides are set to 1 in all dimensions.
    # The first and last stride must always be 1,
    # because the first is for the image-number and
    # the last is for the input-channel.
    # But e.g. strides=[1, 2, 2, 1] would mean that the filter
    # is moved 2 pixels across the x- and y-axis of the image.
    # The padding is set to 'SAME' which means the input image
    # is padded with zeroes so the size of the output is the same.
    layer = tf.nn.conv2d(input=input,
                         filter=weights,
                         strides=[1, 1, 1, 1],
                         padding='SAME')

    # Add the biases to the results of the convolution.
    # A bias-value is added to each filter-channel.
    layer += biases

    # Use pooling to down-sample the image resolution?
    if use_pooling:
        # This is 2x2 max-pooling, which means that we
        # consider 2x2 windows and select the largest value
        # in each window. Then we move 2 pixels to the next window.
        layer = tf.nn.max_pool(value=layer,
                               ksize=[1, 2, 2, 1],
                               strides=[1, 2, 2, 1],
                               padding='SAME')

    # Rectified Linear Unit (ReLU).
    # It calculates max(x, 0) for each input pixel x.
    # This adds some non-linearity to the formula and allows us
    # to learn more complicated functions.
    layer = tf.nn.relu(layer)

    # Note that ReLU is normally executed before the pooling,
    # but since relu(max_pool(x)) == max_pool(relu(x)) we can
    # save 75% of the relu-operations by max-pooling first.

    # We return both the resulting layer and the filter-weights
    # because we will plot the weights later.
    return layer, weights

下面的幫助函數(shù)將一個(gè)4維張量轉(zhuǎn)換到2維漾肮，因此我們可以在卷積層之后添加一個(gè)全連接層厂抖。

def flatten_layer(layer):
    # Get the shape of the input layer.
    layer_shape = layer.get_shape()

    # The shape of the input layer is assumed to be:
    # layer_shape == [num_images, img_height, img_width, num_channels]

    # The number of features is: img_height * img_width * num_channels
    # We can use a function from TensorFlow to calculate this.
    num_features = layer_shape[1:4].num_elements()

    # Reshape the layer to [num_images, num_features].
    # Note that we just set the size of the second dimension
    # to num_features and the size of the first dimension to -1
    # which means the size in that dimension is calculated
    # so the total size of the tensor is unchanged from the reshaping.
    layer_flat = tf.reshape(layer, [-1, num_features])

    # The shape of the flattened layer is now:
    # [num_images, img_height * img_width * num_channels]

    # Return both the flattened layer and the number of features.
    return layer_flat, num_features

接下來的幫助函數(shù)創(chuàng)建一個(gè)全連接層。

def new_fc_layer(input,          # The previous layer.
                 num_inputs,     # Num. inputs from prev. layer.
                 num_outputs,    # Num. outputs.
                 use_relu=True): # Use Rectified Linear Unit (ReLU)?

    # Create new weights and biases.
    weights = new_weights(shape=[num_inputs, num_outputs])
    biases = new_biases(length=num_outputs)

    # Calculate the layer as the matrix multiplication of
    # the input and weights, and then add the bias-values.
    layer = tf.matmul(input, weights) + biases

    # Use ReLU?
    if use_relu:
        layer = tf.nn.relu(layer)

    return layer

構(gòu)造圖（Graph）

現(xiàn)在將會(huì)用上面的幫助函數(shù)來創(chuàng)建卷積神經(jīng)網(wǎng)絡(luò)克懊。如果沒有這些函數(shù)的話忱辅，代碼將會(huì)又長(zhǎng)又難以理解七蜘。

注意，我們并不會(huì)運(yùn)行下面的代碼耕蝉。寫在這里只是為了與PrettyTensor的代碼進(jìn)行比較崔梗。

之前的教程使用定義好的常量，因此很容易改變（變量）垒在。比如蒜魄，我們沒有將 filter_size=5 當(dāng)作 new_conv_layer()的參數(shù)，而是令filter_size=filter_size1场躯，然后在其他地方定義filter_size1=5谈为。這樣子就很容易改變所有的常量。

if False:  # Don't execute this! Just show it for easy comparison.
    # First convolutional layer.
    layer_conv1, weights_conv1 = \
        new_conv_layer(input=x_image,
                       num_input_channels=num_channels,
                       filter_size=5,
                       num_filters=16,
                       use_pooling=True)

    # Second convolutional layer.
    layer_conv2, weights_conv2 = \
        new_conv_layer(input=layer_conv1,
                       num_input_channels=16,
                       filter_size=5,
                       num_filters=36,
                       use_pooling=True)

    # Flatten layer.
    layer_flat, num_features = flatten_layer(layer_conv2)

    # First fully-connected layer.
    layer_fc1 = new_fc_layer(input=layer_flat,
                             num_inputs=num_features,
                             num_outputs=128,
                             use_relu=True)

    # Second fully-connected layer.
    layer_fc2 = new_fc_layer(input=layer_fc1,
                             num_inputs=128,
                             num_outputs=num_classes,
                             use_relu=False)

    # Predicted class-label.
    y_pred = tf.nn.softmax(layer_fc2)

    # Cross-entropy for the classification of each image.
    cross_entropy = \
        tf.nn.softmax_cross_entropy_with_logits(logits=layer_fc2,
                                                labels=y_true)

    # Loss aka. cost-measure.
    # This is the scalar value that must be minimized.
    loss = tf.reduce_mean(cross_entropy)

PrettyTensor實(shí)現(xiàn)

這一節(jié)演示如何用PrettyTensor來實(shí)現(xiàn)一個(gè)相同的卷積神經(jīng)網(wǎng)絡(luò)踢关。

基本思想就是用一個(gè)PrettyTensor object封裝輸入張量x_image伞鲫，它有一個(gè)添加新卷積層的幫助函數(shù)，以此來創(chuàng)建整個(gè)神經(jīng)網(wǎng)絡(luò)签舞。這有點(diǎn)像我們之前實(shí)現(xiàn)的那些幫助函數(shù)秕脓，但它更簡(jiǎn)單一些，因?yàn)镻rettyTensor記錄每一層的輸入和輸出維度等等儒搭。

x_pretty = pt.wrap(x_image)

現(xiàn)在我們已經(jīng)將輸入圖像裝到一個(gè)PrettyTensor的object中吠架，再用幾行代碼就可以添加卷積層和全連接層。

注意搂鲫，在with代碼塊中傍药，pt.defaults_scope(activation_fn=tf.nn.relu)把 activation_fn=tf.nn.relu當(dāng)作每個(gè)的層參數(shù)，因此這些層都用到了 Rectified Linear Units (ReLU) 魂仍。defaults_scope使我們能更方便地修改所有層的參數(shù)拐辽。

with pt.defaults_scope(activation_fn=tf.nn.relu):
    y_pred, loss = x_pretty.\
        conv2d(kernel=5, depth=16, name='layer_conv1').\
        max_pool(kernel=2, stride=2).\
        conv2d(kernel=5, depth=36, name='layer_conv2').\
        max_pool(kernel=2, stride=2).\
        flatten().\
        fully_connected(size=128, name='layer_fc1').\
        softmax_classifier(num_classes=num_classes, labels=y_true)

就是這樣！現(xiàn)在我們用幾行代碼就創(chuàng)建了一個(gè)完全一樣的卷積神經(jīng)網(wǎng)絡(luò)擦酌，如果直接用TensorFlow實(shí)現(xiàn)的話需要一大段非常復(fù)雜的代碼俱诸。

用PrettyTensor來代替TensorFlow，我們可以清楚地看到網(wǎng)絡(luò)的構(gòu)造以及數(shù)據(jù)如何在網(wǎng)絡(luò)中流通赊舶。這讓我們可以專注于神經(jīng)網(wǎng)絡(luò)的關(guān)鍵思想而不是底層的實(shí)現(xiàn)細(xì)節(jié)睁搭。它十分簡(jiǎn)單優(yōu)雅！

獲取權(quán)重

不幸的是锯岖，使用PrettyTensor時(shí)并非所有的事都那么優(yōu)雅介袜。

下面甫何，我們想要繪制出卷積層的權(quán)重出吹。在用TensorFlow實(shí)現(xiàn)時(shí)，我們自己創(chuàng)建了變量辙喂，所以可以直接訪問它們捶牢。但使用PrettyTensor構(gòu)造網(wǎng)絡(luò)時(shí)鸠珠，所有的變量都是間接地由PrettyTensor創(chuàng)建。因此我們不得不從TensorFlow中找回變量秋麸。

我們用layer_conv1 和 layer_conv2代表兩個(gè)卷積層渐排。這也叫變量作用域（不要與上面描述的defaults_scope混淆了）。PrettyTensor會(huì)自動(dòng)給它為每個(gè)層創(chuàng)建的變量命名灸蟆，因此我們可以通過層的作用域名稱和變量名來取得某一層的權(quán)重驯耻。

函數(shù)實(shí)現(xiàn)有點(diǎn)笨拙，因?yàn)槲覀儾坏貌挥肨ensorFlow函數(shù)get_variable()炒考，它是設(shè)計(jì)給其他用途的可缚，創(chuàng)建新的變量或重用現(xiàn)有變量。創(chuàng)建下面的幫助函數(shù)很簡(jiǎn)單斋枢。

def get_weights_variable(layer_name):
    # Retrieve an existing variable named 'weights' in the scope
    # with the given layer_name.
    # This is awkward because the TensorFlow function was
    # really intended for another purpose.

    with tf.variable_scope(layer_name, reuse=True):
        variable = tf.get_variable('weights')

    return variable

借助這個(gè)幫助函數(shù)我們可以獲取變量帘靡。這些是TensorFlow的objects。你需要類似的操作來獲取變量的內(nèi)容： contents = session.run(weights_conv1) 瓤帚，下面會(huì)提到這個(gè)描姚。

weights_conv1 = get_weights_variable(layer_name='layer_conv1')
weights_conv2 = get_weights_variable(layer_name='layer_conv2')

優(yōu)化方法

PrettyTensor給我們提供了預(yù)測(cè)類型標(biāo)簽(y_pred)以及一個(gè)需要最小化的損失度量，用來提升神經(jīng)網(wǎng)絡(luò)分類圖片的能力戈次。

PrettyTensor的文檔并沒有說明它的損失度量是用cross-entropy還是其他的轩勘。但現(xiàn)在我們用AdamOptimizer來最小化損失。

優(yōu)化過程并不是在這里執(zhí)行朝扼。實(shí)際上赃阀，還沒計(jì)算任何東西，我們只是往TensorFlow圖中添加了優(yōu)化器擎颖，以便后續(xù)操作榛斯。

optimizer = tf.train.AdamOptimizer(learning_rate=1e-4).minimize(loss)

性能度量

我們需要另外一些性能度量，來向用戶展示這個(gè)過程搂捧。

首先我們從神經(jīng)網(wǎng)絡(luò)輸出的y_pred中計(jì)算出預(yù)測(cè)的類別驮俗，它是一個(gè)包含10個(gè)元素的向量。類別數(shù)字是最大元素的索引允跑。

y_pred_cls = tf.argmax(y_pred, dimension=1)

然后創(chuàng)建一個(gè)布爾向量王凑，用來告訴我們每張圖片的真實(shí)類別是否與預(yù)測(cè)類別相同。

correct_prediction = tf.equal(y_pred_cls, y_true_cls)

上面的計(jì)算先將布爾值向量類型轉(zhuǎn)換成浮點(diǎn)型向量聋丝，這樣子False就變成0索烹，True變成1，然后計(jì)算這些值的平均數(shù)弱睦，以此來計(jì)算分類的準(zhǔn)確度百姓。

accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

運(yùn)行TensorFlow

創(chuàng)建TensorFlow會(huì)話（session）

一旦創(chuàng)建了TensorFlow圖，我們需要?jiǎng)?chuàng)建一個(gè)TensorFlow會(huì)話况木，用來運(yùn)行圖垒拢。

session = tf.Session()

初始化變量

我們需要在開始優(yōu)化weights和biases變量之前對(duì)它們進(jìn)行初始化旬迹。

session.run(tf.global_variables_initializer())

用來優(yōu)化迭代的幫助函數(shù)

在訓(xùn)練集中有50,000張圖。用這些圖像計(jì)算模型的梯度會(huì)花很多時(shí)間求类。因此我們利用隨機(jī)梯度下降的方法奔垦，它在優(yōu)化器的每次迭代里只用到了一小部分的圖像。
如果內(nèi)存耗盡導(dǎo)致電腦死機(jī)或變得很慢尸疆，你應(yīng)該試著減少這些數(shù)量椿猎，但同時(shí)可能還需要更優(yōu)化的迭代。

train_batch_size = 64

函數(shù)執(zhí)行了多次的優(yōu)化迭代來逐步地提升網(wǎng)絡(luò)層的變量寿弱。在每次迭代中鸵贬，從訓(xùn)練集中選擇一批新的數(shù)據(jù)，然后TensorFlow用這些訓(xùn)練樣本來執(zhí)行優(yōu)化器脖捻。每100次迭代會(huì)打印出相關(guān)信息阔逼。

# Counter for total number of iterations performed so far.
total_iterations = 0

def optimize(num_iterations):
    # Ensure we update the global variable rather than a local copy.
    global total_iterations

    # Start-time used for printing time-usage below.
    start_time = time.time()

    for i in range(total_iterations,
                   total_iterations + num_iterations):

        # Get a batch of training examples.
        # x_batch now holds a batch of images and
        # y_true_batch are the true labels for those images.
        x_batch, y_true_batch = data.train.next_batch(train_batch_size)

        # Put the batch into a dict with the proper names
        # for placeholder variables in the TensorFlow graph.
        feed_dict_train = {x: x_batch,
                           y_true: y_true_batch}

        # Run the optimizer using this batch of training data.
        # TensorFlow assigns the variables in feed_dict_train
        # to the placeholder variables and then runs the optimizer.
        session.run(optimizer, feed_dict=feed_dict_train)

        # Print status every 100 iterations.
        if i % 100 == 0:
            # Calculate the accuracy on the training-set.
            acc = session.run(accuracy, feed_dict=feed_dict_train)

            # Message for printing.
            msg = "Optimization Iteration: {0:>6}, Training Accuracy: {1:>6.1%}"

            # Print it.
            print(msg.format(i + 1, acc))

    # Update the total number of iterations performed.
    total_iterations += num_iterations

    # Ending time.
    end_time = time.time()

    # Difference between start and end-times.
    time_dif = end_time - start_time

    # Print the time-usage.
    print("Time usage: " + str(timedelta(seconds=int(round(time_dif)))))

用來繪制錯(cuò)誤樣本的幫助函數(shù)

函數(shù)用來繪制測(cè)試集中被誤分類的樣本。

def plot_example_errors(cls_pred, correct):
    # This function is called from print_test_accuracy() below.

    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # correct is a boolean array whether the predicted class
    # is equal to the true class for each image in the test-set.

    # Negate the boolean array.
    incorrect = (correct == False)

    # Get the images from the test-set that have been
    # incorrectly classified.
    images = data.test.images[incorrect]

    # Get the predicted classes for those images.
    cls_pred = cls_pred[incorrect]

    # Get the true classes for those images.
    cls_true = data.test.cls[incorrect]

    # Plot the first 9 images.
    plot_images(images=images[0:9],
                cls_true=cls_true[0:9],
                cls_pred=cls_pred[0:9])

繪制混淆（confusion）矩陣的幫助函數(shù)

def plot_confusion_matrix(cls_pred):
    # This is called from print_test_accuracy() below.

    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # Get the true classifications for the test-set.
    cls_true = data.test.cls

    # Get the confusion matrix using sklearn.
    cm = confusion_matrix(y_true=cls_true,
                          y_pred=cls_pred)

    # Print the confusion matrix as text.
    print(cm)

    # Plot the confusion matrix as an image.
    plt.matshow(cm)

    # Make various adjustments to the plot.
    plt.colorbar()
    tick_marks = np.arange(num_classes)
    plt.xticks(tick_marks, range(num_classes))
    plt.yticks(tick_marks, range(num_classes))
    plt.xlabel('Predicted')
    plt.ylabel('True')

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()

展示性能的幫助函數(shù)

函數(shù)用來打印測(cè)試集上的分類準(zhǔn)確度地沮。

為測(cè)試集上的所有圖片計(jì)算分類會(huì)花費(fèi)一段時(shí)間嗜浮，因此我們直接用這個(gè)函數(shù)來調(diào)用上面的結(jié)果，這樣就不用每次都重新計(jì)算了摩疑。

這個(gè)函數(shù)可能會(huì)占用很多電腦內(nèi)存危融，這也是為什么將測(cè)試集分成更小的幾個(gè)部分。如果你的電腦內(nèi)存比較小或死機(jī)了雷袋，就要試著降低batch-size吉殃。

# Split the test-set into smaller batches of this size.
test_batch_size = 256

def print_test_accuracy(show_example_errors=False,
                        show_confusion_matrix=False):

    # Number of images in the test-set.
    num_test = len(data.test.images)

    # Allocate an array for the predicted classes which
    # will be calculated in batches and filled into this array.
    cls_pred = np.zeros(shape=num_test, dtype=np.int)

    # Now calculate the predicted classes for the batches.
    # We will just iterate through all the batches.
    # There might be a more clever and Pythonic way of doing this.

    # The starting index for the next batch is denoted i.
    i = 0

    while i < num_test:
        # The ending index for the next batch is denoted j.
        j = min(i + test_batch_size, num_test)

        # Get the images from the test-set between index i and j.
        images = data.test.images[i:j, :]

        # Get the associated labels.
        labels = data.test.labels[i:j, :]

        # Create a feed-dict with these images and labels.
        feed_dict = {x: images,
                     y_true: labels}

        # Calculate the predicted class using TensorFlow.
        cls_pred[i:j] = session.run(y_pred_cls, feed_dict=feed_dict)

        # Set the start-index for the next batch to the
        # end-index of the current batch.
        i = j

    # Convenience variable for the true class-numbers of the test-set.
    cls_true = data.test.cls

    # Create a boolean array whether each image is correctly classified.
    correct = (cls_true == cls_pred)

    # Calculate the number of correctly classified images.
    # When summing a boolean array, False means 0 and True means 1.
    correct_sum = correct.sum()

    # Classification accuracy is the number of correctly classified
    # images divided by the total number of images in the test-set.
    acc = float(correct_sum) / num_test

    # Print the accuracy.
    msg = "Accuracy on Test-Set: {0:.1%} ({1} / {2})"
    print(msg.format(acc, correct_sum, num_test))

    # Plot some examples of mis-classifications, if desired.
    if show_example_errors:
        print("Example errors:")
        plot_example_errors(cls_pred=cls_pred, correct=correct)

    # Plot the confusion matrix, if desired.
    if show_confusion_matrix:
        print("Confusion Matrix:")
        plot_confusion_matrix(cls_pred=cls_pred)

優(yōu)化之前的性能

測(cè)試集上的準(zhǔn)確度很低，這是由于模型只做了初始化楷怒，并沒做任何優(yōu)化蛋勺，所以它只是對(duì)圖像做隨機(jī)分類。

print_test_accuracy()

Accuracy on Test-Set: 9.1% (909 / 10000)

1次迭代后的性能

做了一次優(yōu)化后鸠删，此時(shí)優(yōu)化器的學(xué)習(xí)率很低抱完，性能其實(shí)并沒有多大提升。

optimize(num_iterations=1)

Optimization Iteration: 1, Training Accuracy: 6.2%
Time usage: 0:00:00

print_test_accuracy()

Accuracy on Test-Set: 8.9% (892 / 10000)

100次迭代優(yōu)化后的性能

100次優(yōu)化迭代之后刃泡，模型顯著地提升了分類的準(zhǔn)確度巧娱。

optimize(num_iterations=99) # We already performed 1 iteration above.

Time usage: 0:00:00

print_test_accuracy(show_example_errors=True)

Accuracy on Test-Set: 83.9% (8393 / 10000)
Example errors:

image.png

1000次優(yōu)化迭代后的性能

1000次優(yōu)化迭代之后，模型在測(cè)試集上的準(zhǔn)確度超過了90%烘贴。

optimize(num_iterations=900) # We performed 100 iterations above.

Optimization Iteration: 101, Training Accuracy: 93.8%
Optimization Iteration: 201, Training Accuracy: 89.1%
Optimization Iteration: 301, Training Accuracy: 85.9%
Optimization Iteration: 401, Training Accuracy: 87.5%
Optimization Iteration: 501, Training Accuracy: 92.2%
Optimization Iteration: 601, Training Accuracy: 95.3%
Optimization Iteration: 701, Training Accuracy: 95.3%
Optimization Iteration: 801, Training Accuracy: 90.6%
Optimization Iteration: 901, Training Accuracy: 98.4%
Time usage: 0:00:03

print_test_accuracy(show_example_errors=True)

Accuracy on Test-Set: 96.3% (9634 / 10000)
Example errors:

image.png

10,000次優(yōu)化迭代后的性能

經(jīng)過10,000次優(yōu)化迭代后禁添，測(cè)試集上的分類準(zhǔn)確率高達(dá)99%。

optimize(num_iterations=9000) # We performed 1000 iterations above.

Optimization Iteration: 1001, Training Accuracy: 98.4%
Optimization Iteration: 1101, Training Accuracy: 95.3%
Optimization Iteration: 1201, Training Accuracy: 98.4%
Optimization Iteration: 1301, Training Accuracy: 96.9%
Optimization Iteration: 1401, Training Accuracy: 100.0%
Optimization Iteration: 1501, Training Accuracy: 95.3%
Optimization Iteration: 1601, Training Accuracy: 96.9%
Optimization Iteration: 1701, Training Accuracy: 96.9%
Optimization Iteration: 1801, Training Accuracy: 98.4%
Optimization Iteration: 1901, Training Accuracy: 96.9%
Optimization Iteration: 2001, Training Accuracy: 98.4%
Optimization Iteration: 2101, Training Accuracy: 95.3%
Optimization Iteration: 2201, Training Accuracy: 98.4%
Optimization Iteration: 2301, Training Accuracy: 98.4%
Optimization Iteration: 2401, Training Accuracy: 98.4%
Optimization Iteration: 2501, Training Accuracy: 93.8%
Optimization Iteration: 2601, Training Accuracy: 98.4%
Optimization Iteration: 2701, Training Accuracy: 98.4%
Optimization Iteration: 2801, Training Accuracy: 95.3%
Optimization Iteration: 2901, Training Accuracy: 98.4%
Optimization Iteration: 3001, Training Accuracy: 98.4%
Optimization Iteration: 3101, Training Accuracy: 100.0%
Optimization Iteration: 3201, Training Accuracy: 96.9%
Optimization Iteration: 3301, Training Accuracy: 100.0%
Optimization Iteration: 3401, Training Accuracy: 98.4%
Optimization Iteration: 3501, Training Accuracy: 96.9%
Optimization Iteration: 3601, Training Accuracy: 98.4%
Optimization Iteration: 3701, Training Accuracy: 96.9%
Optimization Iteration: 3801, Training Accuracy: 100.0%
Optimization Iteration: 3901, Training Accuracy: 98.4%
Optimization Iteration: 4001, Training Accuracy: 96.9%
Optimization Iteration: 4101, Training Accuracy: 98.4%
Optimization Iteration: 4201, Training Accuracy: 100.0%
Optimization Iteration: 4301, Training Accuracy: 100.0%
Optimization Iteration: 4401, Training Accuracy: 100.0%
Optimization Iteration: 4501, Training Accuracy: 100.0%
Optimization Iteration: 4601, Training Accuracy: 98.4%
Optimization Iteration: 4701, Training Accuracy: 96.9%
Optimization Iteration: 4801, Training Accuracy: 95.3%
Optimization Iteration: 4901, Training Accuracy: 100.0%
Optimization Iteration: 5001, Training Accuracy: 96.9%
Optimization Iteration: 5101, Training Accuracy: 100.0%
Optimization Iteration: 5201, Training Accuracy: 98.4%
Optimization Iteration: 5301, Training Accuracy: 98.4%
Optimization Iteration: 5401, Training Accuracy: 100.0%
Optimization Iteration: 5501, Training Accuracy: 98.4%
Optimization Iteration: 5601, Training Accuracy: 96.9%
Optimization Iteration: 5701, Training Accuracy: 100.0%
Optimization Iteration: 5801, Training Accuracy: 96.9%
Optimization Iteration: 5901, Training Accuracy: 100.0%
Optimization Iteration: 6001, Training Accuracy: 98.4%
Optimization Iteration: 6101, Training Accuracy: 98.4%
Optimization Iteration: 6201, Training Accuracy: 98.4%
Optimization Iteration: 6301, Training Accuracy: 98.4%
Optimization Iteration: 6401, Training Accuracy: 100.0%
Optimization Iteration: 6501, Training Accuracy: 100.0%
Optimization Iteration: 6601, Training Accuracy: 100.0%
Optimization Iteration: 6701, Training Accuracy: 100.0%
Optimization Iteration: 6801, Training Accuracy: 96.9%
Optimization Iteration: 6901, Training Accuracy: 100.0%
Optimization Iteration: 7001, Training Accuracy: 100.0%
Optimization Iteration: 7101, Training Accuracy: 100.0%
Optimization Iteration: 7201, Training Accuracy: 100.0%
Optimization Iteration: 7301, Training Accuracy: 96.9%
Optimization Iteration: 7401, Training Accuracy: 100.0%
Optimization Iteration: 7501, Training Accuracy: 100.0%
Optimization Iteration: 7601, Training Accuracy: 96.9%
Optimization Iteration: 7701, Training Accuracy: 100.0%
Optimization Iteration: 7801, Training Accuracy: 100.0%
Optimization Iteration: 7901, Training Accuracy: 100.0%
Optimization Iteration: 8001, Training Accuracy: 98.4%
Optimization Iteration: 8101, Training Accuracy: 100.0%
Optimization Iteration: 8201, Training Accuracy: 100.0%
Optimization Iteration: 8301, Training Accuracy: 100.0%
Optimization Iteration: 8401, Training Accuracy: 100.0%
Optimization Iteration: 8501, Training Accuracy: 98.4%
Optimization Iteration: 8601, Training Accuracy: 100.0%
Optimization Iteration: 8701, Training Accuracy: 100.0%
Optimization Iteration: 8801, Training Accuracy: 100.0%
Optimization Iteration: 8901, Training Accuracy: 100.0%
Optimization Iteration: 9001, Training Accuracy: 98.4%
Optimization Iteration: 9101, Training Accuracy: 98.4%
Optimization Iteration: 9201, Training Accuracy: 100.0%
Optimization Iteration: 9301, Training Accuracy: 100.0%
Optimization Iteration: 9401, Training Accuracy: 98.4%
Optimization Iteration: 9501, Training Accuracy: 100.0%
Optimization Iteration: 9601, Training Accuracy: 100.0%
Optimization Iteration: 9701, Training Accuracy: 100.0%
Optimization Iteration: 9801, Training Accuracy: 98.4%
Optimization Iteration: 9901, Training Accuracy: 100.0%
Time usage: 0:00:27

print_test_accuracy(show_example_errors=True,
                    show_confusion_matrix=True)

Accuracy on Test-Set: 98.8% (9881 / 10000)
Example errors:

image.png

Confusion Matrix:
[[ 975 0 0 0 0 0 1 1 3 0]
[ 0 1127 2 0 0 0 1 2 3 0]
[ 2 2 1019 1 1 0 1 2 4 0]
[ 0 0 0 1005 0 1 0 1 3 0]
[ 0 0 0 0 977 0 1 0 1 3]
[ 2 0 0 13 0 870 1 0 6 0]
[ 5 2 0 0 1 3 943 0 4 0]
[ 0 2 8 2 1 0 0 1007 1 7]
[ 2 0 2 3 1 1 0 0 964 1]
[ 0 2 0 4 5 1 0 1 2 994]]

image.png

權(quán)重和層的可視化

當(dāng)我們直接用TensorFlow來實(shí)現(xiàn)卷積神經(jīng)網(wǎng)絡(luò)時(shí)桨踪，可以很容易地畫出卷積權(quán)重和不同層的輸出圖像老翘。當(dāng)使用PrettyTensor的時(shí)候，我們也可以通過上面提到過的方法取得權(quán)重，但我們無法簡(jiǎn)單得到卷積層的輸出（圖像）酪捡。因此下面只繪制了權(quán)重。

繪制卷積權(quán)重的幫助函數(shù)

def plot_conv_weights(weights, input_channel=0):
    # Assume weights are TensorFlow ops for 4-dim variables
    # e.g. weights_conv1 or weights_conv2.

    # Retrieve the values of the weight-variables from TensorFlow.
    # A feed-dict is not necessary because nothing is calculated.
    w = session.run(weights)

    # Get the lowest and highest values for the weights.
    # This is used to correct the colour intensity across
    # the images so they can be compared with each other.
    w_min = np.min(w)
    w_max = np.max(w)

    # Number of filters used in the conv. layer.
    num_filters = w.shape[3]

    # Number of grids to plot.
    # Rounded-up, square-root of the number of filters.
    num_grids = math.ceil(math.sqrt(num_filters))

    # Create figure with a grid of sub-plots.
    fig, axes = plt.subplots(num_grids, num_grids)

    # Plot all the filter-weights.
    for i, ax in enumerate(axes.flat):
        # Only plot the valid filter-weights.
        if i<num_filters:
            # Get the weights for the i'th filter of the input channel.
            # See new_conv_layer() for details on the format
            # of this 4-dim tensor.
            img = w[:, :, input_channel, i]

            # Plot image.
            ax.imshow(img, vmin=w_min, vmax=w_max,
                      interpolation='nearest', cmap='seismic')

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()

卷積層 1

現(xiàn)在繪制第一個(gè)卷積層的濾波權(quán)重纳账。

其中正值權(quán)重是紅色的逛薇，負(fù)值為藍(lán)色。

plot_conv_weights(weights=weights_conv1)

image.png

卷積層 2

現(xiàn)在繪制第二個(gè)卷積層的濾波權(quán)重疏虫。

第一個(gè)卷積層有16個(gè)輸出通道永罚，代表著第二個(gè)卷基層有16個(gè)輸入。第二個(gè)卷積層的每個(gè)輸入通道也有一些權(quán)重濾波卧秘。我們先繪制第一個(gè)通道的權(quán)重濾波呢袱。

同樣的，正值是紅色翅敌，負(fù)值是藍(lán)色羞福。

plot_conv_weights(weights=weights_conv2, input_channel=0)

image.png

第二個(gè)卷積層共有16個(gè)輸入通道，我們可以同樣地畫出15張其他濾波權(quán)重圖像蚯涮。這里我們?cè)佼嬕幌碌诙€(gè)通道的圖像

plot_conv_weights(weights=weights_conv2, input_channel=1)

image.png

關(guān)閉TensorFlow會(huì)話

現(xiàn)在我們已經(jīng)用TensorFlow完成了任務(wù)治专，關(guān)閉session，釋放資源遭顶。

總結(jié)

相比直接使用TensorFlow张峰，PrettyTensor可以用更簡(jiǎn)單的代碼來實(shí)現(xiàn)神經(jīng)網(wǎng)絡(luò)。這使你能夠?qū)Ｗ⒂谧约旱南敕ǘ皇堑讓拥膶?shí)現(xiàn)細(xì)節(jié)棒旗。它讓代碼更易于理解喘批，也減少犯錯(cuò)的可能。

然而铣揉，PrettyTensor中有一些矛盾和笨拙的設(shè)計(jì)饶深，它的文檔簡(jiǎn)短而又令人疑惑，也不易于學(xué)習(xí)逛拱。希望未來會(huì)有所改進(jìn)（本文寫于2016七月）粥喜。
還有一些PrettyTensor的替代品，包括TFLearn和Keras橘券。

練習(xí)

下面使一些可能會(huì)讓你提升TensorFlow技能的一些建議練習(xí)额湘。為了學(xué)習(xí)如何更合適地使用TensorFlow，實(shí)踐經(jīng)驗(yàn)是很重要的旁舰。

在你對(duì)這個(gè)Notebook進(jìn)行修改之前锋华，可能需要先備份一下。

• 將所有層的激活函數(shù)改成sigmod箭窜。
• 在一些層中使用sigmod毯焕，一些層中使用relu。這里能用defaults_scope嗎？
• 在所有層里使用12loss纳猫。然后試著只在某些層里使用這個(gè)婆咸。
• 用PrettyTensor的reshape函數(shù)代替TensorFlow的。其中某一個(gè)會(huì)更好嗎芜辕？
• 在全連接層后面添加一個(gè)dropout-layer尚骄。如果你在訓(xùn)練和測(cè)試的時(shí)候想要一個(gè)不同的keep_prob，就需要在feed-dict中設(shè)置一個(gè)placeholder變量侵续。
• 用stride=2來代替 2x2 max-pooling層倔丈。分類準(zhǔn)確率會(huì)有所不同么？你多次優(yōu)化它們之后呢状蜗？差異是隨機(jī)的需五，你如何度量是否真實(shí)存在差異呢？在卷積層中使用max-pooling和stride的優(yōu)缺點(diǎn)是什么轧坎？
• 改變層的參數(shù)宏邮，比如kernel、depth缸血、size等等蜀铲。耗費(fèi)的時(shí)間以及分類準(zhǔn)確率有什么差別？
• 添加或刪除某些卷積層和全連接層属百。
• 你設(shè)計(jì)的表現(xiàn)良好的最簡(jiǎn)網(wǎng)絡(luò)是什么记劝？
• 取回卷積層的bias-values并打印出來。參考一下get_weights_variable()的實(shí)現(xiàn)族扰。
• 不看源碼厌丑，自己重寫程序。
• 向朋友解釋程序如何工作渔呵。