文章目录

一、多输入多输出网络1.多标签分类2.多输出分类（1）数据预处理（2）网络结构（3）loss3.使用keras处理回归问题4.使用CNN对输入图片进行回归问题5.多种类输入二、关于改进模型的一些研究1.使用keras搭建高级网络结构2.fit、fit_generator和train_on_batch（1）fit（2）fit_generator（3）train_on_batch3.多GPU训练4.使用opencv实现神经风格迁移5.使用OpenCV将黑白图片转换为彩色图片三、自动机器学习

本篇内容主要是多输入多输出网络、卷积神经网络进阶、自动机器学习

注：本文内容及代码均转自 pyimagesearch

一、多输入多输出网络

在卷积神经网络训练中，有时需要输出多个不同类别的特征，例如同时获得颜色和物品类别，更详细地来说判断这是个蓝色牛仔裤还是红色衬衫。这是使用多标签分类或者多输出分类的方法得到结果。

1.多标签分类

原文链接

多标签分类的重点是对y值的预处理，其余部分则和普通的神经网络训练相同。对于分类的特征单一的问题，很多时候使用one-hot编码即可将不同类别进行向量表示，但对于具有多个需要得到的特征，则需要进行进一步处理。这里举个例子，假设数据集包含六个类别，分别是黑色牛仔裤、蓝色裙子、蓝色牛仔裤、蓝色衬衫、红色裙子、红色衬衫。具体的处理方法是通过sklearn的MultiLabelBinarizer类进行处理，代码如下（部分）：

from sklearn.preprocessing import MultiLabelBinarizerlabels = [("blue", "jeans"),("blue", "dress"),("red", "dress"),("red", "shirt"),("blue", "shirt"),("black", "jeans")]mlb = MultiLabelBinarizer()labels = mlb.fit_transform(labels)

labels为可能存在的不同标签，首先初始化MultiLabelBinarizer函数，接着使用fit_transform方法即可获得多标签的向量表示，fit_transform可拆分为fit和transform两部分

mlb.fit(labels)mlb.classes_mlb.transform([("red", "dress")])

fit方法可以统计labels的种类，对于该数据集则为[‘black’, ‘blue’, ‘dress’, ‘jeans’, ‘red’, ‘shirt’]，通过classes_属性可以输出种类，transform方法则是对labels中每个数据进行转换，例如(“red”, “dress”)可以转换为array([[0, 0, 1, 0, 1, 0]])

2.多输出分类

原文链接

多输出分类和多标签分类都是应对多种类别输出的方法，区别在于多标签分类在网络末尾只有一个用来进行分类的线性层，而多输出分类则有至少两个分支。还是以上面的例子为例，多输出分类关键部分主要有三个，分别是数据预处理、网络模型和loss

（1）数据预处理

数据预处理时要将颜色信息和类别信息单独列出来，也就是说最后训练的时候需要有三个列表，输入图像、颜色标签信息和类别标签信息，要一一对应。

data = []categoryLabels = []colorLabels = []

（2）网络结构

在之前的网络搭建中，我们使用的是Sequential方式，即初始化一个Sequential（），再使用add方法添加所需要的层结构，这种方法比较简单，但无法实现复杂的层结构，比如多输入或者多输出结构，所以我们采用functional模型来进行多输入多输出结构。

在原有的神经网络结构基础上，多输出分类网络有两个分支，如图所示：

部分代码（省略了大部分中间层）参考：

class FashionNet:@staticmethoddef build_category_branch(inputs, numCategories,finalAct="softmax", chanDim=-1):# 灰度转换，减弱颜色影响x = Lambda(lambda c: tf.image.rgb_to_grayscale(c))(inputs)x = Conv2D(32, (3, 3), padding="same")(x)x = Activation("relu")(x)x = BatchNormalization(axis=chanDim)(x)x = MaxPooling2D(pool_size=(3, 3))(x)x = Dropout(0.25)(x)x = Flatten()(x)x = Dense(256)(x)x = Activation("relu")(x)x = BatchNormalization()(x)x = Dropout(0.5)(x)x = Dense(numCategories)(x)x = Activation(finalAct, name="category_output")(x)# return the category prediction sub-networkreturn xdef build_color_branch(inputs, numColors, finalAct="softmax",chanDim=-1):x = Conv2D(32, (3, 3), padding="same")(x)x = Activation("relu")(x)x = BatchNormalization(axis=chanDim)(x)x = MaxPooling2D(pool_size=(3, 3))(x)x = Dropout(0.25)(x)x = Flatten()(x)x = Dense(256)(x)x = Activation("relu")(x)x = BatchNormalization()(x)x = Dropout(0.5)(x)x = Dense(numCategories)(x)x = Activation(finalAct, name="category_output")(x)# return the category prediction sub-networkreturn xdef build(width, height, numCategories, numColors,finalAct="softmax"):# initialize the input shape and channel dimension (this code# assumes you are using TensorFlow which utilizes channels# last ordering)inputShape = (height, width, 3)chanDim = -1# construct both the "category" and "color" sub-networksinputs = Input(shape=inputShape)categoryBranch = FashionNet.build_category_branch(inputs,numCategories, finalAct=finalAct, chanDim=chanDim)colorBranch = FashionNet.build_color_branch(inputs,numColors, finalAct=finalAct, chanDim=chanDim)# create the model using our input (the batch of images) and# two separate outputs -- one for the clothing category# branch and another for the color branch, respectivelymodel = Model(inputs=inputs,outputs=[categoryBranch, colorBranch],name="fashionnet")# return the constructed network architecturereturn model

（3）loss

由于有两个分支，所以就会有两个loss，在定义多个loss时，需要通过两个分支最后输出层的名字，所以要提前定义一下，另外可以定义lossWeights 来选择两个loss所占的比重。在训练时需要一个字典来传递标签。

model = FashionNet.build(96, 96,numCategories=len(categoryLB.classes_),numColors=len(colorLB.classes_),finalAct="softmax")losses = {"category_output": "categorical_crossentropy","color_output": "categorical_crossentropy",}lossWeights = {"category_output": 1.0, "color_output": 1.0}opt = Adam(lr=INIT_LR, decay=INIT_LR / EPOCHS)pile(optimizer=opt, loss=losses, loss_weights=lossWeights,metrics=["accuracy"])H = model.fit(x=trainX,y={"category_output": trainCategoryY, "color_output": trainColorY},validation_data=(testX,{"category_output": testCategoryY, "color_output": testColorY}),epochs=EPOCHS,verbose=1

3.使用keras处理回归问题

原文链接

回归问题，不同于分类问题，需要预测的结果是一个数值，以房价数据集为例，该数据集输入有四个属性，分别是卧室数量、浴室数量、面积、邮政编码，输出为房子的价格。使用keras处理单一的回归问题比较简单，只需要一个神经网络模型，最后一层只有一个神经元即可。

from sklearn.preprocessing import LabelBinarizerfrom sklearn.preprocessing import MinMaxScalerimport pandas as pdimport numpy as npimport globimport cv2import osdef load_house_attributes(inputPath):# initialize the list of column names in the CSV file and then# load it using Pandascols = ["bedrooms", "bathrooms", "area", "zipcode", "price"]df = pd.read_csv(inputPath, sep=" ", header=None, names=cols)# determine (1) the unique zip codes and (2) the number of data# points with each zip codezipcodes = df["zipcode"].value_counts().keys().tolist()counts = df["zipcode"].value_counts().tolist()# loop over each of the unique zip codes and their corresponding# countfor (zipcode, count) in zip(zipcodes, counts):# the zip code counts for our housing dataset is *extremely*# unbalanced (some only having 1 or 2 houses per zip code)# so let's sanitize our data by removing any houses with less# than 25 houses per zip codeif count < 25:idxs = df[df["zipcode"] == zipcode].indexdf.drop(idxs, inplace=True)# return the data framereturn dfdef process_house_attributes(df, train, test):# initialize the column names of the continuous datacontinuous = ["bedrooms", "bathrooms", "area"]# performin min-max scaling each continuous feature column to# the range [0, 1]cs = MinMaxScaler()trainContinuous = cs.fit_transform(train[continuous])testContinuous = cs.transform(test[continuous])# one-hot encode the zip code categorical data (by definition of# one-hot encoing, all output features are now in the range [0, 1])zipBinarizer = LabelBinarizer().fit(df["zipcode"])trainCategorical = zipBinarizer.transform(train["zipcode"])testCategorical = zipBinarizer.transform(test["zipcode"])# construct our training and testing data points by concatenating# the categorical features with the continuous featurestrainX = np.hstack([trainCategorical, trainContinuous])testX = np.hstack([testCategorical, testContinuous])# return the concatenated training and testing datareturn (trainX, testX)

以上代码为简单的数据预处理，load_house_attributes函数读入文件兵删除了zipcode中出现次数较少的项，process_house_attributes函数对三个数值型变量进行了归一化转换，对zipcode进行了one-hot编码。

def create_mlp(dim, regress=False):# define our MLP networkmodel = Sequential()model.add(Dense(8, input_dim=dim, activation="relu"))model.add(Dense(4, activation="relu"))# check to see if the regression node should be addedif regress:model.add(Dense(1, activation="linear"))# return our modelreturn model

以上为网络结构，处理回归问题需要在最后加上一个神经元。

pile(loss="mean_absolute_percentage_error", optimizer=opt)

在编译时loss不能使用之前的categorical_crossentropy，要改为mean squared error, mean absolute error, mean absolute percentage error等。

之后的训练过程和之前大同小异，不再赘述。

4.使用CNN对输入图片进行回归问题

原文链接

上例介绍了简单的回归过程，当数据集输入改为图片时，还是以房价数据集为例，输入为四种图片，分别为卧室图片、厨房图片、卫生间图片和整体外观，输出依旧为房价。使用四种图片训练一个回归网络主要有三种思路：

通过CNN一次传递一个图像，并使用房子的价格作为每个图像的目标值创建四个CNN分支最终合并为一个输出将四张图片利用拼接的方式混合到一起送入CNN

方法一显然不行，四张图片对应一个输出，如果一次只训练一张，很难训练好的网络。

方法二虽然理论上可行，但训练四个CNN分支有很大的计算花销而且难以训练。

所以我们选择方法三将四张图片混合成一张图片，例如这样：

拼接方式如下：

inputImages = []outputImage = np.zeros((64, 64, 3), dtype="uint8")for housePath in housePaths:image = cv2.imread(housePath)image = cv2.resize(image, (32, 32))inputImages.append(image)outputImage[0:32, 0:32] = inputImages[0]outputImage[0:32, 32:64] = inputImages[1]outputImage[32:64, 32:64] = inputImages[2]outputImage[32:64, 0:32] = inputImages[3]

其余的网络结构就是在网络的最后一层加一个神经元，loss更改和上例相同。

5.多种类输入

原文链接

本节内容将上面两个例子结合到一起，即输入为多种图像数据和多种数值数据。读入数据的方式也之前相同，难点主要在于网咯的搭建。

网络结构主要包含两个分支，一个为处理数值类型数据的多层感知机，另一个为处理图像数据的CNN，结构如下图：

类似于之前处理多输出时采用的网络结构，在处理多输入问题时也采用functional模型来搭建网络

# import the necessary packagesfrom tensorflow.keras.models import Sequentialfrom tensorflow.keras.layers import BatchNormalizationfrom tensorflow.keras.layers import Conv2Dfrom tensorflow.keras.layers import MaxPooling2Dfrom tensorflow.keras.layers import Activationfrom tensorflow.keras.layers import Dropoutfrom tensorflow.keras.layers import Densefrom tensorflow.keras.layers import Flattenfrom tensorflow.keras.layers import Inputfrom tensorflow.keras.models import Model

导入包

def create_mlp(dim, regress=False):# define our MLP networkmodel = Sequential()model.add(Dense(8, input_dim=dim, activation="relu"))model.add(Dense(4, activation="relu"))# check to see if the regression node should be addedif regress:model.add(Dense(1, activation="linear"))# return our modelreturn model

处理数值型数据的多层感知机，regress设置为False为之后的拼接做准备

def create_cnn(width, height, depth, filters=(16, 32, 64), regress=False):# initialize the input shape and channel dimension, assuming# TensorFlow/channels-last orderinginputShape = (height, width, depth)chanDim = -1# define the model inputinputs = Input(shape=inputShape)# loop over the number of filtersfor (i, f) in enumerate(filters):# if this is the first CONV layer then set the input# appropriatelyif i == 0:x = inputs# CONV => RELU => BN => POOLx = Conv2D(f, (3, 3), padding="same")(x)x = Activation("relu")(x)x = BatchNormalization(axis=chanDim)(x)x = MaxPooling2D(pool_size=(2, 2))(x)# flatten the volume, then FC => RELU => BN => DROPOUTx = Flatten()(x)x = Dense(16)(x)x = Activation("relu")(x)x = BatchNormalization(axis=chanDim)(x)x = Dropout(0.5)(x)# apply another FC layer, this one to match the number of nodes# coming out of the MLPx = Dense(4)(x)x = Activation("relu")(x)# check to see if the regression node should be addedif regress:x = Dense(1, activation="linear")(x)# construct the CNNmodel = Model(inputs, x)# return the CNNreturn model

处理图像数据的CNN，同样regress设置为false

mlp = create_mlp(trainAttrX.shape[1], regress=False)cnn = create_cnn(64, 64, 3, regress=False)# create the input to our final set of layers as the *output* of both# the MLP and CNNcombinedInput = concatenate([mlp.output, cnn.output])# our final FC layer head will have two dense layers, the final one# being our regression headx = Dense(4, activation="relu")(combinedInput)x = Dense(1, activation="linear")(x)# our final model will accept categorical/numerical data on the MLP# input and images on the CNN input, outputting a single value (the# predicted price of the house)model = Model(inputs=[mlp.input, cnn.input], outputs=x)

将两个网络的输出用concatenate拼接起来送入之后的线性层得到最终模型。之后的训练过程与之前无异。

二、关于改进模型的一些研究

1.使用keras搭建高级网络结构

原文链接

在keras中内置了比较流行的网络模型以及预训练参数，需要的话可以直接调用

from tensorflow.keras.applications import ResNet50from tensorflow.keras.applications import InceptionV3from tensorflow.keras.applications import Xception # TensorFlow ONLYfrom tensorflow.keras.applications import VGG16from tensorflow.keras.applications import VGG19MODELS = {"vgg16": VGG16,"vgg19": VGG19,"inception": InceptionV3,"xception": Xception, # TensorFlow ONLY"resnet": ResNet50}Network = MODELS[args["model"]]model = Network(weights="imagenet")preds = model.predict(image)

2.fit、fit_generator和train_on_batch

原文链接

在keras中，训练模型有三种方式，分别是fit、fit_generator和train_on_batch，当然，在TensorFlow2.2及以上的版本中，fit也可以接受生成器的参数，所以使用高版本的建议直接用fit。

（1）fit

model.fit(trainX, trainY, batch_size=32, epochs=50)

以上为使用fit时候的参数，这里需要将所有的数据存入fit中且无法使用数据增强（在TensorFlow高版本中可以实现）

（2）fit_generator

EPOCHS = 100BS = 32# construct the training image generator for data augmentationaug = ImageDataGenerator(rotation_range=20, zoom_range=0.15,width_shift_range=0.2, height_shift_range=0.2, shear_range=0.15,horizontal_flip=True, fill_mode="nearest")# train the networkH = model.fit_generator(aug.flow(trainX, trainY, batch_size=BS),validation_data=(testX, testY), steps_per_epoch=len(trainX) // BS,epochs=EPOCHS)

fit_generator的第一个参数需要输入为一个生成器，这样做可以极大的减轻内存的消耗，设置steps_per_epoch参数是由于生成器会不断运行，keras需要知道什么时候进入了下一个epoch

def csv_image_generator(inputPath, bs, lb, mode="train", aug=None):# open the CSV file for readingf = open(inputPath, "r")# loop indefinitelywhile True:# initialize our batches of images and labelsimages = []labels = []# keep looping until we reach our batch sizewhile len(images) < bs:# attempt to read the next line of the CSV fileline = f.readline()# check to see if the line is empty, indicating we have# reached the end of the fileif line == "":# reset the file pointer to the beginning of the file# and re-read the linef.seek(0)line = f.readline()# if we are evaluating we should now break from our# loop to ensure we don't continue to fill up the# batch from samples at the beginning of the fileif mode == "eval":break# extract the label and construct the imageline = line.strip().split(",")label = line[0]image = np.array([int(x) for x in line[1:]], dtype="float32")image = image.reshape((64, 64, 3))# update our corresponding batches listsimages.append(image)labels.append(label)# one-hot encode the labelslabels = lb.transform(np.array(labels))# if the data augmentation object is not None, apply itif aug is not None:(images, labels) = next(aug.flow(np.array(images),labels, batch_size=bs))# yield the batch to the calling functionyield (np.array(images), labels)

以上为一个生成器实例

（3）train_on_batch

model.train_on_batch(batchX, batchY)

train_on_batch 函数接收单独一个batch的数据，进行反向传播然后更新参数，很少使用不详述

3.多GPU训练

原文链接

from keras.utils import multi_gpu_model# 将 `model` 复制到 8 个 GPU 上。# 假定你的机器有 8 个可用的 GPU。parallel_model = multi_gpu_model(model, gpus=8)pile(loss='categorical_crossentropy',optimizer='rmsprop')# 这个 `fit` 调用将分布在 8 个 GPU 上。# 由于 batch size 为 256，每个 GPU 将处理 32 个样本。parallel_model.fit(x, y, epochs=20, batch_size=256)

keras利用多GPU训练很简单，以上为官方文档一个简单的例子

4.使用opencv实现神经风格迁移

原文链接

神经风格迁移的意思是将一张图片的风格加在另一张图片上形成一张新的图片，例子如下：

利用OpenCV实现神经风格迁移需要用的其中的dnn类

import argparseimport imutilsimport timeimport cv2# load the neural style transfer model from diskprint("[INFO] loading style transfer model...")net = cv2.dnn.readNetFromTorch(args["model"])# load the input image, resize it to have a width of 600 pixels, and# then grab the image dimensionsimage = cv2.imread(args["image"])image = imutils.resize(image, width=600)(h, w) = image.shape[:2]# construct a blob from the image, set the input, and then perform a# forward pass of the networkblob = cv2.dnn.blobFromImage(image, 1.0, (w, h),(103.939, 116.779, 123.680), swapRB=False, crop=False)net.setInput(blob)output = net.forward()

首先net为加载的预训练的风格模型，之后使用blobFromImage函数对图像进行减均值和缩放两种操作，最后通过forward函数得到输出结果

# reshape the output tensor, add back in the mean subtraction, and# then swap the channel orderingoutput = output.reshape((3, output.shape[2], output.shape[3]))output[0] += 103.939output[1] += 116.779output[2] += 123.680output /= 255.0output = output.transpose(1, 2, 0)

将输出结果进行逆变换和缩放操作，将矩阵变为能够送入神经网络的结构

# show information on how long inference tookprint("[INFO] neural style transfer took {:.4f} seconds".format(end - start))# show the imagescv2.imshow("Input", image)cv2.imshow("Output", output)cv2.waitKey(0)

输出结果

5.使用OpenCV将黑白图片转换为彩色图片

原文链接

这里介绍一种图片编码方法——LAB色彩空间，其中L代表亮度，A代表绿-红色，B代表蓝-黄色，由于LAB颜色空间中的L通道不带有颜色信息，所以可以当做黑白图片作为输入，A和B通道可以作为将要预测的结构

# load our serialized black and white colorizer model and cluster# center points from diskprint("[INFO] loading model...")net = cv2.dnn.readNetFromCaffe(args["prototxt"], args["model"])pts = np.load(args["points"])# add the cluster centers as 1x1 convolutions to the modelclass8 = net.getLayerId("class8_ab")conv8 = net.getLayerId("conv8_313_rh")pts = pts.transpose().reshape(2, 313, 1, 1)net.getLayer(class8).blobs = [pts.astype("float32")]net.getLayer(conv8).blobs = [np.full([1, 313], 2.606, dtype="float32")]

首先载入用于黑白色转彩色的配置文件和预训练模型，后面没太看懂T_T

image = cv2.imread(args["image"])scaled = image.astype("float32") / 255.0lab = cv2.cvtColor(scaled, cv2.COLOR_BGR2LAB)resized = cv2.resize(lab, (224, 224))L = cv2.split(resized)[0]L -= 50

读入文件并获取L通道

'print("[INFO] colorizing image...")'net.setInput(cv2.dnn.blobFromImage(L))ab = net.forward()[0, :, :, :].transpose((1, 2, 0))# resize the predicted 'ab' volume to the same dimensions as our# input imageab = cv2.resize(ab, (image.shape[1], image.shape[0]))# grab the 'L' channel from the *original* input image (not the# resized one) and concatenate the original 'L' channel with the# predicted 'ab' channelsL = cv2.split(lab)[0]colorized = np.concatenate((L[:, :, np.newaxis], ab), axis=2)# convert the output image from the Lab color space to RGB, then# clip any values that fall outside the range [0, 1]colorized = cv2.cvtColor(colorized, cv2.COLOR_LAB2BGR)colorized = np.clip(colorized, 0, 1)# the current colorized image is represented as a floating point# data type in the range [0, 1] -- let's convert to an unsigned# 8-bit integer representation in the range [0, 255]colorized = (255 * colorized).astype("uint8")# show the original and output colorized imagescv2.imshow("Original", image)cv2.imshow("Colorized", colorized)cv2.waitKey(0)

forward函数获取得到的A、B通道数据，组合在一起得到最终结果

三、自动机器学习

原文链接

自动机器学习（AutoML）：自动机器学习能够自动的调整超参数，以Google AutoML为例，自动机器学习的主要步骤为：

在训练集上训练一个网络在测试集上评估该网络调整神经网络结构调整超参数重复以上过程

神经架构搜索（Neural Architecture Search）：神经架构搜索的作用是给定输入数据集，可以自动寻找最优的网络结构和对应的参数，在计算机视觉中，神经架构搜索将会：

接受输入的数据集优化并寻找结构块（cells），这些块能够自动的被学习并且类似于inception、residual结构继续训练并寻找最优的块

对于一个自动机器学习系统，应该使用如下过程进行训练：

用神经架构搜索方法在整个数据集的一个小子集中进行训练寻找一个最优的结构块集合使用这些块并手动的定义更深层版本的网络最后用整个数据集进行训练

Auto-Keras：Auto-Keras是一个可以运行自动机器学习的包。 以下为代码示例：

from sklearn.metrics import classification_reportfrom keras.datasets import cifar10import autokeras as akimport os

def main():# initialize the output directoryOUTPUT_PATH = "output"# initialize the list of training times that we'll allow# Auto-Keras to train forTRAINING_TIMES = [60 * 60,# 1 hour60 * 60 * 2,# 2 hours60 * 60 * 4,# 4 hours60 * 60 * 8,# 8 hours60 * 60 * 12,# 12 hours60 * 60 * 24,# 24 hours]# load the training and testing data, then scale it into the# range [0, 1]print("[INFO] loading CIFAR-10 data...")((trainX, trainY), (testX, testY)) = cifar10.load_data()trainX = trainX.astype("float") / 255.0testX = testX.astype("float") / 255.0# initialize the label names for the CIFAR-10 datasetlabelNames = ["airplane", "automobile", "bird", "cat", "deer","dog", "frog", "horse", "ship", "truck"]

首先定义TRAINING_TIMES列表中包含[1, 2, 4, 8, 12, 24]小时，为Auto-Keras运行的时间

之后导入数据

# loop over the number of seconds to allow the current Auto-Keras# model to train forfor seconds in TRAINING_TIMES:# train our Auto-Keras modelprint("[INFO] training model for {} seconds max...".format(seconds))model = ak.ImageClassifier(verbose=True)model.fit(trainX, trainY, time_limit=seconds)model.final_fit(trainX, trainY, testX, testY, retrain=True)# evaluate the Auto-Keras modelscore = model.evaluate(testX, testY)predictions = model.predict(testX)report = classification_report(testY, predictions,target_names=labelNames)# write the report to diskp = os.path.sep.join(OUTPUT_PATH, "{}.txt".format(seconds))f = open(p, "w")f.write(report)f.write("\nscore: {}".format(score))f.close()# if this is the main thread of execution then start the process (our# code must be wrapped like this to avoid threading issues with# TensorFlow)if __name__ == "__main__":main()

for循环是为了对比Auto-Keras运行不同时间的结构。

在训练过程中，首先初始化模型，不需要定义网络结构，不需要设置超参数，之后fit方法开始训练；

当时间限制即seconds达到限制后，final_fit会选取最好的模型和参数；

最后评估和构建结构。

当前，AutoML的训练效果并没有远超用普通神经网络进行训练的结果，所以这个方法还有待进一步发掘。