一、簡介
筆者閱讀的Pose-Transfer代碼為https://github.com/tengteng95/Pose-Transfer的Pytorch_v1.0分支柴底，適應(yīng)于pytorch1.x的版本匀哄。以下講的流程為ReadMe中給出的運行參數(shù)的情況下的流程召烂，它是論文Progressive Pose Attention for Person Image Generation in CVPR19 (Oral)的代碼贡避。
二继准、網(wǎng)絡(luò)結(jié)構(gòu)
神經(jīng)網(wǎng)絡(luò)相關(guān)的代碼閱讀入口為models/PATN.py中的class TransferModel
2.1 生成網(wǎng)絡(luò)netG
定義了生成網(wǎng)絡(luò)netG的的代碼為：
netG = PATNetwork(input_nc, output_nc, ngf, norm_layer=norm_layer, use_dropout=use_dropout,
n_blocks=9, gpu_ids=gpu_ids, n_downsampling=n_downsampling)
參數(shù)：input_nc(輸入channel數(shù))躬审，output_nc(輸出channel數(shù)）棘街，ngf(channel數(shù)相關(guān)，可理解為特征圖個數(shù)承边，生成網(wǎng)絡(luò)基于此數(shù)的倍數(shù)進行channel變化）, n_blocks（PATB個數(shù)）遭殉，n_downsampling(下采樣卷積個數(shù))
網(wǎng)絡(luò)結(jié)構(gòu)相關(guān)代碼如下：
先看PATNetwork的forward前向計算代碼：
def forward(self, input): # x from stream 1 and stream 2
# here x should be a tuple
x1, x2 = input
# down_sample
x1 = self.stream1_down(x1)
x2 = self.stream2_down(x2)
# att_block
for model in self.att:
x1, x2, _ = model(x1, x2)
# up_sample
x1 = self.stream1_up(x1)
return x1
也就是生成網(wǎng)絡(luò)大致可分為下采樣部分、att_block部分博助、上采樣部分恩沽。att_block的上面分支的最后輸出經(jīng)過上采樣為最終結(jié)果。
2.1.1 下采樣部分
1.先是Padding層：
model_stream1_down = [nn.ReflectionPad2d(3),
nn.Conv2d(self.input_nc_s1, ngf, kernel_size=7, padding=0,
bias=use_bias),
norm_layer(ngf),
nn.ReLU(True)]
model_stream2_down = [nn.ReflectionPad2d(3),
nn.Conv2d(self.input_nc_s2, ngf, kernel_size=7, padding=0,
bias=use_bias),
norm_layer(ngf),
nn.ReLU(True)]
2.n_downsampling個下采樣卷積層：
for i in range(n_downsampling):
mult = 2**i
model_stream1_down += [nn.Conv2d(ngf * mult, ngf * mult * 2, kernel_size=3,
stride=2, padding=1, bias=use_bias),
norm_layer(ngf * mult * 2),
nn.ReLU(True)]
model_stream2_down += [nn.Conv2d(ngf * mult, ngf * mult * 2, kernel_size=3,
stride=2, padding=1, bias=use_bias),
norm_layer(ngf * mult * 2),
nn.ReLU(True)]
3.鏈接起層翔始，賦值
self.stream1_down = nn.Sequential(*model_stream1_down)
self.stream2_down = nn.Sequential(*model_stream2_down)
2.1.2 att block部分罗心，即對應(yīng)論文中的PATB里伯，即Pose-Attentional Transfer Network。
貼一張論文里的圖：
后文中講的PATB的第一分支就是上面的分支渤闷，第二分支就是下面的分支疾瓮。
mult = 2**n_downsampling
cated_stream2 = [True for i in range(n_blocks)]
cated_stream2[0] = False
attBlock = nn.ModuleList()
for i in range(n_blocks):
attBlock.append(PATBlock(ngf * mult, padding_type=padding_type, norm_layer=norm_layer, use_dropout=use_dropout, use_bias=use_bias, cated_stream2=cated_stream2[i]))
也就是n_blocks個PATB塊，一個PATB塊構(gòu)成為：
首先是變量定義飒箭，conv_blocks用于存儲各個層：
conv_block = []
p = 0
前向計算forward函數(shù)代碼如下：
def forward(self, x1, x2):
x1_out = self.conv_block_stream1(x1)
x2_out = self.conv_block_stream2(x2)
# att = F.sigmoid(x2_out)
att = torch.sigmoid(x2_out)
x1_out = x1_out * att
out = x1 + x1_out # residual connection
# stream2 receive feedback from stream1
x2_out = torch.cat((x2_out, out), 1)
return out, x2_out, x1_out
可以看出是兩個輸入狼电，兩個輸出，結(jié)合論文圖示看更易理解弦蹂。在整個生成網(wǎng)絡(luò)的forward函數(shù)中肩碟，取out、x2_out進行下一步運算凸椿。
其中削祈，conv_block_stream1與conv_block_stream2的構(gòu)建代碼為：
self.conv_block_stream1 = self.build_conv_block(dim, padding_type, norm_layer, use_dropout, use_bias, cal_att=False)
self.conv_block_stream2 = self.build_conv_block(dim, padding_type, norm_layer, use_dropout, use_bias, cal_att=True, cated_stream2=cated_stream2)
conv_block_stream1與conv_block_stream2結(jié)構(gòu)具體的網(wǎng)絡(luò)結(jié)構(gòu)如下：
1.Padding
if padding_type == 'reflect':
conv_block += [nn.ReflectionPad2d(1)]
elif padding_type == 'replicate':
conv_block += [nn.ReplicationPad2d(1)]
elif padding_type == 'zero':
p = 1
2.Normalize
if cated_stream2:
conv_block += [nn.Conv2d(dim*2, dim*2, kernel_size=3, padding=p, bias=use_bias),
norm_layer(dim*2),
nn.ReLU(True)]
else:
conv_block += [nn.Conv2d(dim, dim, kernel_size=3, padding=p, bias=use_bias),
norm_layer(dim),
nn.ReLU(True)]
其中cated_stream2，在第一個分支為False脑漫，在第二個分支第一個PATB中為False髓抑，在第二個及以后中為True，因為PATB的第二個分支的最后輸出為為第一個分支卷積結(jié)果和第二個分支卷積結(jié)果的拼接（x2_out = torch.cat((x2_out, out), 1)）
3.dropout層（可選）
if use_dropout:
conv_block += [nn.Dropout(0.5)]
4.再次Padding
代碼與第一次Padding相同
5.卷積層
if cal_att:
if cated_stream2:
conv_block += [nn.Conv2d(dim*2, dim, kernel_size=3, padding=p, bias=use_bias)]
else:
conv_block += [nn.Conv2d(dim, dim, kernel_size=3, padding=p, bias=use_bias)]
else:
conv_block += [nn.Conv2d(dim, dim, kernel_size=3, padding=p, bias=use_bias),
norm_layer(dim)]
cal_att為False時是第一分支优幸，cal_att為第二分支吨拍。第一分支輸入輸出channel均為dim，第二個分支則需要把dim*2的輸入channel轉(zhuǎn)成dim的輸出channel网杆，方便與第一分支進行拼接羹饰。
2.1.3 上采樣部分
model_stream1_up = []
for i in range(n_downsampling):
mult = 2**(n_downsampling - i)
model_stream1_up += [nn.ConvTranspose2d(ngf * mult, int(ngf * mult / 2),
kernel_size=3, stride=2,
padding=1, output_padding=1,
bias=use_bias),
norm_layer(int(ngf * mult / 2)),
nn.ReLU(True)]
大致就是n_downsampling個反卷積層的上采樣。
2.2 分類網(wǎng)絡(luò)
有兩個分類網(wǎng)絡(luò)碳却，分別為netD_PB和netD_PP队秩。netD_PB用于評判輸出圖片Pg和目標(biāo)姿態(tài)St的的匹配程度（英文原文：how well Pg align with the target pose St(shape consistency).），netD_PP用于評判輸出圖片Pg是否包含輸入圖片Pc中的同一個人（英文原文：judge how likely Pg contains the same person in Pc (appearance consistency)）
netD_PB和net_PP結(jié)構(gòu)相同：
use_sigmoid = opt.no_lsgan
if opt.with_D_PB:
self.netD_PB = networks.define_D(opt.P_input_nc+opt.BP_input_nc, opt.ndf,
opt.which_model_netD,
opt.n_layers_D, opt.norm, use_sigmoid, opt.init_type, self.gpu_ids,
not opt.no_dropout_D,
n_downsampling = opt.D_n_downsampling)
if opt.with_D_PP:
self.netD_PP = networks.define_D(opt.P_input_nc+opt.P_input_nc, opt.ndf,
opt.which_model_netD,
opt.n_layers_D, opt.norm, use_sigmoid, opt.init_type, self.gpu_ids,
not opt.no_dropout_D,
n_downsampling = opt.D_n_downsampling)
參數(shù)：input_nc(輸入channel數(shù))追城，output_nv(輸出channel數(shù)）刹碾，ndf(channel數(shù)相關(guān)，可理解為特征圖個數(shù)座柱，分類網(wǎng)絡(luò)基于此數(shù)的倍數(shù)進行channel變化）,which_model_netD（分類器的基礎(chǔ)網(wǎng)絡(luò)迷帜，如resnet），n_layers_D（分類器中的block個數(shù)）色洞，norm（instance normalization or batch normalization）戏锹，n_downsampling(下采樣卷積個數(shù)）
代碼中define_D提供的是ResnetDiscriminator。首先看ResnetDiscriminator的forward函數(shù)：
def forward(self, input):
if self.gpu_ids and isinstance(input.data, torch.cuda.FloatTensor):
return nn.parallel.data_parallel(self.model, input, self.gpu_ids)
else:
return self.model(input)
就是只有一個self.model跑輸入得到輸出即可火诸。
self.model的結(jié)構(gòu)為:
1.Padding
model = [nn.ReflectionPad2d(3),
nn.Conv2d(input_nc, ngf, kernel_size=7, padding=0,
bias=use_bias),
norm_layer(ngf),
nn.ReLU(True)]
2.下采樣部分
if n_downsampling <= 2:
for i in range(n_downsampling):
mult = 2 ** i
model += [nn.Conv2d(ngf * mult, ngf * mult * 2, kernel_size=3,
stride=2, padding=1, bias=use_bias),
norm_layer(ngf * mult * 2),
nn.ReLU(True)]
elif n_downsampling == 3:
mult = 2 ** 0
model += [nn.Conv2d(ngf * mult, ngf * mult * 2, kernel_size=3,
stride=2, padding=1, bias=use_bias),
norm_layer(ngf * mult * 2),
nn.ReLU(True)]
mult = 2 ** 1
model += [nn.Conv2d(ngf * mult, ngf * mult * 2, kernel_size=3,
stride=2, padding=1, bias=use_bias),
norm_layer(ngf * mult * 2),
nn.ReLU(True)]
mult = 2 ** 2
model += [nn.Conv2d(ngf * mult, ngf * mult, kernel_size=3,
stride=2, padding=1, bias=use_bias),
norm_layer(ngf * mult),
nn.ReLU(True)]
if n_downsampling <= 2:
mult = 2 ** n_downsampling
else:
mult = 4
就是湊出n_downsampling個下采樣卷積層
3.殘差塊部分
for i in range(n_blocks):
model += [ResnetBlock(ngf * mult, padding_type=padding_type, norm_layer=norm_layer, use_dropout=use_dropout,
use_bias=use_bias)]
ResnetBlock就是resnet中的Identity Block锦针，不再展開敘述了
4.sigmoid層
if use_sigmoid:
model += [nn.Sigmoid()]
三、損失函數(shù)計算
在train.py中，調(diào)用的model.optimize_parameters()調(diào)整網(wǎng)絡(luò)權(quán)重函數(shù)具體代碼如下：
# forward
self.forward()
self.optimizer_G.zero_grad()
self.backward_G()
self.optimizer_G.step()
# D_P
if self.opt.with_D_PP:
for i in range(self.opt.DG_ratio):
self.optimizer_D_PP.zero_grad()
self.backward_D_PP()
self.optimizer_D_PP.step()
# D_BP
if self.opt.with_D_PB:
for i in range(self.opt.DG_ratio):
self.optimizer_D_PB.zero_grad()
self.backward_D_PB()
self.optimizer_D_PB.step()
其中的forward函數(shù)為：
def forward(self):
G_input = [self.input_P1,
torch.cat((self.input_BP1, self.input_BP2), 1)]
self.fake_p2 = self.netG(G_input)
總結(jié)一下就是分為以下幾步：
3.1 前向計算生成網(wǎng)絡(luò)G得到生成圖片self.fake_p2
3.2 給G網(wǎng)絡(luò)調(diào)參奈搜，即向后傳播
backward_G()代碼為：
def backward_G(self):
if self.opt.with_D_PB:
pred_fake_PB = self.netD_PB(torch.cat((self.fake_p2, self.input_BP2), 1))
self.loss_G_GAN_PB = self.criterionGAN(pred_fake_PB, True)
if self.opt.with_D_PP:
pred_fake_PP = self.netD_PP(torch.cat((self.fake_p2, self.input_P1), 1))
self.loss_G_GAN_PP = self.criterionGAN(pred_fake_PP, True)
# L1 loss
if self.opt.L1_type == 'l1_plus_perL1' :
losses = self.criterionL1(self.fake_p2, self.input_P2)
self.loss_G_L1 = losses[0]
self.loss_originL1 = losses[1].item()
self.loss_perceptual = losses[2].item()
else:
self.loss_G_L1 = self.criterionL1(self.fake_p2, self.input_P2) * self.opt.lambda_A
pair_L1loss = self.loss_G_L1
if self.opt.with_D_PB:
pair_GANloss = self.loss_G_GAN_PB * self.opt.lambda_GAN
if self.opt.with_D_PP:
pair_GANloss += self.loss_G_GAN_PP * self.opt.lambda_GAN
pair_GANloss = pair_GANloss / 2
else:
if self.opt.with_D_PP:
pair_GANloss = self.loss_G_GAN_PP * self.opt.lambda_GAN
if self.opt.with_D_PB or self.opt.with_D_PP:
pair_loss = pair_L1loss + pair_GANloss
else:
pair_loss = pair_L1loss
pair_loss.backward()
self.pair_L1loss = pair_L1loss.item()
if self.opt.with_D_PB or self.opt.with_D_PP:
self.pair_GANloss = pair_GANloss.item()
文字表述就是：
1.分別計算分類器生成的D_PP悉盆，D_PB（鏈上文2.2)的分類損失（生成目標(biāo)為了混淆分類器，理想值應(yīng)為True)馋吗，分別記做loss_G_GAN_PB焕盟、loss_G_GAN_PP
2.（l1_plus_perL1）將目標(biāo)圖片與生成圖片做l1_plus_perL1的損失函數(shù)計算。來看L1_plus_perceptualLoss的具體代碼：
首先是該loss層的forward函數(shù)：
def forward(self, inputs, targets):
if self.lambda_L1 == 0 and self.lambda_perceptual == 0:
return torch.zeros(1).cuda(), torch.zeros(1), torch.zeros(1)
# normal L1
loss_l1 = F.l1_loss(inputs, targets) * self.lambda_L1
# perceptual L1
mean = torch.FloatTensor(3)
mean[0] = 0.485
mean[1] = 0.456
mean[2] = 0.406
mean = mean.resize(1, 3, 1, 1).cuda()
std = torch.FloatTensor(3)
std[0] = 0.229
std[1] = 0.224
std[2] = 0.225
std = std.resize(1, 3, 1, 1).cuda()
fake_p2_norm = (inputs + 1)/2 # [-1, 1] => [0, 1]
fake_p2_norm = (fake_p2_norm - mean)/std
input_p2_norm = (targets + 1)/2 # [-1, 1] => [0, 1]
input_p2_norm = (input_p2_norm - mean)/std
fake_p2_norm = self.vgg_submodel(fake_p2_norm)
input_p2_norm = self.vgg_submodel(input_p2_norm)
input_p2_norm_no_grad = input_p2_norm.detach()
if self.percep_is_l1 == 1:
# use l1 for perceptual loss
loss_perceptual = F.l1_loss(fake_p2_norm, input_p2_norm_no_grad) * self.lambda_perceptual
else:
# use l2 for perceptual loss
loss_perceptual = F.mse_loss(fake_p2_norm, input_p2_norm_no_grad) * self.lambda_perceptual
loss = loss_l1 + loss_perceptual
return loss, loss_l1, loss_perceptual
l1_plus_perL1包括兩種loss:一種是普通的L1 loss宏粤，即直接將input和target做L1 loss脚翘，記做loss_l1。
另一種 loss_perceptual的計算過程如下：（1）對input和target分別做normalize绍哎，其實就是將他們從[-1,1]的范圍變到[0,1]来农，然后減去mean再除以方差std（2）將normalize后的input和target送給vgg網(wǎng)絡(luò)得到輸出fake_p2_norm,和input_p2_norm_no_grad（3）將兩個輸出做L1 loss得到loss_perceptual
loss_perceptual是為了讓圖片更加平滑和自然，引入論文原文：
將兩種loss相加就得到了最后的loss崇堰。
回到backward_G()沃于，三種loss分別記為loss，loss_originL1赶袄，loss_perceptual
3.計算總loss
鏈接原文公式：
在上面的代碼中揽涮，4式中的α為2抠藕，也就是Lcomb除以2之后加上Lgan為總loss饿肺。
最后調(diào)用總loss.backward()跟新參數(shù)
3.3給兩個D網(wǎng)絡(luò)調(diào)參（鏈上文2.2節(jié)）
在上面更新了一次G網(wǎng)絡(luò)之后，更新DG_ratio次分類網(wǎng)絡(luò)D_PP和D_PB
3.3.1 給D_PP網(wǎng)絡(luò)調(diào)參
if self.opt.with_D_PP:
for i in range(self.opt.DG_ratio):
self.optimizer_D_PP.zero_grad()
self.backward_D_PP()
self.optimizer_D_PP.step()
def backward_D_PP(self):
real_PP = torch.cat((self.input_P2, self.input_P1), 1)
# fake_PP = self.fake_PP_pool.query(torch.cat((self.fake_p2, self.input_P1), 1))
fake_PP = self.fake_PP_pool.query( torch.cat((self.fake_p2, self.input_P1), 1).data )
loss_D_PP = self.backward_D_basic(self.netD_PP, real_PP, fake_PP)
self.loss_D_PP = loss_D_PP.item()
def backward_D_basic(self, netD, real, fake):
# Real
pred_real = netD(real)
loss_D_real = self.criterionGAN(pred_real, True) * self.opt.lambda_GAN
# Fake
pred_fake = netD(fake.detach())
loss_D_fake = self.criterionGAN(pred_fake, False) * self.opt.lambda_GAN
# Combined loss
loss_D = (loss_D_real + loss_D_fake) * 0.5
# backward
loss_D.backward()
return loss_D
總結(jié)步驟如下：
1盾似、將生成圖片fake_p2和輸入原圖片input_P1傳給fake_PP_pool.query函數(shù)敬辣，這個query函數(shù)的代碼如下：
def query(self, images):
if self.pool_size == 0:
return Variable(images)
return_images = []
for image in images:
image = torch.unsqueeze(image, 0)
if self.num_imgs < self.pool_size:
self.num_imgs = self.num_imgs + 1
self.images.append(image)
return_images.append(image)
else:
p = random.uniform(0, 1)
if p > 0.5:
random_id = random.randint(0, self.pool_size-1)
tmp = self.images[random_id].clone()
self.images[random_id] = image
return_images.append(tmp)
else:
return_images.append(image)
return_images = Variable(torch.cat(return_images, 0))
return return_images
這在干啥咱也不知道咱也不敢問，根據(jù)默認的配置的話跑的話是不超過50張圖時將fake_p2和input_p1拼接起來返回零院，超過了就是取之前訓(xùn)練的前49張的某張圖片來跟當(dāng)前圖片交換溉跃。
2、將輸入原圖片和目標(biāo)圖片拼接起來得到real_PP告抄，拿D_PP網(wǎng)絡(luò)去預(yù)測real_PP撰茎，計算預(yù)測結(jié)果與理想結(jié)果（TRUE）之間的loss，記為loss_D_real打洼。拿D_PP網(wǎng)絡(luò)去預(yù)測fake_PP龄糊，計算預(yù)測結(jié)果與理想結(jié)果（FALSE）之間的loss，記為loss_D_fake募疮。這里提醒一下D_PP網(wǎng)絡(luò)用于預(yù)測兩張圖是否包含同一個人炫惩。
3.總loss loss_D= (loss_D_real + loss_D_fake) * 0.5，loss_D.backward()更新參數(shù)
3.3.2給D_PB網(wǎng)絡(luò)調(diào)參
def backward_D_PB(self):
real_PB = torch.cat((self.input_P2, self.input_BP2), 1)
# fake_PB = self.fake_PB_pool.query(torch.cat((self.fake_p2, self.input_BP2), 1))
fake_PB = self.fake_PB_pool.query( torch.cat((self.fake_p2, self.input_BP2), 1).data )
loss_D_PB = self.backward_D_basic(self.netD_PB, real_PB, fake_PB)
self.loss_D_PB = loss_D_PB.item()
跟D_PP類似阿浓，只不過real_PB拼接的是目標(biāo)圖片和目標(biāo)姿勢他嚷，fake_PB拼接的是生成圖片和目標(biāo)姿勢。提醒一下D_PB用于判斷圖中的人的姿勢是否為目標(biāo)姿勢。
四筋蓖、總結(jié)
本文主要講了訓(xùn)練邏輯卸耘，筆者覺得弄懂了訓(xùn)練代碼，看測試代碼就簡單多了粘咖，就不再在文里分析了鹊奖。
