關(guān)于源碼還是沒看懂
問題：
Q: 對pytorch里GAN更新G的過程疑問擦酌？
fake=G(x)
G.zero_gard()
out = D(fake)
loss = cri(out,label)
loss.backward()
opt_G.step()
這里集绰，經(jīng)過了D網(wǎng)絡(luò)結(jié)構(gòu)揭保，怎么可以就之更新G了呢？叹螟？鹃骂？
A: 經(jīng)過了前向傳播之后，pytorch就構(gòu)建了計算圖罢绽，可以想像G計算圖的輸出作為了D計算圖的輸入畏线，當(dāng)loss.backward的時候雖然計算了所有計算圖的梯度，但是由于只更新了G參數(shù)的數(shù)值良价，所以只利用了G計算圖的梯度寝殴，這是可以理解的蒿叠。之前糾結(jié)的點是跳過D更新G,但計算梯度的時候確實是經(jīng)過了D的。只是GAN這里沒有去更新D杯矩。
code
敲了敲代碼模擬了一下這個過程
import torch
import torch.nn as nn
import torch.optim as optim
net1 = nn.Conv2d(2,1,kernel_size=3)
net2 = nn.Linear(3,1)
x = torch.randn((1,2 ,5, 5), requires_grad=True)
opt_1 = optim.SGD(net1.parameters(),lr=0.01,momentum=0)
opt_2 = optim.SGD(net2.parameters(),lr=0.01,momentum=0)
cri = nn.MSELoss()
#打印參數(shù)（ out_channels x in_channels x kernel_size x kernel_size )
for p in net1.parameters():
print(p)
>>>
Parameter containing:
tensor([[[[-0.1586, 0.0571, -0.0606],
[-0.0712, -0.0753, 0.1583],
[-0.1438, 0.0217, 0.2169]],
[[ 0.0830, -0.2124, 0.1965],
[ 0.2107, 0.0655, -0.2229],
[ 0.0809, 0.1761, -0.2056]]]], requires_grad=True)
Parameter containing:
tensor([0.2213], requires_grad=True)
for p in net2.parameters():
print(p)
>>>
Parameter containing:
tensor([[0.1985, 0.0381, 0.3382]], requires_grad=True)
Parameter containing:
tensor([0.1597], requires_grad=True)
#through net1
y = net1(x)
print('y',y)
#through net2
z = net2(y)
print('z',z,z.shape)
>>>
y tensor([[[[ 0.7343, 0.1432, -0.3362],
[ 0.5525, 0.5968, 0.6297],
[ 1.0429, -0.0552, 0.6783]]]], grad_fn=<MkldnnConvolutionBackward>)
z tensor([[[[0.1972],
[0.5050],
[0.5940]]]], grad_fn=<AddBackward0>) torch.Size([1, 1, 3, 1])
#reshape z and compute loss
net1.zero_grad()
new_z = z.reshape(1,-1)
target = torch.tensor([[1,1,0]],dtype=torch.float)
loss = cri(new_z,target)
#before backward
print(net1.bias.grad)
print(net2.bias.grad)
#==========compute gradient======
loss.backward()
#================================
#after
print(net1.bias.grad)
print(net2.bias.grad)
for f in net1.parameters():
print(f.grad.data)
>>>
None
None
tensor([-0.2697])
tensor([-0.4691])
tensor([[[[-0.5155, -0.3812, -0.3589],
[ 0.2562, -0.3376, 0.1517],
[ 0.2322, -0.2214, 0.0431]],
[[ 0.1007, 0.2636, 0.0203],
[ 0.0201, 0.0348, -0.2474],
[-0.0404, 0.0261, 0.0283]]]])
tensor([-0.2697])
for p in net1.parameters():
print(p)
print('manully updata params:')
tmp_p = p.data.clone()
print(tmp_p.sub_(p.grad.data*0.01))#weight = weight-lr*d_p
print('---')
print(p.data)
for p in net2.parameters():
print(p)
Parameter containing:
tensor([[[[-0.1586, 0.0571, -0.0606],
[-0.0712, -0.0753, 0.1583],
[-0.1438, 0.0217, 0.2169]],
[[ 0.0830, -0.2124, 0.1965],
[ 0.2107, 0.0655, -0.2229],
[ 0.0809, 0.1761, -0.2056]]]], requires_grad=True)
manully updata params:
tensor([[[[-0.1535, 0.0609, -0.0570],
[-0.0738, -0.0719, 0.1568],
[-0.1461, 0.0239, 0.2165]],
[[ 0.0820, -0.2150, 0.1963],
[ 0.2105, 0.0652, -0.2204],
[ 0.0813, 0.1758, -0.2059]]]])
---
tensor([[[[-0.1586, 0.0571, -0.0606],
[-0.0712, -0.0753, 0.1583],
[-0.1438, 0.0217, 0.2169]],
[[ 0.0830, -0.2124, 0.1965],
[ 0.2107, 0.0655, -0.2229],
[ 0.0809, 0.1761, -0.2056]]]])
Parameter containing:
tensor([0.2213], requires_grad=True)
manully updata params:
tensor([0.2240])
---
tensor([0.2213])
Parameter containing:
tensor([[0.1985, 0.0381, 0.3382]], requires_grad=True)
Parameter containing:
tensor([0.1597], requires_grad=True)
#upadate
opt_1.step()
for p in net1.parameters():
print(p)
print('=======')
for p in net2.parameters():
print(p)
Parameter containing:
tensor([[[[-0.1535, 0.0609, -0.0570],
[-0.0738, -0.0719, 0.1568],
[-0.1461, 0.0239, 0.2165]],
[[ 0.0820, -0.2150, 0.1963],
[ 0.2105, 0.0652, -0.2204],
[ 0.0813, 0.1758, -0.2059]]]], requires_grad=True)
Parameter containing:
tensor([0.2240], requires_grad=True)
=======
Parameter containing:
tensor([[0.1985, 0.0381, 0.3382]], requires_grad=True)
Parameter containing:
tensor([0.1597], requires_grad=True)
大概就是這么個流程栈虚，不過具體的自動求導(dǎo)機制暫時還看不懂源碼袖外，之后如果看懂的話史隆，會在這里補充的。
