反向传播算法的推导

如图为2-layers CNN，输入单元下标为i，数量d；隐层单元下表j，数量$n_H$；输出层下表k，单元数量c

1.目标

调整权系数$w_{ji}$,$w_{kj}$，使得输出$(x_i,z_i)$尽可能等于样本$(x_i,t_i)$
即定义误差函数$J(w)$最小
\[
J(w)=\sum_{x} J_x(w)
\\
J_x(w)=\frac{1}{2} \sum _{k=1}^c(t_k-z_k(x))^2
\]

2.节点表示

对于隐层中的节点，定义权值和为$net_j$，则
\[
net_j=\sum _{i=1}^d w_{ji}x_i+w_{j0}
\]
同理，则输出层的节点权值和为$net_k$，有
\[
net_k=\sum _{j=1}^{n_H} w_{kj}y_j+w_{k0}
\]
对于隐层节点输出$y_j=f(net_j)$，输出层节点输出$z_k=f(net_k)$

3.权系数的调整方法

按照梯度下降的方法，对误差函数$J(w)求导，调整 $ $\frac {\partial J} {\partial w_{kj} } $和 $ \frac{\partial J} {\partial w_{ji}} $
\[
w_{kj} \to w_{kj}-\eta \frac{\partial J} {\partial w_{kj} }
\\
w_{ji} \to w_{ji} - \eta \frac{\partial J} {\partial w_{ji}}
\]
其中$\eta$控制下降速率

3.1对输出层权系数的微分

\[
\frac{\partial J}{\partial w_{kj}}=\frac{\partial J}{\partial net_k} \frac{\partial net_k}{\partial w_{kj}}
\\
其中\frac{\partial J}{\partial net_k}=\frac{\partial J}{\partial z_k} \frac {\partial z_k}{\partial net_k}=-(t_k-z_k)f'(net_k)
,\quad
\frac{\partial net_k}{\partial w_{kj}}=y_j
\]

通常令$\frac{\partial J}{\partial net_k}=\delta_k$，则$\frac {\partial J}{\partial w_{kj}}=\delta_k y_j$

3.2对隐层权系数的微分

\[
\frac{\partial J}{\partial w_{ji}}=\frac{\partial J}{\partial net_j} \frac{\partial net_j}{\partial w_{ji}}
\\
其中\frac{\partial J}{\partial net_j}=\frac{\partial J}{\partial y_j} \frac {\partial y_j}{\partial net_j}=\sum _{k=1}^c \delta _k w_{kj}f'(net_j)
,\quad
\frac{\partial net_j}{\partial w_{ji}}=x_i
\]

同样令$\frac{\partial J}{\partial net_j}=\delta_j$，则$\frac {\partial J}{\partial w_{ji}}=\delta_j x_i$

关于$\frac{\partial J}{\partial y_j}$的推导如下：
\[
\frac{\partial J}{\partial y_j}=\sum_{k=1}^c \frac{\partial J}{\partial net_k} \frac{\partial net_k}{\partial y_j}=\sum_{k=1}^c\delta_k w_{kj}
\]
故对权系数的调整变为
\[
w_{kj} \to w_{kj}-\eta \frac{\partial J} {\partial w_{kj} }= w_{kj}-\eta \delta_k y_j
\\
w_{ji} \to w_{ji} - \eta \frac{\partial J} {\partial w_{ji}}= w_{ji}-\eta_j\delta x_i
\]