# Retropropagação com Softmax / Cross Entropy

40

Estou tentando entender como a retropropagação funciona para uma camada de saída softmax / entropia cruzada.

A função de erro de entropia cruzada é

$E\left(t,o\right)=-\sum _{j}{t}_{j}\mathrm{log}{o}_{j}$

com $t$$t$ e $o$$o$ como alvo e saída no neurônio $j$$j$ , respectivamente. A soma é sobre cada neurônio na camada de saída. ${o}_{j}$$o_j$ em si é o resultado da função softmax:

${o}_{j}=softmax\left({z}_{j}\right)=\frac{{e}^{{z}_{j}}}{\sum _{j}{e}^{{z}_{j}}}$

Novamente, a soma está sobre cada neurônio na camada de saída e ${z}_{j}$$z_j$ é a entrada do neurônio $j$$j$ :

${z}_{j}=\sum _{i}{w}_{ij}{o}_{i}+b$

Que é a soma sobre todos os neurónios na camada anterior, com a sua saída correspondente ${o}_{i}$$o_i$ e o peso ${w}_{ij}$$w_{ij}$ no sentido de neurónio $j$$j$ mais um viés $b$$b$ .

Agora, para atualizar um peso que conecta um neurônio na camada de saída com um neurônio${w}_{ij}$$w_{ij}$$j$$j$ na camada anterior, preciso calcular a derivada parcial da função de erro usando a regra da cadeia:$i$$i$

$\frac{\mathrm{\partial }E}{\mathrm{\partial }{w}_{ij}}=\frac{\mathrm{\partial }E}{\mathrm{\partial }{o}_{j}}\frac{\mathrm{\partial }{o}_{j}}{\mathrm{\partial }{z}_{j}}\frac{\mathrm{\partial }{z}_{j}}{\mathrm{\partial }{w}_{ij}}$

com como entrada para o neurônio .${z}_{j}$$z_j$$j$$j$

O último termo é bastante simples. Uma vez que há apenas um peso entre e , o derivado é:$i$$i$$j$$j$

$\frac{\mathrm{\partial }{z}_{j}}{\mathrm{\partial }{w}_{ij}}={o}_{i}$

O primeiro termo é a derivação da função de erro em relação à saída :${o}_{j}$$o_j$

$\frac{\mathrm{\partial }E}{\mathrm{\partial }{o}_{j}}=\frac{-{t}_{j}}{{o}_{j}}$

O termo do meio é a derivação da função softmax em relação à sua entrada é mais difícil:${z}_{j}$$z_j$

$\frac{\mathrm{\partial }{o}_{j}}{\mathrm{\partial }{z}_{j}}=\frac{\mathrm{\partial }}{\mathrm{\partial }{z}_{j}}\frac{{e}^{{z}_{j}}}{\sum _{j}{e}^{{z}_{j}}}$

Digamos que temos três neurônios de saída correspondentes às classes então $a,b,c$$a,b,c$${o}_{b}=softmax\left(b\right)$$o_b = softmax(b)$ é:

${o}_{b}=\frac{{e}^{{z}_{b}}}{\sum {e}^{z}}=\frac{{e}^{{z}_{b}}}{{e}^{{z}_{a}}+{e}^{{z}_{b}}+{e}^{{z}_{c}}}$

e sua derivação usando a regra do quociente:

$\frac{\mathrm{\partial }{o}_{b}}{\mathrm{\partial }{z}_{b}}=\frac{{e}^{{z}_{b}}\ast \sum {e}^{z}-\left({e}^{{z}_{b}}{\right)}^{2}}{\left(\sum _{j}{e}^{z}{\right)}^{2}}=\frac{{e}^{{z}_{b}}}{\sum {e}^{z}}-\frac{\left({e}^{{z}_{b}}{\right)}^{2}}{\left(\sum {e}^{z}{\right)}^{2}}$
$=softmax\left(b\right)-softma{x}^{2}\left(b\right)={o}_{b}-{o}_{b}^{2}={o}_{b}\left(1-{o}_{b}\right)$
Back to the middle term for backpropagation this means:
$\frac{\mathrm{\partial }{o}_{j}}{\mathrm{\partial }{z}_{j}}={o}_{j}\left(1-{o}_{j}\right)$

Putting it all together I get

$\frac{\mathrm{\partial }E}{\mathrm{\partial }{w}_{ij}}=\frac{-{t}_{j}}{{o}_{j}}\ast {o}_{j}\left(1-{o}_{j}\right)\ast {o}_{i}=-{t}_{j}\left(1-{o}_{j}\right)\ast {o}_{i}$

which means, if the target for this class is ${t}_{j}=0$$t_j=0$, then I will not update the weights for this. That does not sound right.

Investigating on this I found people having two variants for the softmax derivation, one where $i=j$$i=j$ and the other for $i\ne j$$i\ne j$, like here or here.

But I can't make any sense out of this. Also I'm not even sure if this is the cause of my error, which is why I'm posting all of my calculations. I hope someone can clarify me where I am missing something or going wrong.

The links you have given are calculating the derivative relative to the input, whilst you're calculating the derivative relative to the weights.
Jenkar

Respostas:

35

Note: I am not an expert on backprop, but now having read a bit, I think the following caveat is appropriate. When reading papers or books on neural nets, it is not uncommon for derivatives to be written using a mix of the standard summation/index notation, matrix notation, and multi-index notation (include a hybrid of the last two for tensor-tensor derivatives). Typically the intent is that this should be "understood from context", so you have to be careful!

I noticed a couple of inconsistencies in your derivation. I do not do neural networks really, so the following may be incorrect. However, here is how I would go about the problem.

First, you need to take account of the summation in $E$$E$, and you cannot assume each term only depends on one weight. So taking the gradient of $E$$E$ with respect to component $k$$k$ of $z$$z$, we have

$E=-\sum _{j}{t}_{j}\mathrm{log}{o}_{j}\phantom{\rule{thickmathspace}{0ex}}⟹\phantom{\rule{thickmathspace}{0ex}}\frac{\mathrm{\partial }E}{\mathrm{\partial }{z}_{k}}=-\sum _{j}{t}_{j}\frac{\mathrm{\partial }\mathrm{log}{o}_{j}}{\mathrm{\partial }{z}_{k}}$

${o}_{j}$$o_j$

${o}_{j}=\frac{1}{\mathrm{\Omega }}{e}^{{z}_{j}}\phantom{\rule{thinmathspace}{0ex}},\phantom{\rule{thinmathspace}{0ex}}\mathrm{\Omega }=\sum _{i}{e}^{{z}_{i}}\phantom{\rule{thickmathspace}{0ex}}⟹\phantom{\rule{thickmathspace}{0ex}}\mathrm{log}{o}_{j}={z}_{j}-\mathrm{log}\mathrm{\Omega }$
$\frac{\mathrm{\partial }\mathrm{log}{o}_{j}}{\mathrm{\partial }{z}_{k}}={\delta }_{jk}-\frac{1}{\mathrm{\Omega }}\frac{\mathrm{\partial }\mathrm{\Omega }}{\mathrm{\partial }{z}_{k}}$
${\delta }_{jk}$$\delta_{jk}$
$\frac{\mathrm{\partial }\mathrm{\Omega }}{\mathrm{\partial }{z}_{k}}=\sum _{i}{e}^{{z}_{i}}{\delta }_{ik}={e}^{{z}_{k}}$
which gives
$\frac{\mathrm{\partial }\mathrm{log}{o}_{j}}{\mathrm{\partial }{z}_{k}}={\delta }_{jk}-{o}_{k}$
or, expanding the log
$\frac{\mathrm{\partial }{o}_{j}}{\mathrm{\partial }{z}_{k}}={o}_{j}\left({\delta }_{jk}-{o}_{k}\right)$
Note that the derivative is with respect to ${z}_{k}$$z_k$, an arbitrary component of $z$$z$, which gives the ${\delta }_{jk}$$\delta_{jk}$ term ($=1$$=1$ only when $k=j$$k=j$).

So the gradient of $E$$E$ with respect to $z$$z$ is then

$\frac{\mathrm{\partial }E}{\mathrm{\partial }{z}_{k}}=\sum _{j}{t}_{j}\left({o}_{k}-{\delta }_{jk}\right)={o}_{k}\left(\sum _{j}{t}_{j}\right)-{t}_{k}\phantom{\rule{thickmathspace}{0ex}}⟹\phantom{\rule{thickmathspace}{0ex}}\frac{\mathrm{\partial }E}{\mathrm{\partial }{z}_{k}}={o}_{k}\tau -{t}_{k}$
where $\tau =\sum _{j}{t}_{j}$$\tau=\sum_jt_j$ is constant (for a given $t$$t$ vector).

This shows a first difference from your result: the ${t}_{k}$$t_k$ no longer multiplies ${o}_{k}$$o_k$. Note that for the typical case where $t$$t$ is "one-hot" we have $\tau =1$$\tau=1$ (as noted in your first link).

A second inconsistency, if I understand correctly, is that the "$o$$o$" that is input to $z$$z$ seems unlikely to be the "$o$$o$" that is output from the softmax. I would think that it makes more sense that this is actually "further back" in network architecture?

Calling this vector $y$$y$, we then have

${z}_{k}=\sum _{i}{w}_{ik}{y}_{i}+{b}_{k}\phantom{\rule{thickmathspace}{0ex}}⟹\phantom{\rule{thickmathspace}{0ex}}\frac{\mathrm{\partial }{z}_{k}}{\mathrm{\partial }{w}_{pq}}=\sum _{i}{y}_{i}\frac{\mathrm{\partial }{w}_{ik}}{\mathrm{\partial }{w}_{pq}}=\sum _{i}{y}_{i}{\delta }_{ip}{\delta }_{kq}={\delta }_{kq}{y}_{p}$

Finally, to get the gradient of $E$$E$ with respect to the weight-matrix $w$$w$, we use the chain rule

$\frac{\mathrm{\partial }E}{\mathrm{\partial }{w}_{pq}}=\sum _{k}\frac{\mathrm{\partial }E}{\mathrm{\partial }{z}_{k}}\frac{\mathrm{\partial }{z}_{k}}{\mathrm{\partial }{w}_{pq}}=\sum _{k}\left({o}_{k}\tau -{t}_{k}\right){\delta }_{kq}{y}_{p}={y}_{p}\left({o}_{q}\tau -{t}_{q}\right)$
giving the final expression (assuming a one-hot $t$$t$, i.e. $\tau =1$$\tau=1$)
$\frac{\mathrm{\partial }E}{\mathrm{\partial }{w}_{ij}}={y}_{i}\left({o}_{j}-{t}_{j}\right)$
where $y$$y$ is the input on the lowest level (of your example).

So this shows a second difference from your result: the "${o}_{i}$$o_i$" should presumably be from the level below $z$$z$, which I call $y$$y$, rather than the level above $z$$z$ (which is $o$$o$).

Hopefully this helps. Does this result seem more consistent?

Update: In response to a query from the OP in the comments, here is an expansion of the first step. First, note that the vector chain rule requires summations (see here). Second, to be certain of getting all gradient components, you should always introduce a new subscript letter for the component in the denominator of the partial derivative. So to fully write out the gradient with the full chain rule, we have

$\frac{\mathrm{\partial }E}{\mathrm{\partial }{w}_{pq}}=\sum _{i}\frac{\mathrm{\partial }E}{\mathrm{\partial }{o}_{i}}\frac{\mathrm{\partial }{o}_{i}}{\mathrm{\partial }{w}_{pq}}$
and
$\frac{\mathrm{\partial }{o}_{i}}{\mathrm{\partial }{w}_{pq}}=\sum _{k}\frac{\mathrm{\partial }{o}_{i}}{\mathrm{\partial }{z}_{k}}\frac{\mathrm{\partial }{z}_{k}}{\mathrm{\partial }{w}_{pq}}$
so
$\frac{\mathrm{\partial }E}{\mathrm{\partial }{w}_{pq}}=\sum _{i}\left[\frac{\mathrm{\partial }E}{\mathrm{\partial }{o}_{i}}\left(\sum _{k}\frac{\mathrm{\partial }{o}_{i}}{\mathrm{\partial }{z}_{k}}\frac{\mathrm{\partial }{z}_{k}}{\mathrm{\partial }{w}_{pq}}\right)\right]$
In practice the full summations reduce, because you get a lot of ${\delta }_{ab}$$\delta_{ab}$ terms. Although it involves a lot of perhaps "extra" summations and subscripts, using the full chain rule will ensure you always get the correct result.

I am not certain how the "Backprop/AutoDiff" community does these problems, but I find any time I try to take shortcuts, I am liable to make errors. So I end up doing as here, writing everything out in terms of summations with full subscripting, and always introducing new subscripts for every derivative. (Similar to my answer here ... I hope I am at least giving correct results in the end!)
GeoMatt22

I personally find that you writing everything down makes it much easier to follow. The results look correct to me.
Jenkar

Although I'm still trying to fully understand each of your steps, I got some valuable insights that helped me with the overall picture. I guess I need to read more into the topic of derivations and sums. But taking your advise to take account of the summation in E, I came up with this:
micha

for two outputs ${o}_{{j}_{1}}=\frac{{e}^{{z}_{{j}_{1}}}}{\mathrm{\Omega }}$$o_{j_1}=\frac{e^{z_{j_1}}}{\Omega}$ and ${o}_{{j}_{1}}=\frac{{e}^{{z}_{{j}_{1}}}}{\mathrm{\Omega }}$$o_{j_1}=\frac{e^{z_{j_1}}}{\Omega}$ with
$\mathrm{\Omega }={e}^{{z}_{{j}_{1}}}+{e}^{{z}_{{j}_{2}}}$
the cross entropy error is
$E=-\left({t}_{1}log{o}_{{j}_{1}}+{t}_{2}log{o}_{{j}_{2}}\right)=-\left({t}_{1}\left({z}_{{j}_{1}}-log\left(\mathrm{\Omega }\right)\right)+{t}_{2}\left({z}_{{j}_{2}}-log\left(\mathrm{\Omega }\right)\right)\right)$
Then the derivative is
$\frac{\mathrm{\partial }E}{\mathrm{\partial }\left({z}_{{j}_{1}}}=-\left({t}_{1}-{t}_{1}\frac{{e}^{{z}_{{j}_{1}}}}{\mathrm{\Omega }}-{t}_{2}\frac{{e}^{{z}_{{j}_{2}}}}{\mathrm{\Omega }}\right)=-{t}_{1}+{o}_{{j}_{1}}\left({t}_{1}+{t}_{2}\right)$
which conforms with your result... taking in account that you didn't have the minus sign before the error sum
micha

But a further question I have is: Instead of
$\frac{\mathrm{\partial }E}{\mathrm{\partial }{w}_{ij}}=\frac{\mathrm{\partial }E}{\mathrm{\partial }{o}_{j}}\frac{\mathrm{\partial }{o}_{j}}{\mathrm{\partial }{z}_{j}}\frac{\mathrm{\partial }{z}_{j}}{\mathrm{\partial }{w}_{ij}}$
which is generally what your introduced to with backpropagation, you calculated:
$\frac{\mathrm{\partial }E}{\mathrm{\partial }{w}_{ij}}=\frac{\mathrm{\partial }E}{\mathrm{\partial }{z}_{j}}\frac{\mathrm{\partial }{z}_{j}}{\mathrm{\partial }{w}_{ij}}$
as like to cancel out the $\mathrm{\partial }{o}_{j}$$\partial o_j$ . Why is this way leading to the right result?
micha

12

While @GeoMatt22's answer is correct, I personally found it very useful to reduce the problem to a toy example and draw a picture:

I then defined the operations each node was computing, treating the $h$$h$'s and $w$$w$'s as inputs to a "network" ($\mathbf{t}$$\mathbf{t}$ is a one-hot vector representing the class label of the data point):

$L=-{t}_{1}\mathrm{log}{o}_{1}-{t}_{2}\mathrm{log}{o}_{2}$
${o}_{1}=\frac{\mathrm{exp}\left({y}_{1}\right)}{\mathrm{exp}\left({y}_{1}\right)+\mathrm{exp}\left({y}_{2}\right)}$
${o}_{2}=\frac{\mathrm{exp}\left({y}_{2}\right)}{\mathrm{exp}\left({y}_{1}\right)+\mathrm{exp}\left({y}_{2}\right)}$
${y}_{1}={w}_{11}{h}_{1}+{w}_{21}{h}_{2}+{w}_{31}{h}_{3}$
${y}_{2}={w}_{12}{h}_{1}+{w}_{22}{h}_{2}+{w}_{32}{h}_{3}$

Say I want to calculate the derivative of the loss with respect to ${w}_{21}$$w_{21}$. I can just use my picture to trace back the path from the loss to the weight I'm interested in (removed the second column of $w$$w$'s for clarity):

Then, I can just calculate the desired derivatives. Note that there are two paths through ${y}_{1}$$y_1$ that lead to ${w}_{21}$$w_{21}$, so I need to sum the derivatives that go through each of them.

$\frac{\mathrm{\partial }L}{\mathrm{\partial }{o}_{1}}=-\frac{{t}_{1}}{{o}_{1}}$
$\frac{\mathrm{\partial }L}{\mathrm{\partial }{o}_{2}}=-\frac{{t}_{2}}{{o}_{2}}$
$\frac{\mathrm{\partial }{o}_{1}}{\mathrm{\partial }{y}_{1}}=\frac{\mathrm{exp}\left({y}_{1}\right)}{\mathrm{exp}\left({y}_{1}\right)+\mathrm{exp}\left({y}_{2}\right)}-{\left(\frac{\mathrm{exp}\left({y}_{1}\right)}{\mathrm{exp}\left({y}_{1}\right)+\mathrm{exp}\left({y}_{2}\right)}\right)}^{2}={o}_{1}\left(1-{o}_{1}\right)$
$\frac{\mathrm{\partial }{o}_{2}}{\mathrm{\partial }{y}_{1}}=\frac{-\mathrm{exp}\left({y}_{2}\right)\mathrm{exp}\left({y}_{1}\right)}{\left(\mathrm{exp}\left({y}_{1}\right)+\mathrm{exp}\left({y}_{2}\right){\right)}^{2}}=-{o}_{2}{o}_{1}$
$\frac{\mathrm{\partial }{y}_{1}}{\mathrm{\partial }{w}_{21}}={h}_{2}$

Finally, putting the chain rule together:

$\begin{array}{rl}\frac{\mathrm{\partial }L}{\mathrm{\partial }{w}_{21}}& =\frac{\mathrm{\partial }L}{\mathrm{\partial }{o}_{1}}\frac{\mathrm{\partial }{o}_{1}}{\mathrm{\partial }{y}_{1}}\frac{\mathrm{\partial }{y}_{1}}{\mathrm{\partial }{w}_{21}}+\frac{\mathrm{\partial }L}{\mathrm{\partial }{o}_{2}}\frac{\mathrm{\partial }{o}_{2}}{\mathrm{\partial }{y}_{1}}\frac{\mathrm{\partial }{y}_{1}}{\mathrm{\partial }{w}_{21}}\\ & =\frac{-{t}_{1}}{{o}_{1}}\left[{o}_{1}\left(1-{o}_{1}\right)\right]{h}_{2}+\frac{-{t}_{2}}{{o}_{2}}\left(-{o}_{2}{o}_{1}\right){h}_{2}\\ & ={h}_{2}\left({t}_{2}{o}_{1}-{t}_{1}+{t}_{1}{o}_{1}\right)\\ & ={h}_{2}\left({o}_{1}\left({t}_{1}+{t}_{2}\right)-{t}_{1}\right)\\ & ={h}_{2}\left({o}_{1}-{t}_{1}\right)\end{array}$

Note that in the last step, ${t}_{1}+{t}_{2}=1$$t_1 + t_2 = 1$ because the vector $\mathbf{t}$$\mathbf{t}$ is a one-hot vector.

This is what finally cleared this up for me! Excellent and Elegant explanation!!!!
SantoshGupta7

2
I’m glad you both enjoyed and benefited from reading my post! It was also helpful for me to write it out and explain it.
Vivek Subramanian

@VivekSubramanian should it be
$=\frac{-{t}_{1}}{{o}_{1}}\left[{o}_{1}\left(1-{o}_{1}\right)\right]{h}_{2}+\frac{-{t}_{2}}{{o}_{2}}\left(-{o}_{2}{o}_{1}\right){h}_{2}\phantom{\rule{0ex}{0ex}}$
koryakinp

You’re right - it was a typo! I will make the change.
Vivek Subramanian

The thing i do not understand here is that you also assign logits (unscaled scores) to some neurons. (o is softmaxed logits (predictions) and y is logits in your case). However, this is not the case normally, is not it? Look at this picture ( o_out1 is prediction and o_in1 is logits) so how is it possible in this case how can you find the partial derivative of o2 with respect to y1?
ARAT

6

In place of the $\left\{{o}_{i}\right\},\phantom{\rule{thinmathspace}{0ex}}$$\{o_i\},\,$ I want a letter whose uppercase is visually distinct from its lowercase. So let me substitute $\left\{{y}_{i}\right\}$$\{y_i\}$. Also, let's use the variable $\left\{{p}_{i}\right\}$$\{p_i\}$ to designate the $\left\{{o}_{i}\right\}$$\{o_i\}$ from the previous layer.

Let $Y$$Y$ be the diagonal matrix whose diagonal equals the vector $y$$y$, i.e.

$Y=\mathrm{D}\mathrm{i}\mathrm{a}\mathrm{g}\left(y\right)$
Using this new matrix variable and the Frobenius Inner Product we can calculate the gradient of $E$$E$ wrt $W$$W$.
$\begin{array}{rll}z& =Wp+b& dz=dWp\\ y& =\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(z\right)& dy=\left(Y-y{y}^{T}\right)\phantom{\rule{thinmathspace}{0ex}}dz\\ E& =-t:\mathrm{log}\left(y\right)& dE=-t:{Y}^{-1}dy\\ \\ dE& =-t:{Y}^{-1}\left(Y-y{y}^{T}\right)\phantom{\rule{thinmathspace}{0ex}}dz\\ & =-t:\left(I-1{y}^{T}\right)\phantom{\rule{thinmathspace}{0ex}}dz\\ & =-t:\left(I-1{y}^{T}\right)\phantom{\rule{thinmathspace}{0ex}}dW\phantom{\rule{thinmathspace}{0ex}}p\\ & =\left(y{1}^{T}-I\right)t{p}^{T}:dW\\ & =\left(\left({1}^{T}t\right)y{p}^{T}-t{p}^{T}\right):dW\\ \\ \frac{\mathrm{\partial }E}{\mathrm{\partial }W}& =\left({1}^{T}t\right)y{p}^{T}-t{p}^{T}\end{array}$

6

Here is one of the cleanest and well written notes that I came across the web which explains about "calculation of derivatives in backpropagation algorithm with cross entropy loss function".

In the given pdf how did equation 22 become equation 23? As in how did the Summation(k!=i) get a negative sign. Shouldn't it get a positive sign? Like Summation(Fn)(For All K) = Fn(k=i) + Summation(Fn)(k!=i) should be happening according to my understanding.
faizan

1

Here's a link explaining the softmax and its derivative.

It explains the reason for using i=j and i!=j.

It is recommended to provide a minimal, stand-alone answer, in case that link gets broken in the future. Otherwise, this might no longer help other users in the future.
luchonacho

0

Other answers have provided the correct way of calculating the derivative, but they do not point out where you have gone wrong. In fact, $$tjtjt_j$$ is always 1 in your last equation, cause you have assumed that $$ojojo_j$$ takes that node of target 1 in your output; $$ojojo_j$$ of other nodes have different forms of probability function, thus lead to different forms of derivative, so you should now understand why other people have treated $$i=ji=ji=j$$ and $$i≠ji≠ji\neq j$$ differently.

Ao utilizar nosso site, você reconhece que leu e compreendeu nossa Política de Cookies e nossa Política de Privacidade.