深度度量学习（Deep Metric Learning）函数求导公式

归一化向量的求导

向量模求导： 定义$\mathbf{x}\in {\mathbb{R}}^{1\times d}$为一个列向量，$||\mathbf{x}||$为向量的模,$\hat{\mathbf{x}}$表示经过L2归一化之后的向量，长度为1

$\frac{\partial }{\partial \mathbf{x}}(||\mathbf{x}||)=\frac{\partial }{\partial \mathbf{x}}\sqrt{{\mathbf{x}}^T\mathbf{x}}=\frac{1}{2}\frac{2{\mathbf{x}}^T}{\sqrt{{\mathbf{x}}^T\mathbf{x}}}=\frac{{\mathbf{x}}^T}{||\mathbf{x}||}={\hat{\mathbf{x}}}^T$

归一化向量求导：
$$\begin{gathered} \frac{\partial }{\partial \mathbf{x}}(\hat{\mathbf{x}})=\frac{\partial }{\partial \mathbf{x}}(\frac{\mathbf{x}}{||\mathbf{x}||})=\frac{\partial }{\partial \mathbf{x}}({\mathbf{x}}^T\frac{1}{||\mathbf{x}||}) \\ =\frac{1}{||\mathbf{x}||}\frac{\partial }{\partial \mathbf{x}}(\mathbf{x})+\mathbf{x}\frac{\partial }{\partial \mathbf{x}}(\frac{1}{||\mathbf{x}||}) \\ =\frac{1}{||\mathbf{x}||}-\mathbf{x}\frac{{\hat{\mathbf{x}}}^T}{||\mathbf{x}|{|}^2} \\ =\frac{||\mathbf{x}||I-\mathbf{x}{\hat{\mathbf{x}}}^T}{||\mathbf{x}|{|}^2} \\ =\frac{I-\hat{\mathbf{x}}{\hat{\mathbf{x}}}^T}{||\mathbf{x}||} \end{gathered}$$

对比误差求导：

对于anchor向量$q$,对比向量为 { k i } i = 0 K \{k_i\}_{i=0}^K {ki​}i=0K​,其中$k_0$是正样本向量， { k i } i = 1 K \{k_i\}_{i=1}^K {ki​}i=1K​为K个负样本向量，基于交叉熵误差函数为：
$$L=-\log \frac{\exp (q^T{k}_0/\tau )}{{\sum }_{i=0}^K\exp (q^T{k}_i/\tau )}$$
这里，可以认为logits $z=[q^T{k}_0/\tau ,q^T{k}_1/\tau ,\cdots ,q^T{k}_K/\tau ]$,这里$\tau$是一个温度系数，可以设为可学习变量，也可以设置为一个常数
根据交叉熵误差梯度计算规则（具体可以参考参考地址），可以得到：
$$\frac{\partial L}{\partial z_i}=p_i-y_i$$
其中$p_i=\frac{\exp (q^T{k}_i/\tau )}{{\sum }_{i=0}^K\exp (q^T{k}_i/\tau )}$为softmax归一化之后的概率值。对于锚样本$q$的梯度，计算为：
$$\frac{\partial L}{\partial q}=\sum _{i=0}^K(\frac{\partial L}{\partial z_i}\frac{\partial z_i}{\partial q})=\sum _{i=0}^K(p_i-y_i)k_i/\tau$$

对比向量梯度为：
$$\frac{\partial L}{\partial k_i}=\frac{\partial L}{\partial z_i}\frac{\partial z_i}{\partial k_i}=(p_i-y_i)q/\tau$$

温度系数的梯度为：
$$\frac{\partial L}{\partial \tau }=\sum _{i=0}^K\frac{\partial L}{\partial z_i}\frac{\partial z_i}{\partial \tau }=\sum _{i=0}^K(p_i-y_i)q^T{k}_i(-1/{\tau }^2)=\sum _{i=0}^K(y_i-p_i)z_i/\tau$$

pytorch代码实现：

import torch
import torch.nn.functional as F
p = torch.tensor([[0.1, 0.2, 0.3],
                  [0.5, 0.6, 0.8]], requires_grad=True)
k = torch.tensor([[0.4, 0.5, 0.7],
                  [0.6, 0.8, 0.6],
                  [0.6, 0.8, 0.6],], requires_grad=True)
tau = torch.tensor(0.01, requires_grad=True)
targets = torch.tensor([0, 0])

class CrossEntropyLoss(torch.autograd.Function):
    @staticmethod
    def forward(ctx, p, k, tau, targets):
        logits = p @ k.T / tau
        targets = F.one_hot(targets, num_classes=logits.size(1)).float()
        prob = F.softmax(logits, 1)
        ctx.save_for_backward(logits, prob, targets, p, k, tau)
        logits = F.log_softmax(logits, 1)
        loss = -(targets * logits).sum(1).mean()
        return loss

    @staticmethod
    def backward(ctx, grad_output):
        logits, prob, targets, p, k, tau = ctx.saved_tensors
        grad_p = grad_output * (prob - targets) @ k / tau / targets.size(0)
        embed_size = p.size(1)
        prob_targets_repeat = (prob - targets).t().repeat(1, embed_size).view(-1,embed_size, p.size(0))
        grad_k = grad_output * (prob_targets_repeat * (p.t() / tau).unsqueeze(0)).sum(-1) / targets.size(0)
        tau_grad = grad_output * torch.sum((targets - prob) * logits / tau, dim=1).mean()
        grad_targets = None 
        return grad_p, grad_k, tau_grad, grad_targets

loss = CrossEntropyLoss.apply(p, k, tau, targets)
loss.backward()

print(p.grad)
print(k.grad)
print(tau.grad)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

输出：

tensor([[ 9.9664, 14.9496, -4.9832],
        [10.0000, 15.0000, -5.0000]])
tensor([[-29.9832, -39.9664, -54.9496],
        [ 14.9916,  19.9832,  27.4748],
        [ 14.9916,  19.9832,  27.4748]])
tensor(-1249.1605)
1
2
3
4
5
6

pytorch函数验证：

import torch
import torch.nn.functional as F
p = torch.tensor([[0.1, 0.2, 0.3],
                  [0.5, 0.6, 0.8]], requires_grad=True)
k = torch.tensor([[0.4, 0.5, 0.7],
                  [0.6, 0.8, 0.6],
                  [0.6, 0.8, 0.6],], requires_grad=True)
tau = torch.tensor(0.01, requires_grad=True)

logits = p @ k.T / tau
targets = torch.tensor([0, 0])

loss = F.cross_entropy(logits, targets)

loss.backward()

print(p.grad)
print(k.grad)
print(tau.grad)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

输出：

tensor([[ 9.9664, 14.9496, -4.9832],
        [10.0000, 15.0000, -5.0000]])
tensor([[-29.9832, -39.9664, -54.9496],
        [ 14.9916,  19.9832,  27.4748],
        [ 14.9916,  19.9832,  27.4748]])
tensor(-1249.1606)
1
2
3
4
5
6