General Idea:

For Classification Task:
Input the feature vector and the corresponding type,use the matrix to calculate the distance/simmilarity between the pairs. With different assumption or view, there're several different design for the loss function.
the code resource is from https://github.com/bnulihaixia/Deep_metric

The Main Scopes

1. dynamic learning rate
2. euclidean distance or similarity or js divergence
3. whether consider the all samples or the select the hard samples
4. how to evaluate the hard samples, absolute?relative? above/below mean? with sampling probability?
   if so, how to calculate the probability?
5. whether use different weight for calculating loss

6. whether use slice?BDW, like a batch normalization

   
   
   image.png

Details

BatchAll (almost same as A BatchAll, without the a_lr)

• dynamic learning rate

            pos_logit = torch.sum(torch.exp(self.alpha * (1 - pos_pair)))
            neg_logit = torch.sum(torch.exp(self.alpha * (1 - neg_pair)))
            a_lr = 1 - (pos_logit / (pos_logit + neg_logit)).data[0]
            ......
            loss_ = a_lr*torch.sum(valid_triplets)

• pos num: neg_num = 1:1 with the repeating operation

            pos_pair = pos_pair.repeat(num_neg_instances, 1)
            neg_pair = neg_pair.repeat((num_instances-1), 1).t()

• self.margin

          triplet_mat = pos_pair - neg_pair + self.margin

A_hard_pair

• dynamic learning rate
• Focus on the hard pair, loss get value only when the pos pair exceeds the limit(too far) and the neg pair is within the margin(dis<1.1) (the absolute distance)

            pos_loss = torch.log(torch.sum(torch.exp(self.beta * (pos_pair - 0.8))))
            neg_loss = torch.log(torch.sum(torch.exp(self.beta * (1.1 - neg_pair))))

A_triplet/Triplet

• dynamic learning rate
• the pos distance exceed neg distance too much(the relative distance)

            triplet_mat = torch.log(torch.exp(self.beta*(pos_pair - neg_pair)) + 1)
            triplet_mask = triplet_mat > 0.65

BatchHard

select the max pos and min( both the hardest examples) by row in a batch;
(much more elegant for the neg_dist/pos_dist calculation)

        hard_pos = torch.max(pos_dist_mat, dim=0)[0]
        hard_neg = torch.min(neg_dist_mat, dim=0)[0]

BDWDistWeightNeighborloss( Slice+ DistWeightNeighborloss)

BinDevianceLoss(Branch)

• use the similarity matrix, not the euclidean distance. The similarity score is high with the distance being short on the regularized case. and only select the neg pair of which similarity score is larger than the pos pair.

             neg_pair = torch.masked_select(neg_pair, neg_pair > pos_pair[0] - 0.05)

• with constant margin to constrain

            pos_loss = torch.mean(torch.log(1 + torch.exp(-2*(pos_pair - self.margin))))
            neg_loss = 0.04*torch.mean(torch.log(1 + torch.exp(50*(neg_pair - self.margin))))

CenterLoss

(ref:https://blog.csdn.net/u014380165/article/details/76946339)

image.png

image.png

image.png

the distance to the center for every feature, smaller distance in the cluster represents the better results
Steps:

1. store the centers and inputs
2. calculate the center dist
3. the closet neighbour center as neg sample and the farthest input to its center as pos sample.(also for selecting the hard samples)

            dist_an.append(centers_dist[i][targets_ != target].min())
            center_diff = inputs_list[i] - centers[i]
            center_diff_norm = torch.cat([torch.norm(temp) for temp in center_diff])
            dist_ap.append(center_diff_norm.max())

CenterNCA

Center+ base(I can't tell the apparent feature of NCA from the view of code, maybe the "base" setting,but here, the pos or neg selecting is the individual sample with the center)

CenterPair

constant limit to select pos/neg samples

       loss = torch.mean(pos_dist.clamp(min=0.15) -
       torch.log(torch.sum(torch.exp(-neg_dist.clamp(max=0.6)), 0)))

ClusterNCA

lable is not from the initialization but the kmean clustering result(need to point out the number of cluster)

Contrastive Loss

take the samples with in same class as positive examples.

image.png

DistWeightLoss

• select the pos samples according to the similarity probability(the weight)
• select the hard neg pair

pos_pair = torch.sort(pos_pair)[0]
sampled_index = torch.multinomial(torch.exp(5*pos_pair), 1)
neg_pair = torch.masked_select(neg_pair, neg_pair > pos_min - 0.01)

DistWeightContrastiveLoss

Gaussian Probability to select, with constant limit.

DistanceMatchLoss

regard the neg/pos samples distribution as Gaussian Distribution, and select the sample use the Gaussian parameters

            neg_pair = neg_dist[i]
            neg_mean, neg_std = GaussDistribution(neg_pair)
            prob = torch.exp(torch.pow(neg_pair - neg_mean, 2) / (2 * torch.pow(neg_std, 2)))
            neg_index = torch.multinomial(prob, 3*num_instances, replacement=False)

and different weight to calculate the loss

               base = [0.95, 1.05, 1.12]
                muls = [4, 8, 16]
                pos_diff = [pos_pair[i] - base[i] for i in range(len(base))]
                pos_diff = torch.cat([1.0 / muls[i] *torch.log(1 + torch.exp(pos_diff[i])) for i in range(len(base))])

Gaussian LDA

different way to calculate the loss

           pos_logit = torch.sum(torch.exp(self.alpha*(1 - pos_neig)))
            neg_logit = torch.sum(torch.exp(self.alpha*(1 - neg_neig)))
            loss_ = -torch.log(pos_logit/(pos_logit + neg_logit))

GaussianMetric

in the code, there's nothing about Gaussian. The main idea is to select the neg/pos pair according to the mean samples.
(it seems that those are all about how to select negative samples and positive samples)

NCA Loss

To select pos/neg in Top-K Neighborhood.
base function: in case the float number operation happened illegally