KL divergence is a natural way to measure the difference between two probability distributions. The entropy of a distribution gives the minimum possible number of bits per message that would be needed (on average) to losslessly encode events drawn from . Achieving this bound would require using an optimal code designed for , which assigns shorter code words to higher probability events. can be interpreted as the expected number of extra bits per message needed to encode events drawn from true distribution , if using an optimal code for distribution rather than . It has some nice properties for comparing distributions. For example, if and are equal, then the KL divergence is 0.

The cross entropy can be interpreted as the number of bits per message needed (on average) to encode events drawn from true distribution p, if using an optimal code for distribution q. Note the difference: measures the average number of extra bits per message, whereas measures the average number of total bits per message. It's true that, for fixed p, will grow as becomes increasingly different from . But, if isn't held fixed, it's hard to interpret as an absolute measure of the difference, because it grows with the entropy of .

KL divergence and cross entropy are related as:

We can see from this expression that, when and are equal, the cross entropy is not zero; rather, it's equal to the entropy of .

Cross entropy commonly shows up in loss functions in machine learning. In many of these situations, is treated as the 'true' distribution, and as the model that we're trying to optimize. For example, in classification problems, the commonly used cross entropy loss (aka log loss), measures the cross entropy between the empirical distribution of the labels (given the inputs) and the distribution predicted by the classifier. The empirical distribution for each data point simply assigns probability 1 to the class of that data point, and 0 to all other classes. Side note: The cross entropy in this case turns out to be proportional to the negative log likelihood, so minimizing it is equivalent maximizing the likelihood.

Note that (the empirical distribution in this example) is fixed. So, it would be equivalent to say that we're minimizing the KL divergence between the empirical distribution and the predicted distribution. As we can see in the expression above, the two are related by the additive term (the entropy of the empirical distribution). Because is fixed, doesn't change with the parameters of the model, and can be disregarded in the loss function. We might still want to talk about the KL divergence for theoretical/philosophical reasons but, in this case, they're equivalent from the perspective of solving the optimization problem. This may not be true for other uses of cross entropy and KL divergence, where might vary.

t-SNE fits a distribution in the input space. Each data point is mapped into the embedding space, where corresponding distribution is fit. The algorithm attempts to adjust the embedding to minimize . As above, is held fixed. So, from the perspective of the optimization problem, minimizing the KL divergence and minimizing the cross entropy are equivalent. Indeed, van der Maaten and Hinton (2008) say in section 2: "A natural measure of the faithfulness with which models is the Kullback-Leibler divergence (which is in this case equal to the cross-entropy up to an additive constant)."

van der Maaten and Hinton (2008). Visualizing data using t-SNE.

原文見StackExchange questions #265966