A review of distances for the Mallows and Generalized Mallows estimation of distribution algorithms

The Mallows (MM) and the Generalized Mallows (GMM) probability models have demonstrated their validity in the framework of Estimation of distribution algorithms (EDAs) for solving permutation-based combinatorial optimisation problems. Recent works, however, have suggested that the performance of these algorithms strongly relies on the distance used under the model. The goal of this paper is to review three common distances for permutations, Kendall’s-$$\tau $$τ, Cayley and Ulam, and compare their performance under MM and GMM EDAs. Moreover, with the aim of predicting the most suitable distance for solving any given permutation problem, we focus our attention on the relation between these distances and the neighbourhood systems in the field of local search optimisation. In this sense, we demonstrate that the performance of the MM and GMM EDAs is strongly correlated with that of multistart local search algorithms when using related neighbourhoods. Furthermore, by means of fitness landscape analysis techniques, we show that the suitability of a distance to solve a problem is clearly characterised by the generation of high smoothness fitness landscapes.