Bayesian mixed effect atlas estimation with a diffeomorphic deformation model

In this paper we introduce a diffeomorphic constraint on the deformations considered in the deformable Bayesian mixed effect template model. Our approach is built on a generic group of diffeomorphisms, which is parameterized by an arbitrary set of control point positions and momentum vectors. This enables us to estimate the optimal positions of control points together with a template image and parameters of the deformation distribution which compose the atlas. We propose to use a stochastic version of the expectation-maximization algorithm where the simulation is performed using the anisotropic Metropolis adjusted Langevin algorithm. We propose also an extension of the model including a sparsity constraint to select an optimal number of control points with relevant positions. Experiments are carried out on the United States Postal Service database, on mandibles of mice, and on three-dimensional murine dendrite spine images.