Efficient algorithm for "on-the-flyψ error analysis of local or distributed serially correlated data

We describe the Dynamic Distributable Decorrelation Algorithm (DDDA) which efficiently calculates the true statistical error of an expectation value obtained from serially correlated data "on-the-fly," as the calculation progresses. DDDA is an improvement on the Flyvbjerg-Petersen renormalization group blocking method (Flyvberg and Peterson, J Chem Phys 1989, 91, 461). This "on-the-fly" determination of statistical quantities allows dynamic termination of Monte Carlo calculations once a specified level of convergence is attained. This is highly desirable when the required precision might take days or months to compute, but cannot be accurately estimated prior to the calculation. Furthermore, DDDA allows for a parallel implementation which requires very low communication, O(log₂ N), and can also evaluate the variance of a calculation efficiently "on-the-fly." Quantum Monte Carlo calculations are presented to illustrate "on-the-fly" variance calculations for serial and massively parallel Monte Carlo calculations.