Variational Principles in Quantum Monte Carlo: The Troubled Story of Variance Minimization

We investigate the use of different variational principles in quantum Monte Carlo, namely, energy and variance minimization, prompted by the interest in the robust and accurate estimation of electronic excited states. For two prototypical, challenging molecules, we readily reach the accuracy of the best available reference excitation energies using energy minimization in a state-specific or state-average fashion for states of different or equal symmetry, respectively. On the other hand, in variance minimization, where the use of suitable functionals is expected to target specific states regardless of the symmetry, we encounter severe problems for a variety of wave functions: as the variance converges, the energy drifts away from that of the selected state. This unexpected behavior is sometimes observed even when the target is the ground state and generally prevents the robust estimation of total and excitation energies. We analyze this problem using a very simple wave function and infer that the optimization finds little or no barrier to escape from a local minimum or local plateau, eventually converging to a lower-variance state instead of the target state. For the increasingly complex systems becoming in reach of quantum Monte Carlo simulations, variance minimization with current functionals appears to be an impractical route.