Small P values may not yield robust findings: an example using REST-meta-PD

Thousands of resting state functional magnetic resonance imaging (RS-fMRI) articles have been published on brain disorders. For precise localization of abnormal brain activity, a voxel-level comparison is needed. Because of the large number of voxels in the brain, multiple comparison correction (MCC) must be performed to reduce false positive rates, and a smaller P value (usually including either liberal or stringent MCC) is widely recommended. The study suggests that a stringent MCC may reduce the number of false positive results and that the brain regions surviving MCC are true positive results and thus can improve the reproducibility between studies. However, determining the clusters that represent true positives is difficult because ground truths have not been established for many brain disorders that do not present visible structural abnormalities, e.g., Parkinson's disease (PD) and many psychiatric disorders. Meta-analysis provides robust results that can be used to investigate reproducibility across studies. However, most existing neuroimaging meta-analytic articles are based on only a few peak coordinates reported in the original articles, namely coordinate-based meta-analysis (CB-meta). The current REST-meta-PD study included raw RS-fMRI data from 15 cohorts of PD patients and performed a meta-analysis of the amplitude of low frequency fluctuations (ALFF). We used the robustness results of the meta-analysis to test whether liberal or stringent MCC could increase reproducibility and reduce false positive rates.