A Novel Spatial Fidelity With Learnable Nonlinear Mapping for Panchromatic Sharpening

The purpose of panchromatic (PAN) sharpening, i.e., pansharpening, is to fuse a low spatial resolution multispectral (LRMS) image with a high spatial resolution PAN image, aiming to obtain a high spatial resolution multispectral (HRMS) image. Pansharpening models based on variational optimization consist of a spectral fidelity term, a spatial fidelity term, and a regularization term. Most of the methods assume that the existing PAN image and the homologous HRMS image satisfy the global or local linear relationship, which could be far from the real case, thus causing suboptimal performance. Inspired by the nonlinear mapping ability of machine learning (ML) techniques, we propose a novel spatial fidelity term with learnable nonlinear mapping (LNM-SF), which trains an implicit functional operator via a specifically designed convolutional neural network (CNN) and efficiently constructs the nonlinear relationship between the known PAN and the latent HRMS images. Relying upon the above description of the spatial fidelity term, a new variational model with a learnable nonlinear mapping in the spatial fidelity term for pansharpening, named LNM-PS, is simply integrated by the conventional spectral fidelity term into the proposed LNM-SF. To effectively solve the resulting optimization problem, we develop an alternating direction method of multipliers (ADMM)-based algorithm with the fast iterative shrinkage-thresholding algorithm (FISTA) as an inner solver. Extensive numerical experiments on different datasets, assessing the performance both at reduced resolution and full resolution, show the superiority of the proposed LNM-PS method. The code is available at https://github.com/liangjiandeng/-LNM-PS.