Enhancement of three-mode optomechanical interaction by feedback-controlled light

We realise a feedback-controlled optical Fabry-Perot cavity in which the transmitted cavity output is used to modulate the input amplitude fluctuations. The resulting phase-dependent fluctuations of the in-loop optical field, which may be either sub-shot or super-shot noise, can be engineered to favourably affect the optomechanical interaction with a nanomechanical membrane placed within the cavity. Here we show that in the super-shot-noise regime ('anti-squashed light') the in-loop field has a strongly reduced effective cavity linewidth, corresponding to an increased optomechanical cooperativity. In this regime, feedback improves the simultaneous resolved-sideband cooling of two nearly degenerate membrane mechanical modes by one order of magnitude.