Maximum Spectral Efficiency With Adaptive MQAM Transmissions Over Terrestrial Coherent FSO Links

Coherent free-space optical (FSO) communication is recognized as a key enabler for ultra-high-capacity fronthaul and backhaul links in next-generation wireless networks. Spectrally efficient $M$-ary quadrature amplitude modulation (MQAM) formats are well-suited for these links. However, theoretical analyses of adaptive MQAM transmissions over terrestrial FSO channels remain limited. In this letter, we first derive the spectral efficiency limit of adaptive unconstrained MQAM over gamma-gamma turbulence with pointing error. We then show that adaptive transmissions using only six square MQAM constellations performs close to the theoretical limit (within $0.10$-$0.12$ bits/s/Hz) across a wide range of signal-to-noise ratios and channel conditions.