Fiber-optic communication systems face various challenges due to the ongoing demand for higher data rates and longer transmission reach. In addition to linear distortions due to chromatic dispersion and bandwidth limitations of devices, the nonlinear fiber impairments, which become more dominant due to transmission with higher launch power, lead to the fiber-channel capacity crunch. With the advancement of high-speed electronics, digital signal processing techniques offer a powerful yet cost-effective solution for the challenges mentioned above. This work investigates potential candidates for pre- and post-compensation of ISI and nonlinear distortions, separated into three stages. In the first stage, the compensation of fiber impairments is investigated with an emphasis on the digital back-propagation (DBP) method as a solution for the capacity-crunch problem. Possible implications of DBP on the future transmission-link design are examined, and its performance in an optical network scenario is evaluated. In the second stage, the capability of feed-forward and feedback equalizer structures to compensate for ISI is investigated. In particular, the suitability of the Tomlinson-Harashima precoding (THP) method for fiber-optic communications is assessed. It is shown experimentally that THP significantly outperforms decision-feedback equalization in transmission scenarios characterized by spectral zeros. The concept of fitting the decision thresholds to the received signal to allow enhanced symbol detection is introduced and discussed. Clustering algorithms are compared with selected post-compensation techniques in a case study regarding their performance and computational complexity to evaluate the potential of enhanced detection to replace different compensation stages.weiterlesen