In a visible light communication (VLC) system, the light emitting diode (LED) is nonlinear for large signals, which limits the transmission power or equivalently the coverage of the VLC system. When the input signal amplitude is large, the nonlinear distortion creates harmonic and intermodulation distortion, which degrades the transmission error vector magnitude (EVM). To evaluate the impact of nonlinearity on system performance, the signal to noise and distortion ratio (SNDR) is applied, defined as the linear signal power over the thermal noise plus the front end nonlinear distortion. At a given noise level, the optimal system performance can be achieved by maximizing the SNDR, which results in high transmission rate or long transmission range for the VLC system. In this paper, we provide theoretical analysis on the optimization of SNDR with a nonlinear Hammerstein model of LED. Simulation results and lab experiments validate the theoretical analysis.