Optical scattering loss coefficient of multimode rectangular waveguide is analyzed in this work. First, the effective refractive index and the mode field distribution of waveguide modes are obtained using the Marcatili method. The influence on scattering loss coefficient by waveguide surface roughness is then analyzed. Finally, the mode coupling efficiency for the SMF-Optical-Waveguide (SOW) structure and MMF-OpticalWaveguide (MOW) structure are presented. The total scattering loss coefficient depends on modes scattering loss coefficients and the mode coupling efficiency between fiber and waveguide. The simulation results show that the total scattering loss coefficient for the MOW structure is affected more strongly by surface roughness than that for the SOW structure. The total scattering loss coefficient of waveguide decreases from 3.97 × 10 ^-2 dB/cm to 2.96 × 10 ^-4 dB/cm for the SOW structure and from 5.24 × 10 ^-2 dB/cm to 4.7 × 10 ^-4 dB/ cm for the MOW structure when surface roughness is from 300nm to 20nm and waveguide length is 100cm.