Ultrafast transient absorption spectroscopy is a powerful technique to study fast photo-induced processes, such as electron, proton and energy transfer, isomerization and molecular dynamics, in a diverse range of samples, including solid state materials and proteins. Many such experiments suffer from signal distortion by scattered excitation light, in particular close to the excitation (pump) frequency. Scattered light can be effectively suppressed by a polarizer oriented perpendicular to the excitation polarization and positioned behind the sample in the optical path of the probe beam. However, this introduces anisotropic polarization contributions into the recorded signal. We present an approach based on setting specific polarizations of the pump and probe pulses, combined with a polarizer behind the sample. Together, this controls the signal-to-scatter ratio (SSR), while maintaining isotropic signal. We present SSR for the full range of polarizations and analytically derive the optimal configuration at angles of 40.5° between probe and pump and of 66.9° between polarizer and pump polarizations. This improves SSR by (or compared to polarizer parallel to probe). The calculations are validated by transient absorption experiments on the common fluorescent dye Rhodamine B. This approach provides a simple method to considerably improve the SSR in transient absorption spectroscopy.