It has recently been shown that Fourier transform infrared spectroscopy has potential for the prediction of detailed milk fat composition, even based on a limited number of observations. Therefore, there seems to be an opportunity for improvement by means of using more observations. The objective of this study was to verify whether the use of more data would add to the accuracy of predicting milk fat composition. In addition, the effect of season on modeling was quantified because large differences in milk fat composition between winter and summer samples exist. We concluded that the use of 3,622 observations does increase predictability of milk fat composition based on infrared spectroscopy. However, for fatty acids with low concentrations, the use of many observations does not increase predictability to a level at which application of the model becomes obvious. Furthermore, the effect of season on validation r-square was limited but was occasionally large on prediction bias. For fatty acids that show large differences in level and standard deviation between winter and summer, a representative sample that includes observations collected in various seasons is critical for unbiased prediction. This research shows that all major fatty acids, combined groups of fatty acids, and the ratio of saturated to unsaturated fatty acids can be predicted accurately.