Iron oxide nanoparticles are omnipresent in nature and of great importance for environmental sciences and technology. The size-dependent magnetic behavior of ferrihydrite (Fh), magnetite (Fe3O4), and maghemite (γ-Fe2O3) has been studied in relation to the surface structure. The selected minerals have in common the presence of tetrahedral Fe. This Fe polyhedron is unstable at the surface when forming singly coordinated ligand(s). This leads to the size-dependency of the polyhedral composition, which is for Fh in excellent agreement with the relative contributions of edge and corner sharing measured with high-energy total X-ray scattering. For Fh, superparamagnetic behavior scales with particle volume in which magnetic coupling is proportional to a fraction of the Fe per particle. Magnetic saturation at low temperature scales with size and is predominantly due to polyhedral surface depletion. The mineral core of Fh may behave ferrimagnetically as well as antiferromagnetically. Both have opposite particle size dependency, for which a surface structural model has been developed. The relative stability of ferrimagnetic and antiferromagnetic Fh is related to a slight difference in the surface Gibbs free energy (∼0.03 J m-2). At the same surface structure, the predicted crossover point is at ∼4 nm, above which the core of Fh shifts from antiferromagnetic to ferrimagnetic. For magnetite and maghemite, the size dependency of the ferrimagnetic behavior can be described with the same model as that developed for Fh by only adjusting the maximum magnetic saturation of the ideal bulk material to its theoretical value. As discussed and quantified, the structural defects of superparamagnetic Fe-oxide nanoparticles (SPION) will lower the magnetic saturation at a given size.