Olfactory loss (OL) affects up to 20% of the general population and is related to changes in olfaction-related brain regions. This study investigated the effect of etiology and duration of OL on gray matter volume (GMV) of these regions in 257 patients. Voxel-based morphometry was applied to measure GMV in brain regions of interest to test the effects of etiology and duration on regional GMV and the relation between olfactory function and regional GMV. Etiology of OL had a significant effect on GMV in clusters representing the gyrus rectus and orbitofrontal cortex (OFC), bilaterally. Patients with congenital anosmia had reduced GMV in the gyrus rectus and an increased OFC volume compared to patients with acquired OL. There was a significant association between volume of the left OFC and olfactory function. This implies that changes in GMV in patients with acquired OL are mainly reflected in the OFC and depend on olfactory function. Morphology of olfactory areas in the brain therefore seems to relate to olfactory function and the subsequent degree of exposure to olfactory input in patients with acquired OL. Differences in GMV in congenital anosmia are most likely due to the fact that patients were never able to smell.