Low yield/hectare gains, food insecurity and environmental unsustainability are challenges experienced by the agriculture sector in Ethiopia despite substantial government investment. Although there are many factors that contribute to the poor performance of the sector, smallholder farmer competence gaps are principal among them. This study aims to examine the effects of Comprehensive Competence-Based Training (CCBT) on the competence development and performance improvement of smallholder farmers using the authentic professional core task during maize planting as a problem context. We applied a 3-week randomized (control group pretest posttest) design and single-blind field experiment to test the impact of CCBT through provision of a training to two comparable farmer groups using conventional ‘Low-CBT’ and innovative ‘High-CBT’ implementation levels. The samples included ‘High-CBT’ (N = 220) and ‘Low-CBT’ (N = 220) groups of smallholder farmers in the West Gojjam Zone in Ethiopia. Data on competence development and performance improvement of farmers were collected from themselves, trainer Development Agents and Trained Assessors. The yield in quintal/hectare gains for each smallholder farmer was collected twice (before and after the intervention). Repeated (pretest, posttest) MANOVA and ANOVA measurements were used to analyze the data. The results revealed that the development of smallholder farmer competence in ‘High-CBT’ was higher than in ‘Low-CBT’ training. Comparisons of performance in both the authentic job situation and in terms of yield in quintal/hectare gains in the two groups revealed a better performance of both groups. However, the ‘High-CBT’ group performed better than the ‘Low-CBT’ group in both the authentic job situation and in terms of yield in quintal/hectare gains. We obtained 31 and 41 quintal/hectares of maize for the ‘Low-CBT’ and ‘High-CBT’ groups, respectively, which are better than the baseline average 22 quintal/hectare for both groups. These findings underscore the relevance of CCBT, especially when the design principles of CBE are integrated well in the training programme (which was called ‘High-CBT level’), for improving performance, in this case gain in yield per hectare of smallholder farmers, which potentially results in the increase of household food security.