Changes in donor priorities have meant that agronomists working in the tropics find themselves in a fundamentally new operational space, one that demands rapid improvements in farmers' livelihoods resulting from the large-scale adoption of new technologies and crop management practices. As a result, on-farm trials in contemporary Agricultural Research for Development (AR4D) are increasingly implemented both to collect data and to spur farmer adoption. We examine the different interpretations and organisational practices of AR4D organisations in this new operational space, and reflect on the usefulness of on-farm trials for agricultural technology scaling. Three case studies are presented to address these questions – two in sub-Saharan Africa and one in South Asia. Each study is considered in light of Science and Technology Studies theory and locates science as a politically situated practice, recognising the tension that scientists face between providing evidence and persuading selected audiences. The case studies show that this tension results in the introduction of several biases that limit the scalability of the technologies under investigation. These include biases at the level of the trial location, host-farmer selection, trial design, management and evaluation. We conclude by discussing how the contemporary political and institutional environment of AR4D produces project beneficiaries and research outcomes on selected farms, but not necessarily impacts at scale.