Deep learning based plant part detection in Greenhouse settings

Precision agriculture challenges such as automatic harvesting, phenotyping, and yield prediction require precise detection of plant parts such as the fruits, leaves or stems. Deep learning has emerged as the state-of-the-art technology for image segmentation and object detection in several domains, notably in self-driving vehicles and medical imaging. In recent years, deep learning methods are being increasingly adopted in vision-based applications for precision agriculture. In previous work, methods were investigated to segment the image for plant parts. However, such an approach did not yield object instances. In this work, we applied the state-of-the-art deep learning object detector, Mask RCNN, to the problem of detecting fruit and other plant parts, in the sweet pepper (capsicum annuum) plant. An extensive study was carried out where we investigated different transfer learning schemes, different convolutional neural network architectures, and varying numbers of training images. Experimentally, we found that Mask RCNN trained with the synthetic data and fine-tuned with very few empirical images is able to detect more than 95% of the sweet pepper fruit. It was also found that training on the synthetic data and then fine-tuning over a few empirical images led to a better performance in the detection of fruit, over training only on the limited set of empirical images. Furthermore, results show that the best model could successfully generalize to different imaging conditions. This work is a necessary step for applying deep learning in high-throughput robotics and phenotyping approaches and will open up many opportunities for smart farming and more efficient use of resources. Currently, training deep learning models is dependent on the knowledge and expertise of the scientists involved. The insights gained from this work should lead to more automatic training protocols, allowing widespread use in very different applications.