Vision principles for harvest robotics: sowing artificial intelligence in agriculture

The objective of this work was to further advance technology in agriculture, specifically by pursuing the research direction of agricultural robotics for harvesting in greenhouses, with the specific use-case of Capsicum annuum, also known as sweet or bell pepper. Within this scope, it was previously determined that the primary cause of agricultural robotics not yet maturing was the complexity of the tasks due to inherent variations of the crops, in turn limiting performance in harvest success and time. As a solution, it was suggested to further enhance robotic systems with sensing, world modelling and reasoning, for example by pursuing approaches like machine learning and visual servo control. In this work, we have followed this suggestion. It was identified that facilitating new levels of artificial intelligence in the domains of sensing and motion control would be one of the ways to improve upon classical mechanization. Specifically, we investigated the means of using machine learning based computer vision guided manipulation towards a basic form of world representation and autonomy. For this, in Chapter 2 we developed an eye-in-hand sensing and visual control framework for dense crops with the goal to overcome issues of occlusion and image registration that were previously introduced when sensing was performed externally from the robot manipulator. Additionally, simultaneous localization and mapping was investigated to aid in forming a world model. In Chapter 3 we aimed to reduce the requirement of annotating empirical images by providing a method to synthetically generate large sets of automatically annotated images as input for convolutional neural network (CNN) based segmentation models. An annotated dataset was created of 10,500 synthetic and 50 empirical images. In Chapter 4 we further investigated how synthetic images can be used to bootstrap CNNs for successful learning of empirical images. We provided computer vision in agriculture a pioneering machine learning based methodology for state-of-the-art plant part segmentation performance, whilst simultaneously reducing the reliance on labor intensive manual annotations. Chapter 5 explored applying a cycle consistent generative adversarial network to our dataset with the objective to generate more realistic synthetic images by translating them to the feature distribution of the empirical domain. We show that this approach can further improve segmentation performance whilst further reducing the requirement of annotated empirical images. In Chapter 6 we aimed to bring all previous chapters into practice. The objective was to estimate angles between fruit and stems from image segmentations to support visual servo control grasping in a sweet-pepper harvesting robot. Our approach calculated angles under unmodified greenhouse conditions that met the accuracy requirement of 25 degrees for 73% of the cases. Combined, the work shows a promising stepping stone towards agricultural robotics which could ensure the quality of meals and nourishment of a growing population. Furthermore, it can become an important technology for societal issues in developed nations, e.g. by solving current labor problems. It can further improve upon the quality of life and contribute to reaching an exemplary equilibrium of sustainable agricultural production.