Multiple object tracking with multi-view active vision to effectively find plant nodes in a cluttered tomato greenhouse

Agricultural robots can alleviate the challenges of increasing food demand and worsening labor shortages. However, agricultural environments are typically cluttered, challenging robotic perception to find all relevant objects, as these are often hidden from view. For instance, for leaf removal and whole-truss tomato harvesting, a robot needs to efficiently detect and accurately localize the petiole and peduncle nodes of a tomato plant, which are often hidden behind leaves or tomato trusses. In this study, an integration of multiview active vision (MAV) and multiple object tracking (MOT) algorithms is proposed to improve the detection and localization of tomato plant nodes in an occluded greenhouse environment. This paper details how the system is able to collect information in a cluttered 3D environment, to generate a 3D representations and based on that effectively pinpoints plant nodes through an active-vision strategy. This study tested the MOT-MAV perception system on ten randomly selected tomato plants in an unmodified cluttered tomato greenhouse. The integration perception system was compared to a baseline experiment, which uses predefined sets of camera viewpoints and a clustering method to combine node observations from different viewpoints. Three different versions of experiments were evaluated: “MOT Exp”, “MAV Exp” and the combined “MOT-MAV Exp”. Under the same camera viewpoints and routes, the “MOT Exp” improved recall by 0.09 over the “Baseline Exp,” confirming MOT's effectiveness in enhancing object detection. After six viewpoints, the “MAV Exp” achieved a PCO of 77%, 11 p.p. higher than the “Baseline Exp,” demonstrating faster object detection with active vision. The “MOT-MAV Exp” outperformed all setups in detection efficiency and accuracy. After six viewpoints, it reached 90% of the final detection result (10 viewpoints), 17 p.p. higher than the “Baseline Exp,” highlighting its faster detection speed. Additionally, its PCO reached 84%, exceeding the “Baseline Exp” by 18 p.p., showcasing improved detection accuracy. These results demonstrate that integrating MOT with next-best-view (NBV) planning enables robots to handle cluttered greenhouse environments more effectively, which improving the perception capabilities of agro-food robotic systems is essential for promoting the development of efficient tomato de-leafing and harvesting robots.