Software design for automatic discards survival prediction

One major problem with the Beam Trawl fisheries is the high amount of bycatch. For this reason, the European Union expanded the Common Fisheries Policy (CFP) with a Landing Obligation (LO)[EU, 2013]. With the LO, discarding certain species, such as Plaice, Sole and Turbot, is prohibited. The reasoning behind the LO is that this enforces developments in fishing technology, creating more selective fishing vessels and fishing gears. An exemption can be given to the LO when high survival for certain species, such as Plaice, Sole and Turbot, is scientifically proven in a survival study. This survival study is done by Wageningen Marine Research (WMR) for the Dutch fisheries [Schram et al., 2023]. This survival study is time-intensive, man-hour-intensive and costly. WMR has done some work in trying to predict the survival based on visual observation with A-D classes, but these classes were not accurate enough to quantify the survival. This EngD aims to take a first step in automating this survival study with computer vision for three species, Plaice, Sole and Turbot. Three components were designed during this EngD. The first component is a camera system comprising a Red Green Blue (RGB) camera and a multispectral camera, photographing in 660 nm, 580 nm, 820 nm and 735 nm bands. The second component is two computer vision algorithms developed in five design iterations. There are two final algorithms. The first is MTLseg, which can do semantic segmentation of various body parts (body, tail, head, and fins) and damages (blood and scale loss) using Mask2Former and predict survival[ Cheng et al., 2022]. This network is only developed for Plaice. MTL stands for Multi- Task Learning (MTL), where a single network learns multiple tasks simultaneously. The second is MTLdamage, which can classify various damages (Fin damage, 50% damage on the Dorsal side, head bleeding, hypodermic bleeding, visible intestines and wound) and predict survival. This network is developed for Plaice, Sole and Turbot. The networks learn normalised survival, between 0 and 1, where 1 means the fish survived the entire survival study. The multispectral images did not give an advantage due to the ImageNET pre-trained weights on RGB. On the test set,MTLseg andMTLdamage fulfil four of the five pre-defined requirements, which consist of requirements comparing the performance of the humans (with the A-D classes) to the AI. Specifically, MTLseg and MTLdamage outperform the human prediction for the Concordance-Index (C-Index) and the R2, while the performance on the Balanced Accuracy (BA) is almost equal. However, the error in the Survival Rate (SR) is lower with human prediction. For Plaice, humans achieve aΔSR of 0.02, compared to 0.07 with MTLseg and 0.03 with MTLdamage. Similarly, on Sole, humans achieve a Δ SR of 0.07, whereas MTLdamage reaches 0.13. On Turbot, humans achieve a Δ SR of 0.11, but this increases to 0.17 withMTLdamage. Therefore, humans outperformMTLdamage and MTLseg in the Δ SR, because the structural error in the A-D classes is known, while the structural error for the AI’s prediction is unknown. Consequently, this structural error with MTLseg and MTLdamage must be discovered in future research. The third component of the EngD design is a Graphical User Interface (GUI). This GUI enables the researchers to run the computer vision algorithm on their PC. The GUI consists of various buttons to select the data, run the models and export the predictions. Also, different graphs are shown, such as the count of the number of damages and the Kaplan- Meier curve. Overall, this EngD took the first step in predicting the survival of flatfish using computer vision technologies. However, more training data is recommended, especially for Sole and Turbot.