REAL-TIME SELECTIVE SPRAYING FOR VIOLA ROPE CONTROL IN SOYBEAN AND COTTON CROPS USING DEEP LEARNING

The cultivation of soy and cotton is of great importance in the Brazilian economic scenario, both of which move billions of reais per year in exports. Weed management is important for obtaining optimal yields. Among the plants that have gained resistance and tolerance are those of the genus Ipomoea spp . These plants affect soybean and cotton crops throughout their cycle, thereby affecting their productivity. In this context, the objective of this work was to develop an embedded system for the selective spraying of rope and viola in cotton and soybean crops using algorithms for the classification and detection of objects in real time (Faster R-CNN and YOLOv3). This project was developed at the Agricultural Machinery Laboratory of the Federal University of Rondonópolis. The algorithms were trained to detect three classes (soybean, viola, and cotton) and were evaluated in terms of precision and sensitivity in the laboratory and field. Control results using faster R-CNN sprays demonstrated that real-time object detection algorithms for the selective control of weeds can be used for soybean and cotton crops.


INTRODUCTION
Weeds are categorized as plants that develop in a place unsuitable for the objectives of humans, interfering negatively with economic crops, both in productivity and in the final quality of harvested products (Vasconcelos et al., 2012).
However, it is widely known that losses caused by weeds exceed the losses of any category of agricultural pests, such as insects, nematodes, diseases, and rodents (Abouziena & Haggag, 2016). In bulk crops like beans, the decrease in yield reaches 71 % (Kozlowski et al., 2002), the average losses in soybean vary from 37% (Fleck & Candemil, 1995) to 58% (Rizzardi et al., 2003), and in some cases, the total loss of the crop.
Because of their fast growth characteristics and their ability to inhabit various environments (Ferreira & Miotto, 2009), these crops are found in vegetable crops (Moreira & Bragança, 2011), sugarcane (Correia & Kronka JR., 2010;Azania et al., 2011), soybean, cotton (Constantin et al., 2011), and cereals. The areas where these species predominate have reduced productivity, for example, there was 34% reduction in the number of end of stalks and 46% reduction in productivity when Ipomoea hederifolia grew in competition with sugarcane crop (Silva et al., 2009), while in the soybean crop the reduction varied from 15.53% to 80% according to the density of cordgrass (Piccinini et al., 2018;Pagnoncelli et al., 2017). Rizzardi et al. (2004) pointed out substantial increase in the incidence and density of cordgrass in recent years, directly affecting soybean crops.
In the area of selective spraying technologies, electronic optical sensors are used to measure reflectance intensities of spectral bands. Light Dependent resistors (LDR) were used to detect the presence of weeds or only the exposed soil from the reflectance of the red spectrum in the visible and near infrared (NIR) range, with 100% accuracy (Viliotti, 2002).
These types of sensors are also employed in commercially available nozzle controllers such as WeedSeeker® (Trimble Navigation Limited®). The second generation WeedSeeker2® has two optical sensors equipped with normalized difference vegetation index (NDVI) readers, which increases detection efficiency.
Another commercially available sensor is the WEED-IT® ², which works based on chlorophyll fluorescence in response to high-intensity near-infrared light (Visser & Timmermans, 1996).
Currently, controllers using image sensors and artificial intelligence have been developed and marketed as See & Spray® (Blue River technology), which process images obtained by cameras used as sensors, detects the presence of weeds in areas of cotton plantations, and allows a more selective spraying.
Recent approaches using deep learning have shown satisfactory results in classification, segmentation, object detection, and image-tracking tasks (Zhou et al., 2019). The employment of machine vision with deep learning is growing in recent researches for developing selective application technologies. For example, Ferreira et al. (2017) developed a software for weed detection in soybean crops using deep convolutional networks and obtained accuracy above 99%.
Considering this, we aimed to evaluate the application of two object detection algorithms (FASTER R-CNN and YOLOv3) in real time and develop an embedded selective spraying system for the control of Ipomoea spp. in soybean and cotton crops.

MATERIAL AND METHODS
This work was conducted in the Agricultural Machines and Mechanization Laboratory of the Federal University of Rondonópolis. The project involved the development of an on-board system for spraying under field conditions, object classification and detection, and valve actuation for spraying.
Faster R-CNN (Ren et al., 2016) is an evolution of FAST R-CNN (Girshick, 2015), which improves the computational performance of the network in terms of processing time and detection in selective search algorithms. YOLO (You Only Look Once) is a real-time object detection algorithm (Redmon & Farhadi, 2018) based on convolutional neural networks for multiple object detection with good generalization ability and simultaneously performing predictions at multiple locations in the image, that predicts a class to which object belongs in an optimized manner and allows working at high frames per second (FPS) rates.
An image bank containing soybean, cotton, and rape leaves images was created as per interest for detection. Images were captured using a semi-professional Nikon P250 camera in areas of the Federal University of Rondonópolis and Mato Grosso Institute of Cotton (IMA) in Rondonópolis (Figure 1). The high variability and quantity of images helped in the learning process of a CNN, and should correspond as closely as possible to its real application (Olsen et al. 2019). FIGURE 1. Photos of leaves of rape-string (1a, 1b, 1c), cotton (2a, 2b, 2c) and soybean (3a, 3b, 3c), at various angles, environments and weather conditions used to form the training and testing data set of the network. Source: the author.
The images were resized to 416 × 416 pixels before training the algorithm networks. The image bank contained 1,612 training images including 1,214 images acquired from crop modules planted at the UFR and 398 obtained at the IMA. Out of this, 708 images were of rape-string leaves, 487 of cotton leaves, and 417 of soybean leaves, and 10% of the images, that is 162 images, were randomly chosen to compose the test set, which was divided into 54 images of each class. The images varied with respect to the number of leaves contained in each sample at different angles, forming a dataset with representative variability.
After resizing all images in the bank, the manual tagging and labeling process was performed with the help of a program with a graphical interface known as Labelimg available in Darrenl's (2019) repository ( Figure 2). The algorithms were trained on a computer with a 3.19 GHz i7-8700 processor, 16 GB memory, GeForce RTX 2070 video card, and 8 GB of dedicated memory. In this study, we used the smaller architecture version of YOLOV3, which is a smaller and faster network called YOLOv3 tiny (ALEXEY, 2019) developed for hardware with medium configurations, which makes it possible to achieve the goal with lower computational cost, instead of using the YOLOV3 backbone, which has a larger number of layers and parameters as the network architecture. Because of the size of memory available on the GPU and the processing capacity of the computers used in this study, the YOLOv3 tiny version was chosen. YOLO-tiny network has the advantage of being faster, although it loses accuracy compared to YOLO network (Partel et al., 2019).
For the training run, the number of classes and batches, filters, and values of the anchor boxes were varied according to the purpose of our study. The number of training classes was changed to three, and the defined batch size was 64 divided into 16 subdivisions, with the learning rate set at 0.001. The convolutional layer filters were changed according to [eq. (1)].
Equation 1: Calculation of the number of filters to be applied to the image in the YOLOv3 tiny network in the detection layers.
c refers to the number of classes that will be submitted to the network for learning; in this study, the filters were set to 24.
The faster R-CNN (Ren et al., 2016) used in this study was made available by Evan (2019), and the pretrained incepton_v2 network architecture made available by Pkulzc (2019) was used. The number of classes to be trained was changed according to YOLOV3 and the learning rate was 0.0002. The faster R-CNN model of this project was developed and executed in TensorFlow's object detection API.
The evaluations of the networks in field conditions were checked for their precision and sensitivity in each class in each algorithm by visually analyzing each frame of video of each repetition recorded at the time of the tests. Subsequently, the number of leaves captured by each camera were counted according to correctly detected leaves (VP), non-detected leaves (FN) and incorrectly detected leaves (FP), and the calculations were proceeded.
A prototype of autonomous sprayer was developed for the field tests, as shown in Figure 3. The developed spray system was composed of a tank with a capacity of 30 L, pressurization pump, and four sections connected by hydraulic connections and highpressure hoses. Each section was composed of a solenoid valve and a spray tip.
The pump and reservoir were positioned on the upper part of the chassis, affixed with screws to a base wood; the nozzles and spray valves located at the rear of the prototype were fixed on a galvanized plate, that was spaced 0.63 m from the chassis by 25 x 25 mm metalon bars.
The solenoid valves were placed between the hoses and the tips of each section allowing the syrup to run through the hose. The spacing between the cameras and spray bar was 0.10 m and the support for the sensors was positioned 0.50 m from the ground, as shown in Figure 4. The tips for applying the product were installed at a height of 0.50 m from the ground, providing an overlap of 22% with a spray range of 0.70 m, JACTO nozzles, yellow color, model JSF, flow rate of 0.76 L.min -1 with an application angle of 110º at a working pressure of 310.26 kPa.
The on-board system for the detection and control of selective spraying was composed of four Logitech C920 HD PRO webcams with maximum resolution of 1080p/30qps -720p/30qps and auto focus feature; a Pentium gold G5400 computer with 3.70 GHz processor, 8.00 GB RAM, NVIDIA GeForce GTX 1050 video card (4GB GDDR5 memory, Windows 10 operating system); a Plotting Board with Arduino Nano with Atmega Atmel 328P microcontroller; DVR Valves of stainless steel with working pressure up to 1172.11 kPa and voltage of 12 V; a Pressure pump (Seaflo brand) with a working pressure of 420 kPa and a flow rate of 4.92 L.min -1 The webcams (sensors) are connected to a computer that performs the processing of the detection algorithm, which sends a signal to the Arduino microcontroller when the leaf of the vine string is detected. The sent signal indicates the sensor that detects the weed, so that the microcontroller sends a signal to each relay channel where the detection occurred. In this way, the valves were actuated individually, because each solenoid valve was connected to a relay channel, which in turn was connected to four different digital ports on the Arduino that performed the actuation only when the port emitted 5V signal. In this case, as soon as the programming detects the leaf, the computer sends a command to the microcontroller, which sends a signal to the relay module that activates the solenoid valves to initiate the spraying. The power to the entire system was supplied using a 12 V direct current (DC) battery with a 60 Ah supply. To power the computer with a bivolt source of alternating current, a 1000 W power voltage inverter that transforms 12 V DC voltage from the battery to 127 V alternating current (AC).
The average flow rate of the nozzles in this study was 0.64 L min-¹ at a 400 kPa working pressure of the pump and was obtained using Chaim's (2019) methodology (Equation (2)). To calculate this, the system was actuated for 1 min, and the sprayed volume was collected in a 1 L volumetric cup.

=
(2) Where: q = the flow rate of the system in L.min -1 , V = volume obtained in the graduated cup in liters for the system actuation time (Ts) in minutes.
To evaluate networks in the field, Partel et al. methodology (2019) was adopted, in which two test modules were set up. The first module contained two rows of the soybean crop of 2 meters in length each, with plant spacing of 0.20 m and inter-row spacing of 0.90 m. The second module contained two rows of cotton crops 4 meters in length each with plant spacing of 0.15 m and inter-row spacing 0.90 m. The spacing adopted in this study does not correspond to the adequate spacing for the soybean crop because of the prototype gauge to avoid crushing the evaluated rows of the crop during the evaluations.
To obtain sufficient number of frames of images per second, the prototype sprayer was moved in the field evaluations with a maximum speed of 0.5 m.s -1 , pump pressure at 400 kPa, and flow rate of the tips at 0.64 L.min-¹. Ten repetitions were performed for each culture in each module. A schematic of the experimental evaluation area is shown in Figure 5. At the time of evaluations, the real-time videos were recorded by each camera for each repetition provided by the networks. These videos were saved, and later converted into images in order to analyze each image frame by frame for counting the number of correctly detected leaves (PV), nondetected leaves (FN), and incorrectly detected leaves (FP). The data were then tabulated, and precision and sensitivity were calculated for each class, camera, and repetition.
The number of sprayed string plants was also counted for each repetition, algorithm, and module. To determine the percentage of control of the target plants in the area, a person walked behind the prototype to take notes every time a spray nozzle was activated on the desired target.

RESULTS AND DISCUSSION
The YOLOv3 tiny during the evaluations carried out in the field on cotton crops detected only one leaf as being correct for the crop in only one out of ten repetitions that were performed. Thus, the average accuracy obtained was 10% ( Figure 6).
The error in the cotton leaf classification may be related to the leaf format and angle at which the images are captured, as the cotton leaves are also cordate and, according to their stage and the angle in relation to the camera, present a format similar to that of a cotton cord. The capture angle also has an effect on detection accuracy, as observed by Quan et al. (2019), in which the authors evaluated different object detection algorithms (YOLOV2, FASTER R-CNN, and FASTER R-CNN with VGG19) for detection of different phenological growth stages in corn plants and weeds as a function of three different positioning angles for image capture (0°, 30°, and 75°), and observed better results at 75° angle for all algorithms. Higher values were observed for FASTER R-CNN with VGG19 delivering accuracy of 98.20% and sensitivity of 97.25%. The authors attributed this difference to the larger image size and view of the corn plants. The views of the corn plants were frontal and superior at 30º and 75º angle , while at 0º angle they were only superior as used in this study. Accuracy of 98.00% in case of Soybean plants shows that the algorithm sometimes misclassified weeds as crops.
Higher precision was observed in the detection of weeds in the cotton crop with 93% compared to 76% in the soybean crop (Table 1). Although the shape of the cotton leaves is more similar to that of the springtails, the algorithm confuses more in the detection because of similarity between leaflets of soybean and leaves of springtail, thus decreasing the precision of weed detection (Figure 7). This fact can perhaps be explained by the features extracted from the images of both plants at the time of training and learning and may present more similar points for the algorithm.  With respect to the sensitivity, for vine-string in both modules and classes, it was observed below 12.00%, as shown in Table 2. These results were below the values obtained in the laboratory test image sets, demonstrating the low capacity of the YOLOv3 tiny algorithm to recover all the objects contained in the image in real time. These sensitivity values may have been influenced by the environmental conditions at the time of the test and by the camera used, which was programmed for use with an automatic zoom and did not present an adequate focus on the images. The constant search for focal adjustment on the move may have contributed to the low detection of all objects in the image and incorrect classification of some plants (Figure 8). According to Zheng et. al. (2017) detection in natural RGB images is difficult because of the complex background, different illumination, weather, shadow regions, and color similarities. According to Hong et al. (2012), lighting conditions are one of the challenges in machine vision, as reflectance can increase on brightly lit days, reducing feature differentiation and making detection difficult because increased reflectance distorts image colors. The faster R-CNN showed excellent precision results for both cotton and soybean crops. For rapeseed, there was a variation of 4.00% between the averages of both modules, as shown in Table 3. The accuracies for cotton and cottonseed in the cotton module were above 90.00%; whereas they were below 88.00% for cottonseed in the soybean module, as shown in Figure 9. Similar results were observed by Olsen et al. (2019) during evaluation of the performance of two convolutional neural network architectures in classifying eight weed species, where they obtained accuracies ranging from 88.50% to 97.60% across weed species, using the ResNet-50 architecture. The authors attributed the lower accuracy in some weeds to the availability of fewer unique visible features to be trained on. They further pointed out that a high degree of confusion in the classification of weeds is related to similar image features.
Thus, as pointed out earlier, the position of the leaf or leaflet at the time of capturing image may be similar to other species to be detected, leading the machine to a misclassification. Another important fact to highlight is the accuracy of the crops of interest, as there was little error in classifying weeds as crops. This result is significant because the aim is to control weeds, and if they are detected and misclassified, they are not sprayed. However, the detection of crop plants of economic interest as weeds leads to a waste of herbicides and consequently a larger quantity of products in the environment.
In terms of sensitivity, faster R-CNN obtained results lower than 26.00% when performing real-time detection for both modules (Table 4). The low capacity to detect all objects in the image is insignificant for crops of economic interest, considering that the objective is to spray only the weeds. As for the weeds, the detection of all leaves is not necessary; however, the detection of at least one leaf from each plant is necessary to activate the nozzles and perform herbicide spraying. In comparison between the algorithms, faster R-CNN showed better results in terms of sensitivity for both modules and both classes; however, the accuracy was close to that of the YOLOv3 tiny, which was superior for the soybean module that detected a smaller number of soybean leaflets as weeds. The accuracy error in the classification of the cotton leaf algorithm may be related to the shape of the leaves and the angle at which the images were captured, as cotton leaves are also cordate and, depending on their stage and the angle in relation to the camera, have similar shape to that of the cotton leaf. (Table 5).  (2019) compared the classic FASTER-CNN, faster R-CNN with VGG19 architecture and the YOLOv2 version, and showed that FASTER R-CNN was superior to YOLOv2 in terms of accuracy, which was found 10.31% in detection of corn seedlings between 6-7 leaves. Although it is an earlier version of YOLO, it shows that faster R-CNN has superior accuracy results but slower processing.
Control through spraying is directly related to the accuracy and sensitivity of detection. In this context, the faster R-CNN showed variations in the control during the repetitions in each module. In the cotton module, higher percentages can be observed, varying from 67.00% to 87.00% between repetitions. As for the soy module, the lowest percentage occurred in Repetition 9, perhaps due to several passes of the prototype in the area where some of the rape leaves were crushed, making its detection and consequently its control difficult. However, the other repetitions showed percentages greater than or equal to 67.00% in the control of weeds (Tables 6 and 7). For the YOLOv3 tiny algorithm, as shown in Table  7, the control of the vine string was below 44% in both modules. This value is reflected by the low number of sensitivities presented by the detection algorithm, although it has good accuracy in detection.
The average number of weeds that are sprayed in the cotton module was 3 (three) plants out of 15, representing an average of 19.00% in the control. As for the soy module, the control was 20.00%, with an average of 2 sprayed plants out of 9 present in the module in 10 repetitions. For the Faster R-CNN algorithm, the best average control of cordgrass plants was obtained in both modules, controlling 12 out of 15 plants present in the cotton module and 7 sprayed plants out of 9 plants present in the soybean module. The average control was 81% for the cotton module and 78% for soybean (Table 8).

CONCLUSIONS
In field tests performed on the selective sprayer prototype, the faster R-CNN showed better results in the control of cotton and soybean weevils present in both the cotton and soybean modules, even with low sensitivity values. The on-board system developed for the spraying prototype was efficient for the individual activation of each nozzle, performing the application selectively according to the detected target. The FASTER R-CNN algorithm demonstrated its applicability as an Ipomoea spp. plant detection tool for variable-rate herbicide spraying.