Extraction of navigation line based on improved grayscale factor in corn field

ABSTRACT: Focusing on the problem that corn plant in different growth periods is grayed out by known methods, the gray scale difference of different part is large or the soil discrimination degree is not high, the navigation path is low in accuracy and speed. This paper proposed a new method for extracting cornfield navigation baselines, which is used to control walking of agricultural robots. Design method included image segmentation, navigation point extraction, and navigation path fitting. Image segmentation is based on a new grayscale factor combined with median filtering, OSTU method and morphological operations to achieve the separation of crops and soil. The extraction of the navigation point is based on the binary image vertical projection map to obtain the region of interest, and the navigation point coordinates are determined by calculating the relative center point of the white pixel points of the sampling line in the region of interest. The Hough transform is used to fit the navigation point obtained by the vertical projection map to determine the navigation path, and then the control parameters are obtained. The gray scale factor that is improved in this paper combined with the vertical projection map can extract the target ridge with an accuracy rate of 92%, and the accuracy of extracting the navigation line is more than 90%. When conducting navigation tracking experiments in corn field, the maximum error is 5cm.


INTRODUCTION
In recent years, with the development of a series of high-tech technologies such as modern information technology, biotechnology, and engineering technology, precision agriculture has rapidly developed and popularized. Automatic navigation of agricultural machinery is the core technology of precision agriculture (HU et al., 2015), which has been widely used in agricultural production activities such as farm furrows, pesticide spraying, and harvesting (ÅSTRAND et al., 2002;JI et al., 2011;WU et al., 2019). Machine vision, as the main research content of automatic navigation of agricultural robots, is widely used in the detection of crop lines in various crops. For example, wheat (ZHAO et al., 2016), soybeans (KISE et al., 2005), cucumbers (REN et al., 2009), potatoes (GARCíA-SANTILLáN et al., 2018), corn (SURIYAKOON et al., 2017;ZHANG et al., 2018;SONG et al., 2017;DIAO et al., 2015;MONTALVO et al., 2012;JIANG et al., 2015;GUIJARRO et al., 2011), cotton (ZHAI Chen et al. et al., 2016) and rice (CHOI et al., 2015) etc other crop line detection and recognition.
At present, the control of corn diseases and insect pests is mainly based on manual spraying of pesticides, which has some problems such as low efficiency, uneven spraying and harm to human body. Therefore, the application of robotics and machine vision technology to corn pesticide spraying is currently the developmental trend. However, the machine visual navigation method studied by many scholars is only suitable for the detection of crop rows during specific growth periods of corn, and can not meet the needs of visual navigation at all stages of the corn growth cycle. Especially after the corn flowering, because the crop line is difficult to extract, it iwas difficult for agricultural robot to realize the automatic navigation. How to accurately extract the navigation lines between corn rows during each growth cycle is the key to realize the automatic navigation of agricultural robots, while the precise extraction of navigation lines depends on the successful segmentation of corn and soil. Many scholars had conducted extensive research on the segmentation of corn crop images. SURIYAKOON et al. (2017) used excess red grayscale factor to separate corn from soil. SONG et al. (2017), DIAO et al. (2015) used differently improved 2G-B-R grayscale factor to separate corn crops from soil. MONTALVO et al. (2012) separated corn plants from soil using a traditional grayscale factor 2G-B-R and double Otsu. JIANG et al. (2015), GUIJARRO et al. (2011) used genetic algorithm to improve the grayscale factor, and then used it to separate corn from soil.
The above methods can successfully separate the corn crop from soil in a specific scene, and extract the crop rows. However, during the different growth cycles of corn, the large difference of grayscale in different parts of corn, or the little gray difference of corn plants and soil, both which could affect the segmentation of corn plants and soil. However, the grayscale factors proposed in the above literature do not considered these situations, so the segmentation effect is not ideal. In order to achieve the segmentation of corn plants and soil in various periods, a new gray-scale factor is proposed in this paper, and on the basis of the new grayscale factor, the overall algorithm of corn field navigation line extraction is proposed. Firstly, in order to achieve the segmentation of corn plants and soil in different growth periods, this paper used the improved grayscale factor, combined with morphological processing and binary algorithm to separate soil and corn plants. Secondly, the region of interest (ROI) is intercepted by the binary image vertical projection map. Third, the coordinates of the discrete points of the navigation are determined by calculating the relative coordinates of the white points of the sampled pixel. Finally, The Hough transform was used to fit the navigation points to obtain the navigation line, which is used to control the automatic walking of the corn spray robot, and achieved good results.

Corn field image pretreatments Intercepting the region area of interest
If the whole image captured by the camera is taken as the research object, it will inevitably lead to a large amount of calculation and interference, which will eventually affect the real-time and accuracy of autonomous navigation. In the actual intelligent navigation process, the region of interest only accounts for a part of the image. Because in the course of the actual investigation, it is reported that not all corn crop rows are straight lines, artificial corn crop rows have a certain degree of arc, especially in some mountain corn fields.in order to reduce the amount of image calculation, and improve navigation accuracy. In this paper, the idea of "turning the curve into straight" was taken, the image that is captured by camera is intercept out a part as the research object, after many interceptions, the decomposition of a curved crop row into multiple straight crop lines. After many field shooting results, it was reported that the last 190 pixel rows of the captured image are intercepted as the image region of interest can not only get good experimental results, but also greatly reduce the processing time of the image.

Propose a new grayscale factor
The images captured by the camera were stored in RGB format. In the RGB color space, the G component of crops and weeds has a large proportion, while the R and B component of soil background have a large proportion. Therefore, the appropriate grayscale factor is very important for the segmentation of crops and soil. In this paper, a new grayscale factor is proposed based on the commonly used graying factor.
At present, most of the graying of farmland plants adopts the 2G-R-B method, which can increase the weight of green plants, highlight the green characteristics of plant targets, enhance the contrast between target and background, and have a good effect on the segmentation of green plant and soil. Using this grayscale factor to process the image; although, the green plant and the soil can be well separated, the grayscale discrimination of different part of the same corn plant is high, which cause many holes in the binarized image, therefore this method affecting the extraction accuracy of navigation discrete points. The algorithm is such as formula (1). (1) In the formula: gray (x,y) Grayscale factor; G(x, y) --Color space green component; Another common method is G-B. In the farmland image, the G value in the R, G, and B components of the green plant is significantly different from the G value of the soil background, so a pair of grayscale images can be obtained by using the two components G and B. However, the disadvantage of using this grayscale factor is that the grayscale image is relatively fuzzy, and the feature point discrimination is insufficient. The algorithm is such as formula (2). (2) In the formula: gray (x, y) --Grayscale factor; G(x, y) --Color space green component; B(x, y) --Color space blue component; Based on the advantages of the above two grayscale factors, a new gray-scale factor is proposed in this paper. Because the R component of the soil is large, this paper through increasing the R component to limit the G component to distinguish between soil and green plants, and can well extract the ridges from the plant background. At the same time, G(x, y) -B(x, y) method is used to determine the gray value which could reduce the gray level difference of different parts of the same plant. The algorithm is such as formula (3). (3) In the formula: gray (x, y) --Grayscale factor; G(x, y) --Color space green component; R(x,y) --Color space red component; B(x, y) --Color space blue component;

Noise removal and image binarization
The main content of this paper is the intelligent navigation of corn spray robot. When the robot works in the field, there is a situation in which the camera lens is contaminated by dew, pesticides, etc., thereby affecting the sharpness of the captured images. In this paper, median filtering is used to deal with noise pollution, and the Otsu method is used to binarize the image. Due to uneven coverage of green leaves and interference of plastic mulch and other debris, there will be some white spots and holes after the binarization of grayscale image. These white spots and holes can affect the accuracy of navigation discrete points, and even causes the occurrence of extreme deviation points, which could affect navigation line accuracy. Therefore, it is necessary to perform morphological processing on the binarized image, which can remove incoherent structure while ensuring the basic shape feature of the image. This paper uses morphological closure operations to process binary images, smooth object contours, break narrow gaps, and eliminate smaller holes. and eliminate smaller holes the process is as in formula (4). (4) 1. Firstly, the structure element g is used to expand the image A, as in formula (5); (5) 2. The structured element g is then used to erode the expanded image, as in formula (6); Get the navigation discrete points In this paper, an algorithm based on vertical projection map is proposed to obtain the discrete points. Firstly, the image region of interest (ROI) is further extracted by the column coordinates of the lowest trough on both sides of the intermediate peak in the vertical projection of the binary image, thereby reducing the influence of surrounding ridges and other factors on the target ridge, and speeding up processing speed of the image. Then, taking the intermediate pixel column of the intercepted region of interest as a reference, the pixel value of the left side of the referencewas negative, and the value of the pixel column of the right side is positive, and then calculating the algebraic sum of the coordinates of the white phase points on the left and right sides of the reference. Finally, the result iwas divided by the number of pixels of the sample row of white pixels to determine the coordinates of the navigation discrete points. Pixel vertical projection is a simple statistical method that can achieve the target ridge extraction. (7) is vertical projection method formula. (8) is navigation discrete point solution formula.
Chen et al.

(7)
In the formula: f(x, y) --The value of the binary image point (x, y) (0 or 1); K --Image height; H --Image width; In the formula: L --The column in which the baseline is located; N --The number of columns of the binary image; K --Represents the total number of white pixels in a sampling line; In order to speed up the fitting of navigation lines, this paper takes one sample line every 10-pixel rows to calculate a navigation discrete point.

Fit the navigation line
Considering that the Hough transform has strong anti-interference ability, which can reduce the influence of error points and improve the fitting accuracy. Therefore, this paper used Hough transform to fit the navigation discrete points obtained by the vertical projection map to determine the navigation line. The Hough transform converts the point (x, y) in the plane Cartesian coordinate system into a corresponding sinusoid . The specific algorithm is as follows: Initialize an accumulator that corresponds to the square associated with the parameter space coordinates, initially set to 0. 2) For each navigation discrete point x k , y k obtained, be equal to the allowed subdivision value on the axis, and then solve the corresponding value by formula . The value of is rounded to the nearest integer value of the axis, and the corresponding accumulator is incremented by 1.
3) Find the maximum value Q in the accumulator, which means that there are Q points on the line that in the xy plane, thereby find the navigation path.

Control system and extraction of navigation parameters
The corn spray robot control system scheme is shown in figure 1. The whole system is composed of human-computer interaction layer and motion control layer. The human-computer interaction layer is composed of a laptop computer, the motion control layer is composed of a controller and a driver. The human-computer interaction layer is mainly responsible for sending commands and other operations. During the working process, the computer transports the images captured by the camera through USB, performs image processing, plans the navigation path of the robot, and sends instructions to the motion controller. The motion control layer acquires a motion command issued by the host computer to drive the DC motor to rotate. When the robot walks in the field, the control command is mainly obtained from the navigation path: Angle deviation and displacement deviation d. The angle deviation refers to the angle between the navigation path and the axis direction of the robot. The displacement deviation refers to the distance d between the navigation path and the camera, as shown in figure 2.

Experiment and analysis
In order to verify the advantages of the grayscale factor proposed in this paper, and the realtime of algorithm for seeking and fitting navigation discrete points. Making experiments in corn fields with different growth cycle and growth environment. Finally, the feasibility of the method is verified by the navigation and tracking experiment.

Experimental platform
The construction of the experimental platform is shown in figure 3. It mainly includes the main control computer, the tracked vehicle part, the camera and its support, the STM 32F407ZGT6 control chip, the DC motor JCF76R-24120-210R and its driver. Experimental data is mainly processed on platforms Python2 and Opencv2. The crawler chassis was produced by Shenzhen Shen lian Technology Co., Ltd. The camera used the industrial camera MER-132-43U3C-L from Daheng Image Co., Ltd, which transmits data by USB3.0. The camera is mounted on the bracket, the angle and height of the camera can be adjusted by the bracket, and the camera is mounted on the crawler vehicle chassis through the bracket.

Farmland experiment
One hundred pictures are collected in four different environments and processed on the master computer. In order to judge the accuracy of the navigation line that is extracted through the method proposed in this paper, the navigation datum is manually fitted as the measurement datum to judge the accuracy of the navigation line. Taking the starting point of the experimental vehicle as the coordinate origin, the artificial coordinate system is established, the coordinates of maize rhizomes between rows are measured, and the gradient is calculated. Finally, the datum equation is solved; therefore, the artificially fitted navigation datum line was obtained. The angle between manually navigation datum line and the navigation line obtained by the algorithm proposed in this paper is the navigation angle, which is used to measure the accuracy of the navigation datum line obtained by the algorithm proposed in this paper. The processes and results are shown in figure 4 to figure 7. The effect is shown in table 1.  It can be seen from the comparison of figures 4c, 4d, 4e, 4h, 4i, 4j: When the image is processed by the grayscale factor 2G-R-B, the large difference of grayscale in different parts of corn, which makes some holes when the image is binarized, and morphological processing could not fill all the holes, which seriously affects the accuracy of the navigation discrete points. When the image is  processed by the grayscale factor G-B, the grayscale image is relatively fuzzy overall, and some plant leaves of the target ridge are lost after binarization, at the same time, there are more holes, which will affect the accuracy of the navigation points and navigation path. When the image is processed by the grayscale factor proposed in this paper, the difference in plant and soil in the grayscale image can be increased  while the difference in grayscale in different parts of the same plant can be reduced. The image after the grayscale processing is relatively clear, and the feature points of the target ridge are not lost in the binarized image. Moreover, the number of holes is less and the area is small, and the small holes can be processed by the morphological closing formula (4). The result is shown in figure 4k. Details of the plant leaves in the target ridge are completely preserved, and the small holes on both sides of the ridge are well filled, which can attenuate the interference of the target ridge lacking seedlings and uneven growth to a certain extent. As shown in figure 4l, the target ridge is highlighted in the vertical projection map. By comparing figure 4m and figure 4l, it can be seen that after processing the image by using the grayscale factor proposed in this paper, the peaks and troughs in the vertical projection map of the binary image are less, which reduces the extraction target ridge cycle algorithm operation time. As shown in figure  4n, by using the vertical projection map to intercept the binary image, the region of interest (target ridge) in the image can be extracted well, which not only reduces other ridge disturbances, but also reduces the amount of image calculation. It can be seen from the result of the navigation discrete points shown in figure  4n, that the navigation discrete points calculated by the method (8)  , it can be seen that for corn fields with different growth periods and environments, the grayscale factor proposed in this paper well separates plants and soil. After the morphological operation, the holes of the target ridge can be basically filled, which makes the vertical projection curve of the binary image smoother and reduced the number of peaks and troughs in the vertical projection. From figure 8, it can be seen that the navigation line obtained through method that proposed in this paper is well distributed in the target row. From table 1, we can see that the accuracy of navigation line is more than 90%, and the processing time of each frame is less than 110 ms, which can be applied to the extraction of navigation lines in different growth periods of corn field.  MONTALVO et al., (2012), SONG et al., (2017, DIAO et al., (2015), Suriyakoon andRuangpayoongsak., (2017), JIANG et al., (2015), GUIJARRO et al. (2011) proposed different grayscale factor algorithms to extract crop line. In order to verify the processing effect of the grayscale factor proposed in this paper, the farmland image taken by the test vehicle is taken as the research object. The judgment basis is whether the target ridge can be extracted. The vehicle runs at a speed of 0.3m/s. One frame of image is extracted every 1s. A total of 100 frames of image are collected as the research object. The method comparison results are shown in table 2. From table 2, we can see that the grayscale algorithm proposed in this paper can effectively separate the target ridge from the plant.

Navigation tracking
The navigation tracking experiment was carried out in the corn field shown in figure 5. The corn is planted in wide and narrow rows, wide row spacing of 65 cm and narrow row spacing of 30 cm. The crawler-mounted car is 45cm wide and the robot can work in a wide-row planting environment. According to the image taken by the camera during walking, the main control computer processes the acquired image in real time to obtain a navigation path, and controls the steering with the displacement deviation and the angular deviation shown in figure 3. When the experimental platform drives at 0.3m/s, the error map is shown in figure 9. The maximum error is less than 5cm, and the crop line will not be encountered. Therefore, the method proposed in this paper can meet the work requirements.

CONCLUSION
In this paper, corn fields with different growth periods and growth environments are taken as research objects. In the image preprocessing stage, only 190-pixel rows are taken as regions of interest,  which reduces the amount of image processing and speeds up the processing. For the image graying process, a new graying factor is proposed in this paper. The image processed by the new gray factor not only retains the feature points of the target ridge, but also weakens the influence of plant coverage unevenness to some extent, so that the binary image has fewer holes, which makes it successful to intercept the region of interest through the vertical projection. The navigation points are obtained based on the vertical projection map of the binary image, and the navigation path is well fitted through the Hough transform. It also takes good results in the process of tracking the navigation path of corn field.
In this paper, a new grayscale factor is proposed, which can be used to well separate corn plants from soil and meet the automatic navigation requirements of corn spray robots in the different growth cycles and environments. 1) This paper proposds that the vertical projection map and Hough transform are used to extract the navigation discrete points and navigation paths, which improved the shortcomings of the traditional Hough transform fitting straight line, to a certain extent, and can satisfies the real-time requirements of the robot. 2) When the robot walks at a speed of 0.3 ms -1 , the maximum error is less than 5 cm, which can satisfy the application of the robot in agriculture.