Chlorophyll content for millet leaf using hyperspectral imaging and an attention-convolutional neural network. Chlorophyll content for millet leaf using hyperspectral imaging and an attention-convolutional neural network

: Chlorophyll is a major factor affecting photosynthesis; and consequently, crop growth and yield. In this study, we devised a chlorophyll-content detection model for millet leaves in different stages of growth based on hyperspectral data. The hyperspectral images of millet leaves were obtained under a wavelength range of 380–1000 nm using a hyperspectral imager. Threshold segmentation was performed with near-infrared (NIR) reflectance and normalized difference vegetation index (NDVI) to intelligently acquire the regions of interest (ROI). Furthermore, raw spectral data were preprocessed using multivariate scatter correction (MSC). A correlation coefficient-successive projections algorithm (CC-SPA) was used to extract the characteristic wavelengths, and the characteristic parameters were extracted based on the spectral and image information. A partial least squares regression (PLSR) prediction model was established based on the single characteristic parameter and multi-characteristic parameter fusion. The determination coefficient (R v 2 ) and the root-mean-square error (RMSEv) of the validation set for the multi-characteristic parameter fusion model were reported to be 0.813 and 1.766, respectively, which are higher than those obtained by the single characteristic parameter model. Based on the multi-characteristic parameter fusion, an attention-convolutional neural network (attention-CNN) (R v 2 = 0.839, RMSEv = 1.451, RPD = 2.355) was established, which is more effective than the PLSR (R v 2 = 0.813, RMSEv = 1.766, RPD = 2.167) and least squares support vector machine (LS-SVM) models (R v 2 = 0.806, RMSEv = 1.576, RPD = 2.061). These results indicated that the combination of hyperspectral imaging and attention-CNN is beneficial to the application of nutrient element monitoring of crops.


INTRODUCTION
Chlorophyll is a vital pigment for photosynthesis in plants. It is well correlated with nitrogen nutrition status at different growth stages of crops and can be used as an essential indicator to characterize crop growth and for nutrient monitoring (MAO et al., 2018). Currently, the chlorophyll content in plants is mainly measured by chemical methods (EVANS et al., 2012;LOH et al., 2012;SCOTTER, 2011) and the chlorophyll meter method. However, these methods are time-consuming and damage leaves. In addition, measurement of chlorophyll is not possible over a large area. In recent years, Xiaoyan et al. hyperspectral imaging technology has been widely used to estimate chlorophyll content of crops because of it is highly efficient, non-destructive, and nonpolluting (KORUS, 2013).
At present, chlorophyll content has been estimated in leaves of large grain crops, but studies on the estimation of chlorophyll in small grain crops such as millet are rare. Previous studies mostly used single type spectral variables to estimate crop nutrition. ZHENG et al. (2018) established a model to estimate chlorophyll content in potato leaves based on red edge position, and the determination coefficient of the model was evaluated to be 0.8682 (ZHENG et al., 2018). GITELSON & SOLOVCHENKO (2017) presented an algorithm to estimate the chlorophyll content in 524 leaves of 45 different species, and their results indicated that red edge chlorophyll index (CI RE ) and meris terrestrial chlorophyll index (MTCI) were very accurate in estimating chlorophyll content (R 2 = 0.95, RMSE = 4.5% and R 2 = 0.898, RMSE = 6.4%) (GITELSON & SOLOVCHENKO, 2017). In this study, we attempted to use multi-characteristic parameter fusion data to predict the chlorophyll content of millet leaves. Classical machine learning methods have been widely used in crop nutrition detection. YU et al. (2014) proposed the ratio of reflectance difference index (RRDI) for estimating chlorophyll content and compared the results with optimum multiple band regression (OMBR), partial least squares regression (PLSR), and support vector machines for regression (SVR). Results indicated that RRDI can efficiently eliminate the effects of various canopy structures on the estimation of leaf chlorophyll (R 2 = 0.76, RMSE = 1141.5) (YU et al., 2014). JIA et al. (2013) used hyperspectral remote sensing to develop normalized difference vegetation indices (NDVI), spectral ratio (SR) indices, stepwise multiple linear regression (SMLR) and back-propagation (BP) neural network for estimation of nicotine content in flue-cured tobacco leaves under different shade conditions. Results indicated that BP neural network had the best accuracy and reproducibility (R 2 = 0.968, RMSE = 0.109) (JIA et al., 2013). A convolutional neural network (CNN) can autonomously learn and deeply extract local features of data (SHARMA et al., 2018), and the attention mechanism can effectively highlight important features.
In this study, we adopted an attention-convolutional neural network (attention-CNN) model to monitor the chemical content during crop growth.
The millets studied in this research were obtained from the Agricultural Station of Shanxi Agricultural University. Leaves were collected at different growth stages, and the average spectral of the leaves was obtained by intelligently extracting the region of interest (ROI). The original spectral and image information were denoised by multiscatter correction (MSC). Furthermore, we used the correlation coefficient-successive projections algorithm (CC-SPA) for data reduction and to extract the characteristic parameters based on spectral and image information. The single characteristic and multi-characteristic parameter fusions were used to build the PLSR model and search for optimal characteristic parameters. Furthermore, an attention-CNN model was established using the multicharacteristic parameter fusion and was compared with conventional PLSR and least squares support vector machine (LS-SVM) algorithms to establish a method for chlorophyll estimation in millet leaves.

Experimental procedure
The study was conducted at the Agricultural Station of Shanxi Agricultural University in June-October, 2018. A quadratic orthogonal rotation design and three factors-nitrogen, phosphorus, and potassium-were adopted in the experiment. Each factor had five different levels, and 23 test treatment combinations were available, in which each test was conducted in triplicate. A total of 69 plots were established (4 m × 4 m), which were arranged in random groups of blocks and planted with a row spacing of 40 cm. Phosphorus and potassium fertilizers were utilized as the base fertilizers and applied once, and the nitrogen fertilizer was applied thrice: 40% as the base fertilizer, 30% at the jointing stage, and 30% at the booting stage. During the experiment, field management was unified to prevent lodging and bird damage.

Determination of leaf spectral data
In the jointing, booting, and filling stages of millet, three representative and healthy growing leaves were collected from each plot. The leaves were stored in ice packs, numbered, and immediately brought to the laboratory for hyperspectral information acquisition and chemical determination of chlorophyll content. We used a Starter Kit indoor mobile scanning platform (Headwall Photonics, USA) to obtain visible near-infrared (NIR) hyperspectral images. The spectral range was 380-1000 nm, and the distance from the lens to the surface of the leaf was 25 cm. Three leaves from each plot were laid on a black-light-absorbing cloth with an exposure time of 39.84 ms and a platform moving speed of 2.721 mm/s.
To reduce the interference of noise caused by uneven distribution of light source and dark current noise (AMBROSE et al., 2016), the system was calibrated before the experiment. The correction formula is as follows: (1) where I represents the corrected hyperspectral image, I 0 denotes the original hyperspectral image, W represents the white background image captured by the scanning standard white correction board, and B is the dark background image captured by covering the lens. The Spectral View software was used to extract the required data from the hyperspectral image files obtained after black and white correction.

Measurement of chlorophyll content
The chlorophyll content of each leaf was determined by spectrophotometry. The whole leaf was selected, and after removing the veins, the millet leaves were cut into silk; 0.1 g of each leaf was weighed; and subsequently, treated with 80% acetone for 24 h. The absorbance at 646 nm and 663 nm was measured using a spectrophotometer and the following formula (CHEN et al., 2018): (2) where A646 and A663 denote the absorbance at wavelengths of 646 nm and 663 nm, respectively, C a and C b represent the chlorophyll contents a and b, respectively, and C T represents the total chlorophyll content. Results of chlorophyll measurement are shown in figure 1, the observed range of chlorophyll content was 4.358-15.272 mg/L in the jointing stage, 19.970-34.598 mg/L in the booting stage, and 13.194-27.613 mg/L in the filling stage.

Selection of sample sets
Two thirds of the data of the three millet growth stages (jointing stage, booting stage, and filling stage) were randomly selected as the calibration set (46 samples per growth period and 138 samples for the whole growth period). The remaining one third (23 samples per growing season and 69 samples in the whole growth period) was used as the validation set. Samples from the three growing stages comprised a full-length sample. Statistical results of chlorophyll content in millet leaves are provided in table 1. The mean, standard deviation, and coefficient of variation of the chlorophyll content of the calibration set and the validation set are similar to those of the whole sample, i.e., the calibration set and the validation set effectively reflect the statistical characteristics of the samples.

Extraction and pretreatment of hyperspectral data
Because the conventional method of extracting ROI is not only time-and energyconsuming but also inefficient, a method of intelligently extracting the ROI was employed in this study. Figure 2 is a single-point spectral curve that reflects the differences in spectral curves in different regions of millet leaves. This clearly shows that the reflectance curves in the background region and the shadow region are relatively stable, and the reflectance is low in the NIR (780-900 nm) band; furthermore, the reflectance of the stem area is significantly higher than that of the object area in the red light (630-680 nm) band, and the hole region is characterized by low reflectance in the NIR (780-900 nm) band. Threshold segmentation was then performed using the NIR band reflectance (cull shadows, holes, and background areas) and the NDVI (cull stem area) based on this spectral change trend to intelligently select the ROI. Equation 3 shows the calculation formula of NDVI, where the range of NIR was 0.350-0.525 and the range of NDVI was 0.760-0.783 in the jointing stage, 0.790-0.814 in the booting stage, and 0.728-0.755 in the filling stage. Finally, the ROI regions without background, shadow, holes, and part of the stem can be obtained.
(3) To reduce noise interference, spectral data of a large number of points were extracted from the ROI, and then, the average spectral curve of samples was obtained via the arithmetic average, as shown in figure 3. Because the spectral reflectance was disturbed by equipment noise at both sides of the range (380 nm and 1000 nm), the bands at both sides were removed; thus, the spectral range 412.368-972.004 nm was used for the following analysis. We employed MSC  to denoise the original spectral data. First, the average spectral of the sample set was calculated as the standard spectral based on equation 4. Then, a unitary linear regression calculation was performed for the spectral and the average spectral information of each sample using equation 5. The linear translation (b i ) and tilt offset (m i ) of each spectrum relative to the average spectrum were obtained; finally, the spectral curve of the sample after MSC correction was calculated using equation 6. (4) (6) where A denotes an n × p dimensional spectral data matrix, n represents the sampling number, p denotes the spectral dimension, A denotes the average spectral vector of all samples, i A represents the average spectrum of the i th sample, and m i and b i represent the slope and intercept of thelinear regression relationship between i A and A , respectively.

Selection of feature parameters
In this study, four different methods were used to extract the feature parameters as follows: (1) Twenty-six vegetation indexes from the original spectral curve that are sensitive to green plant chlorophyll content were extracted, including NDVI, ratio vegetation index (RVI), difference vegetation index (DVI), CI RE , and leaf chlorophyll index (LCI) (GITELSON & MERZLYAK, 1994).
(2) Twenty-one trilateral parameters of the leaf from the first-order differential spectrum were extracted, including the trilateral position, amplitude, area, kurtosis, and skewness. Then, twenty-five peakvalley parameters were constructed by utilizing the rising or falling rates of the characteristic sideband regions in the reflected spectral curve and the angle between the two sides of the peak-to-valley curve, which included the green edge (k g ), red edge (k r ), NIR (k nir ) rate of change, green peak to red valley bottom rate (k gprv ), angle between the reflected green peaks (GPA1), absorption angle between red valleys (RVA), red edge and NIR angle (GPA2), and their combination (XU et al., 2011).
(3) The ENVI software was used to segment the image of the millet leaf to remove the background, excluding only the main part of the blade (without veins). The mean and variance of the red, green, and blue components of the blade image were calculated using the RGB color system, and 27 color features were obtained using the combination of R, G, and B (CHENG et al., 2017). (4) Seven common texture features were extracted using the gray level co-occurrence matrix (GLCM), including the energy, entropy, moment of inertia, correlation, mean, standard deviation, and smoothness.

Method of analysis Selection of characteristic wavelength
Because the dimensions of the hyperspectral data were large, the sensitive bands that play a key role in the model had to be screened to reduce the interference from insignificant wavelengths on the accuracy of the model. The correlation between the spectral data vector and the chlorophyll content was calculated using the correlation coefficient method (CC), and the sensitive band with greater correlation was selected. We employed the successive projections algorithm (SPA) to extract the characteristic wavelengths in the sensitive band because this model can improve the prediction accuracy and stability based on the reduced model (HONG et al., 2010).

Chemometrics methods
In this study, a millet-leaf chlorophyll prediction model based on the characteristic wavelength and multi-characteristic parameter fusion was established using the attention-CNN model; in addition, the PLSR and LS-SVM models were used for comparative analysis to determine the better model. CNN (SAIKIA et al., 2019) is a conventional model that is widely used in deep learning, which extracts the effective features of the data of each layer via multi-layer convolution and pooling operation. Moreover, the convergence speed of this model is high, and it provides good prediction results; however, this model is weak in highlighting the important convolution features, which is not sufficiently intuitive, and is poorly interpretable. Attention mechanism is a commonly used model, which has a long-term memory mechanism, in the field of natural language processing; this mechanism directly showed the contribution of each feature to the results. Thus, we used the attention mechanism to establish an attention-CNN model, as shown in figure 4.
The attention-CNN model mainly comprises an input layer, convolution layer, attention layer, pooling layer, fully connected layer, and output layer.
The input layer is an p n × grain leaf feature fusion data matrix X, where n denotes the number of samples and p denotes the data dimension.
The convolution layer acts as the sample feature extraction layer. The input sample matrix X is convoluted by a convolution kernel of size d m × to obtain the characteristic value i g , where i g is given by, where f denotes the activation function, b denotes the bias item, and , and is given as follows: (8) Then, the attention-weighted feature K is input to the pooling layer. After performing the maximum pooling operation, the computational complexity was simplified to retain the essential features, and the over-fitting phenomenon is alleviated.
Finally, the feature map output from the pooling layer was rasterized and fully connected with the multi-layer perceptron (MLP), and the back-propagation algorithm was used to update the model parameters.

Analysis of the effect of MSC on the average spectral curve
As shown in figure 3, characteristics of the original spectral curve were consistent with those of typical green plants. Furthermore, absorption valleys were found in the range of 390-450 nm of violet light, reflection peaks in the green light range of 500-580 nm, absorption valleys in the red-light range of 630-770 nm, and a sharp rise in reflectivity in the range of 680-750 nm. In addition, the curve rises and forms a remarkable high reflection platform in the NIR band of 750-900 nm.
As shown in figure 3, the millet spectral curves greatly overlap in the visible light band, and the spectral reflectance varies greatly in the NIR band. To address these problems, the original spectral curve was pretreated with MSC. As shown in figure 5 , the shift in the spectral curve baseline was effectively eliminated. Then, the correlation between the spectral information of the millet leaves and the chlorophyll content before and after MSC pretreatment was analyzed; the correlation coefficient curve is shown in figure 6.  in value. Thus, the correlation between spectroscopic results and chlorophyll content was clearly improved after MSC pretreatment.

Selection and modeling analysis of the sensitive band
The CC method was used to analyze the spectral data of millet leaves for the whole growth period, and the wavelengths 647-690 nm and 700-743 nm were selected as the sensitive bands. When the characteristic wavelength was extracted using SPA in the wavelength ranges of 647-690 nm and 700-743 nm, three (686 nm, 652 nm, and 690 nm) and four characteristic wavelengths (737 nm, 701 nm, 725 nm, and 707 nm) were selected when the root-mean-square errors were 2.104 and 1.977, respectively. These extracted characteristic wavelengths belong to the red-light range, which is consistent with the spectral sensitivity band of green-plant-leaf chlorophyll content. An analytical model to establish the chlorophyll content of millet leaves based on PLSR is shown in table 2, where R c 2 and R v 2 denoted the determination coefficients of the calibration set and the validation set, respectively, and RMSEc and RMSEv represented the root-mean-square errors of the calibration and validation sets, respectively.
Table 2 clearly indicates that chlorophyll estimation based on the full-band model was better than the characteristic wavelength model; furthermore, R c 2 was 0.748, which was 1.7 % higher than that of the characteristic wavelength model. The R v 2 of the characteristic wavelength model was 0.725, which is 0.9 % higher than that of the fullband model. These results indicated that the CC-SPA method can accurately extract the sensitive band of chlorophyll, effectively reduce the number of wavelength variables, and accurately predicted chlorophyll content in millet leaves based on the simplified model.

Analysis of the influence of different characteristic parameters on PLSR modeling
A correlation analysis was then performed between the extracted characteristic parameters and the chlorophyll content. The selected characteristic parameters are provided in table 3, where the correlation coefficient of NDVI is the largest at 0.831. The accuracy of chlorophyll estimation by NDVI is illustrated in figure 7, where it is seen that the R 2 and RMSE values for 207 leaves were 0.690 and 4.188, respectively. The characteristic parameters were fused to establish a prediction model to improve the prediction accuracy, whose results are provided in table 4.
As seen in the table, the prediction results were obtained for five different characteristic parameters (R v 2 = 0.421-0.767, RMSEv = 1.726-2.858), among which the trilateral parameter model exhibited better prediction. More importantly, the best prediction of chlorophyll content was achieved by the multi-characteristic parameter fusion model (R v 2 = 0813, RMSEv = 1.766). Previous studies have mostly chosen a single type of vegetation index, trilateral parameters, and peak-valley parameters for crop nutrition detection (JAY et al., 2017;XU et al., 2011), ignoring the characteristic information of the image itself. However, in this study, a "spectral-image" fusion index was constructed, and the chlorophyll content of millet leaves was estimated based on multi-characteristic parameter fusion data, which provided a new method for quantitative estimation of chlorophyll.

Attention-CNN model with multi-characteristic parameters
The attention-CNN chlorophyll inversion model was established by multi-characteristic parameter fusion. The specific parameters of the model were established with two convolution layers. The first layer contained five convolution kernels of size 5 × 5, and the second layer contained 10 convolution kernels of the same size. Different convolution kernels were used to convolute all the feature maps in the former layer, following which the corresponding elements were accumulated and biased, and each output feature map was activated using the ReLu function. The size of the pooling layer was 2 × 2, and the feature vector dimension of the attention layer was 100. The loss function used the Mahalanobis distance, which is given by (9) where p y is the predicted value and t y is the measured value of chlorophyll content. Furthermore, the mini-batch gradient descent method was used in the model. The network learning rate was 0.5, batch was 23, and the maximum number of iterations was set to 1300. The loss function curve and the accuracy value were used as the basis of convergence.
The common PLSR and LS-SVM models were selected for comparative analysis. The LS-SVM model used a radial basis function, and the optimal parameters were determined by cross-validation. The gam and sig2 of the calibration were set at 2.355 and 111.100, respectively, in this study. The chlorophyll prediction results of millet obtained using different models are listed in table 5. Table 5 shows that the R c 2 and RMSEc values for the calibration set using the PLSR model were 0.848 and 1.447, respectively, and those using the LS-SVM model were 0.837 and 1.445, respectively. The results of the calibration set indicated that the modeling accuracy and the stability were effectively improved by the attention-CNN method (R c 2 = 0.865, RMSEc = 1.290). In the verification set, compared = 0.839, RMSEv = 1.451, RPD = 2.355), and the difference between the modeling accuracy and the actual prediction ability was the lowest at 0.026.
Traditional machine learning algorithms have been mainly used for data analysis. For example, FÉRET et al. (2011) used spectral indices and PLSR to develop statistical relationships between reflectance spectral and leaf chemical properties (FÉRET et al., 2011). This study represented the first application of attention-CNN to measure millet chlorophyll content, which has a higher fitting accuracy and generalization ability than the conventional models. As discussed by KARTHIK et al. (2019), CNN can automatically extract the infection features in tomato leaves, and the attention mechanism can learn significant features for disease detection (KARTHIK et al., 2019).
Results demonstrated that the attention-CNN model can deeply mine the internal features of spectral data and simultaneously simplify preprocessing, which was effective in predicting the chlorophyll content of millet leaves; and was therefore, a high-performance chlorophyll-content prediction model.

CONCLUSION
In this study, hyperspectral imaging technology was used to obtain the spectral and image information of millet leaves at different growth stages, and the average spectra of the millet leaves were extracted by intelligently extracting the ROI. The original spectral data were preprocessed by MSC and were subsequently analyzed to study the effect and correlation of pretreatment. We used the CC-SPA model for data reduction and extracted the characteristic parameters based on the spectral and image information. Furthermore, single characteristic and multi-characteristic parameter fusion were used to build the PLSR model. The multi-characteristic parameter fusion achieved accurate prediction results (R v 2 = 0.813, RMSEv = 1.766) and exhibited better prediction accuracy than the single characteristic parameter model. Based on the multi-characteristic parameter fusion model, the attention-CNN model yielded more accurate results (R v 2 , RMSEv, and RPD were 0.839, 1.451, and 2.355, respectively) than the PLSR model (R v 2 , RMSEv, and RPD were 0.813, 1.766, and 2.167, respectively) and the LS-SVM model (R v 2 , RMSEv, and RPD were 0.806, 1.576, and 2.061, respectively). In addition, the difference between the modeling accuracy and the actual prediction ability of the attention-CNN model was the smallest (0.026). These results demonstrated that the attention-CNN model has a higher prediction accuracy and regression fit than the conventional models and exhibits better adaptability to the sample data. Therefore, the attention-CNN model is a highly advantageous novel method for non-invasive measurement of chlorophyll content in millet plants.

DECLARATION OF CONFLICT OF INTERESTS
The authors declare no conflict of interest. The founding sponsors had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; and in the decision to publish the results.