Chlorophyll content for millet leaf using hyperspectral imaging and an attention-convolutional neural network

Estimativa do teor de clorofila para folhas de milheto usando imagens hiperespectrais e uma rede neural atenção-convolucional

ABSTRACT:

Chlorophyll is a major factor affecting photosynthesis; and consequently, crop growth and yield. In this study, we devised a chlorophyll-content detection model for millet leaves in different stages of growth based on hyperspectral data. The hyperspectral images of millet leaves were obtained under a wavelength range of 380-1000 nm using a hyperspectral imager. Threshold segmentation was performed with near-infrared (NIR) reflectance and normalized difference vegetation index (NDVI) to intelligently acquire the regions of interest (ROI). Furthermore, raw spectral data were preprocessed using multivariate scatter correction (MSC). A correlation coefficient-successive projections algorithm (CC-SPA) was used to extract the characteristic wavelengths, and the characteristic parameters were extracted based on the spectral and image information. A partial least squares regression (PLSR) prediction model was established based on the single characteristic parameter and multi-characteristic parameter fusion. The determination coefficient (R_v ²) and the root-mean-square error (RMSEv) of the validation set for the multi-characteristic parameter fusion model were reported to be 0.813 and 1.766, respectively, which are higher than those obtained by the single characteristic parameter model. Based on the multi-characteristic parameter fusion, an attention-convolutional neural network (attention-CNN) (R_v ² = 0.839, RMSEv = 1.451, RPD = 2.355) was established, which is more effective than the PLSR (R_v ² = 0.813, RMSEv = 1.766, RPD = 2.167) and least squares support vector machine (LS-SVM) models (R_v ² = 0.806, RMSEv = 1.576, RPD = 2.061). These results indicated that the combination of hyperspectral imaging and attention-CNN is beneficial to the application of nutrient element monitoring of crops.

Key words:
chlorophyll content; multi-characteristic parameters fusion; attention-CNN; hyperspectral imaging technology.

RESUMO:

A clorofila é um fator importante que afeta a fotossíntese e, consequentemente, o crescimento e o rendimento das culturas. Neste estudo, um modelo de detecção de conteúdo de clorofila é construído para folhas de milheto em diferentes estágios de crescimento, com base em dados hiperespectrais. As imagens hiperespectrais dos diferentes estágios de crescimento das folhas de milheto foram obtidas para 380-1000 nm, utilizando um gerador de imagens hiperespectrais. Uma segmentação de limiar foi realizada com refletância no infravermelho próximo (NIR) e índice de vegetação com diferença normalizada (NDVI) para adquirir de forma inteligente as regiões de interesse (ROI). Além disso, os dados espectrais brutos foram pré-processados usando o método de correção de dispersão multivariada (MSC). Um algoritmo de projeção de coeficiente de correlação sucessivo (CC-SPA) foi utilizado para extrair os comprimentos de onda característicos, e os parâmetros característicos foram extraídos com base nas informações espectrais e de imagem. O modelo de previsão de regressão parcial dos mínimos quadrados (PLSR) foi estabelecido com base nos parâmetros de característica única e na fusão de parâmetros de característica múltipla. O coeficiente de determinação (Rv2) e o erro quadrático médio da raiz (RMSEv) do conjunto de validação para o modelo de fusão de parâmetros com várias características foram obtidos como 0,813 e 1,766, sendo melhores do que os do modelo de parâmetro de característica única. Com base na fusão de parâmetros com várias características, foi estabelecida uma rede neural atenção-convolucional (atenção-CNN) (Rv2 = 0,839, RMSEv = 1,451, RPD = 2,355) mais eficaz que o PLSR (Rv2 = 0,813, RMSEv = 1,766, RPD = 2,167) e mínimos quadrados que suportam modelos de máquina de vetores (LS-SVM) (Rv2 = 0,806, RMSEv = 1,576, RPD = 2,061). Estes resultados indicam que o modelo atenção-CNN atinge uma previsão efetiva do teor de clorofila nas folhas de milheto usando os dados hiperespectrais. Além disso, esta pesquisa demonstra que a combinação de imagens hiperespectrais e a atenção-CNN se mostra benéfica para a aplicação do monitoramento dos elementos nutricionais das culturas.

Palavras-chave:
teor de clorofila; fusão de parâmetros com várias características; CNN da atenção; tecnologia de imagem hiperespectral

INTRODUCTION:

Chlorophyll is a vital pigment for photosynthesis in plants. It is well correlated with nitrogen nutrition status at different growth stages of crops and can be used as an essential indicator to characterize crop growth and for nutrient monitoring (MAO et al., 2018). Currently, the chlorophyll content in plants is mainly measured by chemical methods (EVANS et al., 2012; LOH et al., 2012; SCOTTER, 2011) and the chlorophyll meter method. However, these methods are time-consuming and damage leaves. In addition, measurement of chlorophy-ll is not possible over a large area. In recent years, hyperspectral imaging technology has been widely used to estimate chlorophyll content of crops because of it is highly efficient, non-destructive, and non-polluting (KORUS, 2013).

At present, chlorophyll content has been estimated in leaves of large grain crops, but studies on the estimation of chlorophyll in small grain crops such as millet are rare. Previous studies mostly used single type spectral variables to estimate crop nutrition. ZHENG et al. (2018) established a model to estimate chlorophyll content in potato leaves based on red edge position, and the determination coefficient of the model was evaluated to be 0.8682 (ZHENG et al., 2018). GITELSON & SOLOVCHENKO (2017) presented an algorithm to estimate the chlorophyll content in 524 leaves of 45 different species, and their results indicated that red edge chlorophyll index (CI_RE) and meris terrestrial chlorophyll index (MTCI) were very accurate in estimating chlorophyll content (R² = 0.95, RMSE = 4.5% and R² = 0.898, RMSE = 6.4%) (GITELSON & SOLOVCHENKO, 2017). In this study, we attempted to use multi-characteristic parameter fusion data to predict the chlorophyll content of millet leaves. Classical machine learning methods have been widely used in crop nutrition detection. YU et al. (2014) proposed the ratio of reflectance difference index (RRDI) for estimating chlorophyll content and compared the results with optimum multiple band regression (OMBR), partial least squares regression (PLSR), and support vector machines for regression (SVR). Results indicated that RRDI can efficiently eliminate the effects of various canopy structures on the estimation of leaf chlorophyll (R² = 0.76, RMSE = 1141.5) (YU et al., 2014). JIA et al. (2013) used hyperspectral remote sensing to develop normalized difference vegetation indices (NDVI), spectral ratio (SR) indices, stepwise multiple linear regression (SMLR) and back-propagation (BP) neural network for estimation of nicotine content in flue-cured tobacco leaves under different shade conditions. Results indicated that BP neural network had the best accuracy and reproducibility (R² = 0.968, RMSE = 0.109) (JIA et al., 2013). A convolutional neural network (CNN) can autonomously learn and deeply extract local features of data (SHARMA et al., 2018), and the attention mechanism can effectively highlight important features. In this study, we adopted an attention-convolutional neural network (attention-CNN) model to monitor the chemical content during crop growth.

The millets studied in this research were obtained from the Agricultural Station of Shanxi Agricultural University. Leaves were collected at different growth stages, and the average spectral of the leaves was obtained by intelligently extracting the region of interest (ROI). The original spectral and image information were denoised by multi-scatter correction (MSC). Furthermore, we used the correlation coefficient-successive projections algorithm (CC-SPA) for data reduction and to extract the characteristic parameters based on spectral and image information. The single characteristic and multi-characteristic parameter fusions were used to build the PLSR model and search for optimal characteristic parameters. Furthermore, an attention-CNN model was established using the multi-characteristic parameter fusion and was compared with conventional PLSR and least squares support vector machine (LS-SVM) algorithms to establish a method for chlorophyll estimation in millet leaves.

MATERIALS AND METHODS:

Experimental procedure

The study was conducted at the Agricultural Station of Shanxi Agricultural University in June-October, 2018. A quadratic orthogonal rotation design and three factors-nitrogen, phosphorus, and potassium-were adopted in the experiment. Each factor had five different levels, and 23 test treatment combinations were available, in which each test was conducted in triplicate. A total of 69 plots were established (4 m × 4 m), which were arranged in random groups of blocks and planted with a row spacing of 40 cm. Phosphorus and potassium fertilizers were utilized as the base fertilizers and applied once, and the nitrogen fertilizer was applied thrice: 40% as the base fertilizer, 30% at the jointing stage, and 30% at the booting stage. During the experiment, field management was unified to prevent lodging and bird damage.

Determination of leaf spectral data

In the jointing, booting, and filling stages of millet, three representative and healthy growing leaves were collected from each plot. The leaves were stored in ice packs, numbered, and immediately brought to the laboratory for hyperspectral information acquisition and chemical determination of chlorophyll content. We used a Starter Kit indoor mobile scanning platform (Headwall Photonics, USA) to obtain visible near-infrared (NIR) hyperspectral images. The spectral range was 380-1000 nm, and the distance from the lens to the surface of the leaf was 25 cm. Three leaves from each plot were laid on a black-light-absorbing cloth with an exposure time of 39.84 ms and a platform moving speed of 2.721 mm/s.

To reduce the interference of noise caused by uneven distribution of light source and dark current noise (AMBROSE et al., 2016), the system was calibrated before the experiment. The correction formula is as follows:

$I=\frac{I_0-B}{W-B}\times 100$(1)

where I represents the corrected hyperspectral image, I ₀ denotes the original hyperspectral image, W represents the white background image captured by the scanning standard white correction board, and B is the dark background image captured by covering the lens. The Spectral View software was used to extract the required data from the hyperspectral image files obtained after black and white correction.

Measurement of chlorophyll content

The chlorophyll content of each leaf was determined by spectrophotometry. The whole leaf was selected, and after removing the veins, the millet leaves were cut into silk; 0.1 g of each leaf was weighed; and subsequently, treated with 80% acetone for 24 h. The absorbance at 646 nm and 663 nm was measured using a spectrophotometer and the following formula (CHEN et al., 2018):

$C_a=12.21A_{663}-2.81A_{646}$

$C_b=20.13A_{646}-5.03{}_{663}$

$C_T=C_a+C_b$ (2)

where A646 and A663 denote the absorbance at wavelengths of 646 nm and 663 nm, respectively, C _a and C _b represent the chlorophyll contents a and b, respectively, and C _T represents the total chlorophyll content. Results of chlorophyll measurement are shown in figure 1, the observed range of chlorophyll content was 4.358-15.272 mg/L in the jointing stage, 19.970-34.598 mg/L in the booting stage, and 13.194-27.613 mg/L in the filling stage.

Figure 1
Measure values of chlorophyll content during the whole growth period.

Selection of sample sets

Two thirds of the data of the three millet growth stages (jointing stage, booting stage, and filling stage) were randomly selected as the calibration set (46 samples per growth period and 138 samples for the whole growth period). The remaining one third (23 samples per growing season and 69 samples in the whole growth period) was used as the validation set. Samples from the three growing stages comprised a full-length sample. Statistical results of chlorophyll content in millet leaves are provided in table 1. The mean, standard deviation, and coefficient of variation of the chlorophyll content of the calibration set and the validation set are similar to those of the whole sample, i.e., the calibration set and the validation set effectively reflect the statistical characteristics of the samples.

Table 1
Basic statistics of chlorophyll content in millet leaves for the entire growth stage.

Type of sample	Observations	Chlorophyll content (mg/L)			Standard deviation (mg/L)	Coefficient of variation (CV)
		Maximum value	Minimum value	Mean value
Whole set	207	34.598	4.358	19.359	7.544	38.970 %
Calibration set	138	34.598	4.358	19.277	7.567	39.219 %
Validation set	69	33.611	5.228	19.524	7.565	38.745 %

Extraction and pretreatment of hyperspectral data

Because the conventional method of extracting ROI is not only time- and energy-consuming but also inefficient, a method of intelligently extracting the ROI was employed in this study. Figure 2 is a single-point spectral curve that reflects the differences in spectral curves in different regions of millet leaves. This clearly shows that the reflectance curves in the background region and the shadow region are relatively stable, and the reflectance is low in the NIR (780-900 nm) band; furthermore, the reflectance of the stem area is significantly higher than that of the object area in the red light (630-680 nm) band, and the hole region is characterized by low reflectance in the NIR (780-900 nm) band. Threshold segmentation was then performed using the NIR band reflectance (cull shadows, holes, and background areas) and the NDVI (cull stem area) based on this spectral change trend to intelligently select the ROI. Equation 3 shows the calculation formula of NDVI, where the range of NIR was 0.350-0.525 and the range of NDVI was 0.760-0.783 in the jointing stage, 0.790-0.814 in the booting stage, and 0.728-0.755 in the filling stage. Finally, the ROI regions without background, shadow, holes, and part of the stem can be obtained.

Figure 2
The spectral characteristics of object, hole, stem, shadow and background.

$NDVI=\frac{NI{R}_{780~900}-R_{630~680}}{NI{R}_{780~900}+R_{630~680}}$ (3)

To reduce noise interference, spectral data of a large number of points were extracted from the ROI, and then, the average spectral curve of samples was obtained via the arithmetic average, as shown in figure 3. Because the spectral reflectance was disturbed by equipment noise at both sides of the range (380 nm and 1000 nm), the bands at both sides were removed; thus, the spectral range 412.368-972.004 nm was used for the following analysis. We employed MSC to denoise the original spectral data. First, the average spectral of the sample set was calculated as the standard spectral based on equation 4. Then, a unitary linear regression calculation was performed for the spectral and the average spectral information of each sample using equation 5. The linear translation (b _i ) and tilt offset (m _i ) of each spectrum relative to the average spectrum were obtained; finally, the spectral curve of the sample after MSC correction was calculated using equation 6.

Figure 3
Original spectral reflectance.

$\overline{A_{i,j}}=\frac{\sum _{i=n}^n{A}_{i,j}}{n}$(4)

$A_i=m_i\overline{A}+b_i$(5)

$A_{i\left(MSC\right)}=\frac{\left(A_i-b_i\right)}{m_i}$(6)

where A denotes an n × p dimensional spectral data matrix, n represents the sampling number, p denotes the spectral dimension, denotes the average spectral vector of all samples, represents the average spectrum of the i ^th sample, and m _i and b _i represent the slope and intercept of thelinear regression relationship between and , respectively.

Selection of feature parameters

In this study, four different methods were used to extract the feature parameters as follows:

(1) Twenty-six vegetation indexes from the original spectral curve that are sensitive to green plant chlorophyll content were extracted, including NDVI, ratio vegetation index (RVI), difference vegetation index (DVI), CI_RE, and leaf chlorophyll index (LCI) (GITELSON & MERZLYAK, 1994).

(2) Twenty-one trilateral parameters of the leaf from the first-order differential spectrum were extracted, including the trilateral position, amplitude, area, kurtosis, and skewness. Then, twenty-five peak-valley parameters were constructed by utilizing the rising or falling rates of the characteristic sideband regions in the reflected spectral curve and the angle between the two sides of the peak-to-valley curve, which included the green edge (k_g), red edge (k_r), NIR (k_nir) rate of change, green peak to red valley bottom rate (k_gprv), angle between the reflected green peaks (GPA1), absorption angle between red valleys (RVA), red edge and NIR angle (GPA2), and their combination (XU et al., 2011).

(3) The ENVI software was used to segment the image of the millet leaf to remove the background, excluding only the main part of the blade (without veins). The mean and variance of the red, green, and blue components of the blade image were calculated using the RGB color system, and 27 color features were obtained using the combination of R, G, and B (CHENG et al., 2017).

(4) Seven common texture features were extracted using the gray level co-occurrence matrix (GLCM), including the energy, entropy, moment of inertia, correlation, mean, standard deviation, and smoothness.

Method of analysis

Selection of characteristic wavelength

Because the dimensions of the hyperspectral data were large, the sensitive bands that play a key role in the model had to be screened to reduce the interference from insignificant wavelengths on the accuracy of the model. The correlation between the spectral data vector and the chlorophyll content was calculated using the correlation coefficient method (CC), and the sensitive band with greater correlation was selected. We employed the successive projections algorithm (SPA) to extract the characteristic wavelengths in the sensitive band because this model can improve the prediction accuracy and stability based on the reduced model (HONG et al., 2010).

Chemometrics methods

In this study, a millet-leaf chlorophyll prediction model based on the characteristic wavelength and multi-characteristic parameter fusion was established using the attention-CNN model; in addition, the PLSR and LS-SVM models were used for comparative analysis to determine the better model.

CNN (SAIKIA et al., 2019) is a conventional model that is widely used in deep learning, which extracts the effective features of the data of each layer via multi-layer convolution and pooling operation. Moreover, the convergence speed of this model is high, and it provides good prediction results; however, this model is weak in highlighting the important convolution features, which is not sufficiently intuitive, and is poorly interpretable. Attention mechanism is a commonly used model, which has a long-term memory mechanism, in the field of natural language processing; this mechanism directly showed the contribution of each feature to the results. Thus, we used the attention mechanism to establish an attention-CNN model, as shown in figure 4.

Figure 4
Schematic diagram of the attention-CNN model.

The attention-CNN model mainly comprises an input layer, convolution layer, attention layer, pooling layer, fully connected layer, and output layer.

The input layer is an grain leaf feature fusion data matrix X, where n denotes the number of samples and p denotes the data dimension.

The convolution layer acts as the sample feature extraction layer. The input sample matrix X is convoluted by a convolution kernel of size to obtain the characteristic value, whereis given by,

$g_i=f\left(w\times X_{ii+h-1}+b\right)$(7)

wheredenotes the activation function, b denotes the bias item, and represents the feature extracted from line i to i+h-1 of X and is expressed as

G = [g₁, g₂,..., g_n]

We introduced the attention mechanism (ZHU et al., 2018) to explain the importance of each sample feature better. In this model, each local convolution feature was assigned an attention weight (BHUNIA et al., 2019) (and satisfies $\sum _{i=1}^n{a}_i=1$), where the importance of the influence of the feature on chlorophyll content prediction of millet leaf increases with the weight. The attention weight matrix and the local convolution feature are dotted with each other to complete the weighting and obtain the final feature vector K, which is expressed as, and is given as follows:

$K=\sum _{i=1}^n{a}_i{g}_i$(8)

Then, the attention-weighted feature K is input to the pooling layer. After performing the maximum pooling operation, the computational complexity was simplified to retain the essential features, and the over-fitting phenomenon is alleviated.

Finally, the feature map output from the pooling layer was rasterized and fully connected with the multi-layer perceptron (MLP), and the back-propagation algorithm was used to update the model parameters.

RESULTS AND DISCUSSION:

Analysis of the effect of MSC on the average spectral curve

As shown in figure 3, characteristics of the original spectral curve were consistent with those of typical green plants. Furthermore, absorption valleys were found in the range of 390-450 nm of violet light, reflection peaks in the green light range of 500-580 nm, absorption valleys in the red-light range of 630-770 nm, and a sharp rise in reflectivity in the range of 680-750 nm. In addition, the curve rises and forms a remarkable high reflection platform in the NIR band of 750-900 nm.

As shown in figure 3, the millet spectral curves greatly overlap in the visible light band, and the spectral reflectance varies greatly in the NIR band. To address these problems, the original spectral curve was pretreated with MSC. As shown in figure 5 , the shift in the spectral curve baseline was effectively eliminated. Then, the correlation between the spectral information of the millet leaves and the chlorophyll content before and after MSC pretreatment was analyzed; the correlation coefficient curve is shown in figure 6.

Figure 5
Multiple scattering correction spectra.

Figure 6
Correlation curves of the spectral information and chlorophyll content of millet leaves before and after MSC pretreatment.

Figure 6 indicates that higher correlation between the original spectrum and the chlorophyll content of the millet leaf is observed at 526-574 nm and 703-727 nm, which is 0.50-0.73 in value; however, after MSC treatment, leaf spectroscopy and chlorophyll content correlated well at the wavelengths of 647-690 nm and 700-743 nm, which is 0.70-0.90 in value. Thus, the correlation between spectroscopic results and chlorophyll content was clearly improved after MSC pretreatment.

Selection and modeling analysis of the sensitive band

The CC method was used to analyze the spectral data of millet leaves for the whole growth period, and the wavelengths 647-690 nm and 700-743 nm were selected as the sensitive bands. When the characteristic wavelength was extracted using SPA in the wavelength ranges of 647-690 nm and 700-743 nm, three (686 nm, 652 nm, and 690 nm) and four characteristic wavelengths (737 nm, 701 nm, 725 nm, and 707 nm) were selected when the root-mean-square errors were 2.104 and 1.977, respectively. These extracted characteristic wavelengths belong to the red-light range, which is consistent with the spectral sensitivity band of green-plant-leaf chlorophyll content. An analytical model to establish the chlorophyll content of millet leaves based on PLSR is shown in table 2, where R_c ² and R_v ² denoted the determination coefficients of the calibration set and the validation set, respectively, and RMSEc and RMSEv represented the root-mean-square errors of the calibration and validation sets, respectively.

Table 2
Chlorophyll-content prediction results of millet leaves based on characteristic wavelengths using PLSR.

Wavelength	Number of variables	Rc 2	RMSEc	Rv 2	RMSEv
686 nm, 652 nm, 690 nm, 737 nm, 701 nm, 725 nm, 707 nm	7	0.731	1.942	0.725	2.858
Entire band (412-972 nm)	771	0.748	1.887	0.716	3.100

Table 2 clearly indicates that chlorophyll estimation based on the full-band model was better than the characteristic wavelength model; furthermore, R_c ² was 0.748, which was 1.7 % higher than that of the characteristic wavelength model. The R_v ² of the characteristic wavelength model was 0.725, which is 0.9 % higher than that of the full-band model. These results indicated that the CC-SPA method can accurately extract the sensitive band of chlorophyll, effectively reduce the number of wavelength variables, and accurately predicted chlorophyll content in millet leaves based on the simplified model.

Analysis of the influence of different characteristic parameters on PLSR modeling

A correlation analysis was then performed between the extracted characteristic parameters and the chlorophyll content. The selected characteristic parameters are provided in table 3, where the correlation coefficient of NDVI is the largest at 0.831. The accuracy of chlorophyll estimation by NDVI is illustrated in figure 7, where it is seen that the R² and RMSE values for 207 leaves were 0.690 and 4.188, respectively. The characteristic parameters were fused to establish a prediction model to improve the prediction accuracy, whose results are provided in table 4.

Table 3
Correlation analysis results of the spectral characteristic parameters and chlorophyll content.

Index	r value	Characteristic parameter
Vegetation index	>0.8	NDVI, SAVI, OSAVI, RVI, DVI, RDVI, MSR, MSAVI, MTCI, TCARI, CCRI, PBRI, R-M, CIRE, LCI, TVI, PSSR, PSNDb, SR713
Trilateral parameters	>0.7	Red edge position, red edge kurtosis, yellow edge amplitude, yellow edge area, blue edge amplitude
Peak-valley parameters	>0.7	Kg, Kgprv, GPA1, Kr/Kg, Kr/Kgprv, GPA1/RVA, GPA1/GPA2
Texture features	>0.5	standard deviation, smoothness
Color features	>0.5	R(variance), R-B(variance), R-G(variance)

Table 4
Chlorophyll content prediction results of millet leaves based on different characteristic parameters using PLSR.

Index	Rc 2	RMSEc	Rv 2	RMSEv
Vegetation index	0.765	1.928	0.754	1.857
Trilateral parameters	0.775	1.774	0.767	1.726
Peak-valley parameters	0.685	2.233	0.654	2.199
Image features	0.458	2.635	0.421	2.847
Characteristic wavelength	0.731	1.942	0.725	2.858
Multi-characteristic parameters fusion	0.848	1.447	0.813	1.766

• Note: Image features include texture features and color features. Multi-characteristic parameters refer to the fusion of characteristic wavelength, vegetation index, trilateral parameters, peak-valley parameters, texture features, and color features according to the correlation coefficient grade.

Figure 7
Relationships between normalized difference vegetation indices (NDVI) and chlorophyll content.

As seen in the table, the prediction results were obtained for five different characteristic parameters (R_v ² = 0.421-0.767, RMSEv = 1.726-2.858), among which the trilateral parameter model exhibited better prediction. More importantly, the best prediction of chlorophyll content was achieved by the multi-characteristic parameter fusion model (R_v ² = 0813, RMSEv = 1.766).

Previous studies have mostly chosen a single type of vegetation index, trilateral parameters, and peak-valley parameters for crop nutrition detection (JAY et al., 2017; XU et al., 2011), ignoring the characteristic information of the image itself. However, in this study, a “spectral-image” fusion index was constructed, and the chlorophyll content of millet leaves was estimated based on multi-characteristic parameter fusion data, which provided a new method for quantitative estimation of chlorophyll.

Attention-CNN model with multi-characteristic parameters

The attention-CNN chlorophyll inversion model was established by multi-characteristic parameter fusion. The specific parameters of the model were established with two convolution layers. The first layer contained five convolution kernels of size 5 × 5, and the second layer contained 10 convolution kernels of the same size. Different convolution kernels were used to convolute all the feature maps in the former layer, following which the corresponding elements were accumulated and biased, and each output feature map was activated using the ReLu function. The size of the pooling layer was 2 × 2, and the feature vector dimension of the attention layer was 100. The loss function used the Mahalanobis distance, which is given by

$\sqrt{{\left(y_p-y_t\right)}^T{\sum }_{}^{-1}\left(y_p-y_t\right)}$(9)

where is the predicted value and is the measured value of chlorophyll content.

Furthermore, the mini-batch gradient descent method was used in the model. The network learning rate was 0.5, batch was 23, and the maximum number of iterations was set to 1300. The loss function curve and the accuracy value were used as the basis of convergence.

The common PLSR and LS-SVM models were selected for comparative analysis. The LS-SVM model used a radial basis function, and the optimal parameters were determined by cross-validation. The gam and sig2 of the calibration were set at 2.355 and 111.100, respectively, in this study. The chlorophyll prediction results of millet obtained using different models are listed in table 5.

Table 5
Modeling and validation of chlorophyll content based on characteristic wavelength and characteristic parameter fusion.

Method	Rc 2	RMSEc	Rv 2	RMSEv	RPD
Attention-CNN	0.865	1.290	0.839	1.451	2.355
PLSR	0.848	1.447	0.813	1.766	2.167
LS-SVM	0.837	1.445	0.806	1.576	2.061

Table 5 shows that the R_c ² and RMSEc values for the calibration set using the PLSR model were 0.848 and 1.447, respectively, and those using the LS-SVM model were 0.837 and 1.445, respectively. The results of the calibration set indicated that the modeling accuracy and the stability were effectively improved by the attention-CNN method (R_c ² = 0.865, RMSEc = 1.290). In the verification set, compared with the PLSR model (R_v ² = 0.813, RMSEv = 1.766, RPD = 2.167) and the LS-SVM model (R_v ² = 0.806, RMSEv = 1.576, RPD = 2.061), the attention-CNN model achieved the best prediction results (R_v ² = 0.839, RMSEv = 1.451, RPD = 2.355), and the difference between the modeling accuracy and the actual prediction ability was the lowest at 0.026.

Traditional machine learning algorithms have been mainly used for data analysis. For example, FÉRET et al. (2011) used spectral indices and PLSR to develop statistical relationships between reflectance spectral and leaf chemical properties (FÉRET et al., 2011). This study represented the first application of attention-CNN to measure millet chlorophyll content, which has a higher fitting accuracy and generalization ability than the conventional models. As discussed by KARTHIK et al. (2019), CNN can automatically extract the infection features in tomato leaves, and the attention mechanism can learn significant features for disease detection (KARTHIK et al., 2019).

Results demonstrated that the attention-CNN model can deeply mine the internal features of spectral data and simultaneously simplify preprocessing, which was effective in predicting the chlorophyll content of millet leaves; and was therefore, a high-performance chlorophyll-content prediction model.

CONCLUSION:

In this study, hyperspectral imaging technology was used to obtain the spectral and image information of millet leaves at different growth stages, and the average spectra of the millet leaves were extracted by intelligently extracting the ROI. The original spectral data were preprocessed by MSC and were subsequently analyzed to study the effect and correlation of pretreatment. We used the CC-SPA model for data reduction and extracted the characteristic parameters based on the spectral and image information. Furthermore, single characteristic and multi-characteristic parameter fusion were used to build the PLSR model. The multi-characteristic parameter fusion achieved accurate prediction results (R_v ² = 0.813, RMSEv = 1.766) and exhibited better prediction accuracy than the single characteristic parameter model. Based on the multi-characteristic parameter fusion model, the attention-CNN model yielded more accurate results (R_v ², RMSEv, and RPD were 0.839, 1.451, and 2.355, respectively) than the PLSR model (R_v ², RMSEv, and RPD were 0.813, 1.766, and 2.167, respectively) and the LS-SVM model (R_v ², RMSEv, and RPD were 0.806, 1.576, and 2.061, respectively). In addition, the difference between the modeling accuracy and the actual prediction ability of the attention-CNN model was the smallest (0.026). These results demonstrated that the attention-CNN model has a higher prediction accuracy and regression fit than the conventional models and exhibits better adaptability to the sample data. Therefore, the attention-CNN model is a highly advantageous novel method for non-invasive measurement of chlorophyll content in millet plants.

ACKNOWLEDGMENTS

This work was funded by National Key Research and Development Program of China (2017YFD0701501), General Program of National Natural Science Foundation of ShanXi Province (201701D121099), and Science and Technology Innovation Fund of Shanxi Agricultural University(2017YJ12).

REFERENCES

• MAO, Z.H. et al. Research on the Application of UAV Multispectral Remote Sensing in the Maize Chlorophyll Prediction. Spectroscopy and Spectral Analysis, v.38, n.9, p.2923-2931, 2018. Available from: <Available from: http://www.gpxygpfx.com/article/2018/1000-0593-38-9-2923.html >. Accessed: Dec. 11, 2018. doi: 10.3964/j.issn.1000-0593(2018)09-2923-09.
  » https://doi.org/10.3964/j.issn.1000-0593(2018)09-2923-09.» http://www.gpxygpfx.com/article/2018/1000-0593-38-9-2923.html
• EVANS, T. et al. Micro-scale chlorophyll analysis and developmental expression of a cytokinin oxidase/dehydrogenase gene during leaf development and senescence. Plant Growth Regulation, v.66, n.1, p.95-99, 2012. Available from: <Available from: https://link.springer.com/article/10.1007/s10725-011-9627-5 >.Accessed: Dec. 23, 2018. doi: 10.1007/s10725-011-9627-5.
  » https://doi.org/10.1007/s10725-011-9627-5.» https://link.springer.com/article/10.1007/s10725-011-9627-5
• LOH, C.H. et al. Determination of Chlorophylls in Taraxacum formosanum by High-Performance Liquid Chromatography-Diode Array Detection-Mass Spectrometry and Preparation by Column Chromatography. Journal of Agricultural and Food Chemistry, v.60, n.24, p.6108-6115, 2012. Available from: Available from: https://pubs.acs.org/doi/10.1021/jf301422m >. Accessed: Sep. 20, 2018. doi: 10.1021/jf301422m.
  » https://doi.org/10.1021/jf301422m.» https://pubs.acs.org/doi/10.1021/jf301422m
• SCOTTER, M.J. Methods for the determination of european union-permitted added natural colours in foods: a review. Food Additives and Contaminants, v.28, n.5, p.527-596, 2011. Available from: <Available from: https://www.tandfonline.com/doi/full/10.1080/19440049.2011.555844 >. Accessed: Sep. 26, 2018. doi: 10.1080/19440049.2011.555844.
  » https://doi.org/10.1080/19440049.2011.555844.» https://www.tandfonline.com/doi/full/10.1080/19440049.2011.555844
• KORUS, A. Effect of preliminary and technological treatments on the content of chlorophylls and carotenoids in kale (Brassica oleracea L. var. Acephala). Journal of Food Processing and Preservation, v.37, p.335-344, 2013. Available from: <Available from: https://onlinelibrary.wiley.com/doi/10.1111/j.1745-4549.2011.00653.x >. Accessed: Dec. 5, 2018. doi: 10.1111/j.1745-4549.2011.00653.x.
  » https://doi.org/10.1111/j.1745-4549.2011.00653.x.» https://onlinelibrary.wiley.com/doi/10.1111/j.1745-4549.2011.00653.x
• ZHENG, T. et al. Estimation of Chlorophyll Content in Potato Leaves Based on Spectral Red Edge Position. IFAC-PapersOnLine, v.51, p.602-606, 2018. Available from: <Available from: https://www.sciencedirect.com/science/article/pii/S2405896318312497 >. Accessed: Dec. 7, 2018. doi: 10.1016/j.ifacol.2018.08.131.
  » https://doi.org/10.1016/j.ifacol.2018.08.131.» https://www.sciencedirect.com/science/article/pii/S2405896318312497
• GITELSON, A., SOLOVCHENKO, A. Generic algorithms for estimating foliar pigment content. Geophysical Research Letters, v.44, n.18, p.9293-9298, 2017. Available from: <Available from: https://agupubs.onlinelibrary.wiley.com/doi/10.1002/2017GL074799 >. Accessed: Nov. 26, 2018. doi: 10.1002/2017GL074799.
  » https://doi.org/10.1002/2017GL074799.» https://agupubs.onlinelibrary.wiley.com/doi/10.1002/2017GL074799
• YU, K. et al. Estimating leaf chlorophyll of barley at different growth stages using spectral indices to reduce soil background and canopy structure effects. ISPRS Journal of Photogrammetry and Remote Sensing, v.97, p.58-77, 2014. Available from: <https://www.researchgate.net/publication/265644981_Estimating_leaf_chlorophyll_of_barley_at_different_growth_stages_using_spectral_indices_to_reduce_soil_background_and_canopy_structure_effects>.
  » https://www.researchgate.net/publication/265644981_Estimating_leaf_chlorophyll_of_barley_at_different_growth_stages_using_spectral_indices_to_reduce_soil_background_and_canopy_structure_effects
• JIA, F.F. et al. Using leaf spectral reflectance to monitor the effects of shading on nicotine content in tobacco leaves. Industrial Crops and Products, v.51, p.444-452, 2013. Available Available from:http://www.doc88.com/p-9119574879588.html >. Accessed: Jun. 15, 2017. doi: 10.1016/j.indcrop.2013.09.027.
  » https://doi.org/10.1016/j.indcrop.2013.09.027.» from:http://www.doc88.com/p-9119574879588.html
• SHARMA, N. et al. An Analysis of Convolutional Neural Networks for Image Classification. Procedia Computer Science, v.132, p.377-384, 2018. Available from: <Available from: https://www.sciencedirect.com/science/article/pii/S1877050918309335 >. Accessed: Dec. 9, 2018. doi: 10.1016/j.procs.2018.05.198.
  » https://doi.org/10.1016/j.procs.2018.05.198.» https://www.sciencedirect.com/science/article/pii/S1877050918309335
• AMBROSE, A. et al. High speed measurement of corn seed viability using hyperspectral imaging. Infrared Physics and Technology, v.75, p.173-179, 2016. Available from: <Available from: https://www.sciencedirect.com/science/article/pii/S1350449515300074 >. Accessed: Aug. 5, 2017. doi: 10.1016/j.infrared.2015.12.008.
  » https://doi.org/10.1016/j.infrared.2015.12.008.» https://www.sciencedirect.com/science/article/pii/S1350449515300074
• CHEN, C.L. et al. Analysis of Chlorophyll Contents in Maize Leaf based on GA-BP Neural Network Hyperspectral Inversion Model. Journal of Shenyang Agricultural University, v.49, n.5, p.626-632, 2018. Available from: <Available from: http://www.cnki.com.cn/Article/CJFDTotal-SYNY201805018.htm >. Accessed: Dec. 13, 2018. doi: 10.3969/j.issn.1000-1700.2018.05.016.
  » https://doi.org/10.3969/j.issn.1000-1700.2018.05.016.» http://www.cnki.com.cn/Article/CJFDTotal-SYNY201805018.htm
• GITELSON, A., MERZLYAK, M.N. Spectral reflectance changes associated with autumn senescence of Aesculus hippocastanum L.and Acer platanoides L.leaves spectral features and relation to chlorophyll estimation. Journal of Plant Physiology, v.143, n.3, p.286-292, 1994. Available from: <Available from: https://www.sciencedirect.com/science/article/pii/S0176161711816330 >. Accessed: Aug. 12, 2017. doi: 10.1016/s0176-1617(11)81633-0.
  » https://doi.org/10.1016/s0176-1617(11)81633-0.» https://www.sciencedirect.com/science/article/pii/S0176161711816330
• CHENG, L.Z. et al. Estimation of Chlorophyll Content in Apple Leaves Based on RGB Model Using Digital Camera. Acta Horticulturae Sinica, v.44, n.2, p.381-390, 2017. Available from: <Available from: http://www.wanfangdata.com.cn/details/detail.do?_type=perio&id=yyxb201702019 >. Accessed: Nov. 5, 2017. doi: 10.16420/j.issn.0513-353x.2016-0529.
  » https://doi.org/10.16420/j.issn.0513-353x.2016-0529.» http://www.wanfangdata.com.cn/details/detail.do?_type=perio&id=yyxb201702019
• HONG, Y. et al. Successive projections algorithm for variable selection in nondestructive measurement of citrus total acidity. Transactions of the Chinese Society of Agricultural Engineering, v.26, n.2, p.380-384, 2010. Available from: <Available from: http://www.docin.com/p-404689103 >. Accessed: Nov. 15, 2017. doi: 10.3969/j.issn.05131002-6819.2010.z2.072.
  » https://doi.org/10.3969/j.issn.05131002-6819.2010.z2.072.» http://www.docin.com/p-404689103
• SAIKIA, A.R. et al. Comparative assessment of CNN architectures for classification of breast FNAC images. Tissue and Cell, v.57, p.8-14, 2019. Available from: <Available from: https://www.sciencedirect.com/science/article/pii/S0040816618304397 >. Accessed: Mar. 25, 2019. doi: 10.1016/j.tice.2019.02.001.
  » https://doi.org/10.1016/j.tice.2019.02.001.» https://www.sciencedirect.com/science/article/pii/S0040816618304397
• ZHU, X.X. et al. Image captioning with triple-attention and stack parallel LSTM. Neurocomputing, v.319, p.55-65, 2018. Available from: <Available from: https://www.sciencedirect.com/science/article/pii/S0925231218310324 >. Accessed: Jan. 20, 2019. doi: 10.1016/j.neucom.2018.08.069.
  » https://doi.org/10.1016/j.neucom.2018.08.069.» https://www.sciencedirect.com/science/article/pii/S0925231218310324
• BHUNIA, A.K. et al. Script identification in natural scene image and video frames using an attention based Convolutional-LSTM network. Pattern Recognition, v.85, p.172-184, 2019. Available from: <Available from: http://www.sciencedirect.com/science/article/pii/S0031320318302590 >. Accessed: May. 2, 2019. doi: 10.1016/j.patcog.2018.07.034.
  » https://doi.org/10.1016/j.patcog.2018.07.034.» http://www.sciencedirect.com/science/article/pii/S0031320318302590
• JAY, S. et al. Estimating leaf chlorophyll content in sugar beet canopies using millimeter- to centimeter-scale reflectance imagery. Remote Sensing of Environment, v.198, p.173-186, 2017. Available from: <Available from: https://s100.copyright.com/AppDispatchServlet?publisherName=ELS&contentID=S0034425717302675&orderBeanReset=true >. Accessed: Jul. 6, 2018. doi: 10.1016/j.rse.2017.06.008.
  » https://doi.org/10.1016/j.rse.2017.06.008.» https://s100.copyright.com/AppDispatchServlet?publisherName=ELS&contentID=S0034425717302675&orderBeanReset=true
• XU, X.G. et al. Study on Relationship between New Characteristic Parameters of Spectral Curve and Chlorophyll Content for Rice. Spectroscopy and Spectral Analysis, v.31, n.1, p.188-191, 2011. Available from: <Available from: http://www.docin.com/p-249286352.html >. Accessed: Jun. 6, 2017. doi: 10.3964/j.issn.1000-0593(2011)01-0188-04.
  » https://doi.org/10.3964/j.issn.1000-0593(2011)01-0188-04.» http://www.docin.com/p-249286352.html
• FÉRET, J.B. et al. Optimizing spectral indices and chemometric analysis of leaf chemical properties using radiative transfer modeling. Remote Sensing of Environment, v.115, n.10, p.2742-2750, 2011. Available from: <Available from: https://www.sciencedirect.com/science/article/abs/pii/S0034425711002409 >. Accessed: Oct. 20, 2018. doi:10.1016/j.rse.2011.06.016.
  » https://doi.org/10.1016/j.rse.2011.06.016» https://www.sciencedirect.com/science/article/abs/pii/S0034425711002409
• KARTHIK, R. et al. Attention embedded residual CNN for disease detection in tomato leaves. Applied Soft Computing, 2019. Available from: <Available from: https://s100.copyright.com/AppDispatchServlet?publisherName=ELS&contentID=S1568494619307148&orderBeanReset=true > . Accessed: Dec. 12, 2019. doi: 10.1016/j.asoc.2019.105933
  » https://doi.org/10.1016/j.asoc.2019.105933» https://s100.copyright.com/AppDispatchServlet?publisherName=ELS&contentID=S1568494619307148&orderBeanReset=true

• CR-2019-0731.R1

Publication Dates

• Publication in this collection
  06 Apr 2020
• Date of issue
  2020

History

• Received
  24 Sept 2019
• Accepted
  17 Dec 2019
• Reviewed
  22 Feb 2020