Statistical evaluation of the PHYGROW model: a case study in Ceará State, Brazil

[Avaliação estatística do modelo PHYGROW: um estudo de caso no estado do Ceará, Brasil]

ABSTRACT

This study was carried out to statistically evaluate the performance of the PHYGROW model in simulating the growth of corn and sorghum plants in different locations in the semi-arid region of Ceará State, Brazil. Corn and sorghum production data were obtained in seven macro-regions, where the characterization of ecological sites was previously performed using soil and vegetation information for model parameterization. Biomass production data were collected in the 2021 and 2022 rainy seasons. The obtained models were statistically analyzed by the following estimators: estimation bias (BIAS), mean absolute error (MAE), estimation efficiency (EE) and Willmott’s index of agreement (d). The results showed the suitability of the PHYGROW model to simulate the growth of corn and sorghum crops in different semi-arid regions of Ceará State; however, the statistical estimators stressed the need for a larger database to improve forecast accuracy.

Keywords:
mathematical estimators; mechanistic modelling; rainfed crops

RESUMO

Este estudo foi realizado com o objetivo de avaliar estatisticamente o desempenho do modelo PHYGROW na simulação do crescimento de plantas de milho e sorgo em diferentes localidades do semiárido cearense. Os dados de produção de milho e sorgo foram obtidos em sete macrorregiões, onde a caracterização dos sítios ecológicos foi previamente realizada utilizando-se informações de solo e vegetação para parametrização do modelo. Os dados de produção de biomassa foram coletados nas épocas chuvosas de 2021 e 2022. Os modelos obtidos foram analisados estatisticamente pelos seguintes estimadores: viés de estimativa (BIAS), erro absoluto médio (MAE), eficiência de estimação (EE) e índice de concordância de Willmott (d). Os resultados mostraram a adequação do modelo PHYGROW para simular o crescimento das culturas de milho e sorgo em diferentes regiões do semiárido cearense; contudo, os estimadores estatísticos sublinharam a necessidade de uma base de dados maior para melhorar a precisão das previsões.

Palavras-chave:
culturas de sequeiro; estimadores matemáticos; modelagem mecanicista

INTRODUCTION

Despite the challenge of expanding detailed information on crop growth and development in the semi-arid, process-based simulation models are a commonly used tool to assess the impact of environmental variation and changes in crop yields (Bouras et al., 2019). Considering the semi-arid environment and its spatial and temporal variability of the rainfall regime, which affect the dynamics of plant growth, stochastic-based mechanistic models are more appropriate for this type of simulation.

In the Brazilian semi-arid region, corn and sorghum crops are important for the nutritional security of people and herds. In 2020, the Northeast region of Brazil produced 6.1 billion tons of corn grains, with Ceará State producing 528.6 thousand tons. The same applies to sorghum, which has been gaining prominence in the region each year, with a planted area 19.2% greater in 2021 (Acompanhamento…, 2022). The data presented show the efforts of the public policies and civil society to make grain and fodder production feasible in the semi-arid region; however, due to the variability of soils and climate, production is heterogeneous, and the local potential is underestimated due to the choice of an inadequate genotype.

To overcome this challenge, the use of mathematical models can help to obtain results from crops in different areas quickly and reliably, minimizing costs and time, increasing decision-making capacity. However, in the case of mechanistic models, field calibration steps are necessary for the model to be of practical use (Brunetti et al., 2021). The mechanistic and hydrologically based model PHYGROW (Stuth et al., 2003), for instance, has demonstrated good accuracy in predicting the growth of native vegetation (Caatinga) (Morais et al., 2021) as well as exotic forage crops adapted to the semi-arid climate (Maranhão, 2021) not using chemical fertilization and irrigation.

When it comes to the statistical evaluation of models, there are conceptual divergences for the terms calibration/validation and evaluation of mathematical models regarding their accuracy. Evaluation is a fundamental step in adjusting forecast models, so that there is a comparison of predicted data with observed data, which uses statistical tools to support the conclusions. Numerous are the statistical techniques used to evaluate the precision and accuracy of the models (Pereira et al., 2018; Botchkarev, 2019; Chakravorty et al., 2021; Hodson, 2022; Robeson and Willmott, 2023), however, no single approach evaluates model performance adequately, so the best way to evaluate the predictive performance of a model is to associate a set of statistical methods.

Therefore, the objective of this study was to statistically evaluate the ability of the PHYGROW model to predict corn and sorghum growth under different edaphic and semi-arid conditions.

MATERIALS AND METHODS

Experimental data were obtained in the macro-regions or milk producing center of Ceará, Brazil. Seven representative sites from each region were selected for corn and sorghum data collection: Sobral (latitude 3º41′S, longitude 40º20′W, altitude 69.5 m a.s.l.), Pentecoste (latitude 3º47′S, longitude 39º16′W, altitude 60.0 m a.s.l.), Quixeramobim (latitude 5º11′S, longitude 39º17′W, altitude 191.7 m a.s.l.), Limoeiro do Norte (latitude 5º08′S, longitude 38º05′W, altitude 140.0 m a.s.l.), Tauá (latitude 6º00′S, longitude 40º17′W, altitude 402.7 m a.s.l.), Quixelô (latitude 6º15′S, longitude 39º12′W, altitude 202.0 m a.s.l.) and Crato (latitude 7º14′S, longitude 39º24′W, altitude 426.9 m a.s.l.).

Field research activities were carried out in 2021 and 2022, with the characterization of the experimental field and collection of crop data. As a criterion for choosing the sample locations (municipality in the Ceará State), those with different edaphoclimatic characteristics, especially rainfall, were selected to expand the model's forecasting capacity (Fig. 1; Table 1). As it is a mechanistic model, the characterization of the ecological site involved a set of soil and vegetation information for parameterization of the PHYGROW model, such as: the type of vegetation cover (native or manipulated) and its classification (aspen, brush, desert, farmsteads, herbaceous-mixture, meadow, pasture, pinyon, sagebrush, woods, woods-grass or undefined); soil textural class (sand, loam, silt, clay or undefined); latitude and longitude; land slope (%); elevation (a.s.l.); type of composition of the bottom layer of soil (permeable, impermeable or groundwater); surface water storage (volume of water temporarily available under the soil surface after a rainfall), exposed soil and rocks on the soil surface (%), among other variables.

Figure 1
Rainfall history (mm) measured in the years 2021 and 2022 in the evaluated municipality of the Ceará State, Brazil.

Table 1
General edaphoclimatic characteristics of selected municipalities of the state of Ceará, Brazil

Municipality in the Ceará State	KCC1	Altitude (a.s.l)2	Soil class3	Temperature (°C)
				Max4	Min5	Avg6
Crato	Aw	426.9	Red yellow podzolic	36.0	19.0	27.5
Limoeiro do Norte	BSh	140.0	Cambisol	38.0	24.0	28.2
Pentecoste	Aw/BSh	60.0	Podzolic/Fluvisol	35.0	22.0	27.8
Quixelô	Aw/BSh	202.0	Podzolic/Fluvisol	36.5	22.0	27.3
Quixeramobim	BSh	191.7	Luvisol	38.0	21.0	27.4
Sobral	BSh	69.5	Luvisol	37.0	23.0	28.0
Tauá	BSh/BWh	402.7	Luvisol	37.5	20.0	28.0

• ¹
  KCC = Köppen Climate Classification (Köppen, 1936); ²At sea level; ³(IUSS Working Group WRB, 2015); ⁴Maximum; ⁵Minimum; ⁶Average.

Regarding crops, information was collected on plant density, height (cm), total forage biomass (kg of dry matter) and other variables obtained indirectly or in the literature, such as leaf area index (ratio), basal, optimal and suppression temperatures (ºC), root length (cm), among other variables. Climatological data (rainfall, radiation, air humidity etc.) are estimated in software that imports the history of climate variables directly from the National Oceanic Atmospheric Administration (NOAA), developed in cooperation with the National Aeronautics and Space Administration (NASA).

The harvest time was determined when the plants had dry matter content around 30%. Total biomass samples present in 1 linear meter were collected and the data were extrapolated to an area of 1 m², a condition required by the PHYGROW model.

The PHYGROW model calibration procedure involved running the model comparing the estimated biomass production of corn and sorghum with those measured in 2021 and 2022. If the model result fell within ± 1 of the standard error of the mean for the measured biomass, the model was considered calibrated. When the simulation results were outside ± 1 of the standard error of the measured data, the parameters were adjusted to move the modelled biomass estimate within the standard error. This process was repeated for each period that data were collected (2021 and 2022) until the model was considered to be in an advanced stage of calibration.

Difference statistics were calculated to examine bias in model estimates and variability between simulated and measured data. These statistics included percentage estimation bias (BIAS) (1), mean absolute error (MAE) (2), estimation efficiency (EE) (3) and Willmott’s index of agreement (d) (4) (Willmott, 1981). BIAS reflects the normalized value (reducing redundancy and increasing integrity) between simulated and observed data, given in percent. Positive values indicate overestimation of the model, while negative values indicate underestimation of the prediction:

$BIAS\left(\%\right)=\frac{\overline{P}-\overline{O}}{\overline{O}}x100$ (1)

Which:

$\overline{P}$ = data predicted by the model;

$\overline{O}$ = measured data.

MAE differs from BIAS regarding the error with a negative value, that is, the forecast with a value greater than the measured number is transformed into a positive one and the mean is calculated posteriori:

$MAE=\frac{{{\displaystyle \sum }}_{i=1}^n\left|Pi-Oi\right|}{n}$ (2)

Which:

Pi = ith predicted value;

Oi = ith measured value;

n = number of simulated and measured data pairs.

EE is a measure of the deviation from a 1:1 line between the simulation model output and the measured data. Values equal to “1” mean perfect estimation between simulated and measured data. Values greater than “0” indicate a positive relationship and can be used as a good estimate of the model, with the reverse also being true:

$EE=1-\frac{{{\displaystyle \sum }}_{i=1}^n\left(Oi-Pi\right)²}{{{\displaystyle \sum }}_{i=1}^n\left(Oi-\overline{O}\right)²}$ (3)

Which:

Pi = ith predicted value;

Oi = ith measured value;

$\overline{O}$ = measured data.

The Willmott’s index of agreement (d) shows the fit between simulated and observed data on a 1:1 line. Values of 1 mean perfect fit (agreement):

$d=1-\frac{{{\displaystyle \sum }}_{i=1}^n\left(Oi-Pi\right)²}{{{\displaystyle \sum }}_{i=1}^n\left(\left|Pi-\overline{O}\right|+\left|Oi-\overline{O}\right|\right)²}$ (4)

Where:

Pi = ith predicted value;

Oi = ith measured value;

$\overline{O}$ = measured data.

RESULTS

During the calibration phase, the PHYGROW accurately simulated corn growth in five of the seven locations studied, with corn growth curves showing smaller oscillations when compared to sorghum (Fig. 2). For sorghum, only one site was not reached for calibration. In relation to the uncalibrated models, the differences between the simulated biomass values and the average of the corn field data were -6.203 and -6.680 kg DM ha^-1 for Pentecoste and Quixelô, respectively. As for sorghum, the difference was -8.297kg DM ha^-1 for Pentecoste (Table 2).

Figure 2
Maize and sorghum growth curves for Ceará State, Brazil, obtained in PHYGROW.

Table 2
Results of maize and sorghum growth simulations by PHYGROW model in Ceará State, Brazil

Maize	Sampling date	MA	SE	EAP	Status*
		kg dry matter ha-1
Crato 1	05/10/2022	11.553	2.435	12.392	C
Limoeiro do Norte 2	05/09/2022	4.823	1.016	4.567	C
Pentecoste 1	06/01/2021	10.000	2.108	3.797	NC
Quixelô 3	05/12/2022	15.283	3.222	8.603	NC
Quixeramobim 4	05/21/2022	8.644	1.822	8.302	C
Sobral 1	10/05/2021;17/05/2022	5.100	1.075	5.550	C
Tauá 1	05/11/2022	7.293	1.537	6.548	C
Sorghum
Limoeiro do Norte 6	05/09/2022	4.470	0.942	4.985	C
Pentecoste 5	06/01/2021	12.333	2.600	4.036	NC
Quixelô 5	05/12/2022	7.473	1.575	8.036	C
Quixeramobim 5	05/21/2022	3.690	0.583	3.139	C
Sobral 5	05/04/2021	5.120	1.079	5.725	C
Tauá 5	05/11/2022	4.654	0.981	5.219	C

• MA = Measured Averages; SE = Standard Error; EAP = Estimated Averages by PHYGROW. ¹ = hybrid BRS 2022; ² = hybrid P3808 VYHR; ³ = hybrid BRS 4104; ⁴ = BRS Caatingueiro; ⁵ = hybrid BRS Ponta Negra; ⁶ = hybrid 84G05. *Status (C = calibration; NC = no calibration).

As for the statistical evaluation, an MAE of less than 550 kg DM ha^-1 is observed for the calibrated models of corn and sorghum. On the other hand, the uncalibrated models underestimate the production capacity in all evaluated locations, with the most pronounced difference being observed in the corn model in Pentecoste (MAE = 6,200.4 kg DM ha^-1). As for the BIAS, the calibrated corn models underestimated the yield of the crop in the municipalities of Limoeiro do Norte (-4.7%), Quixeramobim (-3.7%) and Tauá (-10.4%) and overestimated it for Crato (7.4%) and Sobral (8.8%). As observed for sorghum, all calibrated models overestimate the sorghum production potential by 12.3, 7.0, 18.3 and 16.6% for the municipalities of Limoeiro do Norte, Quixelô, Sobral and Tauá, respectively (Table 3). As for the results of EE and d, the best efficiency indicators were found in the models of corn in Tauá (EE = 0.5; d = 0.8) and sorghum in Quixelô (EE = 0.6; d = 0.8) and Quixeramobim (EE = 0.6; d = 0.9).

Table 3
Statistical parameters to assess the ability of the PHYGROW model to simulate maize and sorghum growth in Ceará State, Brazil

Maize	MAE	BIAS	EE	d	n	Status*
	kg DM ha-1	%	index		-	-
Crato 1	257.6	7.4	3.5	0.6	12	C
Limoeiro do Norte 2	199.3	-4.7	3.0	0.5	12	C
Pentecoste 1	6,200.4	-61.9	-5.8	0	12	NC
Quixelô 3	2,045.8	-43.7	-3.3	0	12	NC
Quixeramobim 4	540.0	-3.7	0.2	0.3	12	C
Sobral 1	178.4	8.8	0.4	0.5	24	C
Tauá 1	66.6	-10.4	0.5	0.8	12	C
Sorghum
Limoeiro do Norte 6	274.5	12.3	0.5	0.5	12	C
Pentecoste 5	997.6	-67.4	-5.6	0	12	NC
Quixelô 5	87.8	7.0	0.6	0.8	12	C
Quixeramobim 5	58.3	-12.1	0.6	0.9	12	C
Sobral 5	358.0	18.3	0.4	0.6	12	C
Tauá 5	306.8	16.6	0.5	0.4	12	C

• MAE = Mean Absolute Error; BIAS = percent estimation bias; EE = Estimation Efficiency; d = Willmott agreement index; n = number of observations. ¹ = hybrid BRS 2022; ² = hybrid P3808 VYHR; ³ = hybrid BRS 4104; ⁴ = BRS Caatingueiro; ⁵ = hybrid BRS Ponta Negra; ⁶ = hybrid 84G05; *Status (C = calibration; NC = no calibration).

DISCUSSION

As an initial calibration, the PHYGROW model showed strong predictive, with its growth curves showing a satisfactory behavior with the rainfall regime of the region. In general, rainfall in Ceará State lasts from December to June (Tinôco et al., 2018), starting in the center-south of the State, with a maximum value in mid-April. The curves, therefore, follow this behavior, showing the good accuracy of the PHYGROW model in estimating climate data. In addition, the use of different cultivars and hybrids of corn and sorghum allowed the visualization of the most effective application of the PHYGROW model, making it possible to identify genotypes that have great production potential in the set of locations or even with adaptation specific to a particular location for the purpose of cultivation recommendation.

The lower accuracy of PHYGROW for some locations, mainly for the corn crop, may be associated with the small database and the generalization of the estimate of soil characteristics by the model, negatively affecting the result of the simulations. In most farms located in the Brazilian semi-arid region, the soils are varied, with different physicochemical characteristics depending on the relief, proximity to watercourses and the original mineralogical material (Silva et al., 2022). In small and medium-sized farms in the Brazilian semi-arid region, agricultural cultivation is carried out first on the patches of soil most suitable for mechanization and then on those with greater natural fertility. These small and scattered areas of cultivation on the farm are not detected by PHYGROW, as the soil sub model operates considering the macro-regions as a single soil class, which leads to under- or overestimation of the local potential.

This statement becomes clear when evaluating the discrepancy between the measured averages (MA) of yield and those estimated by PHYGROW (EAP). The corn and sorghum cultivation areas of all evaluated locations are smaller than 0.05 km², while the soil area of the Pentecoste sub model, for example, corresponds to approximately 6,606.38 km². The generalization of the soil class contemplates the average of the region, that is, the fertile soil patches, the humid shallows and other small, differentiated zones are not precisely detected by PHYGROW.

Regarding the assessment of the predictive capacity of PHYGROW, the calibrated models showed low MAE and BIAS, despite the reduced database of the initial calibration phase. MAE is one of the methods used to measure forecast model accuracy. The MAE value shows the mean absolute error between the forecast results and the actual value, assigning the same weight to all data (Suryanto and Muqtadir, 2019), being an important estimator of the evaluation of forecasting models in agriculture (Getahun et al., 2018).

As for BIAS, it exists in the numbers of any data analysis process, varying between its source, the chosen estimator (Rainey and McCaskey, 2021) and between the forms of data analysis (Gourgoulias et al., 2020). The BIAS should not be confused with the degree of precision of an estimator, as the degree of precision is a measure of the sampling error (Mathur and Vanderweele, 2020); for example, if the sample size is not large enough, the results may not be representative, that is, there may be discrepancies between estimated and measured results. Despite the initial research, the BIAS values found point to a high probability of model validation.

It is important to point out that the data in this study is obtained for each season (annual data). In the calibration phase, each set of data is used to investigate the simulation curve shape and it’s fit to field data. This operation can last several seasons, depending on local edaphoclimatic factors that are not easy to input into the model. However, when a good fit is found at the beginning of the modelling process, the trend is for the model to obtain validation with the need for a smaller database.

Regarding EE and d, the results for the calibrated models provide evidence that the PHYGROW simulation shows moderate ability to predict the biomass of corn and sorghum crops. EE is sensitive to discrepant data, which helps to explain the responses achieved in this first year of evaluations. The same can be said for the index of agreement (d), whose results were moderate for six of the nine calibrated models. Willmott (1981) considers that lower indices at the beginning of the modelling process are acceptable, given that it is a sensitive statistical test. Thus, it is expected that the results of the estimators will improve as new sets of information are added.

CONCLUSION

Statistical analysis is an important procedure during model calibration and evaluation, but there is no standard way indicating how many and which statistics should be used. The PHYGROW model can simulate the cultivation of annual crops in the semi-arid region; however, given the edaphoclimatic variability of the region, a larger set of annual data is necessary for a better forecasting capacity of the models.

ACKNOWLEDGEMENTS

To Agriculture Chief Scientist Program of the Ceará State Government (covenant 14/2022 SDE/ADECE/FUNCAP and FUNCAP 08126425/2020 process) for the financial support provided for this research and the award of scholarships.

REFERENCES

• ACOMPANHAMENTO da Safra Brasileira de Grãos. Observatório Agrícola, Brasília, DF. Brasília: CONAB, 2022. Available in: https://observatorio.agropecuaria.inmet.gov.br/biblioteca/. Accessed in: 2 Dec. 2022.
• BOURAS, E.; JARLAN, L.; KHABBA, S. et al. Assessing the impact of global climate changes on irrigated wheat yields and water requirements in a semi-arid environment of Morocco. Sci. Rep., v.9, p.1-14, 2019.
• BOTCHKAREV, A. Performance metrics (error measures) in machine learning regression, forecasting and prognostics: properties and typology. Interdiscipl. J. Inf. Knowledge Manag., v.14, p.45-79, 2019.
• BRUNETTI, H.B.; BOOTE, K.J.; SANTOS, P.M. et al. Improving the CROPGRO perennial forage model for simulating growth and biomass partitioning of guineagrass. Agron. J., v.113, p.1-16, 2021.
• CHAKRAVORTY, A.; GOGOI, R.B.; KUNDU, S.S.; RAJU, P.L.N. Investigating the efficacy of a new symmetric index of agreement for evaluating WRF simulated summer monsoon rainfall over northeast India. Meteorol. Atmosph. Phys., v.133, p.479-493, 2021.
• GETAHUN, M.A.; SHITOTE, S.M.; GARIY, Z.C.A. Artificial neural network based modelling approach for strength prediction of concrete incorporating agricultural and construction wastes. Constr. Building Mater., v.190, p.517-525, 2018.
• GOURGOULIAS, K.; KATSOULAKIS, M.A.; REY-BELLET, L.; WANG J. How biased is your model? Concentration inequalities, information and model bias. IEEE Trans. Inf. Theor., v.66, p.3079-3097, 2020.
• HODSON, T.O. Root-mean-square error (RMSE) or mean absolute error (MAE): when to use them or not. Geosci Model Dev., v.15, p.5481-5487, 2022.
• KÖPPEN, W. Das geographische system der klimate. In: KÖPPEN, W.; GEIGER, R. Handbuch der klimatologie. Berlin: Gebruder Borntraeger, 1936. v.1, part C, p.1-44.
• MARANHÃO, S. Modelagem de sistemas aplicada à produção de forragem e ao uso da água, nas condições atuais e sob mudanças climáticas, no semiárido brasileiro. 2021. 116f. Tese (Doutorado em Zootecnia) - Universidade Federal do Ceará, Fortaleza, CE.
• MATHUR, M.B.; VANDERWEELE, T.J. Sensitivity analysis for publication bias in meta‐analyses. J. R. Stat. Soc. Ser. C Appl. Stat., v.69, p.1091-1119, 2020.
• MORAIS, L.F.; CAVALCANTE, A.C.R.; AQUINO, D.N.; NOGUEIRA, F.H.M.; CANDIDO, M.J D. Spectral responses in rangelands and land cover change by livestock in regions of the Caatinga Biome, Brazil. Sci. Rep., v.11, p.18261, 2021.
• PEREIRA, H.R.; MESCHIATTI, M.C.; PIRES, R.C.M.; BLAIN, G.C. On the performance of three indices of agreement: an easy-to-use r-code for calculating the Willmott indices. Bragantia, v.77, p.394-403, 2018.
• RAINEY, C.; MCCASKEY, K. Estimating logit models with small samples. Polit. Sci. Res. Methods, v.9, p.549-564, 2021.
• ROBESON, S.M.; WILLMOTT, C.J. Decomposition of the mean absolute error (MAE) into systematic and unsystematic components. PLoS One, v.18, p.e0279774, 2023.
• SILVA, R.J.A.B.; SILVA, Y.J.A.B.; VAN STRAATEN, P. et al. Influence of parent material on soil chemical characteristics in a semi-arid tropical region of Northeast Brazil. Environ. Monit. Assess., v.194, p.1-21, 2022.
• STUTH, J.; SCHMITT, D.; ROWAN, R.C.; ANGERER, J.P.; ZANDER, K. PHYGROW users guide and technical documentation. Texas: Department of Rangeland Ecology and Management, 2003. Available in: https://research.agrilife.org/cnrit/wp-content/uploads/sites/21/2022/04/PHYGROW_user_guide.pdf Accessed in: 13 Sep. 2022.
  » https://research.agrilife.org/cnrit/wp-content/uploads/sites/21/2022/04/PHYGROW_user_guide.pdf
• SURYANTO, A.A.; MUQTADIR, A. Penerapan metode mean absolute error (MEA) dalam algoritma regresi linear untuk prediksi produksi padi. Saintekbu, v.11, p.78-83, 2019.
• TINÔCO, I.C.M.; BEZERRA, B.G.; LUCIO, O.S.; MELO, B.L. Characterization of rainfall patterns in the semiarid Brazil. Anuário Inst. Geociênc., v.41, p.397-409, 2018.
• WILLMOTT, C.J. Some comments on the evaluation of model performance. BAMS, v.63, p.1309-1313, 1982.

Publication Dates

• Publication in this collection
  14 July 2025
• Date of issue
  Jul-Aug 2025

History

• Received
  16 May 2024
• Accepted
  28 Oct 2024