Development of a digital image classification system to support technical assistance for Black Sigatoka detection

,


Introduction
In Colombia, plantain is one of the most important food products of the market basket; consumption per capita in the country is approximately 155 kg/person/year (RUÍZ; URUEÑA, 2009). Creating 286.000 permanent direct jobs a year and generating the livelihood of 57.000 families (ESPINAL et al. 2005).
Over the last decade, the plantain production sector's production has duplicated, going from 2. 604.550 tons in 2008604.550 tons in to 4.316.726 tons in 2018604.550 tons in (SIOC, 2018; however, this is conceived by an increase in the area planted, not by the improvement in the production of crops, since generally there are low yields, being the national average of 8 Ton/ha/year. Colombia produces 1.800.000 tons of high quality plantain, however, only 24% is produced by small farmers (JIMENEZ, 2018), who are faced with the challenge of generating production with few technological tools.
Another limiting factor is the incidence of diseases like the black sigatoka (Mycosphaerella fijiensis) (ALVAREZ et al. 2013), which reduces bunch weight by up to 50% and if control measures are not applied, it can cause losses of 100% of production, due to the deterioration of the quality of the fruit. (STOVER, 1980;MARÍN;ROMERO, 1992, MARÍN et al. 2003RODRÍGUEZ;GAVIRIA;CAYÓN, 2008). In addition, the control of the disease reaches 27% of the total cost of production. Black sigatoka produces small pinches that become greyish spots covering the entire leaf, decreasing the photosynthetic area and the plant's production capacity. Therefore, it is essential to detect the disease and generate an adequate control from the early stages of occurrence. For this, farmers in Colombia perform leaf removal processes seeking to reduce the incidence of the fungus; however, most decisions are made based on empirical visual reviews which results are not standardized with a measurement pattern.
An adequate prediction requires trained personnel, generating a technical criteria from a standardized metric, such as the Fouré scale; that divides the disease in 6 stages in accordance to the visual symptoms presented by the leaf affected. This standardization allows taking adequate control measures, however, the technical assistance service in Colombia has serious problems regarding coverage, quality, financing, institutional disarticulation, absence of information and little technological innovation (AGRONET, 2012;CASTAÑO, 2016;BONILLA, 2020).
In order to solve these problems, the National Agricultural Innovation System (SNIA) is created, under the law 1876 of 2017, which role is to generate support services to manage the development of capacities of agricultural processes and their articulation with the environment; proposing the access to information technologies as an approach (BONILLA, 2020).
Convolutional neural networks (CNN) have also been widely implemented in the disease detection processes (MOHANTY, et al. 2016). The Inception V3 network has been used to classify diseases in yucca (RAMCHARAN, et al. 2017). It is also common to use architectures such as AlexNet and Google that identify 26 diseases in 14 types of crops with a 99.35% efficiency (PRASANNA, et al. 2018) and the LeNet network reporting efficiencies of 98.67% classifying sigatoka.
However, all the techniques listed have two big problems. First, studies only validate their operation based on statistical tests in laboratory controlled conditions, do not contemplate validating the models in productive crops with real situations, the use and perception by the user is not verified either; if it is intended to generate an agricultural impact, it is necessary to carry out these validations. Second, there's no report of the construction of a database taken under a capture protocol that allows sampling all stages of the disease in real conditions (VARGAS, 2012;MODENESE, 2008). Nor is a labeling process reported with an agricultural criterion under a standardized metric such as the Fouré scale (OWOMUGISHA, 2019). For this reason, the classifiers listed only determine between healthy and diseased leaves, omitting that the disease presents different levels of severity, which requires a differentiated management.
In this paper, a support system for monitoring black sigatoka disease is presented, introducing a robust system that is validated from the design of the methodology to the final stage with the user. The main contributions of this article are: • Creation of a database with a capture and labeling protocol under expert criteria on four levels of severity.
• Design and validation of a methodology classifying black sigatoka on 4 levels of severity.
• Validation of the diagnosis system in real conditions with producers in the region.

Materials and methods
In this section the methodology designed for the construction of the tool to diagnose black sigatoka is described in detail. First, a collection of requirements by the farmers of the department of Risaralda (Colombia's Midwest) was performed. Then, a classification system was designed where the model that generated the highest efficiency of a set of methods proposed by the state of the art was used. The classification model was implemented in a mobile application that was socialized and validated with plantain producers from usability surveys. On Figure  1, the methodological diagram that will be explained in detail in this section is observed.

Collection of user requirements
The support system is intended to be a tool for the farmer with a correct validation on field; therefore, it was necessary to characterize the needs of the agricultural environment. For this, a requirement collection process was performed with the plantain producers associations of the department of Risaralda registered in the Secretaries of Agriculture of Apia, Balboa, Belén de Umbría, la Celia, Marsella, Mistrató, Pereira, Santa Rosa de Cabal, Santuario, surveying 220 producers. Questions related to the access level, management and perception of information technologies were asked.

Database
In the literature there are databases with images of the black sigatoka disease, however, they weren't built under an adequate capture and sampling protocol that allows to classify the leaves on different levels of severity. On the other hand, these have a reduced number of samples, ignoring the statistical relevance necessary to train machine learning models, or they were captured in laboratory settings, omitting all the noise that occurs when capturing a sample in real conditions. (VARGAS, 2012). Furthermore, the databases were not labeled with expert criteria from a standardized scale of visual patterns. Which makes it impossible to apply a structured methodology like the one proposed in this article, in order to replicate and compare similar methods. Due to this, an annotated database was built that fulfilled the required characteristics, for this, productive plantain crops were visited which characteristics are mentioned in Table 1. The proposed study is intended to take samples in productive crops under real conditions, complying with user requirements. Therefore, a protocol was defined to allow capturing the patterns generated by black sigatoka and to control the noise produced by elements unrelated to the disease. Below, the components of the protocol are listed: -Cell phone camera in automatic mode.
-Photographs of the plantain leaf taken against the light.
-Capture distance approximately 30 cm from the leaf.
-Sunny day from 10:00 am to 15:00 pm -Avoid any aspect that is not the plantain leaf, such as mud, shadows, drops and background.
To guarantee a correct understanding of this procedure, an illustrative video is performed where the operation of the application is explained in detail and which link is presented in the following link: https://www. youtube.com/watch?v=gW2YdBYwGpU&t=77s.
Sampling was carried out with medium to low-end cell phones, which cameras have a 9.6MP (4128x2322) and 13Mp (3120x4160) resolution. This image size is adequate as it allows sampling all possible stages of black sigatoka.
The images were then labeled following the Fouré scale as a technical criterion. From this metric, 4 classes are generated according to the control actions to avoid economic damage. The early levels of infection demand cultural management (partial cuts of the leaf), without affecting economic performance. Here the learning system should give higher priority, as patterns are not easily detected.
The "Healthy" class corresponds to the absence of the disease. In this stage the farmer must carry out constant monitoring to warn in time the appearance of the disease. The "Low" class corresponds to stages 1 and 2 under Fouré. In this stage the farmer must perform low-cost control practices, with the purpose of avoiding a greater advance of the disease, at this point no economic losses are generated for the producer. The "Medium" class corresponds to stages 3 and 4 under Fouré. In this stage the farmer must remove the leaves of the affected parts more regularly and carry out chemical control. Finally, the high class corresponds to stages 5 and 6 under Fouré, at this point the leaf must be cut completely, generating economic losses. This class is easily detectable by the farmer; therefore, priority should be given to the appearance of the disease in the early stages (ITT2, 2016). Figure 2 presents a general diagram of the capture and labeling of the database.

Image processing
This methodological instrument processes the images with the purpose of highlighting the patterns produced by black sigatoka, for this an image-processing algorithm was implemented that increases the contrast of the pixels where the disease is found. In this, a Gaussian filter with sigma 5 is applied. Then a conversion from RGB to HSV is carried out and the V channel is selected, dividing the image into windows of 500x500 pixels. For each window, every pixel outside the µ ± 3σ range is considered sick and a gain of 3 is applied saturating its value. Finally, the windows are joined, obtaining the image in HSV, and they are also transformed to gray scale. In Figure 3 the result of applying the proposed algorithm is observed. The reader may consult (OPENCV, 2014) to see in detail the calculation and conversion process of color spaces.

Descriptor extraction
In this stage, the main descriptors reported by the state of the art are computed to identify patterns in an image that has lighting variations as a result of an uncontrolled environment such as a crop (PRABUKUMAR et al. 2010;OWOMUGISHA, 2014). This process not only extracts the color properties in the image but also transforms space frequency, generating a robust process to lighting changes. This section is presented in Figure 4 and is explained below.
Rev. Bras. Frutic., Jaboticabal, 2020, v. 43, n. 2 (e-681)  The first descriptor is the Discrete Wavelet Transformed (DWT) in two dimensions using the fourth order Symlets family, from this the horizontal (j), vertical (m) and diagonal (u) coefficient matrixes are extracted (PYWAVELETS, 2019). The mean statistical moments are calculated for each matrix (u), standard deviation (s), bias (d) and kurtosis (k) obtaining the descriptor DWT (j,m,u) given by: The second descriptor is the Fast Fourier Transformed in two dimensions (SCIPY, 2019), that compute the matrix (y). The same statistical moments of DWT are calculated to this matrix, thus obtaining the descriptor F y given by: The last calculation performed at this stage is a histogram with 15 regular intervals (bins) (SCIPY, 2019), from which the descriptor H is obtained. Finally, the x i data vector is obtained, resulting from concatenating the three proposed descriptors for each proposed representation system, in this manner:

Classification of the disease
In this stage, the aim is to establish a classification method with the best relationship between efficiency and computational cost that allows it to be implemented in a user interface. For this the Ensemble Boosted Trees machine learning is proposed using the adaptive boosting methodology, with 30 learning cycles altogether (SCIKITLEARN, 2019). This classifier is chosen because the assembled models have obtained high efficiencies in classifying diseases. Another aspect to be taken into account is the imbalance possessed by the database due to the exponential growth of the disease. In these cases the state of the art suggests using weighted classification methods, therefore, a hierarchical classifier is used. For this, a bi-class model is trained to classify samples in "High" and "Not High" status. Then another model with a multiclass approach is trained using a number of balanced samples to generate labels "Medium," "Low," "Healthy." This avoids generating classifiers biased towards the classes with the highest number of samples.

Mobile application
A mobile application for Android is designed, with client-server architecture, using Kotlin language. The design was carried out following the requirement surveys; therefore, the application presents an intuitive design, simple to use and low computational cost.
The client (mobile device) has a visual interface for interaction with the users, where sampled images are entered with the stipulated capture protocol. The application, takes as reference the Stover sampling process (CUEVAS, et al. 2015) to select the images to capture, for this reason it requests to sample the leaves 3,4 and 5.
Where i is the color channel to compute.
Rev. Bras. Frutic., Jaboticabal, 2020, v. 43, n. 2: (e-681) Every image is sent to a server using the TCP/IP protocol through the 8080 port, where they're processed using the classification methodology introduced in this article. The predictions obtained are sent to the client, presenting the status of the analyzed region and an agronomic management recommendation is made to the farmer. In Figure 5 the application's general diagram is observed.
This architecture allows running the application on low-end devices, since the client is only responsible for the sampling and presentation of results, while the processing or the information is performed in the server.

Validation
The support system for the detection of Sigatoka needs to be validated from three aspects: efficiency of the classification methodology, operation of the tool in the field and perception by agricultural users. Each section will be explained below.

Statistical validation
In this stage the choice of the classification model is corroborated, comparing our classifier to machine learning methods widely used in the state of the art in disease detection, such as SVM, KNN, tree decision. In addition, the machine learning method is compared against a deep learning methodology that contemplates the use of convolutional neural networks. This modern method, divides the image in small segments of 96x96, to be classified by a convolutional neural network LeNet (AMARA, et al. 2017). The previous allows having a map of severity of the leaf, later a decision tree is used that weights the frequency of appearance to obtain the overall severity of the image. The Lenet architecture was chosen because of the good results generated in black sigatoka classification on state of the art works (JIHEN, et al. 2017).
All classification tests are validated using crossvalidation with 100 folds where the samples are fractioned in a random manner following a Gaussian distribution. In this article 70% of the data is used for training and the remainder for validation. All the tests are carried out in a computer with an Intel Core i7 5500U processor and 8Gb of RAM memory using Python language.

Field Validation
Once the classification methodology is verified and implemented in the application, a socialization process is followed, where the operation of the system is validated on field, for this, workshops were arranged with the producer associations in the abovementioned municipalities and surveys were conducted with 56 small plantain producers. Attendees were explained the capture protocol with the system generated, the Stover sampling procedure and they were proposed to validate the status of a crop sector using the diagnostic tool, in total farmers captured 192 samples, in the results section the confusion matrix resulting from this test is presented.

User validation
After performing the use and image capture tests, TAM (technology acceptance model) surveys were applied allowing to evaluate the utility, effectiveness, productivity and performance of the support system. Rev. Bras. Frutic., Jaboticabal, 2020, v. 43, n. 2 (e-681)

Results and discussion
In this section, the results obtained in the collection of user requirements, the design of the classification methodology and the validation of the system with farmers in the region are presented.
Characterization revealed that 91% of the surveyed or some member of their family, owns a device with a camera and Internet access. Also, 86% uses mobile software, however, only 20% has used mobile applications related to agriculture. 96% of the surveyed considers easy to have an application that selects the images and to have a button to open the camera. While 97% consider useful a mobile application that indicates the level of disease and helps making management decisions.
They were also asked what processes should the system support through the new information technologies, to which 33% considered useful to indicate the status of the disease and 36% generate recommendations for taking measures in accordance to the diagnosis. These low results reveal the high ignorance about technologies and corroborate the need to include agricultural tools that improve this perception. These technological elements should be developed for use in mobile devices already owned by the farmers, which facilitates the use and avoids incurring in high technology acquisition costs. The graphs summarizing the results obtained are presented on Figure  6. The characterization surveys confirm that the farmers have basic technological equipment and Internet access; they also have a great disposition and know the need to include technologies to support the country. However, ignorance of technological tools is common. Support systems need to be made from agricultural needs, easy access, simple use, ease of installation on mobile devices.
Regarding the classification methodology, on Table 2, the main diagonals of each classifier and also the computing time needed to carry out the crossed validation of each model are presented.
The results show that the proposed model presents a greater efficiency on the High and Healthy classes, but the SVM and the CNN LeNet surpass it on the Medium class. Despite the foregoing, the Ensemble Boosted Tree classifier generates the most consistency in all the classes and generates the best separation between the Low and Healthy class.
Separability in the first classes is fundamental because it's the point of the disease where the farmer can apply all the adequate management tools and avoid economic losses from happening. If the disease is detected in an advanced stage like the medium class, control actions are less effective.
Rev. Bras. Frutic., Jaboticabal, 2020, v. 43, n. 2: (e-681) The disease advances rapidly in tropical rainy weathers like Colombia, therefore, few leaves are in a totally healthy state in the region's crops. This leads to the database presenting imbalance between its classes and requires applying techniques to avoid training biased models. The use of a hierarchical classification model was configured as an effective method to treat imbalance and allowed obtaining the best results in the detection of the disease.
The CNN LeNet presents the lowest efficiency in the healthy class, in this study this efficiency is to the fact that the differentiating traces between the low and healthy class are barely perceptible to the human eye and the resizing process produces loss of information, which makes it impossible for the network to generate separability. Given the importance of separation between the low and healthy class the CNN Lenet isn't adequate for the help system proposed in this article. In addition, it is highlighted that using a set of images with higher resolution for training increases the number of weights and operations that the model must compute, making its use unfeasible in a low-cost device such as a smartphone (NUMERENTUR, 2018).
After verifying the laboratory results, field validation is performed. The captures made by the farmers which tested the system in productive crops, corroborate efficiency in the laboratory. The tool correctly classified 82.02% of the images processed. On Table 3 the confusion matrix is presented presenting in detail the test's result.
The support system was designed bearing in mind all field conditions, this allowed achieving similar efficiencies between laboratory validation and field tests. It is necessary that the classification methodologies have verification stages in the field and from the user where the efficiency and operation are corroborated under all possible real conditions. The above allows this research to be useful for this sector, it becomes a product with high potential of use and generate impact on the target population of the research. The TAM survey results reveal a high acceptability of the support system, 80% of the surveyed agree or highly agree with the utility perception, 87% has a high perception of affectivity, 87% consider a high perception in terms of performance, finally 84% positively graded the perception of productivity; as presented on Figure 7.

Conclusions
Mass use of smart phones, including their penetration in agricultural contexts, and the vertiginous advance in its software capacities, have made them attractive for the government programs, such as agricultural technical assistance, to be supported by mobile applications that maximize their utility and make the resources destined for this purpose more efficient, primarily in countries like Colombia where access to certain rural areas is very difficult.
In this article, a support system is presented for the black sigatoka detection process that allows classifying stages of the disease from its earliest stages, which would allow a farmer to take corrective action, improving product quality and avoiding economic losses. This study contributes with the creation of a public database, labeled by an expert criterion on 4 levels of severity. This generates a new approach for classification studies where more information for the control of the disease is produced. It also allows researchers on machine learning, to have truthful information that allows research to be directed to useful products for the farmer.
The classification method proposed allows maintaining an adequate efficiency in the first stages of the disease, which is not achieved using other classification techniques proposed in the state of the art, as observed in Table 3. Detecting the disease in the first stages is of vital importance, since it allows the farmer to take preventive action, avoiding economic losses.
The Ensemble Boosted Tree model, following an adaptive boosting methodology, generated an 80.58%±1.05% efficiency. As future work, it is necessary to evaluate the performance of methodologies such as bagging, in addition to comparing it regarding methods composed by different types of classifiers.
Given the problems of technical assistance in the Colombian territory, technological elements become an important tool to improve the productive process. However, these tools do not substitute the evaluation and diagnostic process performed by an expert. The application presented generates a suggestion from a diagnosis an 80.58±1.05 efficiency. This becomes an important help for low-income farmers who have no access to a technical assistance service.