Validation of a Flying Competence Scale for Aircraft Pilots

Harold Toring Ranel Merino Benatiro Nonita Legaspi Maria Luz Cahayagan Ronaldo Felix Adriane Adaptar Sylvester Tan Cortes About the authors

ABSTRACT

This study aims to validate a scale that evaluates the flying competence of aircraft pilots. The scale was developed by pilots in an aerospace university and was approved by the Civil Aviation Authority of the Philippines. To do so, the scale was administered to 288 pilots holding different levels of licenses. The data obtained were subjected to Exploratory Factor Analysis (EFA) and created a three-factor model. The factors are set of flying skills named as instrument flight (Factor 1), basic attitude flying (Factor 2), and instrument landing system (Factor 3). The model was confirmed utilizing the resulting values of five goodness of fit indices (GFIs) generated by the Confirmatory Factor Analysis (CFA). Only comparative fit index, Tucker–Lewis index, and Standardized Root Mean Squared Residual resulted to values falling within the thresholds. These three GFIs are already adequate to confirm that the model is relatively good fit. The standardized factor loadings (SFLs) and composite reliability (CR) were also excellent, thus, establishing convergent validity. Also, the estimated average variance extracted and Cronbach’s alpha of all factors provided evidence of discriminant validity and reliability, respectively. In conclusion, this scale is valid and reliable to evaluate the pilot’s performance in flying an aircraft.

Keywords
Factor analysis; Flight training; Scale development

INTRODUCTION

The growing demand for air transport will result to the recruitment of 635,000 new pilots in the next 20 years by the commercial airlines (Wall and Tangel 2018Wall R, Tangel A (2018) Facing a Critical Pilot Shortage, Airlines Scramble to Hire New Pilots. The Wall Street Journal. [accessed August 2 2021]. https://www.wsj.com/articles/pilot-shortage-spurs-hiring-spree-1533720602
https://www.wsj.com/articles/pilot-short...
). However, the number of graduating pilots over the years has not been growing correspondingly with the estimated number of pilots to be recruited in the future (Smith 2016Smith P (2016) Fact and fallacy of the “pilot shortage”. Aerophilia Enterprises. [accessed August 1 2021]. https://askthepilot.com/pilot-shortage/
https://askthepilot.com/pilot-shortage/...
). There are several reasons that have been pointed out in the declining interest of young individuals to pursue careers in Science, Technology, Engineering, and Mathematics, particularly in aviation, such as unstable remuneration, high training cost (Valenta 2018Valenta V (2018) Effects of airline industry growth on pilot training. Magazine of Aviation Development 6(4):52-56. https://doi.org/10.14311/MAD.2018.04.06
https://doi.org/10.14311/MAD.2018.04.06...
), risky battle associated with occupational health concerns, job exploitation (Bennett 2006Bennett SA (2006) A longitudinal ethnographic study of aircrews’ lived experience offlying operations at a low-cost airline. Risk Manag 8:92-117. https://doi.org/10.1057/palgrave.rm.8250009
https://doi.org/10.1057/palgrave.rm.8250...
; Butcher 2003Butcher JN (2003) Assessing pilots with ‘the wrong stuff’: A call for research on emotional health factors in commercial aviators. Int J Sel Assess 10(1-2):168-184. https://psycnet.apa.org/doi/10.1111/1468-2389.00204
https://psycnet.apa.org/doi/10.1111/1468...
), and perceptions regarding the desirability of the airline pilot as a career path (Lutte and Lovelace 2016Lutte R, Lovelace K (2016) Airline pilot supply in the US: Factors influencing the collegiate pilot pipeline. JATE 6(1):53-63. https://doi.org/10.7771/2159-6670.1148
https://doi.org/10.7771/2159-6670.1148...
). This declining number of graduating pilots may consequently lead to a shortage in the workforce in the following years, and it would have a domino effect on the decreasing number of flights, loss of revenues, and closing of some regional airlines (Crouch 2020Crouch V (2020) Analysis of the airline pilot shortage. Scientia et Humanitas 10:92-103.). In this regard, airline companies have implemented various strategies aimed at reattracting individuals to become pilots, such as increasing base salary and bonuses (Tulis 2017Tulis D (2017) Enrollment up at many aviation colleges. [accessed August 2 2021]. https://www.aopa.org/news-and-media/all-news/2017/december/11/enrollment-up-at-many-aviation-colleges
https://www.aopa.org/news-and-media/all-...
), addressing lifestyle and career path concerns (Lutte 2018Lutte B (2018) Pilot supply at the regional airlines: Airline response to the changing environment and the impact on pilot hiring. J Aviat/Aerosp Educ Res 27(1):1. https://doi.org/10.15394/jaaer.2018.1749
https://doi.org/10.15394/jaaer.2018.1749...
), and innovative training and recruitment programs (Crouch 2020Crouch V (2020) Analysis of the airline pilot shortage. Scientia et Humanitas 10:92-103.).

Meanwhile, flying schools and universities have their roles as well in attracting young people to pursue careers in aviation and eventually help them complete the degree program. Kearns (2017)Kearns S (2017) Attracting and informing the next generation of aviation professionals: The online ‘Fundamentals of Air Transport System’ course. Montreal: International Civil Aviation Organization (ICAO). reported that a crucial issue rarely discussed in aviation education is the high attrition rate of students spurring at more than 50%, meaning more than half of the student pilots do not graduate from the degree. She cited three probable causes of attrition, namely: (a) lack of financial resources, (b) difficulties in learning materials of the courses, and (c) realization that they are not suited to the lifestyle of their future professional role. Meanwhile, to attract young people to enroll in flying schools, Jin (2019)Jin J (2019) An examination of student pilot school choice factors for flight schools in California. Ann Arbor, USA: ProQuest LLC. https://doi.org/10.13140/RG.2.2.17545.19040
https://doi.org/10.13140/RG.2.2.17545.19...
examined the factors affecting students’ decision to enroll in aviation school, including but not limited to (a) quality of training, (b) availability of flying opportunities, (c) cost of training, (d) safety records of the programs, and (e) reputation of certificated flight instructors. While all five factors are deemed relevant in attracting enrollees in flying schools, the quality of training has been consistently reported in the literature as a major factor influencing their decisions (e.g., Aircraft Owners and Pilots Association 2010Aircraft Owners and Pilots Association (2010) The flight training experience: A survey of students, pilots, and instructors. Frederick: AOPA. [accessed August 1 2021]. http://download.aopa.org/epilot/2011/AOPA_Research-The_Flight_Training_Experience.pdf
http://download.aopa.org/epilot/2011/AOP...
; Steckel et al. 2010Steckel R, Lercel D, Matsuo H (2010) Factors that influence an undergraduate student to choose a career in aviation, and enroll in the Aviation Science Program at Parks College of Engineering, Aviation and Technology. Coll Aviat Rev 28(2):69-83. https://doi.org/10.22488/okstate.18.100412
https://doi.org/10.22488/okstate.18.1004...
). Rony (2019)Rony HAZ (2019) Explorative study on the factors influencing aviation education in developing countries. ICAO Scientific Review 1:3-31. argued that this quality training might be attained by improving the curriculum, pedagogical competencies of training instructors, and updating resources. In addition, sound evaluation and assessment tools measuring students’ competence in flying are equally important. How students perform well during assessment speaks of the quality of training they received from schools (Malagas et al. 2017Malagas K, Fragoudaki A, Kourousis K, Nikitakos N (2017) Higher education aviation programs in Greece: a missed opportunity or a challenge to meet? J Aerosp Technol Manag 9(4):510-518. https://doi.org/10.5028/jatm.v9i4.888
https://doi.org/10.5028/jatm.v9i4.888...
). These assessment tools should particularly evaluate students’ essential knowledge, skills, and attitudes related to the intended program outcomes (Letassy et al. 2015Letassy NA, Medina MS, Britton ML, Dennis V, Draugalis JR (2015) A progressive, collaborative process to improve a curriculum and define an assessment program. Am J Pharm Educ 79(4):55. https://doi.org/10.5688/ajpe79455
https://doi.org/10.5688/ajpe79455...
).

However, the issue regarding the limited availability of valid and reliable assessment tools in aviation education is evident. This, in effect, poses a greater challenge when ensuring that the offered pilot trainings are of the highest quality. The existing assessment tools such as Aviation Safety Climate Scale (Evans et al. 2007Evans B, Glendon AI, Creed PA (2007) Development and initial validation of an Aviation Safety Climate Scale. J Safety Res 38(6):675-682. https://doi.org/10.1016/j.jsr.2007.09.005
https://doi.org/10.1016/j.jsr.2007.09.00...
), Factors Affecting Situation Awareness (FASA) Questionnaire (Banbury et al. 2007Banbury S, Dudfield H, Hoermann HJ, Soll H (2007) FASA: Development and validation of a novel measure to assess the effectiveness of commercial airline pilot situation awareness training. Int J Aviat Psychol 17(2):131-152. https://doi.org/10.1080/10508410701328557
https://doi.org/10.1080/1050841070132855...
), Willingness to Fly in an Aircraft Scale, and Willingness to Pilot an Aircraft Scale (Rice et al. 2020Rice S, Winter SR, Capps J, Trombley J, Robbins J, Milner M, Lamb TL (2020) Creation of two valid scales: Willingness to fly in an aircraft and willingness to pilot an aircraft. International Int J Aviat Aeronaut Aerosp 7(1):5. https://doi.org/10.15394/ijaaa.2020.1440
https://doi.org/10.15394/ijaaa.2020.1440...
) are not enough. These could not assess all essential competencies that pilots are expected to acquire after the training, particularly maneuvering an aircraft. There may be proposed objective test, execution test, observation techniques, and atittude-opinion survey by Ruiz et al. (2014)Ruiz S, Aguado C, Moreno R (2014) Educational simulation in practice: A teaching experience using a flight simulator. J Technol Sci Educ 4(3):181-200. https://doi.org/10.3926/jotse.129
https://doi.org/10.3926/jotse.129...
, which aimed to evaluate basic knowledge, skills, and attitude of student pilots on air navigation, but these have validity and reliability issues. In addition, the number of items in these scales are limited; thus, some other essential knowledge and skills which pilots are expected to demonstrate are excluded.

In this regard, the present study aims to contribute to the literature by validating a locally-made scale intended to evaluate basic competence in flying of aircraft pilots. This competence scale is approved by the Civil Aviation Authority of the Philippines (CAAP), which is the country’s national aviation authority responsible for implementing policies on civil aviation to ensure safe, economical, and efficient air travel of airline passengers, and has been used for instructor rating, peer rating, personal rating to pilots undergoing training on flying in one aerospace university in Central Visayas, Philippines. However, reports concerning its validation and reliability tests are not yet communicated. The scale aims to facilitate effective assessment and evaluation of pilots undergoing training on flying an aircraft.

METHODOLOGY

Setting and Participants

The participants comprised 288 pilots from various aviation colleges and universities in the Visayan region of the Philippines. They had satisfied one or more of the following inclusion criteria in order to be part of the study: (a) had an actual flying experience, (b) had an educational simulation flying experience, or (c) acquired at least a student pilot license. Figure 1 shows then the complete distribution of qualified participants when grouped according to age, sex, number of license(s) held, type of license(s) acquired, whether experienced in actual flying or not, the total number of flying hours for pilots with actual flying experience, whether experienced in flying through educational simulation or not, and total number of flying hours through flight simulation.

Figure 1
Distribution of the personal and professional profiles of participating pilots.

The majority of those who participated in the survey were at ages between 21 to 24 years old (n = 185), followed by ages between 17 to 20 years old (n = 83). In terms of the distribution of participants when grouped according to sex, 234 were males and 54 were females. The types of licenses held by these pilots are majority student pilot (n = 219) followed by private pilot (n = 50), commercial pilot (n = 27), and instrument rated (n = 18). There were only four with ground and flight instructor licenses. As to the number of licenses acquired, generally, 264 had one license. The rest acquired two (n = 16), three (n = 6), and four licenses (n = 2). Finally, 234 had actual flying experience, and 250 had educational simulation flying experience. It can be noted that the number of pilots having actual flying experience is less than the total number of pilots participating in the survey. The explanation rests on the fact that a student pilot license is granted even with flying through flight simulation only. In addition, the number of pilots with experience flying through educational simulation are less than the total number of pilot participants. The explanation for this is that it was only in 2016 when pilots were required to do flight simulation training before actual flying. Some of the pilots who responded in the survey got their licenses before the forecited year.

Instrument

The Flying Competence Scale for Pilots was developed by a group of pilots in an aerospace university. It was approved by the CAAP. With its approval in 2014, the experts’ validity has been theoretically established, meaning the competences or skills listed as items in the scale was checked and approved by CAAP’s personnel. Thus, the scale has been used by the university as an assessment tool to evaluate training performance and for licensing of pilots, such as granting the following licenses: student pilot, private pilot, commercial pilot, among others. However, reports on reliability and other forms of validity are not yet available. Thus, the study aims to establish its psychometric properties.

The scale consists of 34 items covering six general competences or skills that aircraft pilots are expected to acquire or master. In particular, these skills refer to interpreting data from several instruments to maneuver an aircraft without outside visual or ground reference properly. These skill sets involve Basic Attitude Flying (BAF) (8 items), Standard Instrument Departure or Standard Terminal Arrival (SID/STAR) Procedure (5 items), Non-Directional Beacon or Radio Magnetic Indicator (NDB/RMI)Procedure (7 items), Very High Frequency Omnidirectional Range or with the Horizontal Situation Indicator (VOR/HSI) (7 items), Localizer and Distance Measuring Equipment (LOC & DME) Approach (3 items), and Instrument Landing System (ILS) (4 items). Each of the general skills has subskills which are listed as items. For example, BAF has a list of the following skills: scanning, airspeed control, climbs, descents, among others. The listing of these specific subskills reflects the scale’s flexibility as it can be adapted in different modes of evaluation (e.g., self, peer, and teacher to student evaluation). In other words, the specific skills are just indicated in a word or phrase manner rather than stating it in a sentence form which usually starts with the phrase “I can.” This method limits the utility of the scale, meaning it can only be used for self-evaluation. Finally, the response scaling of each item ranged from 1 = fail to 6 = excellent.

Procedure

The researchers provided oral and written information to the prospected participants regarding the purpose and background, procedures, risks or discomforts, the extent of confidentiality, and the benefits of the study. Meanwhile, the parents of prospected participants aged 17 were also given the same information. The participants with legal age who wished to participate signed an informed consent form, while participants below legal age signed an assent form and submitted an informed consent form signed by their parents. Then, an online survey through Google Form was conducted between August and September 2021. Within the survey form, it was indicated in the preliminary portion before the participants could answer the scale that their participation in the study was entirely voluntary with no incentives, and they could withdraw without penalty or loss on their end. The survey had three parts: (a) instructions, (b) eliciting the pilot’s demographic and professional profiles, and (c) eliciting participating pilot’s self-perceived levels of flying skills. It took approximately 10 to 20 min for each pilot participant to complete the survey. Upon completion, participants were assured about the confidentiality and anonymity of their responses. Their responses were generated as a spreadsheet, and there were no manipulation or statistical correction made prior to data analysis.

Data Analysis

The determination of whether Exploratory Factor Analysis (EFA) could proceed was done through performing two preliminary tests: Bartlett’s test of sphericity (BTS) and Kaiser–Meyer–Olkin (KMO) Measure of Sampling Adequacy. To justify factor analysis, Bartlett’s test should be significant (p < 0.05) and KMO values should exceed 0.60 (Tabachnick and Fidell 2019Tabachnick BG, Fidell LS (2019) Using Multivariate Statistics (7th ed.). Boston: Pearson.). Next, EFA was performed to explore the structure of the underlying relationships between the items in the scale and to determine whether subscales or factors could be created. Principal Component Analysis (PCA) using Varimax rotation and scree plots were used to extract the factors. The factors with Eigen values > 1.0 were included. Meanwhile, an item was retained in a construct if its criterion loading was > 0.6. Taherdoost et al. (2014)Taherdoost H, Sahibuddin S, Jalaliyoon N (2014) Exploratory factor analysis; concepts and theory. Adv Appl Pure Math 27:375-382. argued that the minimum loading should be 0.5 or higher. Hence, it was increased by 0.1 for a pragmatic reason. Finally, the reliability coefficients of the entire scale and its subscales were determined using Chronbach’s alpha, of which 0.7 should be achieved (Tavakol and Dennick 2011Tavakol M, Dennick R (2011) Making sense of Cronbach’s alpha. Int J Med Educ 2:53-55. https://doi.org/10.5116/ijme.4dfb.8dfd
https://doi.org/10.5116/ijme.4dfb.8dfd...
). The program IBM SPSS Statistics version 25 was used to perform all these tests.

Subsequently, Confirmatory Factor Analysis (CFA) using the maximum likelihood method was used to estimate parameters. The analysis was performed in IBM SPSS AMOS version 22. It began with determining the estimation results through the t-value and standardized factor loading (SFL) of each item. The acceptable t-value is ≥ 1.96 or practically 2.00, while the SFL is ≥ 0.7 (Gefen et al. 2000Gefen D, Straub D, Boudreau MC (2000) Structural Equation Modeling and Regression: Guidelines for research practice. Communications of the Association for Information Systems 4:7. https://doi.org/10.17705/1CAIS.00407
https://doi.org/10.17705/1CAIS.00407...
; Hair Jr. et al. 2014Hair Jr. JF, Black WC, Babin BJ, Anderson RE (2014) Multivariate data analysis (7th ed.). Harlow: Pearson Education Limited.; Kline 2016Kline RB (2016) Principles and practice of structural equation modeling (4th ed.). New York: Guilford Press.). Once these values were met, the scale was available for examining overall model data fit using several goodness of fit indices (GFIs) recommended by Hu and Betler (1999)Hu LT, Betler PM (1999) Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal 6(1):1-55.. These are Comparative Fit Index (CFI), Tucker–Lewis Index (TLI) or Non-Normed Fit Index (NNFI), Standardized Root Mean Squared Residual (SRMR), root mean square error of approximation (RMSEA), and chi-square/df ratio. The following values of five selected indices are desired to be achieved to have a relatively good model-data fit. The CFI should be > 0.80 (Garson 2006Garson GD (2006) Structural equation modeling. North Carolina: G. David Garson and Statistical Associates Publishing.), TLI should be > 0.85 (Sharma et al. 2005Sharma S, Mukherjee S, Kumar A, Dillon WR (2005) A simulation study to investigate the use of cutoff values for assessing model fit in covariance structure models. J Bus Res 58(7):935-943. https://doi.org/10.1016/j.jbusres.2003.10.007
https://doi.org/10.1016/j.jbusres.2003.1...
), RMSEA should be < 0.08 (Kenny et al. 2014Kenny DA, Kaniskan B, McCoach DB (2014) The performance of RMSEA in models with small degrees of freedom. Sociological Methods and Research 44(3):1-22.), SRMR should be ≤ 0.08 (Hu and Betler 1999Hu LT, Betler PM (1999) Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal 6(1):1-55.), and chi-square/df ratio should be < 2.0 (Kline 1998Kline RB (1998) Principles and practice of structural equation modeling. New York: Guilford Press.).

Finally, the convergent and the discriminant validities of the scale were established. The evidences for convergent validity were SFL and composite reliability (CR) of which a value ≥ 0.7 for both measures is considered good (Gefen et al. 2000Gefen D, Straub D, Boudreau MC (2000) Structural Equation Modeling and Regression: Guidelines for research practice. Communications of the Association for Information Systems 4:7. https://doi.org/10.17705/1CAIS.00407
https://doi.org/10.17705/1CAIS.00407...
). With regard to the discriminant validity of the scale, the average variance extracted (AVE) estimate should be higher than the squared correlation between the two constructs (Hair Jr. et al. 2014Hair Jr. JF, Black WC, Babin BJ, Anderson RE (2014) Multivariate data analysis (7th ed.). Harlow: Pearson Education Limited.).

RESULTS AND DISCUSSION

Exploratory Factor Analysis

The KMO statistic resulted to a value of 0.960, which is much higher than 0.6 and very close to 1.0. This value indicates that the required sample size for CFA of the scale was satisfied. Meanwhile, the Bartlett’s test was significant (p < 0.000). In this regard, both KMO and Bartlett’s tests suggested that the scale was available for factor analysis. Table 1 shows that the EFA of the 34-item scale identified three factors, which explained 86.064% of the variance in the data. The first factor accounted for 48.05% of the total variance with an eigenvalue of 23.01 and loaded 22 items. The criterion loading and communality values of items loading in this factor, respectively, range from 0.643 to 0.895 and 0.718 to 0.929. The items previously assigned in LOC & DME Approach, V.O.R/H.S.I., NDB/RMI Procedure, and SID/STAR Procedure all loaded under this factor. As such, the factor can be named “Instrument Flight.” The second factor explained 26.91% of the total variance with an eigenvalue of 3.76. The criterion loading and communality values of items loading in this factor, respectively, range from 0.801 to 0.881 and 0.836 to 0.913. The eight items loading under this factor were those earlier assigned in basic attitude flying, hence, the name of the factor is named after it. The third factor accounted for 11.102% of the total variance with an eigen value of 2.50. The criterion loading and communality values of items loading in this factor, respectively, range from 0.957 to 0.973 and 0.919 to 0.951. All four items under instrument landing system load in this factor, hence, the name of the factor is also named after it.

Table 1
The EFA rotated extracted factors with item loadings, communality values, and Cronbach’s alpha.

Reliability

Reliability pertains to the property of the scale to measure consistently and is closely related to scale validity. Tavakol and Dennick (2011)Tavakol M, Dennick R (2011) Making sense of Cronbach’s alpha. Int J Med Educ 2:53-55. https://doi.org/10.5116/ijme.4dfb.8dfd
https://doi.org/10.5116/ijme.4dfb.8dfd...
reasoned that a scale could not be valid if it is not reliable, but it could be reliable even if it is not valid. Hence, reliability should be established. In the present study, the scale’s reliability coefficient was calculated using Cronbach’s alpha. The reference value to determine whether the entire scale and subscales is/are reliable or not should be 0.70 or higher. Table 1 shows the Cronbach’s alpha coefficient of each subscale or factor and item. In the event that a factor or an item within the factor obtains a Cronbach’s alpha of below 0.7 only, it will be deleted to increase the reliability of the scale. The results reflect that the largest value recorded was in Factor 1 (Instrument Flight) with 0.990, while the lowest value recorded was in Factor 3 (Instrument Landing System) with 0.977. This range of Cronbach’s alpha coefficients reveals satisfactory reliability of the subscales with reference to the proposed value indicated above. Meanwhile, the overall Cronbach’s alpha coefficient of the entire scale is 0.979, suggesting that it is very reliable.

Confirmatory Factor Analysis

Estimation Results

The analysis of factor load estimation was done on each observed variable using its resulting critical ratio or t-value and SFL before testing the overall model-data fit. Table 2 shows the results of factor load estimation of the model of which all t-values are greater than 1.96 while all SFL values are greater than 0.7. The observed minimum t-value is 16.104 (SIDSTAR1 – Factor 1) and the maximum is 38.127 (ILS3 – Factor 3). Meanwhile, the recorded minimum SFL value is 0.795 (SIDSTAR2 – Factor 1) and the maximum is 0.974 (ILS3 – Factor 3). In summary, there is no offending estimate observed in the factor load estimation results. This justifies the analysis of the overall model data fit of the scale.

Table 2
Factor load estimation results from the model.

Overall Model Fit

There were five GFIs examined to determine the overall model fit of the CFA results: CFI, TLI, SRMR, RMSEA, and chi-square/df ratio as indicated in Fig. 2. The CFI value of this sample, 0.817, is above 0.8. Meanwhile, the TLI is 0.804, a little below the criterion of 0.85. As to the SRMR, the observed value of 0.0474 is less than the margin of 0.8. These three model fit indices suggest an acceptable fit to the three-factor model. However, the chi-square/df ratio resulted to 7.699. This resulting value is beyond the minimum criteria. Nonetheless, this may be considered tolerable because chi-square/df ratio is sensitive to a large sample size (Kline 2016Kline RB (2016) Principles and practice of structural equation modeling (4th ed.). New York: Guilford Press.). Its value can be high even if the model is good (Karakaya-Ozyer and Aksu-Dunya 2018Karakaya-Ozyer K, Aksu-Dunya B (2018) A Review of Structural Equation Modeling Applications in Turkish Educational Science Literature, 2010-2015. International Journal of Research in Education and Science 4(1):279-291.). The overemphasis on chi-square model could also result in a preference for a smaller sample size in which the null hypothesis is not rejected. This instance may likely lead to the acceptance of poor models and yield inaccurate estimation of parameters (Alavi et al. 2020Alavi M, Visentin DC, Thapa DK, Hunt GE, Watson R, Cleary M (2020) Chi-square for model fit in confirmatory factor analysis. J Adv Nurs 76(9):2209-2211. https://doi.org/10.1111/jan.14399
https://doi.org/10.1111/jan.14399...
). In the same manner, the RMSEA resulted in a poor or offending estimate at 0.158. Lai and Green (2016)Lai K, Green S (2016) The problem with having two watches: Assessment of fit when RMSEA and CFI disgaree. Multivar Behav Res 51(2-3):220-239. explained that this index may sometimes give a contrasting evaluation of model fit with CFI. Thus, the CFI value may be selected as a reference when evaluating model fit over RMSEA. Further, RMSEA is sensitive to factor loading specification, meaning its value is influenced by the level of factor loadings. The decrease in mean RMSEA value monotonically leads to the decrease in factor loading values (Heene et al. 2011Heene M, Hilbert S, Draxler C, Ziegler M, Bühner M (2011) Masking misfit in confirmatory factor analysis by increasing unique variances: A cautionary note on the usefulness of cutoff values of fit indices. Psychol Methods 16(3):319-336. https://doi.org/10.1037/a0024917
https://doi.org/10.1037/a0024917...
; Sun 2005Sun J (2005) Assessing goodness of fit in confirmatory factor analysis. Meas Eval Couns Dev 37(4):240-256. https://doi.org/10.1080/07481756.2005.11909764
https://doi.org/10.1080/07481756.2005.11...
). In the present study, the SFLs range from 0.795 to 0.974 and majority are in 0.90 and above, hence, affecting the RMSEA value. Therefore, both indices could not reject the model. In summary, the model may still be considered as relatively good fit based on the three indices.

Figure 2
Results of the CFA of the scale.

Convergent Validity

Three indices establish the overall model-data fit, leading to the evaluation of the convergent validity of the scale. Figure 2 shows the CFA results, which provide evidence of the convergent validity. It can be noted that the factor loadings of all items are above 0.7. Factors 1, 2 and 3 have SFL loadings ranging from 0.795 to 0.97, 0.897 to 0.957, and 0.938 to 0.974, respectively. Further evidence of convergent validity is the resulting CR values ranging from 0.98 to 0.99. These values indicate a higher inherent consistency of all items in the scale.

Discriminant Validity

For good discriminant validity, the AVE of one factor should be greater than any correlation coefficients between the factors. In the event that any factor has smaller AVE over correlation coefficients, it suggests that the factors are correlated or do not measure well-separated latent concepts. In the present study, Table 3 shows the squared correlation of each factor versus the AVE of a particular factor as evidence of scale’s discriminant validity. It is indicated that all estimated AVE values are greater than the squared correlation of other factors where they are compared, suggesting that the factors are not associated with one another.

Table 3
Discriminant validity of the scale

CONCLUSION

The EFA clustered the 34 items contained in the scale into three factors, namely: Instrument Flight (Factor 1), Basic Attitude Flying (Factor 2), and Instrument Landing System (Factor 3). These factors were supported or confirmed by the resulting values of three GFIs generated by the CFA. These GFIs were CFI (0.817), TLI (0.804), and SRMR (0.0474). There may be issues with respect to the resulting chi-square/df ratio (7.699) and RMSEA (0.153) values but these are affected by sample size and factor loading specification, respectively. The three GFIs are already adequate to confirm that the models are relatively good fit. The other analyses, specifically convergent validity, discriminant validity, and reliability, also provide further evidence of the scale’s validity. In particular, SFL and CR have resulted in values exceedingly more than the criterion values. Also, the estimated AVE value of each factor is greater than the squared correlation of other factors compared. Finally, the Cronbach’s alpha of the entire scale and within subscales is/are greater than the predefined criteria.

This valid and reliable Likert-scale can be used to evaluate the training performance of pilots in flying an aircraft, particularly on basic attitude flying, instrument flight, and instrument landing system. The developed scale can be used for instructor to trainee evaluation, peer evaluation, and self-evaluation of pilots undergoing training on flying aircrafts. In line with this, the future direction of this study may involve utilizing the scale for skill evaluation of other communities of pilot abroad. Also, other GFIs may be used to evaluate the goodness of fit of the model to further provide evidences of the validity of the scale particularly those insensitive to sample size and unaffected by factor loading specification.

ACKNOWLEDGEMENTS

Not applicable.

  • Peer Review History: Single Blind Peer Review.
  • DATA AVAILABILITY STATEMENT

    Not applicable.
  • FUNDING

    Not applicable.

REFERENCES

  • Aircraft Owners and Pilots Association (2010) The flight training experience: A survey of students, pilots, and instructors. Frederick: AOPA. [accessed August 1 2021]. http://download.aopa.org/epilot/2011/AOPA_Research-The_Flight_Training_Experience.pdf
    » http://download.aopa.org/epilot/2011/AOPA_Research-The_Flight_Training_Experience.pdf
  • Alavi M, Visentin DC, Thapa DK, Hunt GE, Watson R, Cleary M (2020) Chi-square for model fit in confirmatory factor analysis. J Adv Nurs 76(9):2209-2211. https://doi.org/10.1111/jan.14399
    » https://doi.org/10.1111/jan.14399
  • Banbury S, Dudfield H, Hoermann HJ, Soll H (2007) FASA: Development and validation of a novel measure to assess the effectiveness of commercial airline pilot situation awareness training. Int J Aviat Psychol 17(2):131-152. https://doi.org/10.1080/10508410701328557
    » https://doi.org/10.1080/10508410701328557
  • Bennett SA (2006) A longitudinal ethnographic study of aircrews’ lived experience offlying operations at a low-cost airline. Risk Manag 8:92-117. https://doi.org/10.1057/palgrave.rm.8250009
    » https://doi.org/10.1057/palgrave.rm.8250009
  • Butcher JN (2003) Assessing pilots with ‘the wrong stuff’: A call for research on emotional health factors in commercial aviators. Int J Sel Assess 10(1-2):168-184. https://psycnet.apa.org/doi/10.1111/1468-2389.00204
    » https://psycnet.apa.org/doi/10.1111/1468-2389.00204
  • Crouch V (2020) Analysis of the airline pilot shortage. Scientia et Humanitas 10:92-103.
  • Evans B, Glendon AI, Creed PA (2007) Development and initial validation of an Aviation Safety Climate Scale. J Safety Res 38(6):675-682. https://doi.org/10.1016/j.jsr.2007.09.005
    » https://doi.org/10.1016/j.jsr.2007.09.005
  • Garson GD (2006) Structural equation modeling. North Carolina: G. David Garson and Statistical Associates Publishing.
  • Gefen D, Straub D, Boudreau MC (2000) Structural Equation Modeling and Regression: Guidelines for research practice. Communications of the Association for Information Systems 4:7. https://doi.org/10.17705/1CAIS.00407
    » https://doi.org/10.17705/1CAIS.00407
  • Hair Jr. JF, Black WC, Babin BJ, Anderson RE (2014) Multivariate data analysis (7th ed.). Harlow: Pearson Education Limited.
  • Heene M, Hilbert S, Draxler C, Ziegler M, Bühner M (2011) Masking misfit in confirmatory factor analysis by increasing unique variances: A cautionary note on the usefulness of cutoff values of fit indices. Psychol Methods 16(3):319-336. https://doi.org/10.1037/a0024917
    » https://doi.org/10.1037/a0024917
  • Hu LT, Betler PM (1999) Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal 6(1):1-55.
  • Jin J (2019) An examination of student pilot school choice factors for flight schools in California. Ann Arbor, USA: ProQuest LLC. https://doi.org/10.13140/RG.2.2.17545.19040
    » https://doi.org/10.13140/RG.2.2.17545.19040
  • Karakaya-Ozyer K, Aksu-Dunya B (2018) A Review of Structural Equation Modeling Applications in Turkish Educational Science Literature, 2010-2015. International Journal of Research in Education and Science 4(1):279-291.
  • Kearns S (2017) Attracting and informing the next generation of aviation professionals: The online ‘Fundamentals of Air Transport System’ course. Montreal: International Civil Aviation Organization (ICAO).
  • Kenny DA, Kaniskan B, McCoach DB (2014) The performance of RMSEA in models with small degrees of freedom. Sociological Methods and Research 44(3):1-22.
  • Kline RB (1998) Principles and practice of structural equation modeling. New York: Guilford Press.
  • Kline RB (2016) Principles and practice of structural equation modeling (4th ed.). New York: Guilford Press.
  • Lai K, Green S (2016) The problem with having two watches: Assessment of fit when RMSEA and CFI disgaree. Multivar Behav Res 51(2-3):220-239.
  • Letassy NA, Medina MS, Britton ML, Dennis V, Draugalis JR (2015) A progressive, collaborative process to improve a curriculum and define an assessment program. Am J Pharm Educ 79(4):55. https://doi.org/10.5688/ajpe79455
    » https://doi.org/10.5688/ajpe79455
  • Lutte B (2018) Pilot supply at the regional airlines: Airline response to the changing environment and the impact on pilot hiring. J Aviat/Aerosp Educ Res 27(1):1. https://doi.org/10.15394/jaaer.2018.1749
    » https://doi.org/10.15394/jaaer.2018.1749
  • Lutte R, Lovelace K (2016) Airline pilot supply in the US: Factors influencing the collegiate pilot pipeline. JATE 6(1):53-63. https://doi.org/10.7771/2159-6670.1148
    » https://doi.org/10.7771/2159-6670.1148
  • Malagas K, Fragoudaki A, Kourousis K, Nikitakos N (2017) Higher education aviation programs in Greece: a missed opportunity or a challenge to meet? J Aerosp Technol Manag 9(4):510-518. https://doi.org/10.5028/jatm.v9i4.888
    » https://doi.org/10.5028/jatm.v9i4.888
  • Rice S, Winter SR, Capps J, Trombley J, Robbins J, Milner M, Lamb TL (2020) Creation of two valid scales: Willingness to fly in an aircraft and willingness to pilot an aircraft. International Int J Aviat Aeronaut Aerosp 7(1):5. https://doi.org/10.15394/ijaaa.2020.1440
    » https://doi.org/10.15394/ijaaa.2020.1440
  • Rony HAZ (2019) Explorative study on the factors influencing aviation education in developing countries. ICAO Scientific Review 1:3-31.
  • Ruiz S, Aguado C, Moreno R (2014) Educational simulation in practice: A teaching experience using a flight simulator. J Technol Sci Educ 4(3):181-200. https://doi.org/10.3926/jotse.129
    » https://doi.org/10.3926/jotse.129
  • Sharma S, Mukherjee S, Kumar A, Dillon WR (2005) A simulation study to investigate the use of cutoff values for assessing model fit in covariance structure models. J Bus Res 58(7):935-943. https://doi.org/10.1016/j.jbusres.2003.10.007
    » https://doi.org/10.1016/j.jbusres.2003.10.007
  • Smith P (2016) Fact and fallacy of the “pilot shortage”. Aerophilia Enterprises. [accessed August 1 2021]. https://askthepilot.com/pilot-shortage/
    » https://askthepilot.com/pilot-shortage/
  • Steckel R, Lercel D, Matsuo H (2010) Factors that influence an undergraduate student to choose a career in aviation, and enroll in the Aviation Science Program at Parks College of Engineering, Aviation and Technology. Coll Aviat Rev 28(2):69-83. https://doi.org/10.22488/okstate.18.100412
    » https://doi.org/10.22488/okstate.18.100412
  • Sun J (2005) Assessing goodness of fit in confirmatory factor analysis. Meas Eval Couns Dev 37(4):240-256. https://doi.org/10.1080/07481756.2005.11909764
    » https://doi.org/10.1080/07481756.2005.11909764
  • Tabachnick BG, Fidell LS (2019) Using Multivariate Statistics (7th ed.). Boston: Pearson.
  • Tavakol M, Dennick R (2011) Making sense of Cronbach’s alpha. Int J Med Educ 2:53-55. https://doi.org/10.5116/ijme.4dfb.8dfd
    » https://doi.org/10.5116/ijme.4dfb.8dfd
  • Taherdoost H, Sahibuddin S, Jalaliyoon N (2014) Exploratory factor analysis; concepts and theory. Adv Appl Pure Math 27:375-382.
  • Tulis D (2017) Enrollment up at many aviation colleges. [accessed August 2 2021]. https://www.aopa.org/news-and-media/all-news/2017/december/11/enrollment-up-at-many-aviation-colleges
    » https://www.aopa.org/news-and-media/all-news/2017/december/11/enrollment-up-at-many-aviation-colleges
  • Valenta V (2018) Effects of airline industry growth on pilot training. Magazine of Aviation Development 6(4):52-56. https://doi.org/10.14311/MAD.2018.04.06
    » https://doi.org/10.14311/MAD.2018.04.06
  • Wall R, Tangel A (2018) Facing a Critical Pilot Shortage, Airlines Scramble to Hire New Pilots. The Wall Street Journal. [accessed August 2 2021]. https://www.wsj.com/articles/pilot-shortage-spurs-hiring-spree-1533720602
    » https://www.wsj.com/articles/pilot-shortage-spurs-hiring-spree-1533720602

Publication Dates

  • Publication in this collection
    24 June 2022
  • Date of issue
    2022

History

  • Received
    19 Oct 2021
  • Accepted
    11 May 2022
Departamento de Ciência e Tecnologia Aeroespacial Instituto de Aeronáutica e Espaço. Praça Marechal do Ar Eduardo Gomes, 50. Vila das Acácias, CEP: 12 228-901, tel (55) 12 99162 5609 - São José dos Campos - SP - Brazil
E-mail: secretary@jatm.com.br