Psychometric assessment of the Instructional Materials Motivation Survey (IMMS) instrument in a remote learning environment

Abstract:

Introduction:

Motivation contributes greatly to learning, being a predictor of student performance. Thus, instruments that assess motivation after exposure to different teaching strategies and materials can contribute to the analysis and decision on its effectiveness. In this sense, the Instructional Materials Motivation Survey (IMMS) instrument measures students’ motivation after instructional activities.

Objective:

This study aimed to evaluate the evidence of validity of the IMMS, previously translated and cross-culturally adapted to Brazilian Portuguese.

Method:

Cross-sectional study used to evaluate the psychometric properties of the IMMS questionnaire, applied to 211 students from the first, third and fourth semesters of the medical course at José do Rosário Vellano University (Unifenas - Campus Belo Horizonte). The principal component analysis (PCA) with Varimax rotation and Cronbach’s alpha coefficient were used to assess the validity and reliability of the instrument.

Results:

The PCA reduced the instrument’s items from 36 to 25 items, distributed in four dimensions. The saturation of the items in the dimensions ranged from 0.529 to 0.790 and the total explained variance was 63.12%. The reliability of the modified IMMS (IMMS-BRV), measured by Cronbach’s alpha, ranged from 0.76 (Attention dimension) to 0.93 (Interest dimension).

Conclusion:

The IMMS application in the remote education scenario, through asynchronous video lectures of human anatomy, resulted in an alternative instrument (IMMS-BRV), with fewer items (more parsimonious) and good internal consistency, demonstrating preliminary evidence of its validity and reliability adequacy.

Keywords:
Medical Education; Motivation; Educational Measurement; Distance Education; Psychometrics

Resumo:

Introdução:

A motivação contribui sobremaneira para a aprendizagem, sendo um fator preditor da performance do estudante. Assim, instrumentos que avaliam a motivação, após exposição a diferentes estratégias e materiais de ensino, podem contribuir para a análise de sua efetividade e decisão sobre esta. Nesse sentido, o instrumento Instructional Materials Motivation Survey (IMMS) mede a motivação dos estudantes após atividades instrucionais.

Objetivo:

Esta pesquisa teve como objetivo avaliar as evidências de validade do IMMS, previamente traduzido e adaptado transculturalmente para o português brasileiro.

Método:

Trata-se de um estudo transversal de avaliação das propriedades psicométricas do questionário IMMS, aplicado a 211 estudantes do primeiro, terceiro e quarto períodos do curso de Medicina da Universidade José do Rosário Vellano (Unifenas - câmpus de Belo Horizonte). Adotaram-se a análise de componentes principais (ACP) com rotação Varimax e o coeficiente alfa de Cronbach para avaliação da validade e da confiabilidade do instrumento.

Resultado:

A ACP reduziu os itens do instrumento de 36 para 25, distribuídos em quatro dimensões. A saturação dos itens nas dimensões variou de 0,529 a 0,790, e a variância total explicada foi de 63,12%. A confiabilidade do IMMS modificado (IMMS-BRV), medida pelo alfa de Cronbach, variou de 0,76 (dimensão atenção) a 0,93 (dimensão interesse).

Conclusão:

A aplicação do IMMS no cenário de ensino remoto, por meio de videoaulas assíncronas de anatomia humana, resultou em instrumento alternativo (IMMS-BRV), modificado com menor número de itens (mais parcimonioso) e boa consistência interna, demonstrando evidências preliminares de adequação de sua validade e confiabilidade.

Palavras-chave:
Educação Médica; Motivação; Avaliação Educacional; Educação a Distância; Psicometria

INTRODUCTION

The purpose of the teaching methods is to provide an environment that facilitates student learning. In this context, motivation is directly related to the efforts made by a certain person to acquire knowledge or develop a certain skill. Highly motivated students actively engage in studies without worrying about external rewards. On the other hand, when poorly motivated, they may have the skills necessary for learning and not achieve good proficiency¹. Thus, instruments based on solid theories, which assess motivation, after exposure to different teaching strategies and materials, can contribute to the analysis and decision about their effectiveness.

Several theories have been conceived to explain the factors and phenomena that interfere with the motivation for learning, such as intrinsic and extrinsic motivation theory, self-determination, social cognitive theory, the expectancy-value theory, as well as the ARCS (Attention, Relevance, Confidence and Satisfaction) model, which supports the construct of the Instructional Materials Motivation Survey (IMMS) instrument, analyzed in the present study^2),(3.

The ARCS model of motivational design was proposed by Keller³, based on cognitivist perspectives. The author’s objective was to find more effective ways to understand the main influences of motivation on learning and create a systematized method to identify and solve the associated problems, building a bridge between the study of motivation and instructional practice, aiming to increase student motivation. Its origin is based on the expectancy-value theory which, in turn, establishes that people become motivated to engage in an activity, if it is associated with the satisfaction of their personal needs (values) and a positive expectancy of success in its performance^4)-(6.

In Keller’s³ original motivational design model, these two dimensions (Value and Expectancy)^5),(6 were expanded to four, as follows:

• Value was subdivided into two dimensions: Interest and Relevance;

• The Interest dimension was more precisely related to factors associated with attention in the educational environment.

• Relevance was linked to activities directed by objectives or goals.

• The Expectancy dimension was maintained, being related to the student’s belief that they would be successful in the task; and the Result dimension was created, which represented the value of reinforcement provided by the instruction, as explained by operant conditioning.

Based on this initial construct (Interest, Relevance, Expectancy, Result), qualitative experiments were conducted to validate the model and then it was modified to the current ARCS (Attention, Relevance, Confidence, Expectation) model. The dimensions of the ARCS model are defined as follows by the author^3),(4: Attention presents characteristics that stimulate and sustain interest and curiosity; Relevance is related to the students’ personal needs and goals, generating a positive learning attitude; Confidence is related to the students’ beliefs and feelings, so that they can be in control of their learning result and will succeed; Satisfaction is generated by the process and/or result of the learning experience and may be related to the intrinsic (sense of competence) or extrinsic (academic degrees, certificates, material reward) factors and the desire to continue learning.

Based on the ARCS theoretical construct, the Instructional Materials Motivation Survey (IMMS) instrument was developed to assess the motivation of students exposed to educational materials or strategies⁴. As a situational instrument, the IMMS does not measure levels of motivation in a generalized way, such as, for instance, motivation regarding school learning. The purpose of this instrument is to measure how much students are motivated for a certain instructional activity, such as video classes, or for a specific strategy such as, for instance, remote learning. Due to the importance of being well interpreted by the respondent, it can be used in populations of students as early as those attending high school¹⁴.

According to Keller⁴, his conception was based on the construction of a pool of items based on ARCS motivational concepts, which were reviewed by 10 undergraduate students well acquainted with motivational literature. Based on the students’ answers, the items were revised and retested or excluded, thus originating the set of variables that currently constitute the IMMS in its four dimensions: Attention, Relevance, Confidence and Satisfaction. Its reliability, measured through internal consistency using Cronbach’s alpha coefficient, was 0.96 for the full scale (Attention = 0.89; Relevance 0.81; Confidence 0.90; Satisfaction 0.92), according to Keller⁴.

Since its conception, the IMMS has been widely used, in several countries, in different teaching-learning contexts, especially in those using technological means in the pedagogical platform, such as web-based courses, e-learning and gamification^7-10. Due to the robustness of the ARCS construct, the wide employability of the IMMS to assess student motivation and the need for an appropriate instrument to perform this measurement in Brazil, Cardoso-Júnior et al.¹¹ carried out the translation and cross-cultural adaptation of the IMMS into Brazilian Portuguese.

However, despite being widely used since its creation to measure student motivation, with the aim of improving teaching strategies, the studies that evaluated the psychometric validity of the IMMS showed heterogeneous results^12)-(14. These studies resulted in the reduction of the number of items and the modification of their distribution into dimensions, emphasizing the instrument’s situational characteristic, as demonstrated by Keller¹⁵, and emphasizing the need to modify the instrument to accommodate the context of its application. In other words, it is suggested that the instrument’s situation is dynamic and varies according to factors related to its implementation, ranging from the students’ characteristics to those of the involved instructional programs.

In this sense, Huang et al.¹² evaluated the validity of the IMMS using first-year engineering students studying a computer-based tutorial called MATLAB. The result of the validation was the maintenance of 20 items, distributed into four dimensions, differently from the original instrument. In turn, Loorbach et al.¹³ conducted a study in a self-directed learning environment. They concluded that, in this context, the instrument could be reduced to 12 items that adequately fit the four original dimensions of the IMMS, thus proposing an instrument called the Reduced Instructional Materials Motivation Survey (RIMMS). Recently, Hauze et al.¹⁴ studied the IMMS validity in a simulated training scenario applied to nursing students. In this study, 19 items were maintained, distributed into the four dimensions of the original IMMS. In conclusion, they emphasized that the validity of its use in other populations and to evaluate other types of instructional methods cannot be predicted by the study.

Based on the analysis of these studies, a lack of homogeneity is observed in relation to the results of the evidence of the IMMS construct validity, when applied in different contexts, related to the students’ characteristics, the evaluated instructional material, the teaching environment and the sociocultural context. Therefore, considering the heterogeneity of the results of the described studies, their situational characteristic and the importance of investigating their applicability to measure medical students’ motivation in the context of remote teaching of basic sciences, this study aimed to analyze the evidence of the IMMS construct validity, previously translated and adapted into Brazilian Portuguese, applied to students attending the first two years of the undergraduate medical course at UNIFENAS (Campus Belo Horizonte), to assess their motivation in relation to asynchronous video classes of human anatomy, attended during the social isolation period due to the COVID-19 pandemic.

The increasing use of this instrument in technological teaching environments, as observed in the literature, emphasizes the importance of having a version with valid parameters known in Brazil, since the use of new technologies in the field of education is a growing trend, which needs evaluation of its motivational properties^7),(9. In addition to its applicability in this context, the IMMS also shows as a particular feature, in relation to other instruments that assess motivation for learning, the fact that it is based on the ARCS model of motivational design, whose concepts and strategies can be applied, based on the diagnosis made through the observation of the results presented by the instrument, for correction and improvement of the instructional activity under investigation⁴.

METHODS

Study design

This was a cross-sectional study that evaluated the validity evidence of the Instructional Materials Motivation Survey (IMMS) questionnaire, using a version previously submitted to translation and cross-cultural adaptation¹¹.

Sample characteristics

The target population consisted of students of any gender or age, attending the first two years of the undergraduate medical course at Universidade José do Rosário Vellano - UNIFENAS, Belo Horizonte campus. The study included, in a convenience sampling, all students who were regularly enrolled in the first semester of 2020, in the Human Anatomy Laboratory Practices (LP) strategy, who wished to participate and did not meet the exclusion criteria. The exclusion criteria were: those who declared they wished to leave the study, lack of adherence to the protocol, prior knowledge of the IMMS, allocation error in relation to the research criteria, and foreign nationality. After applying the inclusion and exclusion criteria, the students who were interested in the study signed the Free and Informed Consent Form, being allocated to the study. This research was approved by the Research Ethics Committee - CEP/UNIFENAS as stated in the Certificate of Presentation for Ethical Approval (CAAE), under number 31795820.4.0000.5143, Opinion 4.074,707.

Asynchronous human anatomy video lessons

The undergraduate medical course at UNIFENAS-BH, founded in 2003, uses the Problem-Based Learning methodology. Human anatomy is studied throughout four semesters, in the first two years. For the teaching of anatomy, synthetic models, a digital anatomical table, imaging exams, videos of surgical procedures and prosection are used. The curriculum consists of thematic blocks that carry out the horizontal and vertical knowledge integration. The course lasts 12 semesters divided into pre-clinical, clinical and internship phases.

During the social isolation period, which occurred as of mid-March 2020, it was decided to use asynchronous video classes to maintain the Human Anatomy educational strategy. The five tutors of this teaching strategy recorded the classes using the Open Broadcaster Software (OBS)® program, version 25.0.4, and uploaded them to the Google Drive® Institutional application, with the weekly access link available in the virtual learning environment (Moodle®), according to the schedule of the thematic blocks. The mean duration of the video lessons comprised 30 minutes. After being included into the Institutional Google Drive®, the video classes were available to students for consultation, on demand, throughout the entire ongoing thematic block.

Application of the Instructional Materials Motivation Survey (IMMS) questionnaire

The original IMMS is derived from the ARCS construct, having been developed with the intention of being applied after students have had an educational experience. The questionnaire consists of 3 sentences related to the instructions for completion and 36 items belonging to the four dimensions of motivation: Attention (12 items); Relevance (9 items); Confidence (9 items) and Satisfaction (6 items). Its purpose is to measure the respondent’s motivation, using the Likert scale, with the following scores: (1) totally disagree; (2) partially disagree; (3) neither disagree nor agree; (4) partially agree; (5) totally agree. The total score ranges from 36 to 180 points, and each dimension or the total score can be evaluated, thus generating the motivation score. The minimum, maximum and mean values ​​for each dimension vary, as the number of items is different and they are independent. Some items, mentioned as inverse ones, should be scored as follows: 5=1; 4=2; 3=3; 2=4 and 1=5. Higher scores in the total score indicate greater motivation¹⁵.

The IMMS used in this research was the version that was previously translated and adapted into Brazilian Portuguese by Cardoso-Júnior et al.¹¹, which maintained all dimensions and items of the original questionnaire. The self-administered instrument was created using the Google Forms® application, preceded by the informed consent and the sociodemographic questionnaire. The access link was posted on the Moodle-UNIFENAS platform. After reading the consent form and accepting to participate in the study, the student was directed to the questionnaire.

As part of the information about the filling out of the questionnaire, it was clear that the instrument was intended to “... assess your motivation in relation to the Human Anatomy video classes, studied in Laboratory Practices (LP) ...” For that purpose, words related to the instructional material were adapted to this context of application in the items’ sentences, according to the IMMS application instructions, without changing their meaning¹⁵.

Students who, after reading the consent form, checked the option “I don’t agree” were directed to a message thanking them for their attention, not having access to the questionnaires. The mean time to complete the forms was 15 minutes.

Principal component analysis

Aiming to verify the evidence of the studied construct validity, a statistical analysis was performed using the principal component analysis (PCA) extraction method and Varimax rotation. The use of the rotation aims to facilitate the interpretation of the dimensions, that is, the saturation of items in the components, finding a solution in which each variable shows high saturation in a few components^16),(17. Items with an inverse direction had their scores inverted.

For PCA, the sample size should be ≥100 observations, preferably reaching five or more observations per instrument item, to avoid extracting sample-specific components, which are not very generalizable¹⁷. In this study, the sample size (n=211) was adequate, comprising six subjects per IMMS item.

To assess the convenience of the PCA model, the Bartlett test of sphericity and the Kaiser-Meyer-Olkin (KMO) sample adequacy measure were used. Values ​​> 0.5 in the KMO test and p<0.05 in the sphericity test were defined as parameters for the PCA to be considered adequate for the set of studied variables.

The establishment of the number of retained dimensions was performed through determination techniques based on eigenvalues. Eigenvalues ​​> 1 were considered as a parameter (Kaiser-Guttman criterion). The loads (item saturation) were considered significant when they were greater than 0.40 (parameter considered adequate for a sample of 211 subjects at a significance level of 0.05 and 80% power)¹⁷. Measures related to the method without rotation were used, followed by the VARIMAX rotation, as performed in previous studies, which allows for a better comparison of results^12),(14.

Measures of commonality (MOC) and measures of sample adequacy were also performed for each of the analyzed items, considering adequate values for commonality and for sample adequacy (Measures of Sampling Adequacy - MSA) > 0.50. Aiming to assess the interdependence of the latent dimensions identified by the PCA, Pearson’s Correlation analysis was performed.

Additionally, the differences between the observed correlations (correlation matrix with all original items) and the reproduced correlations (correlations estimated by the construct matrix) were assessed, where such differences, called residuals, indicate a good adjustment of the generated model, when they show very low values.

The parameters adopted in this study are based on publications by Landis and Koch¹⁶ and Hair Jr et al.¹⁷. The statistical software SPSS (IBM), version 17.0, was used to perform the analyses.

Cronbach’s Alpha

The Cronbach’s alpha coefficient was the measure used to assess the reliability of the internal consistency of each of the dimensions determined by the PCA. This coefficient ranges from 0 to 1, with values ≥0.6 indicating satisfactory reliability of the internal consistency^16),(17.

Probability of significance

All results were considered significant for a probability of significance < 5% (p < 0.05), therefore having at least a 95% confidence in the presented conclusions.

RESULTS

A total of 211 students aged 17 to 46 years, with a mean age of 21.4 years, participated in this study. Of these, 33% were aged between 17 and 19 years, 44.5% between 20 and 22 years, 13% between 23 and 25 years, 6% between 26 and 30 years old and 3.5% aged at least 31 years. There was a predominance of women (67.8%). In relation to the course period, 46.4% of the students were attending the first period, 35.1% the third period and 18.5% the fourth period. The students attending the second-period were excluded from the study because, unlike the others, they participated in synchronous online classes. Only 5.7% of students had another university degree.

Analysis of the IMMS Construct Validity Evidence

The initial PCA was performed with all 36 items that constitute the original IMMS questionnaire, translated and adapted into Brazilian Portuguese¹¹. Table 1 shows the evaluation of the students’ behavior towards each of the IMMS items.

Table 1
Assessment of students’ behavior towards each of the IMMS questions.

Question	Answers					Total	Mean	SD
	1	2	3	4	5
1. When I watched these video lessons for the first time, I had the impression that it would be easy for me.	11.4	19.9	25.6	29.4	13.7	100.0	3.1	1.2
2. There was something interesting at the beginning of these activities that caught my attention	9.5	14.7	28.0	32.2	15.6	100.0	3.3	1.2
3. These video lessons were harder to understand than I would have liked.	20.4	28.4	14.2	26.6	10.4	100.0	2.8	1.3
4. After listening to the initial information, I felt confident that I knew what was expected to be learned from this activity.	12.8	19.4	25.6	27.5	14.7	100.0	3.1	1.2
5. Completing the exercises in this activity gave me a satisfying sense of accomplishment.	10.4	15.2	18.5	34.1	21.8	100.0	3.4	1.3
6. It is clear to me how the content of this material is related to knowledge I already have.	7.1	15.2	32.7	29.8	15.2	100.0	3.3	1.1
7. Many slides had so much information that it was difficult to choose and remember the important points.	27.0	25.1	18.0	18.5	11.4	100.0	2.6	1.4
8. These video lessons are interesting.	10.0	11.4	18.5	32.2	27.9	100.0	3.6	1.3
9. There were stories, pictures or examples that showed me how these video lessons could be important to some people.	5.2	8.5	22.3	33.2	30.8	100.0	3.8	1.1
10. Successfully completing this activity was important to me.	4.3	4.3	21.8	27.0	42.6	100.0	4.0	1.1
11. Video quality helped to keep my attention.	7.1	13.3	16.1	31.7	31.8	100.0	3.7	1.2
12. These video lessons are so abstract that it was hard to keep my attention on them.	25.1	28.4	15.7	18.5	12.3	100.0	2.6	1.4
13. As I worked on this activity, I was confident that I could learn the content.	8.0	16.6	26.1	32.2	17.1	100.0	3.3	1.2
14. I enjoyed these video lessons so much that I would like to know more about this subject.	15.2	22.7	27.5	25.1	9.5	100.0	2.9	1.2
15. The slides in this material look uninteresting.	30.8	31.8	16.1	12.8	8.5	100.0	2.4	1.3
16. The content of these video lessons is relevant to my interests.	1.4	3.3	10.0	27.5	57.8	100.0	4.4	0.9
17. The way the information is organized on the slides helped keep my attention.	8.5	12.8	19.9	34.1	24.7	100.0	3.5	1.2
18. There are explanations or examples of how people use the knowledge of these video lessons	6.2	13.7	30.8	21.8	27.5	100.0	3.5	1.2
19. The exercises in these video lessons were very difficult.	18.5	33.6	38.9	7.6	1.4	100.0	2.4	0.9
20. These video lessons have elements that stimulated my curiosity.	8.0	17.5	20.4	37.0	17.1	100.0	3.4	1.2
21. I really enjoyed studying this activity.	11.4	18.9	25.6	28.9	15.2	100.0	3.2	1.2
22. The amount of repetition in these video lessons made me eventually bored.	27.0	20.4	23.7	19.4	9.5	100.0	2.6	1.3
23. The content and style of the slides in this material give the impression that it is worth getting to know them.	6.6	15.2	20.4	37.4	20.4	100.0	3.5	1.2
24. I learned some things that were surprising or unexpected	6.2	11.8	17.1	38.4	26.5	100.0	3.7	1.2
25. After working on this activity for some time, I was confident that I would be able to pass a test on it.	13.8	18.5	23.2	28.4	16.1	100.0	3.1	1.3
26. These video lessons were not relevant to my needs because I already knew most of them.	63.0	29.4	4.7	1.9	1.0	100.0	1.5	0.8
27. The way feedback was given after the exercises or other comments on the activity helped me feel rewarded for my effort.	19.9	18.5	30.8	19.9	10.9	100.0	2.8	1.3
28. The variety of reading excerpts, exercises, illustrations, etc., helped keep my attention on the activity.	10.5	16.1	21.3	32.2	19.9	100.0	3.4	1.3
29. The writing style is boring.	27.9	24.2	23.2	17.1	7.6	100.0	2.5	1.3
30. I was able to correlate the content of these video lessons to things I have seen, done or thought about in my own life.	8.0	16.1	27.5	31.8	16.6	100.0	3.3	1.2
31. There are so many words on each slide that it is annoying.	47.4	28.0	13.7	8.5	2.4	100.0	1.9	1.1
32. It was good to successfully complete this activity.	4.2	3.8	22.3	30.8	38.9	100.0	4.0	1.1
33. The content of these video lessons will be useful to me.	0.5	3.3	7.6	30.3	58.3	100.0	4.4	0.8
34. I really did not understand much of the material in these video lessons.	23.2	29.4	16.6	20.8	10.0	100.0	2.6	1.3
35. Good content organization helped me make sure I learned this material.	5.7	16.1	29.4	35.1	13.7	100.0	3.4	1.1
36. It was a pleasure working on such a well-planned activity.	12.8	16.6	25.6	24.2	20.8	100.0	3.2	1.3

• SD: standard deviation.
• Note: The wording of the items underwent minor changes to adapt to the instructional activity evaluated (video lessons), without changing its meaning.

The initial model showed the convenience of applying this statistical technique in terms of its adequacy (Kaiser-Meyer-Olkin ENT#091;KMOENT#093; sample adequacy measure) =0.943; Bartlett’s sphericity test p < 0.0001). Table 2 shows the result of the number of factors extracted by the initial PCA.

Table 2
Result of the number of factors extracted based on the initial Principal Component Analysis.

FACTORS	Sum of squares of factorial loads WITHOUT Rotation			Sum of squares of factorial loads WITH Rotation
	Eigenvalue	% of variance	% accumulated	Eigenvalue	% of variance	% accumulated
A	14,500	40,279	40,279	5,767	16,021	16,021
B	1,852	5,145	45,424	5,379	14,942	30,963
C	1,698	4,717	50,141	2,954	8,206	39,169
D	1,325	3,681	53,822	2,700	7,501	46,669
E	1,194	3,317	57,139	2,626	7,295	53,965
F	1,081	3,003	60,142	1,943	5,398	59,363
G	1,015	2,818	62,960	1,295	3,598	62,960
Total	22,665	62,961	--------	22,664	62,961	---------

• Database: 211 students. Rotation Method: Varimax.

However, during the PCA rounds, 11 items were removed from the original instrument for different reasons:

Items 6, 16, 18, 30 and 33 for presenting a low proportion of explained variance (commonality < 0.50) in the initial or subsequent PCA;

Items 10 and 32 due to similar loads shown in various components (dimensions);

Item 19 because it was presented as a component with a single item, that is, it did not show significant correlations with the other variables.

Items 1, 2 and 26 for showing low reliability of internal consistency in the dimensions to which they were allocated.

Table 3 shows the items taken from the original IMMS and the causes of exclusion.

Table 3
Items extracted from the original IMMS and causes of exclusion.

IMMS Dimension	Excluded Item	Cause of exclusion
Attention	02 - There was something interesting at the beginning of these activities that got my attention.	IC
Relevance	06 - It is clear to me how the content of this material is related to things I already know.	MOC < 0.5
	10 - Successfully completing this lesson was important to me.	SL
	16 - The content of these video lessons is relevant to my interests.	MOC < 0.5
	18 - There are explanations or examples of how people use the knowledge of these video lessons.	MOC < 0.5
	26 - These video lessons were not relevant to my needs because I already knew most of them.	IC
	30 - I was able to correlate the content of these video lessons with things I have seen, done or thought about my own life.	MOC < 0.5
	33 - The content of these video lessons will be useful to me.	MOC < 0.5
Confidence	01 - When I first watched these video lessons, I had the impression that it would be easy for me.	IC
	19 - The exercises in these video lesson were too difficult.	SIC
Satisfaction	32 - It felt good to successfully complete this lesson.	SL

• IMMS: Instructional Materials Motivation Survey translated and cross-culturally adapted to Brazilian Portuguese; MOC: Measure of Commonality; SL: similar loads in several components; SIC: Single-item component; IC: low internal consistency in the component.

Consequently, the final PCA model was performed after the removal of these 11 items, which were considered inadequate. In this model, the sample adequacy measure (KMO=0.947) shows convenience for the use of the PCA. Furthermore, the Bartlett test shows that there is a set of significant correlations (p < 0.0001) between the 25 remaining items of the questionnaire. Thus, based on these assumptions, the statistical technique of PCA can be applied. Therefore, based on the Kaiser-Guttman criterion (eigenvalues > 1), 4 components were extracted, as shown in the scree plot illustrated in Figure 1 and in the results depicted in Table 4.

Figure 1
Final Principal Components Analysis Scree plot.

Table 4
Results of the final model of the IMMS-BRV Principal Component Analysis.

Items*/dimensions	Loads (saturation of items in the dimensions)				MOC	MSA	Mean
	A	B	C	D
3				0.753	0.678	0.894	2.8
4		0.661			0.608	0.959	3.1
5		0.529			0.503	0.954	3.4
7			0.717		0.645	0.871	2.6
8	0.779				0.728	0.960	3.6
9	0.589				0.503	0.958	3.8
11	0.663				0.625	0.939	3.7
12				0.569	0.621	0.939	2.6
13		0.734			0.691	0.958	3.3
14	0.625				0.614	0.955	2.9
15	0.599				0.581	0.948	2.4
17	0.551				0.621	0.963	3.5
20	0.586				0.704	0.958	3.4
21		0.555			0.686	0.961	3.2
22			0.631		0.560	0.939	2.6
23	0.671				0.630	0.973	3.5
24	0.627				0.579	0.940	3.7
25		0.790			0.700	0.930	3.1
27	0.556				0.510	0.965	2.8
28	0.709				0.625	0.945	3.4
29			0.693		0.626	0.918	2.5
31			0.776		0.661	0.910	1.9
34				0.595	0.678	0.946	2.6
35		0.533			0.642	0.938	3.4
36	0.622				0.764	0.950	3.2
Cronbach’s Alpha	0.93	0.87	0.76	0.78		global MSA 0.947
% of explained variance	24.299	17.416	12.149	9.256
Eigenvalue	6.075	4.354	3.037	2.314
Total explained variance	63.12%

• * Items numbered as in the IMMS - version translated into Brazilian Portuguese (Cardoso-Júnior et al.11).
• MOC = measure of commonality; MSA = Measure of Sampling Adequacy.
• NOTE: The fit of the model was verified by examining the differences between the observed correlations and the reproduced correlations obtained from the reproduced correlation matrix. It is noteworthy that there were few cases (< 10%) with residuals considered “large”; therefore, the obtained model was considered to be valid.

The results of the final Principal Component Analysis model generated the modified alternative version of the IMMS (IMMS-BRV). Thus, the loads (item saturation in the dimensions), commonalities, measures of sample adequacy (MSA), explained variances, eigenvalues and Cronbach’s alpha coefficients of the IMMS-BRV are summarized in Table 4.

The composition of the 4 latent dimensions generated by PCA with Varimax rotation is shown in Table 4. The coefficients (saturation of loads in the dimensions) represent the levels of correlations between the dimensions with each of the variables (items) that constitute them. Therefore, higher absolute values indicate that the dimensions and their respective variables are closely related. Moreover, differences between the observed correlations (correlation matrix with all original items) and reproduced correlations (correlations estimated by the dimension matrix) were examined. These differences, called residuals, showed very low values, confirming a good fit of the generated model.

Therefore, based on these results, we will work with 4 dimensions that encompass blocks of items capable of explaining dimensions of motivation in the face of asynchronous video lessons on human anatomy, according to the medical students’ opinion, instead of analyzing 25 items individually. The total score generated by the sum of the values assigned to the 25 items represents the overall motivation score.

In this study, significant correlations (p < 0.05) were observed between all dimensions generated by PCA. However, the correlation observed between the Interest and Confidence dimensions showed a value > 0.80, showing that they are intrinsically linked, and may be measuring similar concepts of motivation. Table 5 shows the correlation analysis between the four components generated by the Principal Component Analysis.

Table 5
Analysis of correlation between the four components generated by the Principal Component Analysis.

Dimension/Pearson		Dimension
		A	B	C	D
A	r	1.00	0.81	0.55	0.65
	p	< 0.001	< 0.001	< 0.001	< 0.001
B	r	---------	1.00	0.50	0.64
	p		< 0.001	< 0.001	< 0.001
C	r	---------	---------	1.00	0.50
	p			< 0.001	< 0.001
D	r	---------	---------	---------	1.00
	p				< 0.001

• Database: 211 students.
• r: Pearson’s correlation coefficient; p: probability of significance of the correlation analysis.

IMMS-BRV reliability assessment

Table 6 shows the reliability measures of the IMMS-BRV internal consistency.

Table 6
IMMS-BRV Internal Consistency Reliability Measures.

Formed dimensions / Items	Cronbach’s alpha if item were removed
Interest (Cronbach’s alpha = 0.93)
8	0.92
9	0.93
11	0.93
14	0.92
15	0.93
17	0.92
20	0.92
23	0.92
24	0.93
27	0.93
28	0.92
36	0.92
Confidence (Cronbach’s Alpha = 0.87)
4	0.86
5	0.86
13	0.84
21	0.85
25	0.85
35	0.85
Attention (Cronbach’s Alpha = 0.76)
7	0.75
22	0.71
29	0.67
31	0.68
Expectancy (Cronbach’s Alpha = 0.78)
3	0.71
12	0.74
34	0.66

• Database: 211 students.

Cronbach’s alpha coefficient was used to assess the reliability of the internal consistency of the items that constitute each of the 4 dimensions generated by the PCA. As observed, all of them showed Cronbach’s Alpha measures > 0.75, indicating good internal consistency of the IMMS-BRV dimensions. Moreover, item removal does not increase Cronbach’s alpha coefficient.

Modified alternative Brazilian version of the IMMS (IMMS-BRV)

When comparing the original version of the IMMS and the results of the PCA performed in this study (IMMS-BRV), it is observed that 4 dimensions were maintained and 11 items were removed. The 25 items that were maintained were redistributed into 4 dimensions, in a different combination than the original instrument. Thus, the names of the dimensions were revised aiming to better explain the items allocated in the same dimension. For this purpose, in the content analysis, the semantics of the items combined with the theoretical reference of the ARCS construct and the expectancy-value theory were observed^3),(5.

Thus, once the PCA process was completed, the final structure of the IMMS-BRV was defined, which now has 25 items, without changing the Likert scale originally used in the IMMS. The items were allocated, according to the PCA, into 4 dimensions (A, B, C and D), named as follows:

Interest (explained variance = 24.3%; Cronbach’s alpha = 0.93): the set of meanings of the assertions allocated in this dimension involve items from the following dimensions of the original IMMS: attention (58.3%); satisfaction (25.0%) and relevance (16.7%). The semantic analysis of the meanings of these items allows their aggregation under the Interest dimension, since all of them are clearly related to the respondents’ interest in the evaluated instructional material. The substitution of item nouns by the noun interest itself shows the semantic adequacy of the name given to this dimension. According to the Oxford dictionary Languages¹⁸, from Oxford University Press, interest is: “what is important, useful or advantageous, morally, socially or materially.” Also: “state of mind that one has towards what is considered worthy of attention.” In turn, in the same dictionary, relevance is defined as “that which stands out in a comparative or value scale; importance, significance.” Finally, satisfaction is: “...pleasure arising from the realization of what is expected, what is desired.” In other words: ... what one is interested in.

Confidence (explained variance = 17.4%; Cronbach’s alpha = 0.87): in the Confidence dimension, 4 (66.6%) items from the same dimension of the original IMMS were agglutinated. The other 2 items (33.3%) come from the original Satisfaction dimension. These two items are intrinsically related to the satisfaction generated by the confidence in learning, by the expectancy of success. In the very definition of the theoretical frameworks of the ARCS construct, from which the IMMS derives, the satisfaction dimension is seen as the result of the interaction between the three others (Attention, Relevance and Confidence). Thus, in the PCA of this study, the items related to satisfaction were agglutinated in different dimensions according to their semantic affinity with each set of items. Therefore, due to the predominance of items related to the confidence dimension of the original IMMS and the intricate confidence/satisfaction correlation, the designation of this dimension as Confidence in the IMMS-BRV was maintained.

Attention (explained variance = 12.1%; Cronbach’s alpha = 0.76): in this dimension, 3 items (75%) are derived from the original IMMS Attention dimension. Only 1(25%) item derives from the original Confidence dimension. Although originally allocated to the Confidence dimension, its semantic analysis demonstrates it is a statement whose interpretation is precisely in line with the Attention dimension. Therefore, the denomination was maintained.

Expectancy (explained variance = 9.3%; Cronbach’s alpha = 0.78): the 3 items distributed in this dimension derive from the Confidence (66.7%) and Attention (33.3%) dimensions. The semantic analysis of the three items shows that they are intrinsically related to the expectancy (expectation) of the occurrence or not of the understanding, therefore, of meaningful learning. Thus, the set of its meaning allows the aggregation of the term Expectancy to this dimension.

It is opportune to emphasize that the IMMS-BRV, as well as the IMMS, can be adapted to suit specific situations. In other words, the “standard” wording of the items contains more generic phrases such as “this course”, “this class” or “many pages”. They must be changed to suit the specific situation being evaluated, such as “this lecture”, “this video lesson” or “many slides”^4),(11.

The modified alternative Brazilian version of the Instructional Materials Motivation Survey (IMMM-BRV) is shown in Chart 1. Although the version applied to the subjects of this survey was adapted to the study material (video lessons), as shown in Table 1, Chart 1 shows the standard wording of the items, according to the original IMMS, translated and cross-culturally adapted into Brazilian Portuguese¹¹.

Chart 1
Validated Brazilian Version of the Instructional Materials Motivation Survey (IMMM-BRV).

Validated Brazilian Version of the Instructional Materials Motivation Survey (IMMS-BRV)					Discordo totalmente		Discordo parcialmente	Não concordo, nem discordo		Concordo parcialmente	Concordo totalmente
1. Há 25 afirmações neste questionário. Por favor, pense sobre cada sentença em relação a cada material didático que você acabou de estudar e indique o quão verdadeira é a sentença. Dê a resposta que verdadeiramente se aplica a você, e não aquela que você gostaria que fosse verdadeira, ou aquilo que você pensa que outros querem ouvir
2. Pense em cada afirmação isoladamente e indique quão verdadeira ela é. Não se deixe influenciar por suas respostas a outras afirmações.
3. Registre suas respostas na folha fornecida e siga as instruções adicionais que podem ser fornecidas em relação à folha de resposta que está sendo usada nesta pesquisa. Obrigado.
1. Este material foi mais difícil de entender do que eu gostaria.					①		②	③		④	⑤
2. Depois de ler as informações iniciais, eu me senti confiante de que sabia o que se esperava que deveria ser aprendido com essa atividade.					①		②	③		④	⑤
3. Completar os exercícios dessa atividade me deu um sentimento satisfatório de realização.					①		②	③		④	⑤
4. Muitas das páginas tinham tanta informação que foi difícil escolher e lembrar os pontos importantes.					①		②	③		④	⑤
5. Esses materiais são interessantes.					①		②	③		④	⑤
6. Havia histórias, figuras ou exemplos que me mostraram como esse material poderia ser importante para algumas pessoas.					①		②	③		④	⑤
7. A qualidade da escrita ajudou a manter minha atenção.					①		②	③		④	⑤
8. Essa atividade é tão abstrata que foi difícil manter minha atenção nela.					①		②	③		④	⑤
9. Enquanto eu trabalhava nessa atividade, estava confiante de que poderia aprender o conteúdo.					①		②	③		④	⑤
10. Gostei tanto dessa atividade que gostaria de saber mais sobre este assunto.					①		②	③		④	⑤
11. As páginas desse material parecem pouco interessantes.					①		②	③		④	⑤
12. A forma como a informação está organizada nas páginas ajudou a manter minha atenção.					①		②	③		④	⑤
13. Esta atividade tem elementos que estimularam minha curiosidade.					①		②	③		④	⑤
14. Gostei muito de estudar essa atividade.					①		②	③		④	⑤
15. A quantidade de repetição nesta atividade me fez ficar, eventualmente, entediado.					①		②	③		④	⑤
16. O conteúdo e o estilo de escrita neste material dão a impressão de que vale a pena conhecê-lo.					①		②	③		④	⑤
17. Aprendi algumas coisas que foram surpreendentes ou inesperadas.					①		②	③		④	⑤
18. Depois de trabalhar nessa atividade por algum tempo, eu estava confiante de que seria capaz de passar em um teste sobre ela.					①		②	③		④	⑤
19. A maneira como foi dado o feedback após os exercícios, ou outros comentários na atividade, ajudou-me a sentir recompensado pelo meu esforço.					①		②	③		④	⑤
20. A variedade de trechos de leitura, exercícios, ilustrações etc., ajudou a manter minha atenção na atividade.					①		②	③		④	⑤
21. O estilo de escrever é entediante.					①		②	③		④	⑤
22. Há tantas palavras em cada página que é irritante.					①		②	③		④	⑤
23. Eu realmente não consegui entender muito o material desta atividade.					①		②	③		④	⑤
24. A boa organização do conteúdo me ajudou a ter certeza de que eu aprenderia este material.					①		②	③		④	⑤
25. Foi um prazer trabalhar em uma atividade tão bem planejada.					①		②	③		④		⑤
Itens distribuídos por dimensão
Dimensão	Itens
Interesse	5	6	7	10	11*	12	13	16	17	19	20	25
Confiança	2	3	9	14	18	24
Atenção	4*	15*	21*	22*
Expectativa	1*	8*	23*

• ^*
  Inverted items: the score of the items must be inverted.

DISCUSSION

The importance of instruments that measure student motivation in different teaching-learning environments, with adequate evidence of validity and reliability, initially motivated the translation and cross-cultural adaptation of the IMMS into Brazilian Portuguese¹¹. The translation and cross-cultural adaptation of instruments based on solid constructs, created in other languages, saves time and resources^19),(20.

In this sense, the present study sought evidence of construct validity of the translated version of the IMMS into Brazilian Portuguese, through the analysis of its internal structure, against a pedagogical strategy that became mandatory during the social isolation period due to the COVID-19 pandemic. which it seems, will be increasingly used even after the need for distancing is overcome: the asynchronous video lessons in the context of remote or hybrid teaching^21)-(23.

The necessary premises to demonstrate the convenience of the PCA were adequately verified by the sample adequacy measure (KMO) and Bartlett’s test of sphericity. After the exclusion of 11 items, the IMMS-BRV instrument now has 25 items divided into 4 dimensions. This behavior was also observed in other studies that empirically assessed the validity of the IMMS, with the exclusion of 16, 24 and 17 items having been reported in the three published studies^12)-(14.

It is observed, as described in the results, that the PCA grouped the items in the IMMS-BRV in two dimensions close to the original IMMS, maintained as Confidence and Attention. However, two new dimensions (Interest and Expectancy) were created according to the content analysis and with the higher loads of the items grouped in them. In this regard, it is worth noting that the ARCS model, from which the IMMS derives, is based on the expectancy-value theory, whose construct focuses on the interest and expectancy for learning^4),(5.

Another relevant fact is that the IMMS was originally validated using undergraduate students and self-directed instructional material printed on paper, quite different from the material used in this and other studies that analyzed its validity¹⁵. Furthermore, there was no empirical data treatment for the distribution of items in the four original dimensions, as in the present study. It is possible that such procedure, had it been carried out, would have generated different aggregation of its items in relation to the dimensions described in the original IMMS.

The PCA, conducted in the present study, showed significant loads (0.529 to 0.790), which characterized the adequate correlation of each variable (item) with the corresponding dimension (construct validity)¹⁷. Regarding the adequacy of content analysis, in redefining the nomenclature of the dimensions, although there are differences between the IMMS-BRV and the IMMS, regarding the distribution of items and the designation of components (dimensions), the coherence with the constructs that support the original IMMS was maintained^3),(24.

These dimensions relate the IMMS-BRV construct to the one of the first motivational design model proposed by Keller³ and described in the introduction. In it, the four dimensions are: Interest, Expectancy, Relevance and Result. Therefore, the IMMS-BRV has two dimensions of the first motivational design model (Interest and Expectancy) and two dimensions of the second motivational design model (Attention and Confidence), showing how the concepts underlying it are in line with the evolution of the motivational design construct postulated by Keller^3),(4),(15. Given this composition of dimensions, the issue is highlighted that, if the initial pool of items, from which the IMMS was derived, had been submitted to multivariate techniques to define its dimensionality, the result would not have been closer to that found in the present study and in the others that evaluated the internal structure of the original instrument. Therefore, the results of the PCA and the adequacy of the content analysis in redefining the dimensions indicate initial evidence of construct validity and the Cronbach’s alpha coefficients, found for the dimensions, disclose the good reliability of the IMMS-BRV.

Correlating with the literature, the studies that analyzed the evidence of the IMMS validity also resulted in a reduction in the number of items and the modification of their distribution in dimensions, once again highlighting the situational characteristic of the instrument and emphasizing the need to modify it to accommodate the context of its application^12)-(14. In other words, it is suggested that the situation of the instrument is a dynamic one and varies according to factors correlated to its implementation, which range from the students’ characteristics to those of the involved instructional programs.

In this sense, Huang et al.¹² evaluated the validity of the IMMS in first-year engineering students who used a computer-based tutorial as supplementary material for learning the use of a computational tool called MATLAB. The result was the maintenance of 20 items distributed in four dimensions, differently from the original instrument, as it occurred in the present study. Also correlating with the present study, Attention was the dimension that maintained the highest number of items in both of them and, of the 20 items maintained in the instrument validated by Huang, 15 (75%) items were also maintained in the IMMS-BRV, representing 62.5% of IMMS-BRV.

Regarding the study by Huang et al.¹², Keller⁴ added that the factorial analysis, when applied to the IMMS, can change its structure, because the dimensions can show very high intercorrelations. He explains that this occurs partly because the instrument was designed to measure situation-specific attitudes rather than psychological constructs. In fact, in the present study, the correlation between the Interest dimension and the Confidence dimension was also high (r = 0.81). In the ARCS construct, the student’s interest in the instructional activity is directly related to their confidence that they will be successful in the task, that is, in learning. Therefore, the items in these dimensions measure behaviors that walk in parallel, because the represented latent traits are highly intercorrelated. However, for the purpose of diagnosis and correction or improvement of the instructional activity being evaluated, it is important to maintain these dimensions separately.

In turn, in the study by Loorbach et al.¹³, carried out in a self-directed learning environment, aiming at teaching tasks related to the use of cell phones by the elderly, the instrument was reduced to 12 items that adequately fit in the four original dimensions of the IMMS, with the proposal of the instrument called the “Reduced Instructional Materials Motivation Survey (RIMMS)”. Of the 12 items present in the RIMMS, 10 (83.3%) items are also allocated in the IMMS-BRV.

Recently, Hauze et al.¹⁴ validated the IMMS in a simulated training scenario applied to Nursing students. In this study, 19 items were maintained, distributed in 4 dimensions, according to the original IMMS.

Based on the analysis of these studies that have been published so far, a lack of homogeneity was observed in relation to the results of the IMMS validation, when applied to different contexts, related to the characteristics of the students and the evaluated instructional material. Thus, in the present study, motivation was analyzed in a different context than those previously researched in studies evaluating the validity of the original instrument. The final version of the IMMS-BRV, reduced to 25 items, makes its application more convenient in the evaluation of remote teaching strategies, such as asynchronous video lessons, as it can measure motivation while requiring less time to fill it out and, consequently, resulting in less fatigue and dispersion of attention when interpretating the different items, therefore, in line with the principle of parsimony of measuring instruments²⁴.

The reliability of the IMMS-BRV, measured by Cronbach’s alpha, ranged from 0.76 (Attention) to 0.93 (Interest) in its dimensions, thus showing good internal consistency²⁵. These findings, compared with other validation studies, show that the items of the IMMS-BRV demonstrate reliability, to the same or to a better degree than those verified in such studies^12)-(14. In addition, the IMMS-BRV maintained 80% of the reverse items contained in the original instrument. A similar finding was verified by Hauze et al.¹⁴, demonstrating the resilience of the IMMS questionnaire to response bias and to the respondents’ fatigue.

This study has limitations that must be considered, especially in relation to the generalization of results. This is a study carried out in a single medical course, using a convenience sample from a very specific population. Moreover, the instructional activity assessed by the instrument was also unique, as it only included video lessons on the human anatomy strategy. Added to this fact is the atypical period of the Covid-19 pandemic, during which data were collected, as well as the online application of the instrument. The way the data analysis of the variables generated by ordinal measures converted into numbers was performed, as in the case of those used in Likert scales, also raises debate in the literature, since the differences between the answers are not equidistant²⁶.

Finally, it should be noted that this study was aimed in evaluating the validity of the IMMS in the remote learning scenario (asynchronous video lessons). Its results and data available in the literature emphasize the situational characteristic, as well as the non-universal nature of the IMMS and reinforce that the validity and reliability of the IMMS-BRV, demonstrated in this study, should be considered preliminary. Thus, potential uses in other contexts must be evaluated and further studies that seek to corroborate the IMMS-BRV validity in other teaching-learning contexts, giving it external validity, must be carried out to expand its application spectrum. Therefore, the accuracy of the generalization of its use, in contexts different from the one used in this study, remains unknown.

CONCLUSIONS

The application of the IMMS in the remote learning scenario, through asynchronous video lessons on human anatomy, resulted in an alternative instrument (IMMS-BRV), modified with a smaller number of items (more parsimonious) and good internal consistency, demonstrating preliminary evidence of adequate validity and reliability in the employed context.

ACKNOWLEDGMENTS

The authors are especially grateful to Prof. John Keller for authorizing and encouraging the translation and validation of the IMMS into Brazilian Portuguese.

REFERENCES

• ¹
  Small R, Gluck M. The relationship of motivational conditions to effective instructional attributes: a magnitude scaling approach. Educ Technol. 1994;34:33-40.
• ²
  Gopalan V, Aida J, Bakar A, Nasir A, Alwi A, Mat RC. A review of the motivation theories in learning. In: AIP Conference Proceedings 1891. 2017. p. 1-7.
• ³
  Keller JM. Development and use of the ARCS model of motivational design. J Instr Dev. 1987;10:2-10.
• ⁴
  Keller JM, editor. Motivational design for learning and performance: the ARCS model aproach. New York: Springer; 2010.
• ⁵
  Wigfield A. Expectancy-value theory of achievement motivation: a developmental perspective. Educ Psychol Rev. 1994;6:49-78.
• ⁶
  Wigfield A, Eccles JS. Expectancy-value theory of achievement motivation. Contemp Educ Psychol. 2000;25:68-81.
• ⁷
  Cook DA, Beckman TJ, Thomas KG, Thompson WG. Measuring motivational characteristics of courses: applying Keller’s instructional materials motivation survey to a web-based course. Acad Med. 2009;84:1505-9.
• ⁸
  Wan Hamzah WMAF, Haji Ali N, Mohd Saman MY, Yusoff MH, Yacob A. Influence of gamification on students’ motivation in using e-learning applications based on the motivational design model. Int J Emerg Technol Learn. 2015;10:30-4.
• ⁹
  Huang B, Hew KF. Measuring learners’ motivation level in Massive Open Online Courses. Int J Inf Educ Technol . 2016;6:759-64.
• ¹⁰
  Julià C, Antolí JÒ. Impact of implementing a long-term STEM-based active learning course on students’ motivation. Int J Technol Des Educ. 2019;29:303-27.
• ¹¹
  Cardoso-Júnior A, Garcia VCS, Coelho DV, Said CC, Strapasson ACP, Resende IS. Tradução e adaptação transcultural do Instructional Materials Motivation Survey (IMMS) para o português do Brasil. Rev Bras Educ Med. 2020;44(4):e179.
• ¹²
  Huang W, Huang W, Diefes-Dux H, Imbrie PK. A preliminary validation of attention, relevance, confidence and satisfaction model-based Instructional Material Motivational Survey in a computer-based tutorial setting. Br J Educ Technol . 2006;37:243-59.
• ¹³
  Loorbach N, Peters O, Karreman J, Steehouder M. Validation of the Instructional Materials Motivation Survey (IMMS) in a self-directed instructional setting aimed at working with technology. Br J Educ Technol . 2015;46:204-19.
• ¹⁴
  Hauze SW, Marshall JM. Validation of the Instructional Materials Motivation Survey : measuring student motivation to learn via mixed reality nursing education simulation. International J on E-Learning 2020;19:49-64.
• ¹⁵
  Keller JM. Development of two measures of learner motivation. Tallahassee FL: Florida State University; 2006. v. 3, p. 1-9 ENT#091;access in 12 jan 2021ENT#093;. Available from: Available from: http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:Development+of+Two+Measures+of+Learner+Motivation#0
  » http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:Development+of+Two+Measures+of+Learner+Motivation#0
• ¹⁶
  Landis JR, Koch GG. Agreement of categorical data. Biometrics. 1977;33:159-74.
• ¹⁷
  Hair Jr. JF, Black WC, Babin BJ, Anderson RE. Multivariate data analysis. 7th ed. Harlow: Pearson; 2014.
• ¹⁸
  Dicionário Oxford. 3a ed. Oxford: Oxford University Press; 2018.
• ¹⁹
  Toledo Júnior A, Duca JGM, Coury MIF. Tradução e adaptação transcultural da versão brasileira do Schutte Self-Report Emotional Intelligence Test. Rev Bras Educ Med . 2018;42:109-14.
• ²⁰
  Sleumer JP, Campos APF, Machado FAS, Moura EP. Adaptação transcultural e avaliação das propriedades psicométricas do instrumento Nutrition in Patient Care Survey (Nips) no Brasil. Rev Bras Educ Med . 2019;43:26-35.
• ²¹
  Bernard RM, Abrami PC, Borokhovski E, Wade CA, Tamim RM, Surkes MA, et al. A meta-analysis of three types of interaction treatments in distance education. Rev Educ Res. 2009;79:1243-89.
• ²²
  Bozkurt A, Sharma R. Emergency remote teaching in a time of global crisis due to CoronaVirus pandemic. Asian J Distance Educ. 2020;15:1-6.
• ²³
  Gelles LA, Lord SM, Hoople GD, Chen DA, Mejia JA. Compassionate flexibility and self-discipline: student adaptation to emergency remote teaching in an integrated engineering energy course during Covid-19. Educ Sci. 2020;10:1-23.
• ²⁴
  Pasquali L. Psicometria: teoria dos testes na psicologia e na educação. 5a ed. Petrópolis: Vozes; 2013.
• ²⁵
  Souza AC, Alexandre NMC, Guirardello EB. Propriedades psicométricas na avaliação de instrumentos: avaliação da confiabilidade e da validade. Epidemiol Serv Saúde. 2017;26:649-59.
• ²⁶
  Sullivan GM, Artino Jr. AR. Analyzing and interpreting data from Likert-Type scales. J Grad Med Educ. 2013;5:541-42.

• Evaluated by double blind review process.

• SOURCES OF FUNDING

  The authors declare no sources of funding.

Edited by

Chief Editor: Rosiane Viana Zuza Diniz. Associate Editor: Pedro Tadao Hamamoto Filho.

Publication Dates

• Publication in this collection
  27 Sept 2021
• Date of issue
  2021

History

• Received
  24 Feb 2021
• Accepted
  23 Aug 2021