What Colors do Undergraduates Associate with Training Courses? Student Evaluations of the Applied Mathematics Educational Program through the Color Selection Method

It is no doubt that today the role of mathematics is increasing and mathematical education requires constant attention. Nevertheless, there are not so many studies devoted to the problems of teaching university students who have chosen mathematics as their profession. The purpose of this article is to research the attitude of undergraduates towards the courses that make up the Applied Mathematics educational program, implemented in one of the technical universities in Russia. The survey was conducted using the Color Selection Method, based on Max Lüscher ideas. Students have associated each course with one of the eight proposed colors. The outcomes of the survey were investigated through correlation and cluster analysis and compared with the results of another survey conducted by a verbal evaluation tool. This research has revealed that the student's assessments obtained through two methods (verbal and imaginative) do not contradict each other. The use of the Color Selection Method helps identifying the problems that arise in the educational process and allows to outline ways of improving teaching quality.


Introduction
In a high-tech society, education becomes a valuable resource for sustainable development. Therefore, the quality assessment of education and the search for ways to improve it are essential tasks both at the national and at the individual pedagogical collectives levels. In the international educational practice, there are various approaches for assessing the quality of university work. So, there are many publications devoted to this problem (for example, GERRITSEN- VAN LEEUWENKAMP et al., 2017;HARVEY;GREEN, 1993;NOVIKOV, 2007;SCHINDLER et al., 2015, TAM, 2001. Essential aspects in determining the quality of university educational activities are assessments of training courses quality and the students' satisfaction with the quality of education. The analysis of publications concludes that studies on the problems of student evaluation of teaching (SET) are conducted by scientists around the world, which is evidence of their relevance for education theorists and practitioners. The articles of Benton and Cashin (2014), Kulik (2001) and Richardson (2005) presented reviews of such studies.
In Russia, SET has not been widely adopted. During the period of social and economic transformations (the end of the 1980s), the Ministry of Higher and Secondary Education of the Russian Federation introduced the practice of student interviews using the questionnaire "The Teacher in the Eyes of Students". The appearance of this questionnaire aroused criticism from scientists and educators (GORBATENKO, 1990;ZELENTSOV, 1999;LEVCHENKO, 1990  Lipetsk State Technical University also began to pay attention to students' satisfaction with the education quality. The main problem of monitoring was the choice of an evaluation tool because due to the lack of broad practice of student evaluations, publications devoted to ISSN 1980-4415 DOI: http://dx.doi.org/10.1590/1980 the analysis of evaluation tools used in Russia are scarce. To assess the students' satisfaction with the quality of the educational program, we developed the questionnaire "The Learning Process in the Eyes of Students (LPES)" and surveyed senior students. Using a 100-point scale, students assessed the quality of the learning process organization, the quality of teaching and their results of training for the 34 study courses that make up the "Applied Mathematics" educational program. The survey outcomes were investigated through correlation, factor, regression, and cluster analysis. The study results are presented in the article Kuznetsova (2019). Next, a survey was conducted using the Color Selection Method (CSM). This method is based on Max Lüscher ideas, according to which students choose the color that they associate with each academic discipline of the educational program.
The purpose of this article is, using CSM, to study the students' attitude to the courses that make up the Applied Mathematics curriculum, and to compare the two methods (verbal LPES and imaginative CSM) to reveal their ability to identify bottlenecks in learning and teaching. This article draws on previous research on the factors affecting the satisfaction of undergraduates with the quality of the educational program in applied mathematics.

Literature review
It is well known that the practice of student surveys has become widespread since the late 1960s (DARWIN, 2016) and is seen as a mean of improving teaching quality and involving students in perfecting the education process (BENTON; CASHIN, 2014;HAMMONDS et al., 2017). As noted by Nilson (2012), students have changed noticeably in recent decades. Therefore, the questions "Are the students telling us the truth?" as well as "How reliable are students' evaluations of teaching quality?" continue to interest researchers (CLAYSON;HALEY, 2011;FEISTAUER;RICHTER, 2017;MCCLAIN et al., 2018).
Indeed, outcomes of SET depend not only on teachers and university administrations but also on the students themselves: their attitude to knowledge and the ways of obtaining this knowledge (O'DONOVAN, 2017), the notion of teaching and its forms (FEISTAUER; RICHTER, 2017), and academic maturity and emotion (LYNAM; CACHIA, 2018). Therefore, it seems essential that along with the Likert-type evaluation tools (such as the Course Experience Questionnaire, or Students' Evaluation of Educational Quality Questionnaire) there are alternative approaches based on associations and imaginative thinking. For example, Gal and Ginsburg (1994) demonstrated an example of an evaluation tool for measuring students' attitudes toward the study of mathematics. There is a set of 12-15 ISSN 1980-4415 DOI: http://dx.doi.org/10.1590/1980 cards showing faces with different emotions ("anxious", "puzzled", "fearful", "frozen", "interested", "indifferent", "confused", and so on). Students must choose one card which most reflects their feelings concerning the academic discipline or situation in the learning process.
According to the authors, despite the limitations, this technique can be useful in identifying bottlenecks and problem situations. "But it is useful to break out of the mold for perceiving students' attitudes as lying along linear paths, and for an "attitude change" as moving students "higher" or "lower" along such paths, as is the case when five-point Likert scales are used (GAL;GINSBURG, 1994).
This characteristic may refer to the Color Selection Method (CSM). The developer of the test's original version (Max Lüscher) postulated that the color choice reflects the mood, functional state, and the most enduring personality traits. The test development is based on an empirical approach and was initially associated with a study of a person's emotional and psychological state (LÜSCHER, 1990). The Russian psychologist Sobchik (2007) characterizes CSM as projective, because, in her opinion, this technique reveals not so much the conscious, subjective attitude of the examinee to the color standards, but their unconscious reactions.
The ideas of diagnosis through color associations appeared in the middle of the 20th century. Since then, there were numerous empirical studies on the effect of color on humans and the possibilities of CSM for the diagnosis of a person's internal state. For example, many researchers believe that this method is not sufficiently reliable as a diagnostic tool in clinical practice (see, CERNOVSKY;FERNANDO, 1988;HOLMES et al., 1985). At the same time, numerous studies show examples of successful use of the test to describe personality and behavior (CARMER et al., 1974;COROTTO;HAFNER, 1980;LANGE;RENTFROW, 2007;NOLAN et al., 1995). Donnelly (1974), in his study of the color preferences of college students, concluded that the reliability of the Lüscher Color Test, "although somewhat low, appears comparable to that reported for other projective techniques". Sobchik (2007) argued that the test based on Lüscher's ideas has the following essential characteristics: it does not provoke (in contrast to other, especially verbal, tests) reactions of a protective nature, and also is consistent with the concept of a holistic multi-level understanding of the individual.
Specialists of the psycho-diagnostics laboratory of Tomsk Polytechnic University have developed a particular modification of color selection method in order to investigate students' attitude to teaching. The interpretation of colors is presented in Table 1.
The study of this method was presented in the dissertation research by Maruhina (2003), who is an employee of this university. She saw CSM as an addition to the Likert-type ISSN 1980-4415 DOI: http://dx.doi.org/10.1590/1980 questionnaire to investigate students' attitudes toward study courses. The joint application of the two methods allows concluding that the students' color associations correspond to the interpretations presented in Table 1. Due to this fact, and that many studies also support the validity of the Lüscher color theory (for example, CARMER et al., 1974;COROTTO;HAFNER, 1980), we hypothesized that CSM could be used as a separate methodology for students' assessment of training courses. Since the further investigation of CSM application in education has not been carried out, we believe that our study will contribute to the disclosure of CSM's capabilities as a substantive student teaching evaluation tool and will help expand the practice of using it.

Methodology
Sixty-six seniors of the Applied Mathematics undergraduate program took part in the survey. Of these, 39 are men and 27 are women. The survey took place in the last month of their university studies. These students were invited to participate in the survey, because they could evaluate all the courses of study that make up the educational program. As the studies confirm, senior students have academic maturity and experience and therefore can give an adequate assessment of teaching quality (LYNAM; CACHIA, 2018; THEALL; FRANCLIN,

2001).
Also, an essential factor in conducting student interviews is the attitude of students to participate in the survey. According to Hoshower and Chen (2003), McClain et al. (2018), and some other researchers, students give more honest assessments if they believe that their ISSN 1980-4415 DOI: http://dx.doi.org/10.1590/1980 opinion will help improve the courses' content and teaching rather than serve administrative interests. Benton and Cashin (2014) argue that in order to increase survey validity, "the instructor should take time to encourage students to take the process seriously." Therefore, not only we discussed with the students the goals of the upcoming survey but also invited them to participate in the questionnaire items discussion and selection of the assessment scale (KUZNETSOVA, 2019).
This study consisted of two consecutive stages. At the first stage, a survey was conducted using the questionnaire "The Learning Process in the Eyes of Students" (LPES).
The questionnaire, developed by us, consists of 10 items and reflects the students' opinions on the following aspects of the learning process: the educational process quality of organization; teaching quality; the results of the process of studying the course. Using the 100-point scale, familiar to them, the students evaluated 34 courses studied by them since their first year. From the survey outcomes, we compiled a summary table, in which each academic discipline corresponds to the average score for each of the ten items. Table 2 presents the results of the internal consistency analysis of this questionnaire. Student evaluations were analyzed using correlation, factor, regression, and cluster analysis.
The results are presented in the article KUZNETSOVA (2019). In the second stage, which took place two weeks later, we used a test based on Max Lüscher's ideas. The same group of students was asked to choose which color from the ones presented in Table 1 (but without deciphering their meaning) they associated with one or another course. The students evaluated the same 34 academic disciplines as in the first stage.
Based on the survey outcomes, a summary table was compiled. Table 3 demonstrates a fragment corresponding to such disciplines as Discrete Math and Sociology. ISSN 1980-4415 DOI: http://dx.doi.org/10.1590/1980

Correlation analysis
First, we considered the correlation between the CSM questionnaire items. The results can be seen in Table 4.  ISSN 1980-4415 DOI: http://dx.doi.org/10.1590/1980 For the interpretation, we will rely on the decoding of the color associations presented in Table 1. The survey shows that the association with blue (thoroughness, reliability, durability) has a statistically significant negative correlation with violet (intuition, search for something unusual in the new information). The association with green (interest, the search for meaning for themselves, usefulness) has a statistically significant negative correlation with colors such as brown (stability, conservatism, inflexibility of a position, stiffness, toughness of thought patterns), black (denial, rejection, negative perceptions) and gray (indifference, uncertainty). The association with red (activity, initiative) has weakly significant (p <0.10) negative correlations with yellow (comfort) and gray (indifference). Somewhat unexpected was the presence of a significant negative correlation between yellow (comfort) and brown (stability, conservatism, the toughness of thought patterns): conservatism, the inflexibility of thinking in teaching and learning often causes discomfort and therefore is not as harmless as it seemed to us earlier. Thus, the analysis shows that the survey results as a whole do not contradict the color interpretations presented in Table 1, which agrees with Marukhina (2003) conclusions.
Next, we considered the correlation of CSM outcomes and outcomes of the LPES questionnaire, reflecting on the shortcomings of the learning process organization and teaching quality. The results can be found in Table 5. First of all, note that the colors blue and yellow do not have significant correlations with these items. However, green (cognitive activity), black (denial) and gray (indifference) have a close enough connection with items describing the teaching quality: Teacher's knowledge on the subject, Teaching skills and Impartial and fair assessment. In addition, the shortcomings in the learning process organization, although not provoking complete denial (there aren't any highly significant correlations of the items Lack of theory and Lack of practice with the item black), cause reduced cognitive activity (negative correlation of the items Lack of theory and Lack of practice with the item green) and contribute to the formation of indifference on the studied subject (a positive correlation of these two items with the item gray).  Table 6.  lack of boredom, rejection, and inflexibility (negative correlation with gray, black and brown). The intrinsic subject difficulty (Factor3) is associated with blue (thoroughness, ISSN 1980-4415 DOI: http://dx.doi.org/10.1590/1980 reliability, and durability), lack of easiness, comfort, and intuition (negative correlations with yellow and violet). The presence of low-significance (p <0.1) correlations of the variable Factor3 can be interpreted as the absence of indifference for demanding disciplines (negative correlation with gray), the presence of inflexibility of thinking and rejection (positive correlation with brown and black).
Thus, the correlation analysis indicates the consistency of the student evaluation by two methods (verbal LPES and imaginative CSM). The fact that the obtained conclusions do not contradict the theory and practice of teaching and learning testifies to the meaningful validity of the CSM. Therefore, we will continue comparing the results of the two surveys using cluster analysis.

Cluster analysis
At first, we researched the outcomes of the LPES survey. Using the K-means method, four clusters were identified: Cluster1, Cluster2, Cluster3, and Cluster4. According to the results presented by Kuznetsova (2019), these groups have the following characteristics.
Cluster1problem courses. These are five disciplines related to programming which, in the students' opinion, are interesting, but require teaching improvement. Cluster2severe problem courses. These are four courses which students described as uninteresting, having a low teaching quality level. Cluster3successful courses. These are 18 courses that are interesting enough for students. The teachers of these courses received high marks. For these courses, students noted the lack of shortcomings in the educational process organization.
Cluster4 united difficult courses. These are seven courses covering abstract sections of pure mathematics. The students rated them as uninteresting, but at the same time, the teachers received high marks.
Let us consider clustering the outcomes of a survey conducted on the basis of the CSM evaluation tool. The groups of courses obtained by applying the K-means method to the CSM outcomes are denoted by Cluster1*, Cluster2*, Cluster3*, and Cluster4*. The average means of variables for each cluster are presented in Table 7. The differences in the mean values of the red and violet variables turned out to be statistically insignificant; therefore these variables were excluded during the analysis. Consider the characteristics of the obtained clusters in more detail. ISSN 1980-4415 DOI: http://dx.doi.org/10.1590/1980  These can be demanding fundamental disciplines (18% association with blue), exciting (18% association with green, and comfortable (17% association with yellow).
Cluster3* contains five courses that are very interesting for students (44% association with green) and comfortable (12% association with yellow). These are Economics,

Mathematical Methods and Models in Economics I, Mathematical Methods and Models in
Economics II, Intelligent Systems, and Application Software. Probably, students associate their future professional activities with studying these courses.
Cluster4* brings together seven courses that cause discomfort when studying them.

There are Functional Analysis, Math Modeling, Differential Equations with Partial
Derivatives, Mathematical Theory of Systems, Computer Architecture, Physics, and Philosophy. As is evident in Table 7, many students associate these courses with such colors as brown, black, and gray, which signal problems in the learning process. In order to understand the causes of discomfort, let us examine in more detail the colors associated with these disciplines, using the fragment of the CSM survey summary table for courses inserted ISSN 1980-4415 DOI: http://dx.doi.org/10.1590/1980 into Cluster4* (see Table 8). The perception of Physics reveals little association with the colors blue, green, yellow or red, reflecting the positive perception on the subject. At the same time, 60% of the students associated this course with the colors brown, black, and gray. It seems that the teaching of these courses poses problems caused by the teaching methods chosen by the instructor. Also, some problems are present in the teaching of Computer Architecture. Here, the main difference is that this course is almost not associated with brown color (conservatism). Some students have no problems with its study (association with yellow 14% and red 14%).
However, there is a large proportion of students who are not just indifferent (14% association with gray) but demonstrate complete denial (association with black in about 27% of respondents). It means, when studying this course, that every 3-4 students faced problems that they could not overcome. Perhaps the reason is the lack of a student-centered approach, the teacher's lack of attention to the students' needs and capabilities.
The second group of uncomfortable disciplines is courses that cover the most abstract branches of mathematics. Studying them requires a high level of theoretical thinking. Some students overcome this intrinsic difficulty of pure mathematics (association with green for Math Modeling, Differential Equations with Partial Derivatives, and association with red for Functional Analysis, Mathematical Theory of Systems, and Math Modeling). Some students see in it the fundamental essence (association with blue takes place for all disciplines of this group). However, almost every fourth student has associated these courses with brown ISSN 1980-4415 DOI: http://dx.doi.org/10.1590/1980 (conservatism, inflexibility of thinking). That is, these students could not see a living meaning within the complexity of mathematical structures and the rigor of the logical inference inherent in these disciplines. Therefore, the teaching of these courses also requires improvement in order to help students overcome the intrinsic difficulty of pure mathematics, to show its meaning, elegance, value, and connection with applied problems. The social and psychological conditions that contribute to the solution of this problem are the personal and cognitive maturity inherent to the age of late adolescence, mathematical giftedness of the students who have chosen mathematics as their future professional activity (KUZNETSOVA, 2018).
According to the cluster analysis of the CSM outcomes, of the 34 courses that make up the educational program, three (Philosophy, Physics, and Computer Architecture) cause a negative attitude in students. The process of studying 22 disciplines (17 comfortable from Clusrer2* and five extremely excited from Cluster3*) is acceptable to the students. So, students in general are satisfied with the educational program in Applied Mathematics.
The cluster comparison for the two surveys (see Table 9) has revealed that none of the members of Cluster2 (severe problem courses) belong to Cluster2* (comfortable courses) or Cluster3* (very interesting courses). None of the members of Cluster3 (successful courses) belongs to Cluster4* (uncomfortable courses). All members of Cluster3* (very interesting courses) are members of Cluster3 (successful courses).
That is, as a whole, the clustering outcomes of each of the two surveys do not contradict each other.

Conclusion
Supporting student feedback, an analysis of survey outcomes, and respect for students' opinion are essential factors in the successful improvement of education quality. Application of CSM for monitoring the educational program quality in applied mathematics has shown its ability not only to reveal the existence of problems but also to define their causes. For example, as a result of the cluster analysis, such courses as Philosophy, Physics, Computer Architecture, and Functional Analysis have been gathered in the group of disciplines uncomfortable for students. The detailed analysis of color associations for each of these courses allows us to conclude that the causes of discomfort are different. It will help us choose the correct strategy for improving teaching and learning.
The research has revealed the most appealing courses for students. They are

Economics, Mathematical Methods and Models in Economics I, Mathematical Methods and
Models in Economics II, Intelligent Systems, and Application Software. The content of these disciplines has a close connection with the students' future professional activity as applied and industrial mathematicians. The fact that on average, 44% of students have these courses associated with green (interest, the search for themselves, usefulness, following the Table 1), proves how important the understanding of the connection between learning and practice is for them.
The cluster analysis of the CSM-based survey outcomes has detected problems with the teaching of social sciences and humanities. Such courses as History, Russian Language, Sociology, and Philosophy are not attractive to future mathematicians. An exception to this group is English (Cluster2*comfortable courses), and Economics (Cluster3*very interesting courses). At first sight, social sciences and humanities have no connection with the future professional activity of bachelors-mathematicians. However, mathematics expands its scope today. Increasingly mathematical methods are applied in human sciences, for example, Sociology or Psychology. Besides, the Russian mathematician Kolmogorov emphasized that for developing creative abilities in mathematics, it is necessary to go beyond mathematics and develop common cultural interests, in particular, interest in art and poetry (Yurkevich, 2001).
Therefore, the problem of improving teaching in social sciences and humanities within the Applied Mathematics educational program also requires care and attention.
We can note other advantages of CSM. First, conducting a survey using this questionnaire does not demand much time from students. This is especially important if we have to estimate not one but several disciplines. Indeed, for the characteristic of a course, a ISSN 1980-4415 DOI: http://dx.doi.org/10.1590/1980 student has to choose a color among the eight offered, instead of responding to numerous Likert-type items. Secondly, as our experience has shown us, CSM has not caused rejection among students; none of them refused to participate in the survey. Students were interested in color associations. This fact is consistent with results obtained by Sobchik (2007). However, this task confused three respondents in the beginning of the survey.
The analysis of the CSM outcomes and their comparison with LPES allows us to conclude that the surveys for each of the two methods do not contradict each other and therefore both questionnaires may be considered as robust. It confirms our hypothesis that CSM can be used as a full-fledged evaluation tool for identifying the problems arising in the process of teaching and learning.
In the future, it would be interesting to continue researching the properties of CSM.
For example, further research should investigate the influence of temperament on its results, to understand the reasons why the same situation in studying the course for some students causes a feeling of comfort (association yellow), indifference for others (association with gray), and rejection (association with black).
It is also necessary to note that the results of the CSM, as well as the results of SET in general, need a balanced attitude as one of the education quality indicators and should be considered in conjunction with other indicators of learning effectiveness.