Collective portfolio : assessment of teaching and learning in health undergraduate courses

Resumo O potencial do portfólio coletivo reflexivo como método de ensino, aprendizagem e avaliação precisa ser enfatizado. Os objetivos consistem em avaliar e validar a qualidade do portfólio coletivo como um método de ensino, aprendizagem e avaliação na formação de profissionais de saúde. A qualidade do portfólio foi analisada utilizando um inventário de análise que foi adaptado e testado para este estudo. No total, foram analisados 70 portfólios. O teste Kappa foi utilizado para analisar a reprodutibilidade, confirmando o nível de concordância entre os avaliadores. Análise descritiva e teste t foram realizados para comparar a construção do portfólio nos diferentes anos. Os construtos (layout e organização, compreensão, reflexão, crítica e pensamento criativo) apresentaram valores de alfa de Cronbach acima de 0,7, confirmando a alta confiabilidade da ferramenta. O teste Kappa ponderado revelou uma concordância elevada e significativa (+ 90%) entre os avaliadores. Notavelmente, certos itens e construtos apresentaram diferenças significativas ao longo dos anos. Todas as construções foram avaliadas melhor a partir de 2011, quando a ferramenta de avaliação e autoavaliação foi implementada. O sucesso do portfólio depende de estratégias que proporcionem aos alunos maior clareza e formas de desenvolver o método. Palavras-chave Avaliação, Educação em saúde, Educação médica Abstract The potential of the reflective collective portfolio as a method of teaching, learning and assessment needs to be emphasized. The objectives is to assess and validate the quality of the collective portfolio as a method of teaching, learning and assessment in the training of health professional. Portfolio quality was analyzed using an inventory of analysis that was adapted and tested for this study. In total, 70 portfolios were analyzed. The Kappa test was used to analyze reproducibility, confirming the level of agreement between the appraisers. Descriptive analysis and the t-test were performed to compare results from the years the portfolios were produced. The constructs (layout and organization, comprehensive, reflective, critical and creative thought) exhibited Cronbach alpha values above 0.7, confirming the high reliability of the tool. The weighted Kappa test revealed an elevated and significant agreement (+90%) between the appraisers. Notably, certain items and constructs exhibited significant differences over the years. All of the constructs were assessed better from 2011 onwards, when the assessment and self-assessment tool was implemented. The success of the portfolio depends on strategies that provide students with greater clarity and forms of developing the method.


Introduction
Portfolios have been defined as a collection of evidence and reflections that students use to demonstrate the results of specific learning and are an effective method of assessing personal and professional development 1 .Thus, the portfolio is a method designed by the students who must identify evidence that allows them to assess their knowledge, skills and attitudes (competence), as well as the learning process itself [2][3][4] .In this context, the students are considered as agents: "somebody who acts and causes change and whose achievements can be judged by their own values and objectives, regardless of assessments associated with external criteria" 5 .
There are many different types of portfolios 6,7 , including the collective portfolio, which involves the acquisition of competence in a group, encouraging teamwork and training future professionals in problem-solving and decision-making.This version of social reality presupposes that professional training leads to a new rationality.Students must be prepared to provide the most adequate, competent, democratic and effective answers to the enormous challenges that await them in contemporary society, particularly in the area of health.Therefore, the development of skills, in terms of making people and organizations more reflective, competent and efficient, is indispensable in the training of new professionals 8 .
This emphasizes the potential of the reflective portfolio as a method of teaching, learning and assessment, through the development of the following skills: decision-making; knowing how to work as part of a team, including inter-disciplinary teams and occasionally trans-disciplinary teams; communication skills; development of critical, reflective and creative thought; strengthening autonomy in the learning process.
However, in spite of all of the positive points in favor of the reflective portfolio, there is a consensus about the fragility of the assessment and validation of this method in the day-to-day operation of universities.Driessen et al. 9 stated that assessing a portfolio is complex due to its open, dynamic and flexible format.The same authors defended more global assessments with trained appraisers and stated that the richness and complexity of portfolios cannot be captured by analytical assessment criteria and extremely technical checklists can easily trivialize the assessment.From this perspective, it is essential to create portfolio analysis tools that ensure each portfolio is designed for the specific learning context of the student or group in question.
With this fragility of the assessment and validation of portfolios in mind, the aim of the present study was to assess and validate the collective portfolio as a method of teaching, learning and assessment, in terms of professional health training, as well as to present a tool created and tested for this purpose.

Context
The construction of a collective portfolio is used as a didactic method of teaching, learning and assessment in the discipline of Health Policy for graduate courses in a Brazilian public university.The aim is to use this active method in large groups.The primary purpose of the portfolio is to promote learning associated with health policies, stimulating critical thought and enabling students to decipher reality, acquire autonomy of judgment and share accountability in knowledge building while part of a team.The portfolios are built collectively in groups composed of approximately six students.
At the beginning of the semester, the learning objectives, related to the construction of the portfolio, are designed jointly by teachers and students.In order to guarantee a clear structure with well-defined criteria and guidelines, students are instructed in all of the phases that should be carried out while constructing a portfolio: planning (before the activity); monitoring activities and the monthly assessment process (four assessments per semester); real-time assessments and feedback, with constant dialogue between teachers and students and between the students themselves 3,4,10 .
The teacher who is accompanying the post-graduate students meets the groups, analyzes and discusses the items in the portfolio and promotes debate and questions about the evidence used.It is important to highlight that the assessment tool for the professor and the self-assessment tool for students was created and implemented in 2011, based on the experience of the professor and broad theoretical foundations, having been tested and discussed with students.Its structure is composed of learning objectives and aspects of classification and assessment (needs to improve, progressed adequately, progressed remarkably, exceptional).
Cotta et al. 10 proposed four stages in the construction of a portfolio: 1) Search and conceptual production: the first proposed activity suggests that students search the scientific literature for the concepts, definition and aims of the portfolio (extra-curricular activity) and elaborate an individual and collective concept based on the literature consulted, including a list of the characteristics that a reflective portfolio should contain; 2) My trajectory: memories written at the beginning and end of the semester; 3) Learning with the group: activities designed and conducted in a group using narratives (reports of experiences in different practical scenarios), according to the theses involved; 4) Creative space: free space where the group can express their creativity using cartoons, poems, songs, photos and artwork from written and electronic media, associated with critical thought.

Assessment inventory
An inventory of analysis of the quality of portfolios was created as a result of the studies conducted by Driessen et al. 9 , Gadbury-Amyot et al. 11 and Cotta et al. 3,10 with the aim of assessing portfolios using documentary analysis.
The items (20 questions, Table 1) in the inventory were formulated based on previously referenced studies, as well as the learning objectives proposed by the professor responsible for the discipline at the beginning of the process and the assessment and self-assessment criteria contained in the assessment tool.A Likert scale was created to measure the criteria.The scale was designed with five points and was ordered as follows: clearly not contemplated; partially not contemplated; neutral; partially contemplated; clearly contemplated.
The constructs were formulated based on the theoretical assumptions of Lizarraga 12 adapted by Cotta et al. in 2011 10 and 2013 3 in which cognitive skills were divided as follows: comprehensive thought, which promotes skills such as comparison, classification, analysis, synthesis and the creation of arguments; critical thought, which promotes skills such as the investigation of the reliability of sources, the interpretation of causes, analogical reasoning and deduction; creative thought, which refers to the possibility of generating ideas, establishing associations, producing images, creating metaphors and establishing goals.In addition, theoretical aspects related to studies by Driessen et al. 9 and Gadbury-Amyot et al. 11 were included.
A pilot study was carried out by the research team in order to adjust the formulation of the inventory items and their respective instructions.Ten post-graduate student portfolios were analyzed from the area of health promotion.It is worth noting that the appraiser used the items of the inventory scale for each activity in the portfolio to help analyze the portfolios.

Procedure
In total, 70 portfolios, designed between 2008 and 2013, were analyzed.Three properly trained appraisers performed the analysis.In order to increase inter-appraiser reliability, the appraisers discussed the items of classification of two portfolios, which were excluded from the present study.The analysis was performed blind by all appraisers.

Data analysis
When assessing the measurement properties of an adapted tool, Guillemin et al. 13 proposed an assessment of the reliability and validity of the tool.
The Kappa test, weighted between the three appraisers, was used to analyze reproducibility, confirming the level of agreement between the appraisers.The Kappa values were divided as follows: 0 -poor agreement; from 0 to 0.20 -mild agreement; from 0.21 to 0.40 -considered agreement; from 0.41 to 0.60 -moderate agreement; from 0.61 to 0.80 -substantial agreement; from 0.81 to 1.0 -excellent agreement.
Cronbach's alpha was used to analyze internal consistency.Index values above 0.70 indicated a good construct.Pearson's correlation coefficient was used to correlate each of the items in relation to each other and their constructs.
Descriptive analysis (mean and standard deviation) and the t-test were performed with the data to compare results from the years the portfolios were produced (2008 to 2013).The data were analyzed using SPSS 20.0 and Stata 11.0 software.

Ethical approval
The present study received approval from the Ethics Committee of the Universidade Federal de Viçosa (UFV).According to resolution 466/2012 of Brazil's National Health Council, which regulates studies involving humans, a statement of informed consent was signed by those who agreed to participate in the research.The participants were guaranteed confidentiality and anonymity.

Results
The most significant finding of the present study was the high reliability of the analysis tool (Inventory of collective portfolio quality), as shown in Table 1.The constructs (layout and organization, comprehensive, reflective, critical and creative thought) exhibited Cronbach alpha values greater than 0.7, which confirmed the high reliability of the tool in terms of analyzing the portfolio.
The weighted Kappa test revealed an elevated and significant agreement between the appraisers, taken two by two.The mean level of agreement was greater than 90%, as shown in Table 2.
Therefore, it is worth noting that the analysis tool and the analysis performed by the appraisers were reliable in terms of the assessment conducted, which involved a careful and detailed pre-test, as well as the knowledge and experience of the appraisers in relation to the construction of portfolios.
With regards to the quality of the portfolio, the constructs present in the inventory (layout and organization, comprehensive, reflective, critical and creative thought) can be seen in Table 3, along with their mean and standard deviation values.
As can be seen in Table 3, most of the items exhibited values higher than 3 (neutral and generally contemplated) and these values increased considerably in the years 2011 and 2012.The items with values below 3 were associated with reflective thought when assessing the students capacity to build and rebuild concepts (item 9), thereby demonstrating the student´s difficulty in presenting literature related to the adequate construction of concepts.Another item with values below 3 (also associated with reflective thought) was the presentation of strengths and weaknesses during the construction of the portfolio (item 11).Some of the portfolios omitted this item entirely or only included it in one phase.The critical thought construct (item 16), which deals with deep and theoretical coherence of the evidence in the portfolio, also exhibited a mean value below 3.This was due to the fact that the bibliographical references were not evident or there were few connections between texts.
In the creative thought construct, item 20 exhibited a value below 2 due to the fact that the creation of a group in virtual communities was not obligatory and was meant to facilitate communication and teamwork, as well as helping decision-making and allowing students to make optimal use of the short time they had for meetings.This decreased the final mean value of this construct somewhat.Notably, this item also exhibited the greatest variability between the portfolios analyzed, particularly in the years after 2010, bearing in mind that no portfolio had previously exhibited this characteristic.When analyzing the entire period of the years studied (2008 to 2103), the greatest variability was associated with items 3, 11 and 19.
In spite of the fact that these items exhibited mean values below 3, the vast majority of the items had a final mean value greater than 3.Over the years, a number of items and constructs have reached values higher than 4, including the following: layout and organization (item 1 from 2009 to 2012, 2 and 3 from 2010 to 2012); comprehensive thought (item 5, 6, 7 and 8 in 2011 and 2012); reflective thought (item 10 in 2011); critical thought (item 15 in 2011 and 2012) and creative thought (item 19 and 20 in 2011 and 2012).Figure 1 displays the mean values for the constructs by year for the portfolios assessed.
One notable finding of the present study is the growth of the total mean value of the constructs over time, which indicates appropriation of the process of portfolio construction.The t-test of the mean values, displayed in Table 4, reveals a significant difference, indicating that all of the constructs were assessed more adequately from 2011 onwards.This in turn indicates the construction of better portfolios, as a result of the acquisition of skills and an improvement in the method.The capacity for comprehensive, reflective, critical and creative thought improved This assessment tool involves learning objectives and assessments for each item, including competence (skills, knowledge and attitudes).This tool enabled a clarification of the method, the assessment criteria and the understanding of the reflective, critical and creative processes.
The portfolios exhibited significantly higher values in the items of reflective and critical thought, which indicates that the students analyzed the views of different studies and scholars to form their own opinions.The analysis of the evidence (cartoons, reports, scientific articles) involves more than a list of facts: it explores themes; it questions the opinions provided; it makes connections and inter-connections between the texts and/or documents of the portfolio; ideas are formed leading to a creative and innovative presentation.
The construct with the greatest final mean value in each year was layout and organization.This demonstrates the ease with which the students organized the portfolio using the required steps, providing supplementary evidence and working in a team to develop and create the portfolio.

Discussion
The findings of the present study indicate improvements in the collective portfolio over the years as a method of teaching, learning and assessment, in terms of the acquisition of important skills on behalf of students.The portfolio contributed to an understanding of the content related to health policy and helped the development of the student's skills and attitudes, particularly in relation to reflective, critical and creative thought 3,10 .
Gadbury-Amyot et al. 11 suggested that portfolios are a valid and reliable method of assessing the competence of a student.Likewise, Driessen et al. 9 stated that the portfolio is a precise tool when assessing reflective competence.
One of the great authors of contemporary pedagogy, Dewey 14 was cited by Rué 15 , who identified four central aspects that are fundamental to all learning: experience; data for reflection; management and generation of ideas; retention of what was learned.Dewey 14 said that reflection is more than simply an ingredient that is necessary for success.It is in fact an intelligent experience in itself.Thus, reflective training becomes indispensable in the context of professional training.However, it demands conditions of effectiveness that incorporate the desire to reflect, as well as the development of knowledge, using determined activities and materials 15 .
The changes noticed in portfolios over the years demonstrate the importance of strategies created to help students in the process of acquiring skills (assessment and self-assessment tool, definition of learning objectives), as demonstrated by Driessen et al. 9 , and highlight the significance of the capacity and work of students during the reflection process.Based on the increased score of the constructs, it is clear that the items of the constructs were enhanced after the implementation of the assessment and self-assessment tool in 2011.The portfolio has improved as a method of learning and is now more than just an archive folder (with significant characteristics in the layout and organization).It now constitutes a method that focuses the triad of critical-reflective-creative thought and has become an effective method of assessing skills.
With regards to the teaching, learning and assessment processes promoted by the portfolio, the student should receive all of the necessary guidelines at the beginning, including the learning objectives to be achieved, the skills to be de-

Conclusion
The collective portfolio is an important method in skills training.It requires organizational and structural resources that provide the quality necessary to achieve the proposed objectives.In addition to the skills exercised in an individual portfolio, the collective portfolio also promotes teamwork, contributing to the reflection process and the management of conflicts that are inherent in the decision making process.It also contributes to the development of leadership skills and strategies that involve the effective participation of all members of a team in collective tasks and activities.These factors help to prepare the student to work in a multi-professional environment.In addition, feedback related to the collective portfolio is not only provided by the professor, as is the case in individual portfolios, but by the students partners as well, thereby contributing to their process of self-assessment.Further studies are required to ascertain significant differences among the skills promoted by collective and individual portfolios.The success of the portfolio depends on strategies that provide the students with greater clarity and forms of developing the method, particularly in terms of reflection and critical thought.The analysis of the 70 portfolios revealed that reflective and critical thought have improved in the years 2011 and 2012.This may be the result of the introduction of the assessment and self-assessment tool.
The structure (layout and organization) exhibited significant values, including some greater than 4 (which is the measurement for the scale that is usually used), during all of the years studied.Particular and specific characteristics were presented in the way ideas were organized and presented, which also indicated the development of creativity.In the portfolio assessments highlighted by the literature, there is little evidence in this category.However, the present study presents this aspect as an essential and determining characteristic, within a traditional curricular context, of creating innovative alternatives.
Thus, the inventory of analysis of portfolio quality, which was designed to analyze collective portfolios, was successful in terms of innovating and guiding the analysis of the portfolio and could be adapted to the realities and contexts of different learning processes.

Collaborations
GD Costa: draft the article, conception and design; acquisition of data, analysis and interpretation of data and final approval of the version to be published.E Driessen: critical revision of the manuscript to improve intellectual content and final approval of the version to be published.LS Silva, AAO Campos and CP Donateli: acquisition of data and final approval of the version to be published.TMT Costa: analysis and interpretation of data and final approval of the version to be published.RMM Cotta: orientation and supervision of the research, revising it critically and final approval of the version to be published.

1 9203 6
The portfolio is easy to use and the data are easily found 0.8345 2 The stages/sub-divisions (set apart) are easily understood 3 The students worked hard to prepare the layout of the portfolio 4 The students went beyond what was expected of them Comprehensive Thought 5 The students compared and assessed the different points of view and the content of the portfolio 0.There is the formation of hypotheses and ideas.arriving at coherent conclusions with concepts and theories related to health policies 7 The portfolio contains evidence of understanding the role of Health Policy in general and of the SUS system in particular 8 The portfolio contains evidence of investment in training for the citizenship of the students Reflective Thought 9 There is revision, construction and reconstruction of concepts 0.8616 10 The analysis performed by the students is extensive and transcends a list of facts or situations 11 The students listed the strengths and weaknesses in the process of creating the portfolio 12 The objectives related to proposed reflective thought at the beginning of the discipline were clearly and logically reached Critical Thought 13 Whenever relevant, statements are accompanied by evidence that supports what is being stated in the portfolio 0.9208 14 The students used different types of sources and evidence in a critical manner 15 There is evidence of the incorporation of the content studied in the discipline 16 There is in-depth and theoretical coherence to the evidence presented in the portfolio 17 The data are useful and significant in the construction of learning and are clearly and coherently related to the aims of the discipline Creative Thought 18 The students produce and develop ideas 0.7307 19 There is evidence of the transference of the creative process to the content associated with the discipline 20 The students used Information and Communication Technology (ICT)

Figure 1 .
Figure 1.Evolution of mean values by construct in the analysis of portfolios created by graduate students from health courses in a federal university between 2008 and 2012.

Table 1 .
Constructs, items and internal reliability of the inventory of the analysis of portfolio quality.

Table 2 .
Kappa tests and agreement between the appraisers by item of the inventory of analysis of portfolio quality.

Table 3 .
Mean and standard deviation per item of each construct of the inventory of analysis of portfolio quality.

SD Mean SD Mean SD Mean SD Mean SD Mean SD
SD -Standard deviation.

Table 4 .
Mean summed scale of each construct before and after 2011 and t-test results for the differences between mean values.