The creation of performance evaluation indicators through a focus group

Abstracts

Este estudo, desenvolvido na perspectiva da pesquisa-ação, buscou construir os prognosticadores de avaliação de desempenho profissional para técnicos e auxiliares de enfermagem atuantes no Hospital Universitário da Universidade de São Paulo. A coleta de dados foi realizada por meio da técnica de grupo focal com a participação de 19 profissionais de nível médio, representantes das unidades do Departamento de Enfermagem. Ao longo de sete encontros, os participantes elaboraram as definições de sete prognosticadores considerados, por eles e seus pares, como relevantes para retratar o adequado desempenho dessas categorias profissionais. Nos seus relatos, evidenciaram que a estratégia adotada possibilitou-lhes a expressão e a reflexão a respeito dos significados e sentimentos atribuídos ao processo de avaliação de desempenho. Na avaliação do trabalho realizado, os integrantes do grupo focal verbalizaram que, além de sentirem-se mais preparados para o enfrentamento dos problemas referentes à avaliação de desempenho, sentiram-se valorizados pela participação na composição do novo instrumento.

avaliação de desempenho; grupos de estudo; recursos humanos em saúde; enfermagem

Este estudio, desarrollado a partir de la perspectiva de investigación-acción, buscó construir pronosticadores de evaluación de desempeño profesional para técnicos y auxiliares de enfermería que trabajan en el Hospital Universitario de la USP. La recolección de datos fue realizada por medio de la técnica de grupo focal, con la participación de 19 profesionales de nivel medio, representantes de las unidades del Departamento de Enfermería. A lo largo de siete encuentros, los participantes elaboraron las definiciones de siete pronosticadores, considerados por ellos y sus colegas como relevantes y que retratan el adecuado desempeño de estas categorías profesionales. Sus relatos evidenciaron que la estrategia adoptada posibilitó la expresión y la reflexión respecto a los significados y sentimientos atribuidos al proceso de evaluación de desempeño. En la evaluación del trabajo realizado, los integrantes del grupo focal expresaron que, además de sentirse mas preparados para enfrentar los problemas sobre la evaluación de desempeño, se sintieron valorizados por la participación en la composición del nuevo instrumento.

evaluación del rendimiento de empleados; grupos focales; recursos humanos en salud; enfermería

ORIGINAL ARTICLE

The creation of performance evaluation indicators through a focus group

Vera Lúcia Mira Gonçalves^I; Antônio Fernandes Costa Lima^II; Nanci Crisitano^III; Martha Rumiko Kaio Hashimoto^IV

^IRN, Doctor Professor, College of Nursing, e-mail: [e-mail]

^IIRN, PhD, Technical Director at the University Hospital Support Service, e-mail: [e-mail]

^IIIRN, Master Student in Nursing, Pediatric Unit Head at the University Hospital, e-mail: [e-mail]

^IVRN, M.Sc., College of Nursing, Child Emergency Medical Service Head at the University Hospital, e-mail: [e-mail]. University of São Paulo

ABSTRACT

This study was developed in an action research perspective and aimed to create professional performance evaluation indicators for nursing technicians and auxiliaries working at the University Hospital of the University of Sao Paulo. Data were collected through the focus group technique, involving 19 secondary-level professionals, representing different Nursing Department units. During seven meetings, participants elaborated definitions of seven indicators they and their peers considered relevant to picture the adequate performance of these professional categories. In their reports, they manifested that the adopted strategy allowed them to express themselves about the meanings and feelings attributed to the performance evaluation process. In assessing the activity, the focus group members verbalized that, besides feeling more prepared to face problems related to performance evaluation, they also felt valued by their participation in the composition of the new instrument.

Descriptors: employee performance appraisal; focus groups; health manpower; nursing

INTRODUCTION

Nowadays, quality has become the survival word, making health institutions rethink their structures, processes and work relations. The search for professionals prepared for team work increases every day, within a participative, cooperative and synergic perspective and, mainly, trained to detect errors and propose solutions to the problems they encounter⁽¹⁾.

The importance of the relationship between health service quality and the personal performance is unquestionable. Thus, professional performance assessment should be a permanent concern in hospital organizations, in order to promote the development of human resources to reach care quality⁽²⁾.

The performance evaluation was originally structured to measure the employee's performance and potential. This systematic assessment is performed by supervisors or other hierarchical superiors who are familiar with the work routines and demands. It is traditionally defined as a process that aims to objectively measure performance and to provide collaborators with information about their own performance, in order to allow them to improve it without compromising their independence and motivation to work⁽³⁾.

For these performance evaluation results to favor all those participating in the process, whether the collaborator being evaluated, evaluator or yet the organization, this assessment must allow for the identification of specific abilities that need to be developed or acquired by the evaluated. Additionally, it must improve quality of life at work. Therefore, it is necessary to transform current evaluation systems into instruments that are useful to all of its users, such as prevention, identification and problem-solving tools⁽⁴⁾.

With this perspective in mind, the Nursing Department (ND) of the University of São Paulo Teaching Hospital (TH-USP) has been developing a performance evaluation process since the beginning of its activities, in 1981, which has been systematically analyzed through research^(5-8). The research results have showed that, despite the appointed difficulties, nurses believe in performance assessment as an important managerial resource for personnel management.

The main problem reported by the nurses was related to the evaluation instrument, a closed instrument that did not attend to the institution's needs and, especially, did not correspond to people's expectations, neither regarding professional requirements nor in terms of the possibility of expression.

In order to deepen investigations and work with more planned interventions to restructure the performance evaluation process, another study was developed⁽⁸⁾ through action-research, involving division Directors and Section Heads at the ND.

The above mentioned study resulted, among other things, in the creation of new nursing performance assessment prognosticators and showed the need to continue the discussion process and the reconstruction of prognosticators for nursing technicians and auxiliaries. This study aims to describe how this reconstruction process happened.

OBJECTIVES

To create professional performance evaluation prognosticators for nursing technicians and auxiliaries.

METHODOLOGY

This study was developed as an action-research. This method was chosen because it is a social research strategy which presumes the participation of the people involved in the research problem, in which there exists broad and explicit interaction between the researcher and those involved in the research situation⁽⁹⁾.

The research was carried out after authorization by the HU-USP Teaching and Research Committee and the Research Ethics Committee, a complementary entity of the University of São Paulo that develops teaching, research and community services and offers integral multidisciplinary care of medium complexity, based on the epidemiological profile of the Butantã Health School District, São Paulo, Brazil.

This study aimed to construct assessment prognosticators for nursing technicians and auxiliaries in the same way as for nurses⁽⁸⁾, privileging expression by the people involved. Considering that there were 474 nursing technicians and auxiliaries, this method made it impossible for everyone to participate in the research. Therefore, qualitative representative sampling was adopted, which values "interpretative or argumentatively controlled" criteria, constituting "intentional samples"⁽⁹⁾.

Thus, a work group was composed with 19 professionals, in which one nursing technician or auxiliary represented each ND unit, constituting the focus group to propose the assessment prognosticators for these functional categories. From this group, 14 participated in most of the meetings and five sporadically, justifying their absences by the unit's dynamics and personal problems. These participants were acknowledged by their group, committed and with spontaneous knowledge about the research situation, which enriched the development of this work.

Considering the proposed objective; the research problem, which constitutes a real work situation; and the adopted method, which allows for the integration of various techniques involving people's participation in group processes, data collection was carried out by means of focus group technique. This method offers the possibility to think collectively about an issue that is part of the participants' routine.

The focus group technique was chosen because, through the group, the individual acquires his/her identity, is able to recognizing his/her own form from reality, as a mirror that reflects his/her image⁽¹⁰⁾.

The group is an instrument to transformation both individual and group reality, because it promotes interaction among its members, who influence each other's opinions, and this exchange promotes the group's growth. In addition, the group can be understood as a learning instrument that promotes a reflection, able to transform every day dilemmas into thoughts that are better adjusted to reality⁽¹¹⁾.

Seven meetings of three hours each were held on March 11^th; April2^nd, 8^th, 16^th and 23^rd; May 20^th and June 17^th 2004. The meetings were recorded to facilitate recovery and analysis of the discussed content. The focus group participants received explanations about the research objectives, which were also the objectives of ND management. The latter's task was limited to the creation of performance evaluation prognosticators. They were also informed about the work strategy that established specific objectives and tasks for all meetings, as well as discussion, the group's proposal to realize the task and work synthesis. After all explanations, participants signed the Free and Informed Consent Term which assured anonymity and voluntary participation.

Considering the complexity of the task and its being a polemic issue, two researchers were responsible for coordinating the meetings. They aimed to promote conditions to stimulate group members' participation and create a favorable and trusty environment to reach the group's objectives.

The coordinators systematically began the discussions by synthesizing the content presented by the group. This procedure aimed to facilitate the teaching-learning process, bearing in mind that the discovery of values and beliefs would result from an eminently personal search⁽¹⁰⁾. Thus, the group's expressions and deliberations were analyzed based on specific literature about professional performance assessment in human resource management.

RESULTS

A synthesis of the meetings is presented, highlighting the most relevant aspects of the construction of prognosticators to evaluate the performance of nursing technicians and auxiliaries who worked at the TH-USP.

The first meeting aimed to: present the work proposal to the group; define the meeting program and contextualize the performance evaluation process of the ND at the TH-USP. Thus, the coordinators emphasized it would be a participative process, expressing that the group's attention would focus on the content of the new instrument, that is, on the choice of prognosticators and their definitions, because the same evaluation technique would be used for the new instrument created to evaluate nursing performance.

It is important to say that this instrument previewed five assessment levels, according to the achievement of requirements: Does not achieve the requirements; Achieves the requirements less than satisfactorily; Achieves the requirements satisfactorily; Totally achieves the requirements and Exceeds the requirements. The concept totally achieves the requirements corresponds to the performance standard required by the institution and is defined in the evaluation prognosticator.

It was emphasized that the success of each focus group meeting depended on the participants' involvement. It would be necessary to share tasks with a view to contemplating medium-level professionals' opinions at the respective units they represented. The participants showed enthusiasm and willingness to assume the challenge. Some of them reported that they considered participating in the process was an important opportunity, making decisions and being responsible for the results.

The objective of the second meeting was to start surveying how the participants conceived the desired characteristics for the functions of nursing technician and auxiliary. Thus, the coordinators asked them to reflect about the activities developed by nursing technicians and auxiliaries and to highlight the professional and personal characteristics needed to exercise these functions.

The discussion continued with participants manifesting themselves about the content under analysis and, at the end of the meeting, group members were oriented to ask their colleagues at their respective units to do the same. The objective was to proceed with a more comprehensive and representative investigation about characteristics they considered important.

Some participants inquired about the need to separate the characteristics of nursing technicians and auxiliaries and create two different instruments, taking into account training differences. This decision was left for the next meeting in order to allow participants to consult their colleagues.

The third, fourth and fifth focus group meetings aimed to deepen the discussion of professionals and personal characteristics surveyed at the units, aggregating them by affinity and representativeness, in order to compose the definitions of the performance evaluation prognosticators.

Before that, at the start of the third meeting, the discussion about the construction of different instruments for nursing technicians and auxiliaries was resumed. After reflecting and pondering, the group concluded that, despite the training differences, both performed the same activities at the TH-USP. Also, as there was no differentiation in their salaries, the requirements should be the same for both functions, represented by a single instrument.

The participants reported a lack of interest and some disbelief by their colleagues towards the performance assessment process, which made it difficult to obtain their collaboration to survey desirable characteristics for their function. Additionally, during the meetings, the group reported difficulties to clearly and objectively define the prognosticators that would picture what professional performance corresponded to the concept Totally achieves the requirements.

Among the characteristics appointed by the group as desirable for care delivery by nursing technicians and auxiliaries, we highlight some groupings that preceded the definition of the prognosticators:

- assuming and complying with appointments, tasks, obligations, time schedule, executing the proposed activities;

- being aware of one's actions; being knowledgeable for one's actions; honesty; respecting and maintaining ethical principles; not being conniving; being professional; assuming errors; being ethical;

- being secure in performing activities; being zealous and protect that and those under one's care; being rational; acknowledging limits and seeking orientation; respecting norms, philosophy and mission (of the HU);

- being solidary; bonding with the patient; being cooperative and accepting other people's help; cooperation;

- friendly; polite; nice; teamwork; being respectful; being constant; emotional balance; not having a bad temper; trustworthy; doing what one says; welcoming; being solidary;

- not talking loud; not making fun of friends; not being ironic; not exposing others; knowing how to listen; adequate verbal and non-verbal communication; having good communication; dialogue and talk; knowing how to make yourself understandable with clear and simple language;

- readable writing; no Portuguese errors; clear, objective, succinct; adequate language to the patient's or professional's understanding; voice tone adequate to the environment, situation or people. Not being aggressive; transmitting trust and credibility; having knowledge, coherence between what one says and does.

- having a good presentation and according to the environment; acting according to the environment; adequate clothing and accessories; asking please, excuse me, saying thanks; respecting patients' fears; education; right attitudes in different situations; socialization with different personalities; emotional control; calm and patience;

- having a good relationship with the work team, respecting hierarchy; attentive; good humored;

- seeking new knowledge; keeping one's functional knowledge updated; attending courses; expanding limits and knowledge

- having technical knowledge about equipment and technologies; knowing norms and tasks to execute efficiently; trustworthy and safe work; knowing how and why one is doing the procedure; asking when in doubt; general perspective of the sector; knowing what priorities to set;

- having good sense to organize; start and finish, if unfinished leave things in the right direction; leave the work one has started in proper conditions to be continued by the next; actions that favor care continuity; define action; favor action continuity; prioritize what must be done; adequate resource use; using without wasting to reduce costs; keeping things organized; broad perspective.

- being committed to the work/institution; participating and accompanying changes; seeking knowledge and information for relatives; favoring personal and professional performance.

In order to define the prognosticators, each grouping was thoroughly discussed and, when a consensus was not reached, the coordinators directed the discussion to another grouping. This procedure aimed to assure enough time for the participants to reflect about exposed doubts. Reflections were always conducted so as to avoid focus group members losing the perspective that the prognosticators would interrelate and mutually complementary in order to compose the desired profile.

No quinto encontro, conforme argumentação e consenso do grupo, foi concluída a elaboração das definições dos prognosticadores, apresentadas a seguir, e novamente discutidas em profundidade nessa ocasião:

During the fifth meeting, according to group discussion and consensus, the definition of prognosticators was concluded. These are presented below and were again thoroughly discussed during that meeting:

Responsibility - Means being conscious of one's acts, assuming them according to ethical and legal principles. Attending prescheduled appointments. Being secure in the performance of one's activities, acknowledging both difficulties and limitations. Being assiduous, not unnecessarily absent from work. Being punctual, rarely late. Communicating ahead in case of delay or absence.

Posture - Is respectful and polite to people keeping oneself calm and patient in difficult situations.

Personal appearance: adequate clothing and accessories; personal hygiene.

Interpersonal relationship - Is friendly, polite, solidary, attentive,welcoming and reliable.

Is constant in terms of emotional balance. Is not bad tempered. Cooperates and collaborates with people and allows for collaboration.

Technical-scientific knowledge - Keeps oneself updated in the function. Knows how to perform activities securely, efficiently and guided by technical-scientific knowledge.

Verbal communication: oral and written - Communicates clearly, objectively and succinctly. Makes oneself understood through language adequate to the person to whom one talks. Adequate voice tone for the environment, situation and person. Transmits security in communication. Talks and writes correctly and has readable writing.

Organization - Knows how to organize and prioritize activities to be performed. Conducts actions in order to favor the continuity of care and teamwork. Uses available resources adequately, avoids wasting. Keeps the work place organized.

Interest - Is committed to the work and institution. Seeks knowledge to facilitate and favor one's professional and personal performance. Participates in and follows institutional changes.

During the sixth meeting, the group gathered to approve the prognosticator definitions. Therefore, these definitions were printed on the new instrument and delivered beforehand to the participants for final approval and testing.

Each group member assumed the commitment to test the new instrument at his/her unit, asking a nurse and a nursing auxiliary or technician to do the evaluation and self-evaluation, respectively. In this test, both evaluator and evaluated were invited to express their opinions with respect to the new instrument, suggesting changes they judged necessary. The instruments tested at different ND units were sent to the Educational Support Service, responsible for coordinating the ND assessment process, and then analyzed by the focus group coordinators and other researchers.

During the seventh and final meeting, the test results were evaluated. It was verified that there were no suggestions to change the definitions of prognosticators. Thus, those present at the meeting decided to use the new instrument for all performance evaluations of both nursing technicians and auxiliaries as from July 1^st, 2004.

At the end of this meeting, the group participants reported that the experience gave them personal and professional motivation, since they felt respected and valued for finding an opportunity to participate.

DISCUSSION

The group reported difficulties to achieve involvement and participation of the colleagues they represented. These were exposed during the meetings in some statements that illustrated the disbelief with respect to the performance evaluation. These statements evidenced that problems not related to the evaluation itself, but due to the interpersonal and power relations culturally established at the institution. The group was then stimulated by the coordinators to reflect about these issues, with a view to a critical analysis of reality and proposals for pertinent and possible changes.

As expected, another difficulty experienced by the group was the fact that the prognosticators fragmented the evaluated professionals in characteristics, although the coordinators continually emphasized that the objective of the prognosticators was to facilitate the diagnosis of specific subjects and that it was people's task, both evaluators and evaluated, to correlate the prognosticators and reintegrate the performance, showing the individual as a whole. Therefore, it was also pointed out that it is essential to facilitate the training of those involved in the evaluation process.

The definition of the Responsibility prognosticator presupposes ethical behavior and commitment to acquire technical knowledge to exercise the function and assumes the intent to detect one's own deficiencies. Finally, it redeems assiduity and punctuality in a more practical way and directs them to viable problems.

In order to assess individual responsibility, it is necessary to know the desired work in order to perform the activities in the established schedule and as required, assessing quality according to the assumed ethical behavior⁽⁸⁾.

Responsibility precedes the assumption of actions and faults, precedes the errors, responsibility is not committing errors, is having knowledge so as not to commit errors. This is established in articles 16 to 20, Chapter III of the Nursing Professionals Ethics Code⁽¹²⁾. This aspect refers to the responsibility for one's own training, showing how much the assessed professional is concerned and acts to solve his/her difficulties, seeking external resources, and the extent to which (s)he uses the resources available at or offered by the institution.

With regard to Posture, greatly discussed in the group, it fundamentally carries the meaning of respect to others, discretion in actions and in personal appearance and emotional balance. Participants were unanimous about the importance of posture in the discussions, demonstrating demonstrate knowledge of the requirement level, although it was not an easy task to define it objectively.

In general, literature restricts posture to personal appearance. Other authors have called this prognosticator "Equilibrium and maturity". They define it as not being impulsive, not fleeing from problems and dealing with pressures and frustrations, besides respecting others and being discrete⁽¹³⁾. This definition partially corresponds to the requirement designed by the group and partially goes beyond the group's requirement, because it imposes other unmentioned requirements.

Discernment and order in the execution of activities was attributed to the prognosticator Organization, including concerns to facilitate care continuity and to be accountable for the rational use of available resources.

Oral and written communication was punctually defined, not as interaction between people but as objectiveness to talk, listen and write. This was the group's option to evaluate communication, knowing that one of the main disruptions in organizations is related to people's difficulties to communicate adequately. Generally, communication problems reflect deeper dysfunctions, such as extremely directive and excessively centered managerial styles and, possibly, badly defined attributions⁽¹⁴⁾.

Interpersonal relationship was one of the prognosticators the group most valued, including relations among professionals, patients and family, as well as concern with teamwork.

Technical-scientific knowledge was one of the first prognosticators highlighted by the group. In literature, this concept inevitably appears and has been evaluated by theoretical and practical knowledge of function-related activities. Some authors include the measurement of professional experience⁽¹³⁾, while others evaluate knowledge updating and knowledge of related areas⁽¹⁵⁾. The group's definition limited it to the evaluated function.

The central idea of the prognosticator Interest is related to commitment with organizational strategies and objectives, because it comprises both the importance of attempting to improve one's performance and participation in organizational changes.

CONCLUSIONS

A research is always richer that its report. It could be perceived from the meetings that the group enriched the discussions, aggregating information that went beyond those necessary to perform the task, improving reflections and contributing to the involved people's training.

In this sense, the adopted research method, i.e. action-research, permitted associating a learning ability with the research process⁽⁹⁾. The group itself manifested increased awareness of reality. In addition, the discussions and explanations from the researchers made it possible to generate descriptive and more critical knowledge of the situation.

The motivation and valorization group participants manifested at the end of the meetings evidenced that the adopted strategy permitted going beyond the creation of evaluation prognosticators, as it resulted in a significant instrument for medium-level professionals and promoted possibilities for expression and reflection about the meanings and feelings participants and their partners attributed to the performance evaluation process.

Focus group participants expressed in the work evaluation that, besides feeling more prepared to cope with evaluation problems, they felt more valued by their participation in the construction of the instrument.

Participation is considered to be a shared process, in which people get emotionally involved, in group situations, which encourage them to contribute to the group's objectives and to assume responsibility to reach them. Thus, decisions must come from the group, through consensus and people's maximum involvement and commitment⁽¹⁶⁾.

This participation implies using people's intellectual potential, to improve the quality of decisions and administration as well as to improve people's satisfaction and motivation⁽¹⁷⁾.

Although this work in itself cannot be considered participative administration, this initiative showed that more democratic management methods can and should be adopted, turning rigid structures more flexible and stimulating people's involvement in the work processes.

Recebido em: 9.11.2005

Aprovado em: 20.2.2006

• 1. Silva MJP. Valor da comunicação para o sucesso dos processos de qualidade. In: Mello JB, Camargo MD. Qualidade de saúde. Práticas e conceitos. Normas ISSO nas Áreas Médico-Hospitalar e laboratorial. São Paulo (SP): Best Seller; 1998. p. 69-76.
• 2. Balbueno EA, Nozawa MR. Levantamento dos tipos de repercussões resultantes da avaliação de desempenho em enfermagem hospitalar. Rev Latino- am Enfermagem. 2004; 12(1): 5-8-64.
• 3. Lara JF, Silva MB. Avaliação de desempenho no modelo de gestão por competências: uma experiência de utilização [monograph on the Internet]. Porto; 2005 [cited 2005 june 23]. Available from: http://www.psicologia.com.pt/artigos
• 4. Castro GAO, Lima GBC, Veiga MRM. Implantação de um sistema de avaliação de desempenho: métodos e estratégias. Rev Admin 1996; 31(3):38-52.
• 5. Leite MMJ, Gualda DMR, Gonçalves VLM, Fugulin FMT, Castilho V, Ortiz DCF et al. Análise do instrumento utilizado no processo de avaliação de desempenho da equipe de enfermagem do Hospital Universitário da USP: relatório de pesquisa. São Paulo (SP): FAPESP; 1997.
• 6. Gonçalves VLM. Crenças, sentimentos e capacitação dos enfermeiros no processo de avaliação de desempenho da equipe de enfermagem. [dissertação]. São Paulo (SP): Escola de Enfermagem/USP; 1998.
• 7. Leite MMJ, Gonçalves VLM, Castilho V, Gualda DMR, Fugulin FMT, Coan TCM, et al. Dificuldades e facilidades do avaliador no processo de avaliação de desempenho da equipe de enfermagem do Hospital Universitário da USP: relatório de pesquisa. São Paulo (SP): FAPESP; 1999.
• 8. Gonçalves VLM. Reconstruindo o processo de avaliação de desempenho da equipe de enfermagem do Hospital Universitário da USP. [Tese]. São Paulo (SP): Escola de Enfermagem/USP; 2003.
• 9. Thiollent M. Metodologia da pesquisa-ação. 5ª ed. São Paulo (SP): Cortez; 1992.
• 10. Castilho A. Dinâmica do trabalho em grupo. Rio de Janeiro (RJ): Quality Mark; 1998.
• 11. Gonçalves VLM, Leite MMJ, Ciampone MHT. Cogitare Enfermagem junho 2004; 9(1):50-9.
• 12. Código de Ética Conselho Regional de Enfermagem de São Paulo. Documentos básicos de enfermagem: enfermeiros, técnicos e auxiliares. São Paulo: COREN-SP; 2001.
• 13. Lacombe F, Heilborn G. Administração: princípios e tendências. São Paulo (SP): Saraiva; 2003.
• 14. Boog GG, Boog MT. Energize sua empresa!: como os florais podem dinamizar seu ambiente de negócios. São Paulo (SP): Gente; 1997.
• 15. Marras JP. Administração de recursos humanos: do operacional ao estratégico. 3ª ed. São Paulo (SP): Futura; 2000.
• 16. Chiavenato I. Gerenciando pessoas: como transformar os gerentes em gestores de pessoas. 4ª ed. São Paulo (SP): Prentice Hall; 2002.
• 17. Maximiano ACA. Teoria geral da administração: da escola científica à competitividade na economia globalizada. 2ª ed. São Paulo: Atlas; 2002.

Publication Dates

• Publication in this collection
  15 Mar 2007
• Date of issue
  Feb 2007

History

• Accepted
  20 Feb 2006
• Received
  09 Nov 2005