Accessibility / Report Error

Assessment of professional competence of nurses in emergencies: created and validated instrument

Evaluación de la competencia profesional del enfermero en urgencias: Instrumento creado y validado

ABSTRACT

Objective:

To create an instrument to assess the professional competence of nurses in emergencies.

Method:

Methodological study carried out between 2013 and 2016 in a Brazilian university. We used the Pasquali's model and Psychometrics as references because they allow subjective phenomena to be measured. The survey covered defining steps of the behavior to be measured, the formulation of the items in the instrument, development of instructions for using the instrument and elements necessary to demonstrate validity evidence. Although we have previously performed, in a distinct and consecutive way, the literature review, definition of the Core (or Matrix) Competence and the Competence Profile, and identification of validity with Delphi, essential components to structure a technology, in this step we continued the conclusion of these Theoretical Procedures.

Result:

We created an instrument for self- and/or hetero-evaluation containing: personal/professional/academic characterization, Competency Scale, fictitious cases and actions representing the practice.

Conclusion:

We created a new soft-hard technology based on the validity evidence of contents according to experts of the five regions of Brazil.

Descriptors:
Assessment of Professional Performance; Professional Competence; Nursing in Emergency; Psychometrics; Validity of Tests

RESUMEN

Objetivo:

Crear Instrumento de Evaluación de la Competencia Profesional del Enfermero en Urgencias.

Método:

Estudio metodológico realizado entre 2013 y 2016 en universidad brasileña. Se utilizó el modelo de Pasquali y la Psicometría como referenciales, pues permiten que fenómenos subjetivos sean medidos. La investigación abarcó etapas de definición del comportamiento a ser medido, la formulación de los ítems de la herramienta, desarrollado de instrucciones para el uso del instrumento y de elementos necesarios para demostrar evidencias de validez. Aunque ya haya sido realizada previamente de manera distinta y consecutiva la revisión de la literatura, la definición de la Matriz y del Perfil de Competencias e identificación de validez con Delphi, los componentes esenciales para estructurar una tecnología, se dio continuidad a la finalización de esos Procedimientos Teóricos en esta etapa.

Resultado:

Instrumento creado para auto y/o heteroevaluación conteniendo: la caracterización personal/profesional/académica, la Escala de Competencias, los casos ficticios y las acciones representativas de la práctica.

Conclusión:

Se construyó una tecnología blanda-dura novedosa basada por las evidencias de validez del contenido consensuado por expertos de las cinco regiones de Brasil.

Descriptores:
Evaluación de Desempeño Profesional; Competencia profesional; Enfermería en Urgencia; Psicometría; Validez de las pruebas

RESUMO

Objetivo:

Criar Instrumento de Avaliação da Competência Profissional do Enfermeiro em Emergências.

Método:

Estudo metodológico realizado entre 2013 e 2016 em universidade brasileira. Usou-se o modelo de Pasquali e a Psicometria como referenciais, permitindo que fenômenos subjetivos sejam medidos. A pesquisa abrangeu etapas de definição do comportamento a ser medido, formulação dos itens da ferramenta, desenvolvimento de instruções para uso do Instrumento e de elementos necessários para demonstrar evidências de validade. Embora realizadas previamente de forma distinta e consecutiva revisão da literatura, definição da Matriz e do Perfil de Competências e identificação de validade com Delphi, componentes essenciais para estruturar uma tecnologia, deu-se continuidade à finalização desses Procedimentos Teóricos nesta etapa.

Resultado:

Instrumento criado para auto e/ou heteroavaliação contendo: caracterização pessoal/profissional/acadêmica, Escala de Competências, casos fictícios e ações representativas da prática.

Conclusão:

Construiu-se tecnologia leve-dura inédita alicerçada pelas evidências de validade do conteúdo consensuado por experts das cinco regiões brasileiras.

Descritores:
Avaliação de Desempenho Profissional; Competência Profissional; Enfermagem em Emergência; Psicometria; Validade dos Testes

INTRODUCTION

In Brazil, as of 2010, there was a reorganization of the Brazilian Unified Health System (SUS) in five Thematic Health Care Systems, which considered the aspect of justice and efficiency, focusing on the situations of greater vulnerability in this system. Specifically the Health Care to Emergencies was organized in components interconnected to each other, to promote health care to acute and chronic conditions, generating various challenges to professionals in this health care(11 Brasil. Ministério da Saúde. Portaria nº 1.600, de 7 de julho de 2011. Reformula a Política Nacional de Atenção às Urgências e institui a Rede de Atenção às Urgências no Sistema Único de Saúde-SUS. Brasília: Ministério da Saúde[Internet]. 2011 [cited 2017 Feb 01]. Available from: http://bvsms.saude.gov.br/bvs/saudelegis/gm/2011/prt1600_07_07_2011.html
http://bvsms.saude.gov.br/bvs/saudelegis...
).

Henceforth, one of the challenges of the nurse manager in emergency services is to identify in each nurse their competencies and if such must be developed, in order to adapt the professionals to the new context. Since nurses who work in the emergency room stand out for their technical responsibility in taking care of their patients both for their work and that of the nursing staff, without the expense of the administrative staff, this multiplicity of actions requires the development of specific capacities to the professional exercise in level of excellence. As a result, to have an instrument that identifies existing competencies and those to be acquired becomes very meaningful to the distinction of the uniqueness of the actions for a safe, humane and professional practice and without risk to the patient, the nurse, the institution and the community. To do so, an instrument to evaluate the specific competence of nurses in their field of expertise is necessary.

The routine and systematic assessment of the level of competence of nurses in the workplace represents an important object of interest among educators, health managers and other professionals in different spheres(22 Finnbakk E, Wangensteen S, Skovdahl K, Fagerstrom L. The professional nurse self-assessment scale: psychometric testing in Norwegian long term and home care context. BMC Nurs[Internet]. 2015 [cited 2016 Nov 03];14:58. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4647290/pdf/12912_2015_Article_109.pdf
https://www.ncbi.nlm.nih.gov/pmc/article...
). It enables tracing the competencies along the nurses' professional career, their path, and the achievement of objectives(22 Finnbakk E, Wangensteen S, Skovdahl K, Fagerstrom L. The professional nurse self-assessment scale: psychometric testing in Norwegian long term and home care context. BMC Nurs[Internet]. 2015 [cited 2016 Nov 03];14:58. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4647290/pdf/12912_2015_Article_109.pdf
https://www.ncbi.nlm.nih.gov/pmc/article...
).

A literature research conducted by European researchers related to such theme identified several self-evaluation instruments for the nurse competence, but most focused on a general context(22 Finnbakk E, Wangensteen S, Skovdahl K, Fagerstrom L. The professional nurse self-assessment scale: psychometric testing in Norwegian long term and home care context. BMC Nurs[Internet]. 2015 [cited 2016 Nov 03];14:58. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4647290/pdf/12912_2015_Article_109.pdf
https://www.ncbi.nlm.nih.gov/pmc/article...
). In our study, we found that in Brazil, so far, there is no published study on a technology to create an instrument capable of measuring the professional competence of the nurse in emergencies in a systematic and scientific way, whether to use it in self- or hetero-evaluation or as a support for the nursing manager or the professionals themselves.

In this context, the creation of a specific instrument capable of measuring the professional competence of the nurse working in emergencies was essential, based on the profiles of nurses, patients, the institution, and also of the Brazilian public policy for the health care provided in this area, following a specific methodology. Considering these facts, we had the following question: which procedures of the reference psychometrics are indispensable for creating an instrument to assess the professional competence of nurses in emergency?

OBJECTIVE

To create an instrument to assess the professional competence of nurses in emergencies, considering psychometrics as the theoretical-methodological reference for its development.

METHOD

Ethical aspects

To prepare the Assessment Instrument of the Professional Competence of Nurses in Emergencies, national and international standards of ethics in research involving humans were met, with approval of the Research Ethics Committee in 01/06/2012.

Study design, location and period

This was a methodological study with psychometric reference, part of a larger survey, carried out between 2013 and 2016 in a public university located in the city of São Paulo, Brazil. Testing of the instrument developed in this research occurred in April 2015 with nurses from a large and of high complexity teaching hospital, reference in the Health Care System, serving primarily patients of the Brazilian Unified Health System (SUS).

Population, sample, inclusion and exclusion criteria

Two groups of nurses took part in the testing of the instrument. The groups differed regarding the time of performance in emergencies and specialized training. The first group had seven nurses enrolled in the first and second years of the Interprofessional Residency Program in Emergency Service, being carried out in a university environment in the afternoon shift. The other group consisted of seven nursing assistants and three managers who worked in the hospital emergency service. Testing of the instrument took place during working hours, in the morning, afternoon, and night shifts in all sectors linked to the emergency service. The average time of the first group was 20 minutes, and 30 minutes for the second group.

Study protocol

The methodological research is justified because it is characterized by developing data collection instruments that can assume different purposes(33 Lobiondo-Wood G, Haber J. Nursing research: methods and critical appraisal for evidence based practice. 8th ed. New York: Mosby Elsevier; 2014.). Considering that all processes that were necessary for the preparation of this research were supported by the behaviors of the daily practice of the nurses, psychometrics has proven to be the most appropriate reference for creating the assessment instrument of the professional performance of nurses in emergencies. Psychometrics is a theory measurement technique that allows the interface of psychology with statistics, enabling the empirical observation of phenomena that cannot be directly observed(44 Pasquali L, (Org.). Instrumentação psicológica: fundamentos e práticas. Porto Alegre: Artmed; 2010.-55 Pasquali L. Princípios de elaboração de escalas psicológicas. Rev Psiquiatr Clin[Internet]. 1998 [cited 2016 Nov 03]; 25(5):206-13. Available from: http://mpet.ifam.edu.br/wp-content/uploads/2017/12/Principios-de-elaboracao-de-escalas-psicologicas.pdf
http://mpet.ifam.edu.br/wp-content/uploa...
). Moreover, it enables the systematization of a plausible explanation for the meaning of the numeric responses given by the subjects to a series of items that represent the defining behaviors of the competencies(44 Pasquali L, (Org.). Instrumentação psicológica: fundamentos e práticas. Porto Alegre: Artmed; 2010.-55 Pasquali L. Princípios de elaboração de escalas psicológicas. Rev Psiquiatr Clin[Internet]. 1998 [cited 2016 Nov 03]; 25(5):206-13. Available from: http://mpet.ifam.edu.br/wp-content/uploads/2017/12/Principios-de-elaboracao-de-escalas-psicologicas.pdf
http://mpet.ifam.edu.br/wp-content/uploa...
).

In addition, within the line established by psychometrics, we seek to create an instrument with descriptive and predictive purposes, of diagnostic classification, whose results may be used to plan interventions(66 Conselho Federal de Psicologia. Avaliação psicológica: diretrizes na regulamentação da profissão[Internet]. Brasília: Conselho Federal de Psicologia. 2010[cited 2016 Nov 03]. Available from: http://site.cfp.org.br/wp-content/uploads/2010/09/avaliacao_psicologica_web_30-08-10.pdf
http://site.cfp.org.br/wp-content/upload...
). While the descriptive purpose is related to interpret the results of the instrument to understand the aspects that highlight the most the strengths or weaknesses in terms of competencies, the predictive purpose refers to the use of the results of the instrument to anticipate other features of people's behavior with the construct assessed by measurement of competencies.

In turn, the purpose of diagnostic classification is related to the detailed examination of the results of the instrument as one of the information sources to identify people's characteristics that may be interpreted within a taxonomic system, which limits the presented levels of competence, for example, low, medium and high. Finally, the results obtained with the implementation of the instrument can also be used to support decision-making regarding interventions that may be needed in the field of competencies.

For the instrument of this study to be prepared and to meet the aforementioned purposes, the following steps were covered: definition of the behavior to be measured, formulation of the items of the instrument, as well as the development of instructions for using the instrument and the elements necessary to demonstrate validity evidence in psychometrics. When creating the Assessment Instrument of the Professional Competence of Nurses in Emergencies we used the model proposed by Pasquali, which is divided in three lines named Theoretical Procedures, Empirical/Experimental Procedures, and Analytical/Statistical Procedures(44 Pasquali L, (Org.). Instrumentação psicológica: fundamentos e práticas. Porto Alegre: Artmed; 2010.-55 Pasquali L. Princípios de elaboração de escalas psicológicas. Rev Psiquiatr Clin[Internet]. 1998 [cited 2016 Nov 03]; 25(5):206-13. Available from: http://mpet.ifam.edu.br/wp-content/uploads/2017/12/Principios-de-elaboracao-de-escalas-psicologicas.pdf
http://mpet.ifam.edu.br/wp-content/uploa...
).

Considering that our study aimed to elaborate a research instrument, we only used as reference the Theoretical Procedures, since they are the only ones needed in this stage of the study. Such procedures were the basis for creating the instrument, since these, in addition to highlighting the theory that justifies the psychological object for which we want to develop a measuring instrument, also highlight the operationalization of the construct in items(44 Pasquali L, (Org.). Instrumentação psicológica: fundamentos e práticas. Porto Alegre: Artmed; 2010.-55 Pasquali L. Princípios de elaboração de escalas psicológicas. Rev Psiquiatr Clin[Internet]. 1998 [cited 2016 Nov 03]; 25(5):206-13. Available from: http://mpet.ifam.edu.br/wp-content/uploads/2017/12/Principios-de-elaboracao-de-escalas-psicologicas.pdf
http://mpet.ifam.edu.br/wp-content/uploa...
). To ground this creation and support it scientifically there was need to meet steps in sequence: review of the Brazilian literature, creation of a mini-theory, definition of the construct and the instrument elements relevant to the assessment instrument, as shown in Figure 1.

Figure 1
Methodological organization chart of theoretical procedures for elaborating the Assessment Instrument of the Professional Competence of Nurses in Emergencies adapted from Pasquali's model, São Paulo, Brazil, 2016

We developed two integrative reviews in the Brazilian literature review, one regarding professional competence of nurses in the world of work and the other regarding emergencies, in order to identify the existence of scientific studies in the field, and from the results, to define guiding elements for creating the research instrument. From the works identified, only two theses have a theme related specifically to our research, and any of them was related to the development of an assessment instrument of nurses' performance in emergencies.

For creating the mini-theory we considered the empirical evidence obtained in the review of the literature, public policy in emergencies, the purpose of the instrument as well as the knowledge and experience of researchers in management and emergencies. We also considered the concept of nurses' professional competence as the nurses' ability to provide, coordinate and ensure proactive and individual human care, in a planned way, based on scientific and technical knowledge to achieve the best result by working on time, as we propose in this research.

First we chose the Basic Competencies and the Associated Competencies essential to the capable performance of the nurse.

It is worth mentioning that the Basic Competence is the ability of nurses to perform their work based on a set of general and specific knowledge associated with skills and attitudes that result in added value to their professional and social actions, differing them within the reality they work; on the other hand, the Associated Competence is the support required for the development of each Basic Competence for the qualified performance of the nurse. After selecting the two types of competencies, we elaborated the constitutive definition of each one of them, in order to delimit the ideas that characterized them in a Professional Core Competence (or Matrix) (77 Holanda FL, Marra CC, Cunha ICKO. Construction of a Professional Competency Matrix of the nurse in emergency services. Acta Paul Enferm[Internet]. 2014 [cited 2016 Nov 03]; 27(4):373-9. Available from: http://www.scielo.br/pdf/ape/v27n4/en_1982-0194-ape-027-004-0373.pdf
http://www.scielo.br/pdf/ape/v27n4/en_19...
). From these concepts we have developed the operational definitions, characterizing the behaviors that demonstrated the proposed competences, named Identifying Questions, to determine a Professional Competence Profile(88 Holanda FL, Marra CC, Cunha ICKO. Professional competency profile of nurses working in emergency services. Acta Paul Enferm[Internet]. 2015 [cited 2016 Nov 03]; 28(4):308-14. Available from: http://www.scielo.br/pdf/ape/v28n4/en_1982-0194-ape-28-04-0308.pdf
http://www.scielo.br/pdf/ape/v28n4/en_19...
). By elaborating this process, we created the mini-theory which allowed us to ground the development of the construct we wanted to include in the assessment instrument of each professional competence of nurses. From this Core Competence and the Competence Profile, we continue the creation process of the Assessment Instrument, using the Delphi method, according to which expert nurses evaluated, using the five-point Likert scale, agreement or disagreement quantitative data, and, in a proper space for comments and suggestions, the interpretative content of relevance, clarity, and compliance. The comments suggested minor adjustments in writing and the inclusion of a new professional competence. Thus, we created and conceptualized the Professional Growth associated competence, as well as its Identifying Question, which were inserted into the Assistance Performance basic competence. The Delphi method was used in four steps with a national sample of expert nurses on the subject: 1st Step: n = 25; 2nd Step: n = 21; 3rd Step: n = 21; and 4th Step: n = 18. In the last step the questions evaluated with the Likert scale obtained 90% consensus of Percentage Score and 98.61 of Content Validity Index, demonstrating validity evidence.

When defining the construct, each of these Identifying Questions were arranged into items to represent only one action, but without disassociating the central idea of each behavior already described. After arranging them in items, their content was submitted to experts to verify understanding, clarity, and objectivity. Then, the items started to integrate the assessment instrument in the part pertaining to the measurable actions in daily practice of the nurse in emergencies, in level of excellence.

For creating this instrument, it was necessary to define all the elements that would compose it: spreadsheet with the actions to be measured on the daily practice of nurses in emergencies and the levels of competence with the possible alternatives of responses; scale of assessment of the degree of professional competence of nurses in emergencies; personal and professional characterization data of the subjects; and the description of cases representing the reality to verify the competence level of nurses in their daily practice.

The spreadsheet contained 81 items arranged at random in its lines, constituting a bank of items. The degree of competence to be marked was in five columns according to the assessment of what each item represented to the nurse, ranging from five, equal to the extremely competent level, to one, corresponding to not competent at all.

To resolve doubts about what each competence level means, which were presented as response alternatives in the spreadsheet, we developed a scale that differentiated the quality of devotion to the care provided in each action within the five levels. The assessment scale of the levels of professional competence of nurses in emergencies was understood as a set of data that define quantitative and qualitative variables to verify the quality of the actions of nurses in their daily practice in emergencies, as showed in Chart 1.

Chart 1
Quantitative and qualitative variables to verify the quality regarding the actions of the daily practice of nurses in emergencies, São Paulo, Brazil, 2016

Another part of the research instrument contemplated data of personal identification and those of the nursing assistant as well as those regarding their education and professional development. These professional data of the sample included variables related to the probability of the nurses giving an answer compatible with a particular item in line with their level of competence at work.

To complement the data to characterize the level of competence at work of the research subjects, we described three fictitious cases about emergency care as a control element. We aimed to verify the ability of the respondent in analyzing each situation and rate it regarding the level of competence of the nurse in the actions described in the respective cases.

Analysis of results

Testing of the instrument with a pilot study had the purpose of identifying possible limitations of the product and the process. Regarding the product, the focus was to verify difficulties as for the language, understanding of the text, and use of the scale of competence level, both in the spreadsheet and in the cases. As for the process, special attention was given to the time taken to fill the assessment instrument and the most suitable place for its application.

With the pilot test we verified the need for making minor changes in the writing of a few items, and in the second group, we deemed necessary for the nurse manager to do the hetero-evaluation of the nursing assistant only after these have done the self-evaluation.

RESULTS

The assessment instrument of the professional competence of nurses in emergencies was divided into five parts. Part A, entitled characterization of nurses to be evaluated, contained data regarding age, sex, marital status, number of children, place, time, working time, and leave of absence. Part B, entitled professional training of nurses, included graduation year, specialization course or graduate program in specialized fields, and emergency programs completed in the last two years. We also verified participation in scientific events, study groups, commissions and committees, and scientific activities already carried out. Parts C, D and E are presented respectively in Charts 2, 3 and 4, totally or in an exemplified way.

Chart 2
Scale of assessment of the levels of competence of nurses in emergency services, São Paulo, Brazil, 2016
Chart 3
Example of one of the fictitious cases representing the reality for assessing the level of competence of nurses, considering their own daily practice, São Paulo, Brazil, 2016
Chart 4
Representative items of actions of nurses in emergencies proposed by Brazilian researchers, São Paulo, Brazil, 2016

DISCUSSION

Creating a measurement instrument for assessing the professional competence of nurses in emergencies was quite a challenge, because this is a new area in Brazilian Nursing, which demands knowledge of other specific areas such as psychometrics. Although this measurement instrument was not a psychological test, doing the research from the perspective of a psychometric reference allowed us to characterize, describe, diagnose, and predict behaviors with defined standards regarding what was expected(66 Conselho Federal de Psicologia. Avaliação psicológica: diretrizes na regulamentação da profissão[Internet]. Brasília: Conselho Federal de Psicologia. 2010[cited 2016 Nov 03]. Available from: http://site.cfp.org.br/wp-content/uploads/2010/09/avaliacao_psicologica_web_30-08-10.pdf
http://site.cfp.org.br/wp-content/upload...
). When defining exactly what the construct predicts was extremely relevant to indicate how care should be managed, how the training of nurses should be planned, including how the performance of such professionals should be evaluated in such a way to intervene in time.

When creating the instrument, two processes had to be involved: the creation of the instrument itself and the demonstration of validity evidence. Although distinct, they are interrelated. None was more important than the other, since both had focused on structures that depict the behavior to be measured. Considering it was a long research with several steps, we verified that the existing processes related to the creation of the instrument were the most emphasized at this step of the research.

According to Pasquali, to survey the scientific production carried out on the subject during this process is important, since the research can corroborate and support the construction of the instrument(44 Pasquali L, (Org.). Instrumentação psicológica: fundamentos e práticas. Porto Alegre: Artmed; 2010.-55 Pasquali L. Princípios de elaboração de escalas psicológicas. Rev Psiquiatr Clin[Internet]. 1998 [cited 2016 Nov 03]; 25(5):206-13. Available from: http://mpet.ifam.edu.br/wp-content/uploads/2017/12/Principios-de-elaboracao-de-escalas-psicologicas.pdf
http://mpet.ifam.edu.br/wp-content/uploa...
). Identifying evidence in the literature to guide the preparation of the construct and systematize it was not a particularity of our study, but it is a global reality(99 Schwirian PM. Six dimension scale of nursing performance. Columbus: Ohio University College of Nursing; 1978.

10 Schwirian PM. Evaluation the performance of nurses: a multidimensional approach. Nurs Res. 1978;27(6):347-51.

11 Benner P. From novice to expert, excellence and power in clinical nursing practice. New Jersey: Prentice-Hall; 1984. p.13-34.

12 Emergency Nurses Association. Competencies for nurse practitioners in emergency care. Des Palines[Internet]. 2011 [cited 2016 Nov 03]. Available from: http://c.ymcdn.com/sites/www.nonpf.org/resource/resmgr/competencies/compsfornpsinemergencycarefinal.pdf
http://c.ymcdn.com/sites/www.nonpf.org/r...

13 The National CNS Competency task force. Clinical Nurse Specialist Core Competencies. Executive summary 2006-2008[Internet]. 2010 [cited 2016 Nov 03]. Available from: http://www.nacns.org/docs/CNSCoreCompetenciesBroch.pdf
http://www.nacns.org/docs/CNSCoreCompete...

14 ENA NP Validation Work Team, Hoyt KS, Coyne EA, Ramirez EG, Peard AS, Gisness C, et al. Nurse practitioner Delphi study: competencies for practice in emergency care. J Emerg Nurs[Internet]. 2010 [cited 2016 Nov 03]; 36(5):439-49. Available from: https://www.ncbi.nlm.nih.gov/pubmed/20837213
https://www.ncbi.nlm.nih.gov/pubmed/2083...

15 Emergency Nurses Association. Competencies for clinical nurse specialists in emergency care. Des Palines[Internet]. 2011 [cited 2016 Nov 03]. Available from: https://www.ena.org/practice-research/Practice/Quality/Documents/CNSCompetencies.pdf
https://www.ena.org/practice-research/Pr...

16 College of Emergency Nursing Australasian. Practice standards for the emergency nursing specialist[Internet]. 2013 [cited 2016 Nov 03]. 3rd ed. Available from: http://www.cena.org.au/wp-content/uploads/2014/10/Practice_Standards_for_the_Emergency_Nurse_Specialist_June_14.pdf
http://www.cena.org.au/wp-content/upload...

17 Meretoja R, Leino-Kilpi H. Comparison of competence assessments made by nurse managers and practising nurses. J Nurs Manag[Internet]. 2003 [cited 2016 Nov 03]; 11(6):404-9. Available from: https://www.ncbi.nlm.nih.gov/pubmed/14641722
https://www.ncbi.nlm.nih.gov/pubmed/1464...

18 Meretoja R, Isoaho H, Leino-Kilpi H. Nurse competence scale: development and psychometric testing. J Adv Nurs[Internet]. 2004 [cited 2016 Nov 03]; 47(2):124-33. Available from: https://www.ncbi.nlm.nih.gov/pubmed/15196186
https://www.ncbi.nlm.nih.gov/pubmed/1519...

19 Zhang J, Ye W, Fan F. Development of a self-assessment tool for measuring competences of obstetric nurses in rooming-in wards in China. Int J Clin Exp Med[Internet]. 2015 [cited 2016 Nov 03]; 8(10):18548-59. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4694368/pdf/ijcem0008-18548.pdf
https://www.ncbi.nlm.nih.gov/pmc/article...

20 Kitreerawutiwong K, Sriruecha C, Laohasiriwong W. Development of the competency scale for primary care managers in Thailand: Scale development. BMC Fam Pract[Internet]. 2015;16(1):174. 2015[cited 2016 Nov 03];16(1):174. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4673780/pdf/12875_2015_Article_388.pdf
https://www.ncbi.nlm.nih.gov/pmc/article...

21 Harding AD, Walker-Cillo GE, Duke A, Campos GJ, Stapleton SJ. A framework for creating and evaluating competences for emergency nurses. J Emerg Nurs[Internet]. 2013 [cited 2016 Jan 15]; 39(3):252-64. Available from: http://www.girardslaw.com/library/ED_Nurse_Competency_Framework_2013.pdf
http://www.girardslaw.com/library/ED_Nur...
-2222 EdCaN. Competency assessment in nursing: a summary of literature published since 2000[Internet]. Melbourne: Allison Evans Consulting. 2008[cited 2016 Nov 03]. Available from: http://edcan.org.au/assets/edcan/files/docs/EdCancompetenciesliteraturereviewFINAL_0.pdf
http://edcan.org.au/assets/edcan/files/d...
). Considering the lack of Brazilian literature focused on the professional competence of nurses in emergencies, we have proposed, based on our extensive experience, Basic and Associated Competencies and Identifying Questions to enable the definition of actions to be measured in the instrument, properly represented in 81 items described in the spreadsheet.

We did not find assessment instruments of the competencies of nurses both in the wide researched Brazilian literature and in international literature. The American nurse Patricia Schwirian can be considered the pioneer in creating an assessment instrument on this issue. After the literature review and consultation with experts, the nurse created a scale of nurses' performance according to the theoretical-methodological procedures of psychometrics(99 Schwirian PM. Six dimension scale of nursing performance. Columbus: Ohio University College of Nursing; 1978.-1010 Schwirian PM. Evaluation the performance of nurses: a multidimensional approach. Nurs Res. 1978;27(6):347-51.).

Considering competence proposals for the nurse to work in emergencies, we verified that in the new millennium(1212 Emergency Nurses Association. Competencies for nurse practitioners in emergency care. Des Palines[Internet]. 2011 [cited 2016 Nov 03]. Available from: http://c.ymcdn.com/sites/www.nonpf.org/resource/resmgr/competencies/compsfornpsinemergencycarefinal.pdf
http://c.ymcdn.com/sites/www.nonpf.org/r...

13 The National CNS Competency task force. Clinical Nurse Specialist Core Competencies. Executive summary 2006-2008[Internet]. 2010 [cited 2016 Nov 03]. Available from: http://www.nacns.org/docs/CNSCoreCompetenciesBroch.pdf
http://www.nacns.org/docs/CNSCoreCompete...

14 ENA NP Validation Work Team, Hoyt KS, Coyne EA, Ramirez EG, Peard AS, Gisness C, et al. Nurse practitioner Delphi study: competencies for practice in emergency care. J Emerg Nurs[Internet]. 2010 [cited 2016 Nov 03]; 36(5):439-49. Available from: https://www.ncbi.nlm.nih.gov/pubmed/20837213
https://www.ncbi.nlm.nih.gov/pubmed/2083...

15 Emergency Nurses Association. Competencies for clinical nurse specialists in emergency care. Des Palines[Internet]. 2011 [cited 2016 Nov 03]. Available from: https://www.ena.org/practice-research/Practice/Quality/Documents/CNSCompetencies.pdf
https://www.ena.org/practice-research/Pr...

16 College of Emergency Nursing Australasian. Practice standards for the emergency nursing specialist[Internet]. 2013 [cited 2016 Nov 03]. 3rd ed. Available from: http://www.cena.org.au/wp-content/uploads/2014/10/Practice_Standards_for_the_Emergency_Nurse_Specialist_June_14.pdf
http://www.cena.org.au/wp-content/upload...

17 Meretoja R, Leino-Kilpi H. Comparison of competence assessments made by nurse managers and practising nurses. J Nurs Manag[Internet]. 2003 [cited 2016 Nov 03]; 11(6):404-9. Available from: https://www.ncbi.nlm.nih.gov/pubmed/14641722
https://www.ncbi.nlm.nih.gov/pubmed/1464...
-1818 Meretoja R, Isoaho H, Leino-Kilpi H. Nurse competence scale: development and psychometric testing. J Adv Nurs[Internet]. 2004 [cited 2016 Nov 03]; 47(2):124-33. Available from: https://www.ncbi.nlm.nih.gov/pubmed/15196186
https://www.ncbi.nlm.nih.gov/pubmed/1519...
) trade associations and nursing researchers from other countries identified and elaborated constitutive and operational definitions of competencies. In the United States of America, for instance, the Emergency Nursing Association (ENA) summoned nurses and, based on the “core competencies” created, established ten entry level competencies and 60 items for Nurse Practitioner (NP), and seven competencies and 25 items for Clinical Nurse Specialist (CNS)(1414 ENA NP Validation Work Team, Hoyt KS, Coyne EA, Ramirez EG, Peard AS, Gisness C, et al. Nurse practitioner Delphi study: competencies for practice in emergency care. J Emerg Nurs[Internet]. 2010 [cited 2016 Nov 03]; 36(5):439-49. Available from: https://www.ncbi.nlm.nih.gov/pubmed/20837213
https://www.ncbi.nlm.nih.gov/pubmed/2083...
-1515 Emergency Nurses Association. Competencies for clinical nurse specialists in emergency care. Des Palines[Internet]. 2011 [cited 2016 Nov 03]. Available from: https://www.ena.org/practice-research/Practice/Quality/Documents/CNSCompetencies.pdf
https://www.ena.org/practice-research/Pr...
). In the Australian continent, expected performance standards were defined for the nurse specialist in emergency containing nine areas of competence and 112 items(1616 College of Emergency Nursing Australasian. Practice standards for the emergency nursing specialist[Internet]. 2013 [cited 2016 Nov 03]. 3rd ed. Available from: http://www.cena.org.au/wp-content/uploads/2014/10/Practice_Standards_for_the_Emergency_Nurse_Specialist_June_14.pdf
http://www.cena.org.au/wp-content/upload...
). In Europe, nurse Riitta Meretoja, with collaboration of other authors, have proposed seven competencies and 73 items for nurses working in different sectors, among them, the hospital emergency(1717 Meretoja R, Leino-Kilpi H. Comparison of competence assessments made by nurse managers and practising nurses. J Nurs Manag[Internet]. 2003 [cited 2016 Nov 03]; 11(6):404-9. Available from: https://www.ncbi.nlm.nih.gov/pubmed/14641722
https://www.ncbi.nlm.nih.gov/pubmed/1464...
-1818 Meretoja R, Isoaho H, Leino-Kilpi H. Nurse competence scale: development and psychometric testing. J Adv Nurs[Internet]. 2004 [cited 2016 Nov 03]; 47(2):124-33. Available from: https://www.ncbi.nlm.nih.gov/pubmed/15196186
https://www.ncbi.nlm.nih.gov/pubmed/1519...
).

In our study, in addition to the content of the items to assess the construct, we defined the scale of assessment of the degree/level of professional competence of nurses in emergencies. We also sought, in addition to grading the competence, to describe the quality of devotion to the provided care, and we did not find similar scales for nurse evaluation in all consulted literature, specifically in this area.

In a more general scope, in the United States of America, in 1982, the American Patricia Benner developed and applied the classic model of the Dreyfus brothers and established a clinical competence scale with five levels of abilities: Novice, Advanced Beginner, Competent, Proficient and Expert. In her research she sought to distinguish the differences between such levels. She theorized that the experience in clinical practice is the basis for nurses' performance, since it allows to continuously expand their knowledge ensuring a holistic and competent care to the patient(1111 Benner P. From novice to expert, excellence and power in clinical nursing practice. New Jersey: Prentice-Hall; 1984. p.13-34.). This scale has been used and approved in several studies. In Finland, nurse Riitta Meretoja, with the collaboration of other authors, proposed an assessment instrument of nurses with a 0-100-value visual analog scale to measure levels of competence, considering the weak, moderate, good, and very good alternatives. She applied it in various sectors of a hospital, including the emergency(1717 Meretoja R, Leino-Kilpi H. Comparison of competence assessments made by nurse managers and practising nurses. J Nurs Manag[Internet]. 2003 [cited 2016 Nov 03]; 11(6):404-9. Available from: https://www.ncbi.nlm.nih.gov/pubmed/14641722
https://www.ncbi.nlm.nih.gov/pubmed/1464...
-1818 Meretoja R, Isoaho H, Leino-Kilpi H. Nurse competence scale: development and psychometric testing. J Adv Nurs[Internet]. 2004 [cited 2016 Nov 03]; 47(2):124-33. Available from: https://www.ncbi.nlm.nih.gov/pubmed/15196186
https://www.ncbi.nlm.nih.gov/pubmed/1519...
). In China, researchers have created a five-point scale with options of strongly agree to disagree of the item described in an assessment instrument of nurses' competence in obstetrics(1919 Zhang J, Ye W, Fan F. Development of a self-assessment tool for measuring competences of obstetric nurses in rooming-in wards in China. Int J Clin Exp Med[Internet]. 2015 [cited 2016 Nov 03]; 8(10):18548-59. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4694368/pdf/ijcem0008-18548.pdf
https://www.ncbi.nlm.nih.gov/pmc/article...
).

Considering the literature found in more than one continent, we could identify several components of the assessment instruments of nurses' competencies. In order to allow a more accurate analysis of the replies given by the research subjects in the instrument, other elements were also included:-demographic characterization of respondents, data regarding their professional training and development, and the use of fictional cases. Such indicators are justified because the respondents, when it comes to an assessment, may underestimate or overestimate their ability, and it is necessary to combine information and resolve inconsistencies, using multimethods and multitrainers.

Although empirical evidence about the most effective or safe indicators are still scarce, there is a general consensus in the literature that the competence assessment should use more than one indicator such as self-evaluation, hetero-evaluation, direct observation, Objective Structured Clinical Examination (OSCES), and realistic simulation, among others(2121 Harding AD, Walker-Cillo GE, Duke A, Campos GJ, Stapleton SJ. A framework for creating and evaluating competences for emergency nurses. J Emerg Nurs[Internet]. 2013 [cited 2016 Jan 15]; 39(3):252-64. Available from: http://www.girardslaw.com/library/ED_Nurse_Competency_Framework_2013.pdf
http://www.girardslaw.com/library/ED_Nur...
-2222 EdCaN. Competency assessment in nursing: a summary of literature published since 2000[Internet]. Melbourne: Allison Evans Consulting. 2008[cited 2016 Nov 03]. Available from: http://edcan.org.au/assets/edcan/files/docs/EdCancompetenciesliteraturereviewFINAL_0.pdf
http://edcan.org.au/assets/edcan/files/d...
).

Regarding the behavior of health professionals in the self-evaluation, in studies that more emphatically address this subject, doctors and nurses are the professionals that most perform such evaluation. In a systematic review conducted in 2006, which aimed to determine the precision with which doctors perform self-evaluation compared with external observations, the researcher grouped the results into methods of self-evaluation, external evaluation, comparison between both of them, and the accuracy of the self-evaluation(2323 Davis DA, Mazmanian PF, Fordis M, Harrison RU, Thorpe KF, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA[Internet]. 2006 [cited 2016 Nov 03];296:(9):1094. Available from: http://innovationlabs.com/r3p_public/rtr3/pre/pre-read/Self-assessment.Systematic%20review.Davis.2006.pdf
http://innovationlabs.com/r3p_public/rtr...
). The researcher identified that self-evaluations were carried out through questionnaires, checklists or research focused on the needs of learning, general clinical skills, knowledge of medical evaluation and critical assessment as well as non-clinical skills. Moreover, the self-evaluation was compared with stable external indicators of measurement such as OSCEs, simulations, training performance, and the ability to explain evidence-based concepts to an interviewer(2323 Davis DA, Mazmanian PF, Fordis M, Harrison RU, Thorpe KF, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA[Internet]. 2006 [cited 2016 Nov 03];296:(9):1094. Available from: http://innovationlabs.com/r3p_public/rtr3/pre/pre-read/Self-assessment.Systematic%20review.Davis.2006.pdf
http://innovationlabs.com/r3p_public/rtr...
). In New Zealand, researchers used an assessment instrument of the performance of nurses and doctors in a realistic simulation of emergency(2424 Overeem K, Wollersheim HC, Arah OA, Cruijsberg JK, Grol RPTM, Lombart KMJMH. Evaluation of physicians' professional performance: an iterative development and validation study of multisource feedback instruments. BMC Health Services Res[Internet]. 2012 [cited 2016 Nov 03]; 12(1):80 Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/pmid/22448816/
https://www.ncbi.nlm.nih.gov/pmc/article...
).

Finally, it is worth mentioning that we are developing new studies for demonstrating validity evidence and accuracy to scientifically analyze the use and interpretation of the results of this assessment instrument of nurses who work in emergencies, to be reported in future publications.

Study limitations

Despite all the care that surrounded the creation of this instrument, there were some limitations because it is a pioneering study in Brazil without the possibility of comparison with equivalent proposals. Such datum has highlighted the need to move forward with the research, and we recommend joining forces between nursing researchers, since this procedure can strengthen them in this area.

Contributions to the nursing field

The survey allowed contributing in the advancement of nursing in creating new technologies regarding professional self-evaluation and/or hetero-evaluation as a starting point for future studies.

CONCLUSION

The Assessment Instrument of the Professional Competence of Nurses in Emergencies was created and developed based on validity evidence of the content of the items, in compliance with all the steps of the theoretical procedures contained in the psychometric reference. We hope this can be a mean for identifying strengths and weaknesses in such a way to improve the existing level of competence and/or the development of new competencies.

  • FUNDING
    This research was funded by the National Council for Scientific and Technological Development (CNPq), process no. 483.449/2013-0.

ACKNOWLEDGEMENTS

We thank all the nurses for the valuable contributions to the development of the Assessment Instrument of the Professional Competence of Nurses in Emergencies.

REFERENCES

  • 1
    Brasil. Ministério da Saúde. Portaria nº 1.600, de 7 de julho de 2011. Reformula a Política Nacional de Atenção às Urgências e institui a Rede de Atenção às Urgências no Sistema Único de Saúde-SUS. Brasília: Ministério da Saúde[Internet]. 2011 [cited 2017 Feb 01]. Available from: http://bvsms.saude.gov.br/bvs/saudelegis/gm/2011/prt1600_07_07_2011.html
    » http://bvsms.saude.gov.br/bvs/saudelegis/gm/2011/prt1600_07_07_2011.html
  • 2
    Finnbakk E, Wangensteen S, Skovdahl K, Fagerstrom L. The professional nurse self-assessment scale: psychometric testing in Norwegian long term and home care context. BMC Nurs[Internet]. 2015 [cited 2016 Nov 03];14:58. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4647290/pdf/12912_2015_Article_109.pdf
    » https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4647290/pdf/12912_2015_Article_109.pdf
  • 3
    Lobiondo-Wood G, Haber J. Nursing research: methods and critical appraisal for evidence based practice. 8th ed. New York: Mosby Elsevier; 2014.
  • 4
    Pasquali L, (Org.). Instrumentação psicológica: fundamentos e práticas. Porto Alegre: Artmed; 2010.
  • 5
    Pasquali L. Princípios de elaboração de escalas psicológicas. Rev Psiquiatr Clin[Internet]. 1998 [cited 2016 Nov 03]; 25(5):206-13. Available from: http://mpet.ifam.edu.br/wp-content/uploads/2017/12/Principios-de-elaboracao-de-escalas-psicologicas.pdf
    » http://mpet.ifam.edu.br/wp-content/uploads/2017/12/Principios-de-elaboracao-de-escalas-psicologicas.pdf
  • 6
    Conselho Federal de Psicologia. Avaliação psicológica: diretrizes na regulamentação da profissão[Internet]. Brasília: Conselho Federal de Psicologia. 2010[cited 2016 Nov 03]. Available from: http://site.cfp.org.br/wp-content/uploads/2010/09/avaliacao_psicologica_web_30-08-10.pdf
    » http://site.cfp.org.br/wp-content/uploads/2010/09/avaliacao_psicologica_web_30-08-10.pdf
  • 7
    Holanda FL, Marra CC, Cunha ICKO. Construction of a Professional Competency Matrix of the nurse in emergency services. Acta Paul Enferm[Internet]. 2014 [cited 2016 Nov 03]; 27(4):373-9. Available from: http://www.scielo.br/pdf/ape/v27n4/en_1982-0194-ape-027-004-0373.pdf
    » http://www.scielo.br/pdf/ape/v27n4/en_1982-0194-ape-027-004-0373.pdf
  • 8
    Holanda FL, Marra CC, Cunha ICKO. Professional competency profile of nurses working in emergency services. Acta Paul Enferm[Internet]. 2015 [cited 2016 Nov 03]; 28(4):308-14. Available from: http://www.scielo.br/pdf/ape/v28n4/en_1982-0194-ape-28-04-0308.pdf
    » http://www.scielo.br/pdf/ape/v28n4/en_1982-0194-ape-28-04-0308.pdf
  • 9
    Schwirian PM. Six dimension scale of nursing performance. Columbus: Ohio University College of Nursing; 1978.
  • 10
    Schwirian PM. Evaluation the performance of nurses: a multidimensional approach. Nurs Res. 1978;27(6):347-51.
  • 11
    Benner P. From novice to expert, excellence and power in clinical nursing practice. New Jersey: Prentice-Hall; 1984. p.13-34.
  • 12
    Emergency Nurses Association. Competencies for nurse practitioners in emergency care. Des Palines[Internet]. 2011 [cited 2016 Nov 03]. Available from: http://c.ymcdn.com/sites/www.nonpf.org/resource/resmgr/competencies/compsfornpsinemergencycarefinal.pdf
    » http://c.ymcdn.com/sites/www.nonpf.org/resource/resmgr/competencies/compsfornpsinemergencycarefinal.pdf
  • 13
    The National CNS Competency task force. Clinical Nurse Specialist Core Competencies. Executive summary 2006-2008[Internet]. 2010 [cited 2016 Nov 03]. Available from: http://www.nacns.org/docs/CNSCoreCompetenciesBroch.pdf
    » http://www.nacns.org/docs/CNSCoreCompetenciesBroch.pdf
  • 14
    ENA NP Validation Work Team, Hoyt KS, Coyne EA, Ramirez EG, Peard AS, Gisness C, et al. Nurse practitioner Delphi study: competencies for practice in emergency care. J Emerg Nurs[Internet]. 2010 [cited 2016 Nov 03]; 36(5):439-49. Available from: https://www.ncbi.nlm.nih.gov/pubmed/20837213
    » https://www.ncbi.nlm.nih.gov/pubmed/20837213
  • 15
    Emergency Nurses Association. Competencies for clinical nurse specialists in emergency care. Des Palines[Internet]. 2011 [cited 2016 Nov 03]. Available from: https://www.ena.org/practice-research/Practice/Quality/Documents/CNSCompetencies.pdf
    » https://www.ena.org/practice-research/Practice/Quality/Documents/CNSCompetencies.pdf
  • 16
    College of Emergency Nursing Australasian. Practice standards for the emergency nursing specialist[Internet]. 2013 [cited 2016 Nov 03]. 3rd ed. Available from: http://www.cena.org.au/wp-content/uploads/2014/10/Practice_Standards_for_the_Emergency_Nurse_Specialist_June_14.pdf
    » http://www.cena.org.au/wp-content/uploads/2014/10/Practice_Standards_for_the_Emergency_Nurse_Specialist_June_14.pdf
  • 17
    Meretoja R, Leino-Kilpi H. Comparison of competence assessments made by nurse managers and practising nurses. J Nurs Manag[Internet]. 2003 [cited 2016 Nov 03]; 11(6):404-9. Available from: https://www.ncbi.nlm.nih.gov/pubmed/14641722
    » https://www.ncbi.nlm.nih.gov/pubmed/14641722
  • 18
    Meretoja R, Isoaho H, Leino-Kilpi H. Nurse competence scale: development and psychometric testing. J Adv Nurs[Internet]. 2004 [cited 2016 Nov 03]; 47(2):124-33. Available from: https://www.ncbi.nlm.nih.gov/pubmed/15196186
    » https://www.ncbi.nlm.nih.gov/pubmed/15196186
  • 19
    Zhang J, Ye W, Fan F. Development of a self-assessment tool for measuring competences of obstetric nurses in rooming-in wards in China. Int J Clin Exp Med[Internet]. 2015 [cited 2016 Nov 03]; 8(10):18548-59. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4694368/pdf/ijcem0008-18548.pdf
    » https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4694368/pdf/ijcem0008-18548.pdf
  • 20
    Kitreerawutiwong K, Sriruecha C, Laohasiriwong W. Development of the competency scale for primary care managers in Thailand: Scale development. BMC Fam Pract[Internet]. 2015;16(1):174. 2015[cited 2016 Nov 03];16(1):174. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4673780/pdf/12875_2015_Article_388.pdf
    » https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4673780/pdf/12875_2015_Article_388.pdf
  • 21
    Harding AD, Walker-Cillo GE, Duke A, Campos GJ, Stapleton SJ. A framework for creating and evaluating competences for emergency nurses. J Emerg Nurs[Internet]. 2013 [cited 2016 Jan 15]; 39(3):252-64. Available from: http://www.girardslaw.com/library/ED_Nurse_Competency_Framework_2013.pdf
    » http://www.girardslaw.com/library/ED_Nurse_Competency_Framework_2013.pdf
  • 22
    EdCaN. Competency assessment in nursing: a summary of literature published since 2000[Internet]. Melbourne: Allison Evans Consulting. 2008[cited 2016 Nov 03]. Available from: http://edcan.org.au/assets/edcan/files/docs/EdCancompetenciesliteraturereviewFINAL_0.pdf
    » http://edcan.org.au/assets/edcan/files/docs/EdCancompetenciesliteraturereviewFINAL_0.pdf
  • 23
    Davis DA, Mazmanian PF, Fordis M, Harrison RU, Thorpe KF, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA[Internet]. 2006 [cited 2016 Nov 03];296:(9):1094. Available from: http://innovationlabs.com/r3p_public/rtr3/pre/pre-read/Self-assessment.Systematic%20review.Davis.2006.pdf
    » http://innovationlabs.com/r3p_public/rtr3/pre/pre-read/Self-assessment.Systematic%20review.Davis.2006.pdf
  • 24
    Overeem K, Wollersheim HC, Arah OA, Cruijsberg JK, Grol RPTM, Lombart KMJMH. Evaluation of physicians' professional performance: an iterative development and validation study of multisource feedback instruments. BMC Health Services Res[Internet]. 2012 [cited 2016 Nov 03]; 12(1):80 Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/pmid/22448816/
    » https://www.ncbi.nlm.nih.gov/pmc/articles/pmid/22448816/

Publication Dates

  • Publication in this collection
    Jul-Aug 2018

History

  • Received
    16 Dec 2016
  • Accepted
    04 Apr 2017
Associação Brasileira de Enfermagem SGA Norte Quadra 603 Conj. "B" - Av. L2 Norte 70830-102 Brasília, DF, Brasil, Tel.: (55 61) 3226-0653, Fax: (55 61) 3225-4473 - Brasília - DF - Brazil
E-mail: reben@abennacional.org.br