Evaluating Psychometric Properties of an Instrument Addressing Comprehensiveness of Care Among Dentists

Abstract

This study aimed to evaluate the psychometric properties of an instrument to assess comprehensiveness of care from dentists using a combination of classical test theory and item response theory. A 46-item instrument was developed and tested by a panel of experts, followed by a pilot test and administration to 187 primary care dentists in a large Brazilian city. The 46 items were evaluated using the following criteria: acceptability, internal consistency, temporal stability, inter-item correlation, and tetrachoric correlation. This evaluation led to a shortened version consisting of 11 items that met all the criteria previously described. The temporal stability was measured using Cohen’s kappa, and all 11 items presented values greater than 0.5. The Cronbach’s alpha value was 0.72. None of the 11 items had missing data on the distribution of responses, and the model considering the discrimination as varying fit the data better than the model considering discrimination as a constant parameter (p<0.001). Item characteristic curves showed that 54.5% of items could be considered difficult, i.e., only dentists with a good understanding of comprehensiveness responded favorably. The 11-item instrument to assess comprehensiveness of care by dentists is considered to have good psychometric properties.

Key Words:
public health dentistry; surveys and questionnaires; psychometrics.

Resumo

Objetivou-se avaliar as propriedades psicométricas de um instrumento para avaliar a integralidade do cuidado por dentistas, usando uma combinação da teoria clássica e teoria da resposta ao item. Um instrumento com 46 itens foi desenvolvido e testado por um painel de experts, seguido por um teste piloto e aplicação em 187 dentistas da atenção primária de uma grande cidade brasileira. Os seguintes critérios foram utilizados para avaliar os 46 itens: aceitabilidade, consistência interna, estabilidade temporal, correlação inter-item e correlação tetracórica. Essa avaliação resultou em uma versão reduzida de 11 itens que preencheram todos os critérios descritos acima. A estabilidade temporal foi medida utilizando o Cohen Kappa. Os 11 itens apresentaram valores maiores que 0,5. O alfa de Cronbach foi de 0,72. Nenhum dos 11 itens apresentou perda na distribuição das respostas, e o modelo que considera a discriminação como variante se adequa melhor aos dados do que o modelo que considera a discriminação como um parâmetro constante (p <0,001). Curvas características mostraram que 54,5% dos itens pode ser considerado difícil, ou seja, apenas os dentistas com uma boa compreensão da integralidade respondeu favoravelmente. O instrumento de 11 itens para avaliar a integralidade do cuidado por dentistas é considerado como tendo boas propriedades psicométricas.

Introduction

Many health systems worldwide consider primary health care (PHC) and its principles to be a priority, and there is no longer any doubt that PHC is essential to an effective and efficient health service ¹ along with the need to make public health systems more inclusive, equitable and fair ². Recent literature shows that fragmented, hierarchically organized health systems worldwide have shown poor sanitary and economic performance. Current theories recommend that health systems should be organized as networks in order to be better prepared to address the growing increase in chronic conditions among the population. These facts reinforce the idea that as in all forms of human diligence, society and science change over time, providing new needs and challenges ³.

In most health systems worldwide, developments in health care are initiated in PHC, and dental care plays an important role in the range of services covered by PHC. The responsibility of dentists in primary care, in addition to clinical interventions, can be extended to strengthening community actions, developing personal skills and reorienting health services ⁴. Comprehensiveness of care is a principle of PHC, along with equity and universal health access ¹. To understand its application in dentistry, it is necessary to understand that oral health not only means having healthy teeth and gums but instead requires a multifaceted approach that is influenced by the individual’s changing perceptions: good oral health really means being free from pain, discomfort and other diseases that affect the craniofacial complex. Oral health implies the possibility of a person fully performing functions, such as chewing, swallowing, speaking, smiling, smelling, touching and tasting, and conveying several emotions through facial expressions with confidence, as well as achieving social and psychological interactions that are fundamental to the quality of life. All of these attributes contribute to overall health, oral health being an essential part of physical and mental good health ⁵.

In recent years, various countries have instituted changes that may improve the traditional model of health care. Health projects and programmes have emerged in an attempt to operationalize the concepts of comprehensiveness of care, equity and universal health access. However, despite considerable development of oral health promotion and prevention actions in recent decades, this attempt is clearly more difficult in the oral health area because a large part of the population has a high need for dental treatment that was excluded from the dental care systems for years (6.7.8.9.10).

The comprehensiveness of care has been defined previously (11.12.13.14). Comprehensiveness means to look after people as whole beings in the way in which they live and to provide all of the care that the individuals may need; it indicates a comprehensive approach to people in which the full range of their health needs is recognized, and it was this direction that this study explored ^1,11,14. However, comprehensiveness is a term that has two aspects or scopes; therefore, as defined by ¹⁵, comprehensiveness of care also means guaranteeing that more health services at all levels are linked and provided to the population, ranging from health promotion/disease prevention to advanced and social care.

Comprehensive dental care is important for patients because dental care is important and is part of general health. Furthermore, the focus of our research is on the comprehensiveness approach and how it is understood by dentists. On the one hand, comprehensiveness is related to shaping health practices that promote multidisciplinary and team care. On the other hand, it is important that professionals who establish these practices understand and take part in this process because using the standards of comprehensiveness applied in PHC, health services are shaped by making a connection between spontaneous and programmed flows of patients, being able to apply diagnostic protocols and identifying risk situations for health, as well as creating locations for health promotion actions in the community ^11,12,13,14.

Human resource management is essential to improving health systems ¹⁶. Therefore, developing instruments to evaluate the views of both health professionals and patients is paramount.

The objective of this study was to evaluate the psychometric properties of an instrument to assess comprehensiveness of care in primary care settings from the understanding of dentists using a combination of classical test theory (CTT) and item response theory (IRT) analysis. This study is part of a larger research project that aimed to construct and validate a questionnaire instrument to assess primary care from the perspective of dentists, based on key concepts related to comprehensiveness of care ¹².

Material and Methods

Ethics Statement

All participants signed a written consent form. The consent form model and the research were approved by the Human Research Ethics Committees of Belo Horizonte and by the Federal University of Minas Gerais, Brazil, under the document number 0437.0.203.410-11A.

Study Location and Reference Population

The research took place in the city of Belo Horizonte, Brazil. It is the sixth largest city in the country, with approximately 2,400,000 inhabitants and an area of 330 km². The Public Health System is organized into nine sanitary districts, and patients receive dental and general PHC through a programme called Family Health Strategy. The local health system has a historical tradition in the development of primary activities and also seeks to structure its system towards the integration of actions, and this involves the Medical Specialties Centre and the Centre for Dental Specialties. At the time of the study, primary health care for dental care consisted of 264 oral health teams, including a minimum of one general dentist and a dental assistant in each of the 147 health centres in the city, with some teams also having an oral health technician ^6,10,17 who provided care similar to that of dental hygienists.

Development of the Instrument

In the first phase, the concept of comprehensiveness and its domains were studied in more depth to provide greater familiarity with the subject in identifying items.

First, we performed a literature review on the subject and established initial parameters for the development of the instrument ^12,13,14.

Four focus groups of primary care dentists were formed and used to discuss the domains of comprehensiveness of care. This stage of the project explored the following three main concepts: ‘patient welcoming’ (the acts of receiving the patients in the clinic and of responding to their problems), ‘bonding’ (the link between the patient and the health service) and ‘quality of care’. Under the umbrella of comprehensiveness, each domain explored specific aspects to be addressed in the assessment of oral health in PHC. The aspects assessed by ‘patient welcoming’ were as follows: access (geographical and organizational); posture (listening, professional-patient attitude and intra-team relationship); technique (teamwork, training of professionals, acquisition of technology/knowledge/practices) and reorientation of services (institutional project, supervision and work process). Those assessed by ‘bonding’ were as follows: affectivity (appreciation for the profession, interest in the patient’s person and professional-patient bond); continuity (expectations and perceptions of the patient, acceptance and trusting interrelation, adherence to treatment and uniqueness of the patient) and therapeutic relationship (efficacy of treatment, patient as a subject in the treatment process, professional-patient trust, attitude of care and abandonment reduction). Those assessed by ‘quality of care’ were as follows: professional conduct (technical qualification, technical/socio-environmental care and continuity of care) and professional-patient relationship (patient satisfaction and expectations). These domains and aspects considered by the framework suggest an operational and structural link between the notions of primary care as a way to assess the daily routine of services, professional practices and their relationship with the population ^{8,9,11,12,13,14}.

A draft of the instrument was developed as a product of the outcomes of the focus groups and literature review. It used a Likert-type ordinal scale with five response options: “always,” “almost always,” “sometimes,” “rarely,” and “never”. For some items, the options were “very good”, “good”, “more or less”, “bad” and “very bad”. Despite using a Likert-type ordinal scale, all the analysis was performed by dichotomizing the responses into two categories: “always” and “almost always” in one and “sometimes”, “rarely,” and “never” in the other. Similarly, the options “very good” and “good” were in one category, and “more or less,” “bad” and “very bad” were in the other.

This first version of the instrument was presented to a panel of experts composed of three professors from the Departments of Community and Preventive Dentistry from different universities in Brazil. The only inclusion criterion pertaining to this committee was a history of practical experience in PHC. The panel assessed the content and the scale that composed the instrument, and they then suggested minor changes via questions and in a participant profile session ¹⁸. Using a second version (after the panel of experts’ evaluation), two pilot tests were performed to assess reliability. The first consisted of 24 dentists, and the second consisted of 20 dentists, all from the same service as the previous group. The criterion for participation was a link with public service for at least two years. This step aimed to ascertain the degree of understanding in relation to the questions that were developed; it was the first step in providing evidence that measurements of individuals on different occasions produced the same or similar results ^19,20. To verify the performance and reliability of the instrument, the test/retest method was employed. The questionnaire was administered twice to the same individuals at an interval of 10-14 days ^{13,19,21,22,23}. The two time points were compared using a Cohen kappa analysis. The questionnaire was self-administered, and the application time varied from 15 to 22 minutes. The literature suggests that instrument items should have a Cohen kappa coefficient of greater than 0.5 to remain on the scale on which they were originally included ^18,19. These criteria led us to conduct two pilot tests. The results of the Cohen kappa coefficient obtained during the first pilot test showed the need to adjust the wording of some questions. After modifications were implemented, the instrument was subjected to a second pilot test, and from this step, we established the final version. The questionnaire had 15 items that characterized the professional and oral health service to which the dentist belonged and 46 items that evaluated primary health care (full version of the instrument).

Out of the 264 dentists in the city, 44 dentists participated in the pilot test and were excluded from the final sample. The other 13 who were part of the 264 teams at the time of data collection were on leave; therefore, they were not included. The reference population consisted of 207 tenured dentists from the Family Health Strategy (primary care level). All participants were free to respond or not respond to any item of the questionnaire instrument during the development of the research. The response rate was 90.4% (n=187 dentists).

For the development of the study, the distribution of questionnaires to the 207 dentists took place with collaboration from the staff of city hall. All envelopes were addressed to the health care clinics via inter-office mail, and the local managers from each clinic reminded the recipients and collected the questionnaires three working days later. The whole process of sending and returning the questionnaires was carried out in two different time periods at an interval of 14 days. Overall, 187 dentists responded to the questionnaire both times.

Classical Test Theory (CTT)

In summary, the 46 items were evaluated using the following criteria: acceptability, internal consistency, temporal stability and inter-item correlation ^{20,21,22,23,24,25,26,27}. In addition to those criteria from CTT, we also evaluated this instrument by tetrachoric correlation.

This evaluation led to a short version of 11 items that met all the criteria previously described. The entire analysis was conducted by dichotomizing the answers, with “always,” “almost always” in one category and “sometimes,”, “rarely” and “never” in the other for the last 10 items. For the first item, the options “very good” and “good” were combined into one category and “more or less,” “bad” and “very bad” into the other.

Item Response Theory (IRT)

The properties of the short version of the questionnaire were evaluated by means of an IRT approach using latent factor model tracing ²⁸.

The assumption of unidimensionality for IRT was evaluated based on the decomposition of the polychoric correlation matrix to identify the dominance of the first eigenvalue and the estimated Cronbach’s alpha coefficient ^27,29,30. To identify the discrimination power of each performance scale item, an item characteristic curve of the 11 items was made. Moreover, the test characteristic curve was constructed ³¹, thus generating a measure that allows for the summation of each item characteristic curve of the 11 items of the scale.

Each dentist received a score that ranged from -4 (lowest performance) to +4 (highest performance).

The software R was used to analyse the data (The R Foundation for Statistical Computing, Vienna, Austria; http://www.r-project.org). For the model adjustment, the ltm package outlined by Ritzopoulos ³² was used.

Results

Among the 187 participants, 13.9% were 60-69 years old, 41.2% were 50-59 years old, 28.3% were 40-49 years old, 12.8% were 30-39 years old and 3.8% were 20-29 years old. Women constituted 70.6% of the respondents. Regarding the amount of experience dentists had working in primary dental care, 72.2% had 11 or more years of experience, 10.7% had 6-10 years, 12.3% had 1-5 years and 4.8% had less than 1 year of experience. Approximately 85% of participants had a postgraduate degree. The frequency of each of the 11 items is described in Table 1.

CTT

Table 1
Distribution of items of the instrument, Belo Horizonte, Brazil, 2012

Item	Very good; good (n; %)	Bad; very bad; more or less (n; %)	No response (n; %)	Cohen Kappa (n)
1. In the last 60 days, how do you evaluate dental care in your Health Unit?	174; 94.57	10; 5.43	3; 1.60	0.7279
	Always; almost always (n; %)	Rarely; never; sometimes (n; %)	No response (n; %)	-
2. Does the oral health team develop actions in conjunction with other professionals in the Family Health Team?	113; 60.43	74; 39.57	_	0.5589
3. Do you plan the care of your patients with the help of other professionals in your Health Unit?	80; 42.78	107; 57.22	_	0.5800
4. Do you receive support from the health service management for the performing of your work?	148; 80.87	35; 19.13	4; 2.14	0.6073
5. Do you capacitate members of the auxiliary staff (dental health assistant - ASB - and/or community health agent - ACS; oral health technician - TSB) to carry out promotion and prevention actions in oral health?	73; 39.25	113; 60.75	1; 0.54	0.5853
6. Are training courses provided to the dentist so that he is cohesive with clinical practice advocated by Family Health?	95; 51.08	91; 48.92	1; 0.54	0.6999
7. Upon referral performing, is important information regarding the patient’s oral health passed on to the professional specialist?	163; 87.17	24; 12.83	_	0.5788
8. In your Health Unit, does the flow between primary and specialized care provide continuity of care?	88; 47.06	99; 52.94	_	0.5822
9. In your Unit, is the same patient treated by the oral health team at different times of his life (childhood, adolescence, adulthood, etc.)?	144; 82.29	31; 17.71	12; 6.42	0.5776
10. Do you know the major health problems of the community and help to face them?	115; 63.19	67; 36.81	5; 2.67	0.6254
11. Do courses and trainings for the oral health team enable to improve the care and services provided to the population?	136; 73.51	49; 26.49	2; 1.07	0.6480

Acceptability

The acceptability of the instrument was evaluated by studying the distribution of responses and of missing data per item. Missing data must not exceed one-third of the unanswered items for a person ²⁴. None of the 11 items had missing data.

Internal Consistency

The Cronbach’s alpha coefficient was used to assess the internal consistency of the scale and the homogeneity of the items belonging to a given dimension. Its value varies from 0 to 1, and values of 0.70-0.89 are considered to show good internal consistency ^19,21,22. The Cronbach’s alpha coefficient was 0.72 after considering the 11 items.

Temporal Stability

The test-retest reliability of the questionnaire was assessed using a Cohen kappa analysis, which was selected for evaluating the temporal stability of the developed scale. The cut-off for selecting items was a Cohen kappa value of 0.5 or greater ^18,19. Values are reported in Table 1.

Inter-Item Correlation

The inter-item correlation was conducted to eliminate item redundancy. The criterion for retaining items in the questionnaire would be values for inter-item correlations less than or equal to 0.75; however, if the item presented higher values, it would be considered redundant ²³. All items with these characteristics were removed.

Assumptions of IRT

The correlations between the 11 activities were positive and statistically significant, ranging from 0.02 to 0.74. In the decomposition of the matrix of polychoric correlation, according to their eigenvalues and eigenvectors, values of 4.37 and 1.38 were observed for the first two eigenvalues, which corresponds to an explanation of the variance of 39.7% and 12.6%, respectively.

Estimation of Parameters and Adjustment of the Model

The model that considers the discrimination to be varying fits the data better than the model that considers discrimination as a constant parameter (p<0.001). The estimated values of q (Perception of comprehensiveness of care) varied between -2.41 and 1.34. The mean was -0.03 (SD = 0.81), and the median was -0.10, indicating a slight asymmetry of the distribution of scores (Fig. 1).

Figure 1
Distribution of dentists’ understanding of comprehensiveness of care, Brazil, 2012. (Mean = -0.03; SD = 0.81; Median = 0.10).

Figures 2 and 3 present the item characteristic curves. Figure 2 shows the items used that are related to dental health care and patient welcoming. Items 1 (“In the last 60 days, how do you evaluate dental care in your Health Unit?”) and 4 (“Do you receive support from the health service management for the performance of your work?”) show that dentists with a q less than 0 reported performing these activities, with a probability near 1. In contrast, dentists with a q of approximately 2 responded positively to items 2 (“Does the oral health team develop actions in conjunction with other professionals in the Family Health Team?”), 3 (“Do you plan the care of your patients with the help of other professionals in your Health Unit?”), 5 (“Do you capacitate members of the auxiliary staff (dental health assistant - ASB - and/or community health agent - ACS; oral health technician) to carry out promotion and prevention actions in oral health?”) and 6 (“Are training courses provided to the dentist so that he/she is cohesive with the clinical practices advocated by Family Health?”), with a probability near 1.

Figure 2
Item curve characteristics. Questions about dental health care and welcoming patients, Brazil, 2012.

Figure 3
Item curve characteristics. Questions about bonding and quality of care, Brazil, 2012.

Figure 3 shows the items used relating to bonding and quality of care. Items 7 (“Upon referral, is important information regarding the patient’s oral health passed on to the professional specialist?”), 9 (“In your unit, is the same patient treated by the oral health team at different times of his life (childhood, adolescence, adulthood, etc.)?”) and 11 (“Do courses and trainings for the oral health team enable you to improve the care and services provided to the population?”) show that dentists with a q less than 0 reported performing these activities, with a probability near 1. In contrast, dentists with a q of approximately 2 responded positively to items 8 (“In your Health Unit, does the flow between primary and specialized care provide continuity of care?”) and 10 (“Do you know the major health problems of the community and help face them?”), with a probability near 1.

Figure 4 shows the test information curve, which is obtained based on the sum of the information of each of the items. It can be seen that the test provides the mostmore information for values between -2 and 2 (71.67%). This scale is estimated with better precision for individuals located between these values.

Figure 4
The test information curve of dentists´ understanding of comprehensiveness of care, Brazil, 2012.

Discussion

In this study, an instrument was developed to assess comprehensiveness of care that presented good psychometric properties, measured both by CTT and by IRT. The level of understanding of the concept by dentists varied across the questions.

The IRT showed that the instrument included items considered “difficult” (only dentists with a good understanding of comprehensiveness responded favorably); these items were numbers 2, 3, 5, 6, 8, and 10. There were also items that were considered “easy” (even dentists with a low understanding of comprehensiveness responded favourably); these were items 1, 4, 7, 9, and 11.

Our findings indicate that this questionnaire can be a valid self-report scale for evaluating comprehensiveness of care from the perspective of dentists. This instrument measures key points under the umbrella of comprehensiveness; these points include teamwork needs, training, flow between primary and secondary dental care, continuity of care and health service management support. The application of this instrument amongst dentists in health services could be useful for diagnoses, knowledge and practice. To ensure an honest assessment by dentists, it is necessary for the results of this evaluation to be constructive, not punitive. Thus, new options for education and in-service training could be planned ¹⁶. Furthermore, external evaluation from the patient’s perspective may be useful to gain an overview of comprehensiveness of care in health services settings.

Despite the importance and potential of combining classical test theory (CTT) and item response theory (IRT) in the evaluation of psychometric properties of new instruments ^27,33, there is a paucity of literature in this field, especially in the dental public health literature. The IRT models, which are expressed based on the response to each item and not the total score on the instrument, enable the development of shorter instruments or even instruments adapted to the respondent’s profile that have good psychometric properties ³³. We developed an instrument with such properties with the aim of assessing the subjective aspects of health care delivery that are not usually addressed by conventional measures. It is a challenging approach, and health systems present difficulties in applying the outcomes of assessments using approaches such as comprehensiveness and equity in health service settings ^1,2,8. However, this does not mean that these approaches are less important. These concepts are being increasingly studied and explored in cost-utility analyses of health care, with the goal of evaluating the medium-term and long-term effects on the health care delivered, as well as considering their impact on access and continuity of care ^1,9,11,13.

This research has certain strengths and limitations that should each be considered. First, we worked with all of the established primary care dentists in the municipality in order to achieve a high response rate (90.34%), which was far higher than the average response rate found in studies with professionals from public health services ³⁴. This was most likely because the questionnaires were distributed in collaboration with the headquarters of primary care management from City Hall. As mentioned in the methods section, all envelopes were addressed to the health care clinics via interoffice mail, and the local managers from each clinic reminded the recipients and collected the questionnaires after three working days ³⁵. The findings can therefore be considered representative of the service surveyed but may not be relevant to other services and systems. Second, statistical tests of convergent/discriminant validity for the instrument scale were limited by the lack of a psychometrically validated instrument that measured a similar construct to use in a side-by-side comparison ^19,22. Finally, the responsiveness of the instrument has not yet been evaluated because the current study was not designed to investigate its performance, and the representativeness of our sample of respondents, as well as the generalizability of findings to the wider population of dentists in Brazil, cannot be determined ^19,22,23. We recommend future research with larger samples from various parts of Brazil. The variations within the group necessitate making the instrument as comprehensive as possible because Brazil is a country of continental dimensions, and the perception of comprehensiveness of care by the professional may change according to the reality in which the dentist works. We also welcome reports of how the questionnaire performs in other samples, even with respondents from other countries, as the items may need to be validated in other languages ^18,19,22. Studying larger and new samples, we could test the possibility of creating cut-off points to categorize the service, or services, in relation to comprehensiveness of care.

The process of instrument development is time-consuming, but it ultimately enhances the quality and richness of the data collected ^19,36. The majority of the present approaches to evaluating the quality of care are based on models developed a half-century ago. The World Health Organization report from 2008 indicated the increasing importance of new experiences, prioritizing person-focused assessments rather than disease-focused assessments. Holistic approaches, such as comprehensiveness, should be studied further and better understood so that health systems can benefit from the applicability of the concepts associated with these approaches. Human resource management could well benefit from the use of this instrument. These key characteristics have increasingly recognized the impact of the mode of delivery of health services ^1,7. The 11-item instrument to assess comprehensiveness of care by dentists is considered to have good psychometric properties, as measured both by CTT and by IRT, and it should be tested in other settings and systems.

References

• ¹
  World Health Organization. The World Health Report: Primary Health Care now more than ever. Geneva: WHO; 2008. Available: Available: http://www.who.int/whr/2008/whr08_en.pdf Accessed27 November 2015.
  » http://www.who.int/whr/2008/whr08_en.pdf
• ²
  Marmot, M. Universal health coverage and social determinants of health. Lancet2013; 382: 1227-1228.
• ³
  Mendes ,EV. Health care networks. Ciênc Saúde Coletiva 2010;15:2297-305.
• ⁴
  Morris, A, Burke, F. Health policy: primary and secondary dental care: the nature of the interface. Brit Dent J 2001;191:660-664.
• ⁵
  Glick, M; Williams, DM; Kleinman, DV; Vujicic, M; Watt ,RG; Weyant, RJ. A new definition for oral health developed by the FDI World Dental Federation opens the door to a universal definition of oral health. Int Dent J 2016;66:322
• ⁶
  World Health Organization. Country cooperation strategy at a glance: Brazil; 2009. Available: Available: http://www.apps.who.int/iris/handle/10665/137005 Latest access November 3, 2017.
  » http://www.apps.who.int/iris/handle/10665/137005
• ⁷
  Starfield, B. New paradigms for quality in primary care. Br J Gen Pract2001;51:303-309.
• ⁸
  Starfield, B; Shi, L. Manual for the Primary Care Assessment Tools. Baltimore: Johns Hopkins University; 2009. Available: Available: http://www.jhsph.edu/research/centers-and-institutes/johns-hopkins-primary-care-policy-center/pca_tools.html Latest access November 3, 2017.
  » http://www.jhsph.edu/research/centers-and-institutes/johns-hopkins-primary-care-policy-center/pca_tools.html
• ⁹
  Haggerty, J; Burge, F; Lévesque, J-F; Gass, D; Pineault, R; Beaulieu, MD, et al.. Operational definitions of attributes of primary health care: consensus among Canadian experts. Ann Fam Med2007;5:336-344.
• ¹⁰
  Paim, J; Travassos, C; Almeida, C; Bahia, L; Macinko, J. The Brazilian health system: history, advances, and challenges. Lancet. 2011;377:1778-1797.
• ¹¹
  Dickinson, WP; Miller, BF. Comprehensiveness and continuity of care and the inseparability of mental and behavioural health from the patient-centered medical home. Fam Syst Health 2010;28:348-355.
• ¹²
  Mattos, GCM; Gallagher, JE; Paiva, SM; Abreu, MHNG. Perception of ‘comprehensiveness of care’: a qualitative study amongst dentists in the Brazilian health system. Braz Oral Res 2015;29:1-7.
• ¹³
  Haggerty, JL; Beaulieu, M-D; Pineault, R; Burge, F; Lévesque, J-F; et al.. Comprehensiveness of care from the patient perspective: comparison of primary healthcare evaluation instruments. Healthc Policy 2011;7:154-166.
• ¹⁴
  Silva Junior, AG; Mascarenhas, MTM; Nascimento, Silva; VM, Carvalho, LC. Evaluation experiences of the supplementary care sector: contributions of integrality. Cienc Saude Colet 2008;13:1489-500.
• ¹⁵
  Starfield, B. Primary care: balancing health needs, services, and technology. Oxford University Press, USA; 1998.
• ¹
  Kabene, SM; Orchard, C; Howard, JM; Soriano, MA; Leduc, R. The importance of human resources management in health care: a global context. Hum Resour Health 2006;4:1-17.
• ¹⁷
  BrazilianBrazilian Institute of Geography and Statistics Census Online; 2010. Available: Available: https://www.cidades.ibge.gov.br/ Accessed 14 February 2015.
  » https://www.cidades.ibge.gov.br/
• ¹⁸
  Behling, O; Law, KS. Translating questionnaires and other research instruments: problems and solutions. Series: Quantitative Applications in the Social Sciences. California: Sage; 2000.
• ¹⁹
  Streiner, DL; Norman, GR. Health measurement scales: a practical guide to their development and use. 4^th ed. Oxford: Oxford University Press; 2008.
• ²⁰
  Peršić, S; Palac, A; Bunjevac, T; Čelebić, A. Development of a new chewing function questionnaire for assessment of a self-perceived chewing function. Community Dent Oral Epidemiol2013;41:1-9.
• ²¹
  Lamping, DL; Schroter, S; Kurz, X; Kahn ,SR; Abenhaim, L. Evaluation of outcomes in chronic venous disorders of the leg: development of a scientifically rigorous, patient-reported measure of symptoms and quality of life. J Vasc Surg 2003;37:410-419.
• ²²
  Lamping, DL; Schroter, S; Marquis, P; Marrel, A; Duprat-Lomon, I; Sagnier, PP. The community-acquired pneumonia symptom questionnaire: a new, patient-based outcome measure to evaluate symptoms in patients with community-acquired pneumonia. Chest2002;122:920-929.
• ²³
  Quirk, A; Smith, S; Hamilton, S; Lamping, D; Lelliott, P; Daniel, Stahl, et al.. Development of the carer well-being and support (CWS) questionnaire. Ment Health Rev J 2012;17:128-138.
• ²⁴
  Terwee, CB; Bot, SD; van der De Boer, MR; Windt, DA; Knol, DL; Dekker, J; et al.. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol 2007;60:34-42.
• ²⁵
  Curtis, LM; Revelle, W; Waite, K; Wilson, EA; Condon, DM; Bojarski, E, et al.. Development and validation of the comprehensive health activities scale: a new approach to health literacy measurement. J Health Commun 2015;20:157-164.
• ²⁶
  Brédart, A; Kop, JL; Efficace, F; Beaudeau, A; Brito, T, Dolbeault, S, et al.. Quality of care in the oncology outpatient setting from patients’ perspective: a systematic review of questionnaires’ content and psychometric performance. Psycho-Oncology 2015;24:382-394.
• ²⁷
  Bourion-Bédès S; Schwan R; Epstein J; Laprevote V; Bédès A; Bonnet JL, et al.. Combination of classical test theory (CTT) and item response theory (IRT) analysis to study the psychometric properties of the French version of the Quality of Life Enjoyment and Satisfaction Questionnaire-Short Form (Q-LES-Q-SF). Qual Life Res 2015;24:287-293.
• ²⁸
  Bartholomew, D, Knott, M. Latent Variable Models and Factor Analysis. 2^nd ed. London: Edward Arnold; 1999.
• ²⁹
  Bhakta, B; Tennant, A; Horton, M; Lawton, G; Andrich, D. Using item response theory to explore the psychometric properties of extended matching questions examination in undergraduate medical education. BMC Med Educ 2005;5:9.
• ³⁰
  Gordon, RA. Measuring constructs in family science: How can item response theory improve precision and validity? J Marriage Fam2015;77:147-176.
• ³¹
  Hays, RD; Morales, LS; Reise, SP. Item response theory and health outcomes measurement in the 21^st century. Med Care 2000;38:II28-II42.
• ³²
  Rizopoulos, D. ltm: an R package for latent variable modelling and item response theory analyses. J Stat Softw 2006;17:1-25.
• ³³
  DeMars, C. Item response theory: understanding statistics measurement. New York: Oxford University Press; 2010.
• ³⁴
  Rocha, KB; Rodríguez-Sanz, M; Pasarín, MI; Berra, S; Gotsens, M; Borrell, C. Assessment of primary care in health surveys: a population perspective. Eur J Public Health 2012;22:14-19.
• ³⁵
  Cook, J; Dickinson, H; Eccles, M. Response rates in postal surveys of healthcare professionals between 1996 and 2005: an observational study. BMC Health Serv Res 2009;9:160.
• ³⁶
  Safran, DG; Kosinski, M; Tarlov, AR; Rogers, WH; Taira, DA; Lieberman, N; et al.. The primary care assessment survey: tests of data quality and measurement performance. Med Care 1998;36:728-739.

Publication Dates

• Publication in this collection
  Sep-Oct 2017

History

• Received
  05 Oct 2016
• Accepted
  25 May 2017