Content validation: clarity/relevance, reliability and internal consistency of enunciative signs of language acquisition

ABSTRACT

Purpose

To analyze the results of the validation of building enunciative signs of language acquisition for children aged 3 to 12 months.

Methods

The signs were built based on mechanisms of language acquisition in an enunciative perspective and on clinical experience with language disorders. The signs were submitted to judgment of clarity and relevance by a sample of six experts, doctors in linguistic in with knowledge of psycholinguistics and language clinic. In the validation of reliability, two judges/evaluators helped to implement the instruments in videos of 20% of the total sample of mother-infant dyads using the inter-evaluator method. The method known as internal consistency was applied to the total sample, which consisted of 94 mother-infant dyads to the contents of the Phase 1 (3-6 months) and 61 mother-infant dyads to the contents of Phase 2 (7 to 12 months). The data were collected through the analysis of mother-infant interaction based on filming of dyads and application of the parameters to be validated according to the child's age. Data were organized in a spreadsheet and then converted to computer applications for statistical analysis.

Results

The judgments of clarity/relevance indicated no modifications to be made in the instruments. The reliability test showed an almost perfect agreement between judges (0.8 ≤ Kappa ≥ 1.0); only the item 2 of Phase 1 showed substantial agreement (0.6 ≤ Kappa ≥ 0.79). The internal consistency for Phase 1 had alpha = 0.84, and Phase 2, alpha = 0.74. This demonstrates the reliability of the instruments.

Conclusion

The results suggest adequacy as to content validity of the instruments created for both age groups, demonstrating the relevance of the content of enunciative signs of language acquisition.

Keywords
Risk Factors; Language; Child Development; Validity of Tests

RESUMO

Objetivo

Analisar os resultados da validação de construção de sinais enunciativos de aquisição da linguagem para crianças de 3 a 12 meses.

Método

Os sinais foram construídos a partir de mecanismos enunciativos e experiência clínica com distúrbios de linguagem e submetidos ao julgamento, quanto à clareza e à pertinência, por seis doutores com conhecimento em psicolinguística e clínica de linguagem. A validação de fidedignidade contou com dois juízes para aplicação dos instrumentos em vídeos de 20% da amostra total das díades mãe-bebê com o método entre avaliadores. O método Consistência Interna foi aplicado no total da amostra constituída de 94 díades mãe-bebê para os sinais da Fase 1 (3 a 6 meses) e de 61 díades mãe-bebê para os sinais da Fase 2 (7 a 12 meses). A coleta de dados ocorreu por meio da interação mãe-bebê feita com base em filmagens e aplicação dos sinais conforme a faixa etária da criança. Os dados foram organizados em planilha eletrônica e convertidos para aplicativos computacionais para análise estatística.

Resultados

Os julgamentos quanto à clareza/pertinência mantiveram os instrumentos sem modificações. O teste de fidedignidade apontou uma concordância entre os juízes quase perfeita (0,8 ≤ Kappa ≥ 1,0), apenas o item 2 da Fase 1 apresentou uma concordância substancial (0,6 ≤ Kappa ≥ 0,79). A consistência interna para a Fase 1 apresentou alpha = 0,84 e, para a Fase 2, alpha = 0,74, demonstrando confiabilidade nos instrumentos.

Conclusão

Os resultados sugerem adequação quanto à validação de conteúdo dos sinais criados para ambas as faixas etárias.

Descritores
Fatores de Risco; Linguagem; Desenvolvimento Infantil; Validade dos Testes

INTRODUCTION

An increasing number of questionnaires and scales have been available to verify and evaluate phenomena studied in the various health areas⁽¹⁾. Researchers have emphasized the importance of these instruments to be reliable and valid in order to minimize the possibility of subjective judgments⁽²⁾. Validation, therefore, is a determining factor for choosing and/or applying assessment instruments. Validity is measured by the extent or degree to which the data represent the concept that the instrument proposes to measure⁽³⁾.

One of the most mentioned methods for obtaining the validity of a measure by psychometrists is the content validity⁽⁴⁾. This method introduces the process of association between abstract concepts with measurable indicators and represents the extent to which each item of the measure confirms the phenomenon of interest and the dimension of each item within what it proposes to investigate. It presents two stages: the first consists in the development of the instrument and the second involves the analysis and judgment by specialists. The analysis by judges or content analysis is necessarily based on the judgment of a group of judges with experience in the area, who will have to analyze if the content is accurate and adequate to what it is proposed⁽⁵⁾.

Twelve criteria related to the Pasquali's methodological framework are also used for judging the items of an instrument^(4,6). These criteria provide support for the content validation because they evaluate psychometric properties of the instrument and indicate if the items are understandable to the target population. Among them are the criteria clarity and relevance⁽⁶⁾.

The term trustworthiness suggests reliability. According to Urbina⁽⁷⁾, reliability is based on the consistency and accuracy of the results in the measurement process. To have some degree of confidence, the tests require evidence that the scores obtained would be consistent if the tests were repeated with the same individuals or groups.

Internal consistency is also a measure of reliability that refers to the degree to which the items in the questionnaire are correlated with each other and with the overall result of the survey, which represents a measure of reliability⁽⁸⁾.

The abovementioned criteria make it evident that the recognition of the quality of the instruments is a fundamental aspect for the legitimacy and credibility of the results produced by a research, which reinforces the importance of the validation process⁽⁹⁾.

Many instruments are used in the international literature on language disorder clinic, for the evaluation of language, but few in the national literature, especially regarding the early detection of risk for language disorders⁽¹⁰⁾. Regarding the risk of progression to language disorder, there are many studies on language and development comparing pre-terms to children born at term in the international literature^(11-14).

In the national literature, some studies have addressed the standardization of tests such as Denver II^(15,16), and other instruments created in Brazil^(17-20). There are also studies that reflect on the effects of psychic risk in the language of babies^(21-23). Few studies use declarative references in the qualitative analysis of language^(22,23). These observations and the pertinence and innovation of the enunciative proposal of language acquisition of Silva⁽²⁴⁾ motivated the creation of the instrument described here.

This article, therefore, aims to analyze the content validation process regarding clarity/relevance, reliability and internal consistency of enunciative signs of language acquisition in the case of children from 3 to 12 months of age.

METHODS

In order to carry out the research, the mandatory ethical standards for research involving human beings were used (Resolution 466/12 of the National Health Council - NHC), and the study was approved by the Ethics Committee of the University in which the study was carried out under protocol number 28586914.0.0000.5346. All individuals involved in the research received clarification about the objectives and procedures of the study, and read and signed the Informed Consent form.

This is a content validation research focusing on the development, evaluation and improvement of an instrument.

The preparation of the instrument initially took into account the enunciative mechanisms and strategies proposed by Silva⁽²⁴⁾ and the clinical experience of the authors in the follow-up of risk infants who developed a risk for language acquisition^(21-23,25). After formulated, the items were submitted to expert judges to judge on their clarity and relevance.

The sample of expert judges was selected based on the criteria of knowledge about the enunciation theory, language acquisition process and language clinical studies, including the evaluation process. Considering these criteria, six PhD specialists in linguistics were selected, including four speech therapists and two psycholinguists, each one with more than 15 years of experience in language analysis.

In the content validation - reliability, two judges/evaluators helped to apply the instruments in videos of mother-infant dyads in 20 of the total sample. Both judges were speech therapists with more than five years of clinical experience with infants and language disorders.

To perform content validation - internal consistency, the sample consisted of 94 mothers and their infants aged between 3 months and 1 day and 6 months and 29 days, and 61 mothers and infants aged between 7 months and 1 day and 12 months and 29 days. The dyads were contacted in basic health units in the countryside of Rio Grande do Sul. Cases of babies with evident signs of neurological impairment, malformations and syndromes that affected communication, and premature babies or those with hearing or visual impairment were excluded from the total sample. Infants whose mothers were suspected of much compromised psychic structure, such as psychosis, were also excluded. In case of doubt regarding the mother's psychic situation, the mother was referred for evaluation with professional psychologists from the research group. Thus, term babies without diagnosis of obvious biological alteration and without serious involvement in the family were included.

The content validation of clarity/relevance was carried out based on a semantic analysis aimed at verifying if the signs could be understood and presented apparent validity. The procedure was carried out using a Likert-type scale with five points to investigate the Relevance and Clarity of each item included in the SEAL, for the individual content analysis by the judges, starting from 1 - totally disagree; 2 - partially disagree; 3 - indifferent; 4 - partially agree; and 5 - totally agree. The percentage of judges who agreed totally and partially with the item of the instrument was calculated based on the answers sent by the judges. This criterion was used for both clarity and relevance of the item. Percentages of agreement less than or equal to 70% were assessed for alteration or exclusion by the researchers⁽²⁶⁾.

For the two other phases of the content validation, data were collected from mother-infant dyads through an initial interview conducted by the research team and lasting 15 minutes on average. During this interview, if the child was of adequate age and privileged a possible observation of the mother-infant interaction, the enunciative signs of language acquisition, according to the child's age group (phase 1 or phase 2), were applied.

Considering the dynamics and facilitation of the mother-infant relationship, and also the possibility of observing the mother with her child without the presence of third parties, a situation of interaction was set up and filmed, trying to do this from a certain distance so as to interfere as little as possible in the relation of the dyad.

The mother was then invited to sing, talk, and play with her child with miniature toys brought by the examiner, related to the initial children's vocabulary (animals, food, transport, objects in the house). The examiner would suggest the mother to act with her child as she used to do at home.

The filming of the dyad lasted about 15 minutes, at which time, in 9, the mother would obey the request to sing, talk or play with the toys, depending on her interest and of the baby, with the baby sitting in a baby seat in front of the mother, in the age groups of 3 and 6 months. In the following 6 minutes, the examiner would return to the room and seek to encourage the dialogue and the playing to continue after placing the baby lying on the mat, in the age group of 3 to 6 months. In the age group of 7 to 12 months, the positioning of the baby depended on its psychomotor conditions and this would not necessarily stay in the baby seat; they could move on the mat placed on the floor and explore more freely the box with the toys.

It is important to make clear that the rooms used had a silent environment, two cameras supported by tripods in established dispositions, a mat, a baby seat to place the child and the mother comfortably, and a mirror to visualize both the child and the mother's face in the filming. There was a camera placed two meters from the mirror in front of the baby and capturing the mother's face in the mirror, and a camera place at the side, at one meter, capturing the mother-infant face-to-face interaction. The cameras were maintained in these positions even without the baby seat.

The researcher attributed value to the signs during the interview and her entry into the room. Then she conferred such attribution in the filming. The instrument of phase 1 was applied to the babies in the filming of 3 months and 6 months and the instrument of phase 2 in the filming of 7 months and 9 months. This application occurred in the whole sample.

Reliability was also verified in the filming of the dyads. The inter-evaluator method was used, considering the analysis of the judges/evaluators. Two speech therapists with experience in children's clinic and knowledge of the enunciative theory to apply the instruments in a random group of dyads participated in this moment of the research. Twenty percent of the total sample was analyzed. The two speech therapists individually watched the same videos of the mother-infant dyads and applied the SEAL as they identified the signs or not. In this analysis, the agreement between evaluators was verified by calculating the Kappa coefficient.

After the analysis of the agreement with the items of the two instruments, the method called Internal Consistency was used, applying the Kuder-Richardson (KR) test in the total sample, that is, Phase 1 = 94 dyads and Phase 2 = 61 dyads. This test analyzes each item individually. This test was selected because the instruments presented questions corrected dichotomously (yes/no).

RESULTS

The Enunciative Signs of Language Acquisition prepared by the researchers are shown in Chart 1.

Chart 1
Enunciative Signs of Language Acquisition

Items 3 to 6 months and 29 days
1. The child reacts to "manhês", by means of vocalizations, body movements or eye contact.
2. The child fills its place in the interlocution with verbal sounds such as vowels and/or consonants.
3. The child fills its place in the interlocution with nonverbal sounds attuned to the enunciative context (smile, yelling, cry, cough, grumbling).
4. The child fills its place in the interlocution quietly, only with bodily movements and looks attuned to the enunciative context.
5. The child initiates the conversation or protoconversation.
6. The child and mother (or her substitute) exchange looks during interaction.
7. The mother (or her substitute) gives meaning to the verbal and non-verbal manifestations of the baby, and supports this protoconversation or conversation when the baby initiates it.
8. The mother (or her substitute) uses the manhês while talking to the child in a tuned way to what is happening in context and awaiting the baby's answers.
Items 7 to 12 months and 29 days
9. The child fills its place in the interlocution (utterance) with verbal sounds (syllables with vowels and varied consonants - at least two points and two articulatory modes of consonants).
10. The child outlines the production of words by mirroring the mother's (or her substitute's) speech.
11. The child outlines the production of protowords spontaneously.
12. When the mother (or her substitute) is summoned to enunciate by the child, she reproduces her statement and awaits the response of the child.

The results of the evaluation of clarity and relevance of the instruments (Tables 1 and 2) by the judges are presented below.

Table 1
Judges' evaluation of clarity by the Likert-type scale

Item	J1	J2	J3	J4	J5	J6	% Total A.	% Par A. + Total A.
Group 3 to 6 months
1	5	5	5	5	5	5	100.0	100.0
2	5	5	5	5	5	5	100.0	100.0
3	5	5	5	1	5	5	83.3	83.3
4	1	5	5	5	5	5	83.3	83.3
5	5	5	5	4	4	5	66.7	100.0
6	5	5	5	5	5	5	100.0	100.0
7	5	5	5	4	5	5	83.3	100.0
8	5	5	4	5	5	5	83.3	100.0
Group 7 to 12 months
9	5	5	5	3	5	5	83.3	83.3
10	5	5	5	4	4	5	66.7	100.0
11	4	5	4	4	4	5	33.3	100.0
12	4	5	5	3	3	5	50.0	66.7

• Caption: J = Judge; % Total A. = percentage of agreement on the Likert-type scale in totally agree; % Par A. + Total A. = percentage of agreement on the Likert-type scale in partially agree or totally agree

Table 2
Judges' evaluation of relevance by the Likert-type scale

Item	J1	J2	J3	J4	J5	J6	% Total A.	% Par A. + Total A.
Group 3 to 6 months
1	5	5	5	5	5	5	100.0	100.0
2	5	5	4	5	5	5	83.3	100.0
3	5	5	4	5	5	2	66.7	83.3
4	4	5	4	5	5	2	50.0	83.3
5	5	5	5	5	5	4	83.3	100.0
6	5	5	5	5	5	5	100.0	100.0
7	5	5	4	5	5	5	83.3	100.0
8	5	5	5	5	5	5	100.0	100.0
Group 7 to 12 months
9	4	5	4	5	5	5	66.7	100.0
10	5	5	5	5	5	5	100.0	100.0
11	5	5	5	5	5	5	100.0	100.0
12	4	5	4	5	5	5	66.7	100.0

• Caption: J = Judge; % Total A. = percentage of agreement on the Likert-type scale in totally agree; % Par A. + Total A. = percentage of agreement on the Likert-type scale in partially agree or totally agree

It is observed in Table 1 that the first instrument was considered more clear than the second when considered the total agreement value. However, when considering the partial agreement, this percentage rises to higher levels, as shown in the last column. A similar dynamic was observed in the analysis of relevance, shown in Table 2.

The analysis of reliability of the Phase 1 (3 to 6 months), in the assignment of signs by two judges, had the results shown in Table 3.

Table 3
Agreement between judges in the Phase 1 instrument

SEAL Item	% agreement	Kappa coefficient	p-value
1	100.0	-	-
2	93.8	0.636	0.006
3	100.0	-	-
4	100.0	-	-
5	87.5	-	-
6	93.8	-	-
7	100.0	1.000	<0.01*
8	100.0	1.000	<0.01*

• ^*
  Significant by Kappa agreement analysis with significance level of 5%
• Caption: SEAL; % = percentage

The results of reliability obtained in the Phase 2 (7 to 12 months) are shown in Table 4.

Table 4
Agreement between judges in the Phase 2 instrument

SEAL Item	% agreement	Kappa coefficient	p-value
9	100.0	1.000	<0.01*
10	100.0	1.000	<0.01*
11	100.0	1.000	<0.01*
12	100.0	-	-

• ^*
  Significant by Kappa agreement analysis with significance level of 5%
• Caption: SEAL; % = percentage

As to the internal consistency of the instruments as a result, for the Phase 1, we obained alpha = 0.84 and for the Phase 2, alpha = 0.74.

DISCUSSION

Considering the literature requirements about the judges being experts in the area of the evaluated construct⁽⁸⁾, and the suggestion of a number of six to 20 judges⁽⁵⁾ and at least three in each professional group⁽²⁷⁾, we believed that the content validation met its objectives. There were not many judges with the required training, in terms of homogeneity of the sample, in the Brazilian reality. The six judges selected met all the theoretical-practical criteria established.

There is a variation between 70% and 80% of agreement between judges as a minimum for acceptance of an item as pertinent⁽⁵⁾, since values lower than 70% would indicate the need for alteration or exclusion by the researchers. Considering the sum between the values of the responses 'totally agree' and 'partially agree', it was possible to notice that most of the signs reached percentages above 80%, which allowed to keep all items for the next stages of the research. Furthermore, when analyzing the results in Tables 1 and 2, both for clarity and relevance, it is possible to notice that none of the items was judged with complete disagreement. The decision to maintain the items also considered the inherent subjectivity of judges in the judgement process, as noted in the literature⁽³⁾, as well as the theoretical and practical relevance of the signs, observed from the clinical experience of the researchers.

The reliability or agreement among evaluators regarding the items of the instrument is a technique that aims to verify the consensus among experts about the application of the instrument. In order to analyze the reliability of the elaborated signs, the agreement between evaluators was verified by calculating the Kappa coefficient. This may vary from -1 to 1, indicating complete disagreement or agreement, and the value 0 indicates an agreement value equal to chance. The ranges of values elaborated by Landis and Koch were used for interpreting the Kappa values⁽²⁸⁾. These authors suggest that values above 0.75 represent excellent agreement, values below 0.40, poor agreement, and values between 0.40 and 0.75 represent a sufficiently good agreement.

The explanation for cases in which the Kappa value was not calculated is related to the non-variability in the evaluation of the items by at least one of the judges.

Considering the results of the Kappa coefficient, the concordance between the judges was almost perfect (0.8 ≤ Kappa ≥ 1.0). Only the item 2 of phase 1 presented a substantial agreement (0.6 ≤ Kappa ≥ 0.79)⁽²⁸⁾.

Data analysis through statistical tests represented the analytical procedures of the studies. The Cronbach's Alpha was chosen in a study⁽²⁹⁾ because of its ability to reflect the degree of agreement of the items between themselves; the closer to one, the closer to 100% is the correspondence of the items. In turn, Yamada and Santos⁽³⁰⁾ adopted the Cronbach's Alpha lower than 0.70 for exclusion of items from the instrument.

The Cronbach's Alpha coefficient estimates the reliability of internal consistency of questionnaires and also the estimation of inter-evaluator reliability. Since all the items of a questionnaire use the same measurement scale, the Alpha coefficient is calculated from the variance of the individual items and the covariance between items⁽²⁹⁾.

The applications of the Alpha coefficient in the various areas of knowledge are broad and comprehensive, but there is still no consensus among researchers regarding the interpretation of the reliability a questionnaire obtained from the value of this coefficient. In general, an instrument of research that reaches an Alpha coefficient greater than or equal to 0.70 is considered satisfactory. In contrast, the expected maximum value is 0.90. Values above 0.90 indicate that there is redundancy or duplication, that is, several items are measuring exactly the same element of a construct. Therefore, redundant items must be eliminated⁽²⁹⁾.

Therefore, the results obtained by the coefficient in the current research indicate that the two instruments evaluated are reliable.

CONCLUSION

Considering the purpose of this article to analyze the content validation of the enunciative signs of language acquisition, we have, at the end of the whole process, two specific instruments for two age groups. The first instrument is called Phase 1 (3 to 6 months) and has eight items and the second one is called Phase 2 (7 to 12 months) with four items. These presented results compatible with the adequacy of scientific criteria regarding content validation.

• Study carried out at Programa de Pós-graduação em Distúrbios da Comunicação Humana, Universidade Federal de Santa Maria – UFSM, Santa Maria (RS), Brazil.
• Financial support: Coordenação de Aperfeiçoamento de Pessoal de Nível Superior – CAPES.

REFERÊNCIAS

• ¹
  Alexandre NMC, Coluci MZO. Validade de conteúdo nos processos de construção e adaptação de instrumentos de medidas. Cien Saude Colet. 2011;16(7):3061-8. PMid:21808894. http://dx.doi.org/10.1590/S1413-81232011000800006
  » http://dx.doi.org/10.1590/S1413-81232011000800006
• ²
  Raymundo VP. Construção e validação de instrumentos: um desafio para a psicolinguística. Let Hoje. 2009;44(3):86-93.
• ³
  Bittencourt HR, Creutzberg M, Rodrigues ACM, Casartelli AO, Freitas ALS. Desenvolvimento e validação de um instrumento para avaliação de disciplinas na educação superior. Estud Avaliação Educ. 2011;22(48):91-114. http://dx.doi.org/10.18222/eae224820111994
  » http://dx.doi.org/10.18222/eae224820111994
• ⁴
  Pasquali L. Psicometria. Rev Esc Enferm USP. 2009;43(Esp.):992-9.
• ⁵
  Moura ERF, Bezerra CG, Oliveira MS, Damasceno MMC. Validação de jogo educativo destinado à orientação dietética de portadores de diabetes mellitus. Revista APS. 2008;11(4):435-43.
• ⁶
  Pasquali L. Instrumentação psicológica: fundamentos e práticas. Porto Alegre: Artmed; 2010.
• ⁷
  Urbina S. Fundamentos da testagem psicológica. Porto Alegre: Artmed; 2007. 320 p.
• ⁸
  Campos JADB, Bonafé FSS, Dovigo LN, Maroco J. Avaliação psicométrica da escala de atitudes em relação à estatística. Rev Bras Biom. 2013;31(2):327-37.
• ⁹
  Medeiros RKS, Ferreira MA Jr, Pinto DPSR, Vitor AF, Santos VEP, Barichello E. Modelo de validación de contenido de Pasquali en las investigaciones en enfermería. Rev Enf Ref. 2015;4(4):127-35. http://dx.doi.org/10.12707/RIV14009
  » http://dx.doi.org/10.12707/RIV14009
• ¹⁰
  Lindau TA, Lucchesi FDM, Rossi NF, Giacheti CM. Instrumentos sistemáticos e formais de avaliação da linguagem de pré-escolares no Brasil: uma revisão de literatura. Rev CEFAC. 2015;17(2):656-62. http://dx.doi.org/10.1590/1982-021620151114
  » http://dx.doi.org/10.1590/1982-021620151114
• ¹¹
  D’Odorico L, Majorano M, Fasolo M, Salerni N, Suttora C. Characteristics of phonological development as a risk fator for language development in Italian-Speaking pre-term children: a longitudinal study. Clin Linguist Phon. 2011;25(1):53-65. PMid:21080829. http://dx.doi.org/10.3109/02699206.2010.511759
  » http://dx.doi.org/10.3109/02699206.2010.511759
• ¹²
  Suttora C, Salerni N. Maternal speech to preterm infants during the first 2 years of life: stability and change. Int J Lang Commun Disord. 2011;46(4):464-72. PMid:21771221. http://dx.doi.org/10.1111/j.1460-6984.2011.00007.x
  » http://dx.doi.org/10.1111/j.1460-6984.2011.00007.x
• ¹³
  Lowe JR, Erickson SJ, MacLean P, Schrader R. Fuller J. Association of maternal scaffolding to maternal education and cognition in toddlers born preterm and full term. Acta Paediatr. 2013;102(1):72-7. PMid:23009657. http://dx.doi.org/10.1111/apa.12037
  » http://dx.doi.org/10.1111/apa.12037
• ¹⁴
  Serenius F, Kallen K, Blennow M, Ewald U, Fellman V, Holmstrom G, et al. Neurodevelopmental outcome in extremely preterm infants at 2.5 years after activeperinatal car in Sweden. JAMA. 2013;309(17):1810-20. PMid:23632725. http://dx.doi.org/10.1001/jama.2013.3786
  » http://dx.doi.org/10.1001/jama.2013.3786
• ¹⁵
  Pinto FCA, Isotani SM, Sabatés AL, Perissinoto J. Denver II: comportamentos propostos comparados aos de crianças paulistanas. Rev CEFAC. 2015;17(4):1262-9. http://dx.doi.org/10.1590/1982-0216201517418214
  » http://dx.doi.org/10.1590/1982-0216201517418214
• ¹⁶
  Costa EF, Cavalcante LIC, Dell’Aglio DD. Perfil do desenvolvimento de linguagem de crianças no município de Belém segundo o teste Denver II. Rev CEFAC. 2015;17(4):1090-103. http://dx.doi.org/10.1590/1982-0216201517418514
  » http://dx.doi.org/10.1590/1982-0216201517418514
• ¹⁷
  Hage SRV, Pereira TC, Zorzi JL. Protocolo de observação comportamental – proc: valores de referência para uma análise quantitativa. Rev CEFAC. 2012;14(4):677-90. http://dx.doi.org/10.1590/S1516-18462012005000068
  » http://dx.doi.org/10.1590/S1516-18462012005000068
• ¹⁸
  Scherer S, Souza APR. Types e tokens na aquisição típica de linguagem por sujeitos de 18 a 32 meses falantes do português brasileiro. Rev CEFAC. 2011;13(5):838-46. http://dx.doi.org/10.1590/S1516-18462011005000058
  » http://dx.doi.org/10.1590/S1516-18462011005000058
• ¹⁹
  Santos ME, Lynce S, Carvalho S, Cacela M, Mineiro A. Extensão média do enunciado-palavras em crianças de 4 a 5 anos com desenvolvimento típico de linguagem. Rev CEFAC. 2015;17(4):1143-51. http://dx.doi.org/10.1590/1982-021620151741315
  » http://dx.doi.org/10.1590/1982-021620151741315
• ²⁰
  Verly FRE, Freire RMAC. Indicadores clínicos de risco para a constituição do sujeito falante. Rev CEFAC. 2015;17(3):766-74. http://dx.doi.org/10.1590/1982-0216201513014
  » http://dx.doi.org/10.1590/1982-0216201513014
• ²¹
  Crestani AH, Moraes AB, Souza APR. Análise da associação entre indicadores de risco ao desenvolvimento infantil e produção inicial de fala entre 13 e 16 meses. Rev CEFAC. 2015;17(1):169-76. http://dx.doi.org/10.1590/1982-021620153514
  » http://dx.doi.org/10.1590/1982-021620153514
• ²²
  Flores MR, Souza APR. Diálogo de pais e bebês em situação de risco ao desenvolvimento. Rev CEFAC. 2014;16(3):840-52. http://dx.doi.org/10.1590/1982-0216201411412
  » http://dx.doi.org/10.1590/1982-0216201411412
• ²³
  Oliveira LD, Ramos-Souza APR. O distúrbio de linguagem em dois sujeitos com risco para o desenvolvimento em uma perspectiva enunciativa de funcionamento de linguagem. Rev CEFAC. 2014;16(5):1700-12. http://dx.doi.org/10.1590/1982-0216201410713
  » http://dx.doi.org/10.1590/1982-0216201410713
• ²⁴
  Silva CLC. A criança na linguagem: enunciação e aquisição. Campinas: Pontes; 2009.
• ²⁵
  Vendrúscolo JF, Souza APR. Intersubjetividade no olhar interdisciplinar sobre o brincar e a linguagem de sujeitos com risco psíquico. Rev CEFAC. 2015;17(3):707-19. http://dx.doi.org/10.1590/1982-0216201514814
  » http://dx.doi.org/10.1590/1982-0216201514814
• ²⁶
  Freitas LV, Teles LMR, Lima TM, Vieira NFC, Barbosa RCM, Pinheiro AKB, et al. Exame físico no pré-natal: construção e validação de hipermídia educativa para a enfermagem. Acta Paul Enferm. 2012;25(4):581-8. http://dx.doi.org/10.1590/S0103-21002012000400016
  » http://dx.doi.org/10.1590/S0103-21002012000400016
• ²⁷
  Vituri DW, Matsuda LM. Validação de conteúdo de indicadores de qualidade para avaliação do cuidado de enfermagem. Rev Esc Enferm USP. 2009;43(2):429-37. PMid:19655686. http://dx.doi.org/10.1590/S0080-62342009000200024
  » http://dx.doi.org/10.1590/S0080-62342009000200024
• ²⁸
  Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-74. PMid:843571. http://dx.doi.org/10.2307/2529310
  » http://dx.doi.org/10.2307/2529310
• ²⁹
  Martins TB, Mejias NA. Adaptação transcultural e validação do instrumento nurse parent support tool para a língua portuguesa. Online Braz J Nurs. 2011;10(2):1-10.
• ³⁰
  Yamada BFA, Santos VLCG. Construção e validação do Índice de Qualidade de Vida de Ferrans & Powers: versão feridas. Rev Esc Enferm USP. 2009;43(Esp.):1105-13.

Publication Dates

• Publication in this collection
  2017

History

• Received
  23 Sept 2016
• Accepted
  31 Mar 2017