Acessibilidade / Reportar erro

Application of Direct and Systematic Observation of Interaction with Teenage Mother-Son Dyads

Aplicação de Observação Direta e Sistemática da Interação com Díades Mãe-Filho Adolescente

Aplicación de la Observación Directa y Sistemática de la Interacción con Díadas Madre-Hijo Adolescente

Abstract

The observation of parent/child interaction is an effective method for identifying, evaluating, and monitoring adolescents and families who need or are undergoing intervention. This study aims to (1) describe a method of direct and systematic observation of dyadic interaction and (2) discuss its application in research with families with adolescents in northeastern Brazil. The data collection and analysis procedures are presented, indicating their possibilities and limits based on the lessons learned from this experience. The experience of applying the method is discussed through potential advances and challenges for the field of observational research in Psychology. It is concluded that the method of direct and systematic observation of dyadic interaction is, on the one hand, expensive in terms of expertise, time, and resources, and, on the other hand, it is a rich source of data for understanding interaction patterns in multiple contexts.

Keywords:
observation methods; qualitative research; mother-child relations; adolescents; interpersonal relationships

Resumo

A observação da interação pai/mãe e filha/o é um método efetivo para identificação, avaliação e monitoramento de adolescentes e famílias que necessitem ou estejam em intervenção. Este artigo tem como objetivos: (1) descrever um método de observação direta e sistemática da interação diádica; e (2) discutir sobre sua aplicação em uma pesquisa junto a famílias com adolescentes do nordeste brasileiro. Apresentam-se os procedimentos de coleta e análise de dados, indicando suas possibilidades e limites a partir das lições aprendidas com esta vivência. Discute-se a experiência de aplicação do método por meio dos potenciais avanços e desafios para o campo da pesquisa observacional em Psicologia. Conclui-se que o método da observação direta e sistemática da interação diádica é, por um lado, dispendioso em termos de expertise, tempo e recursos e, por outro, consiste em rica fonte de dados para a compreensão dos padrões de interação em múltiplos contextos.

Palavras-chave:
métodos de observação; pesquisa qualitativa; interação mãe-filho; adolescentes; relações interpessoais

Resumen

La observación de la interacción padre/hijo es un método efectivo para identificar, evaluar y monitorear a los adolescentes y familias que necesitan o están siendo intervenidos. Objetivos: (1) describir un método de observación directa y sistemática de la interacción diádica y (2) discutir su aplicación en una investigación junto a familias con adolescentes en el noreste brasileño. Se presentan los procedimientos de recolección y análisis de datos, indicando sus posibilidades y límites a partir de las lecciones aprendidas de esta experiencia. Se discute la experiencia de aplicación del método a través de potenciales avances y desafíos para el campo de la investigación observacional en Psicología. Se concluye que el método de observación directa y sistemática de la interacción diádica es, por un lado, costoso en términos de experiencia, tiempo y recursos y, por otro, es una rica fuente de datos para comprender los patrones de interacción en múltiples contextos.

Palabras clave:
métodos de observación; pesquisa cualitativa; relaciones madre-hijo; adolescentes; relaciones interpersonales

The observational method can be considered one of the most modern in science, as it allows a high degree of precision in obtaining and extracting data (Gil, 2019Gil, A. C. (2019). Métodos e técnicas de pesquisa social [Methods and techniques of social research] (6a ed.). Atlas.). Through it, it is possible to understand how individuals react to others, both within their group and to people outside it (Foster & Stewart, 2017Foster, B. J., & Stewart, P. A. (2017). Observational research methods and politics. In S. A. Petersen, & A. Somit (Eds.), Handbook of Biology and Politics (pp. 324-342). Edward Elgar.), whether in dyadic, triadic, or larger group interactions. In the case of dyadic interaction, the use of observation is invaluable for obtaining unique and independent information about the subjects’ actions and reactions (Lindahl et al., 2019Lindahl, K. M., Malik, N. M., & Wigderson, S. (2019). Observational methods: Overview. In B. H. Fiese, M. Celano, K. Deater-Deckard, E. N. Jouriles, & M. A. Whisman (Eds.), APA handbook of contemporary family psychology: Foundations, methods, and contemporary issues across the lifespan (pp. 281-297). American Psychological Association. ), planning and evaluating the results of interventions, and for understanding the mechanisms involved in social interaction (Aspland & Gardner, 2003Aspland, H., & Gardner, F. (2003). Observational measures of parent-child interaction: An introductory review. Child and Adolescent Mental Health, 8(3), 136-143. https://doi.org/10.1111/1475-3588.00061
https://doi.org/10.1111/1475-3588.00061...
).

The direct and systematic observation of behavior can be considered the most effective strategy for evaluating parent-child interaction (Hawes & Dadds, 2006Hawes, D. J., & Dadds, M. R. (2006). Assessing parenting practices through parent-report and direct observation during parent-training. Journal of Child and Family Studies , 15(5), 555-568. https://doi.org/10.1007/s10826-006-9029-x
https://doi.org/10.1007/s10826-006-9029-...
). It is useful both for identifying and monitoring adolescents and families who need or are in the process of intervention, and for evaluating the effects of intervention programs (Pope & Allen, 2020Pope, C., & Allen, D. (2020). Observational methods. In C. Pope, & N. Mays (Eds.), Qualitative research in health care (4th ed., pp. 67-81). Wiley Blackwell.). However, despite the method’s potential, its use is arduous and challenging, as it requires time and technological resources (Foster & Stewart, 2017Foster, B. J., & Stewart, P. A. (2017). Observational research methods and politics. In S. A. Petersen, & A. Somit (Eds.), Handbook of Biology and Politics (pp. 324-342). Edward Elgar.).

Although the direct and systematic observation of behavior has been widely used in psychological science in order to understand human interactions, the scientific improvement of the method, including standards of validity and reliability, has a relatively recent history (Cummings et al., 2014Cummings, E. M., Bergman, K. N., & Kuznicki, K. A. (2014). Emerging methods for studying families as systems. In S. M. McHale, P. Amato, & A. Booth (Eds.), Emerging methods in family research (pp. 95-108). Springer. ). The creation and coding of category systems to be observed, the evaluation of the use of natural or structured scenarios, the use of technologies, the reduction of possible biases in data extraction and analysis through the training of judges, and the use of independent and blind coding are procedures that have gained prominence in national and international research. In addition, studies that focus on deconstructing criticisms aimed at the method, notably those aimed at the small sample size and the overestimation of effect sizes (Concato et al., 2000Concato, J., Shah, N., & Horwitz, R. I. (2000). Randomized, controlled trials, observational studies, and the hierarchy of research designs. New England Journal of Medicine, 342(25), 1887-1892. https://doi.org/10.1056/nejm200006223422507
https://doi.org/10.1056/nejm200006223422...
), as well as understanding the experiences of the participants observed and possible artificialities of the method (Bennetts et al., 2017Bennetts, S. K., Mensah, F. K., Green, J., Hackworth, N. J., Westrupp, E. M., & Reilly, S. (2017). Mothers’ experiences of parent-reported and video-recorded observational assessments. Journal of Child and Family Studies, 26(12), 3312-3326. https://dx.doi.org/10.1007/s10826-017-0826-1
https://dx.doi.org/10.1007/s10826-017-08...
), have been the product of research.

Brazilian studies that have applied different methods of observing dyads have indicated their scope and limits. Among the limits, the number and duration of sessions were pointed out as insufficient to apprehend changes in behavior, the unknown setting and distracting stimuli, limited filming conditions (Pontes & Brino, 2022Pontes, L. B., & Brino, R. F. (2022). Programa Parental ACT: Um piloto com medidas observacionais da interação cuidador-criança [ACT Parenting Program: A pilot study with observational measures of parent-child interaction]. Psicologia: Teoria e Pesquisa , 38, e38313. https://doi.org/10.1590/0102.3772e38313.pt%20
https://doi.org/10.1590/0102.3772e38313....
), difficulty in data collection and extraction stages (Aguiar et al., 2022Aguiar, L., Carlini, N. C., Jr., & Portes, J. R. M. (2022). Características das interações das díades mãe-criança com deficiência auditiva durante contação de história de livro infantil [Characteristics of hearing-impaired child-mother interactions during story time with children’s book]. Revista Educação Especial, 35. http://dx.doi.org/10.5902/1984686X42598
http://dx.doi.org/10.5902/1984686X42598...
), variability of technological resources and reliable access to data in risky situations (Palazzi et al., 2021Palazzi, A., Schmidt, B., & Piccinini, C. A. (2021). Observação on-line das interações familiares: Considerações para a pesquisa [Online Observation of Family Interactions: Considerations for Research]. Revista de Psicologia da IMED, 13(2), 159-174. https://doi.org/10.18256/2175-5027.2021.v13i2.4350
https://doi.org/10.18256/2175-5027.2021....
). The following potentialities were found: analysis of verbal and non-verbal behaviors (Pontes & Brino, 2022Pontes, L. B., & Brino, R. F. (2022). Programa Parental ACT: Um piloto com medidas observacionais da interação cuidador-criança [ACT Parenting Program: A pilot study with observational measures of parent-child interaction]. Psicologia: Teoria e Pesquisa , 38, e38313. https://doi.org/10.1590/0102.3772e38313.pt%20
https://doi.org/10.1590/0102.3772e38313....
) and relational patterns, analysis of how the behaviors of family members feed back into each other (Palazzi et al., 2021Palazzi, A., Schmidt, B., & Piccinini, C. A. (2021). Observação on-line das interações familiares: Considerações para a pesquisa [Online Observation of Family Interactions: Considerations for Research]. Revista de Psicologia da IMED, 13(2), 159-174. https://doi.org/10.18256/2175-5027.2021.v13i2.4350
https://doi.org/10.18256/2175-5027.2021....
), the use of recorded material for feedback and analysis by the dyad of their own interaction (Oliveira et al., 2020Oliveira, J. J. M. D., Schmidt, C., & Pendeza, D. P. (2020). Intervenção implementada pelos pais e empoderamento parental no Transtorno do Espectro Autista [Intervention implemented by parents and parental empowerment in Autism Spectrum Disorder] Psicologia Escolar e Educacional, 24, 1-10. http://dx.doi.org/10.1590/2175-35392020218432
http://dx.doi.org/10.1590/2175-353920202...
).

Despite being recognized as an outstanding method for research with families, no recent methodological or empirical studies were found that applied the dyadic observation strategy with non-clinical adolescents as the target population, which was adopted in this study. With a view to addressing the first gap, i.e. the methodological gap, this article aims to: (1) describe a method of direct and systematic observation of dyadic interaction and (2) discuss its application in a survey of families with adolescents in northeastern Brazil.

More specifically, it is the method of direct and systematic observation of dyadic interaction developed by Kreppner and Ulrich (1996Kreppner, K., & Ulrich, M. (1996). Familien-Codier-System (FCS): Beschreibung eines codiersystems zur beurteilung von kommunikationsverhalten in familiendyaden [Family Coding System (FCS): Description of a coding system for assessing communication behavior in family dyads]. Max-Planck Institut für Bildungsforschung.), adapted for Brazil by Villas Boas (2013Villas Boas, A. C. V. B. (2013). Violência física contra a criança: Fatores de risco e proteção e padrões de interação na família. [Physical violence against children: Risk and protective factors and patterns of interaction in the family] [Tese de doutorado não publicada]. Universidade de Brasília. https://repositorio.unb.br/handle/10482/15387
https://repositorio.unb.br/handle/10482/...
), which was used within the scope of the research project “Evaluation of the Effectiveness, Quality of Implementation and Social Validity of the Strengthening Families Program”, coordinated by the second author of this article. The method is based on theories from Developmental and Family Psychology and was chosen because of its scientific maturity for assessing the dynamics of family relationships and because it has already been adapted and validated nationally (Kreppner, 2012Kreppner, K. (2012). Aplicando a metodologia de observação em psicologia do desenvolvimento e da família [Applying observational methodology in developmental and family psychology]. Juruá.; Villas Boas 2013Villas Boas, A. C. V. B. (2013). Violência física contra a criança: Fatores de risco e proteção e padrões de interação na família. [Physical violence against children: Risk and protective factors and patterns of interaction in the family] [Tese de doutorado não publicada]. Universidade de Brasília. https://repositorio.unb.br/handle/10482/15387
https://repositorio.unb.br/handle/10482/...
).

From this perspective, the research locus where the method was applied will be discussed first. Then, its methodological procedures will be presented, articulating them with the real case illustrated here. Finally, the method application will be discussed, indicating limits and possibilities, as well as highlighting the potential and challenges of its use in psychological science.

Contextualization of the Research Locus

It is not the purpose of this study to go into the research results obtained by applying the dyadic observation method, but it is necessary to contextualize the scenario in which it was applied. In 2013, the Brazilian Federal Government, through the Coordenação de Saúde Mental, Álcool, e Outras Drogas of the Ministério da Saúde, decided to expand the offer of evidence-based Alcohol, Tobacco, And Other Drugs (ATOD) prevention programs for adolescents. Not only for this specific purpose but also with a view to improving relationships, strengthening family ties, and reducing health inequalities. One of the programs chosen was the Strengthening Families Program (SFP 10-14), a program aimed at families with children between the ages of 10 and 14 (Kumpfer et al., 1996Kumpfer, K. L., Molgaard, V., & Spoth, R. (1996). The Strengthening Families Program for prevention of delinquency and drug use. In Peters, R. D., & McMahon, R. J. (Eds.), Preventing childhood disorders, substance abuse, and delinquency (pp. 241-268). Sage.).

The Programa Famílias Fortes (PFF 10-14), as it has been called in Brazil, is based on a set of theories and models: Social Learning Theory, Social Ecology Theory, Family Systems Theory, Social Cognitive Theory, Attachment Theory, Biopsychosocial Vulnerability Model, Family Resilience Model and Family Process Model. Its ultimate goal is to reduce, delay, and prevent the use of ATOD by improving parenting practices, young people’s social, academic, and problem-solving skills, and fostering family values, communication, and organization (Kumpfer et al., 1996Kumpfer, K. L., Molgaard, V., & Spoth, R. (1996). The Strengthening Families Program for prevention of delinquency and drug use. In Peters, R. D., & McMahon, R. J. (Eds.), Preventing childhood disorders, substance abuse, and delinquency (pp. 241-268). Sage.). It consists of seven two-hour sessions, in which in the first hour parents and children are attended simultaneously in separate sessions; and in the second hour, everyone is assisted in a joint session. Booster sessions are also recommended. Unlike most of the program’s offerings around the world (Pinheiro-Carozzo et al., 2021Pinheiro-Carozzo, N. P., Murta, S. G., Vinha, L. G. A., Silva, I. M., & Fontaine, A. M. G. V. (2021). Beyond effectiveness of the Strengthening Families Program (10-14): A scoping REAIM-based review. Psicologia: Reflexão e Crítica, 34, 1-16. https://doi.org/10.1186/s41155-021-00182-z
https://doi.org/10.1186/s41155-021-00182...
), PFF 10-14 was not delivered in schools, but in Centros de Referência em Assistência Social (CRAS), public social facilities that serve individuals and families in situations of social vulnerability.

Considering the partnership between government entities at the time, the northeast of Brazil was the region that proved most favorable to the implementation of the PPF 10-14. With information on the size of the CRAS and the volume of services provided throughout the year, the social facilities that would take part in the study were chosen for convenience. For this stage of the research, 107 families with children aged 10 to 14 who benefited from some form of continued assistance were invited to take part in the program; 40 of them accepted the invitation. Subsequently, participants were lost for one or more of the following reasons: failure to complete the program (dropping out), loss of contact with the family, moving to a geographical area not served by CRAS, loss of ties with CRAS, withdrawal from the study and technical recording difficulties.

The team that applied the method was selected via a call for tenders advertised in the main psychology faculties in the states where the collection was planned, and received technical training on the observation and video preservation methodology. This team reported to, and was supported by, a regional coordinator, who was linked to the national coordinator of the project.

Given the exploratory nature of the study, a single-group design was adopted, with the dyads participating in the observation method, which is the subject of this article, in the pre and post-test. There were variations in the time interval for data collection in the post-test, which occurred due to the limited availability of physical space and scheduling restrictions between the collection team, the family, and CRAS. However, as this effect of improved family interaction is expected to be short-term and long-lasting, the difference in time interval did not seem to influence the results.

The research followed all the ethical parameters set out by the National Health Council and was approved by the Comitê de Ética em Pesquisa em Ciências Humanas e Sociais of the Universidade de Brasília (CAAE: 53103516.1.0000. 5540/Approval 2.191.408). Despite efforts to standardize data collection procedures, local particularities and political (dis)articulations were impediments (Abdala et al., 2020Abdala, I. G., Murta, S. G., Menezes, J. C. L., Nobre-Sandoval, L. A., Gomes, M. S. M., Duailibe, K. D., & Farias, D. A. (2020). Barriers and facilitators in the Strengthening Families Program (SFP 10-14) implementation process in Northeast Brazil: A retrospective qualitative study. International Journal of Environmental Research and Public Health, 17(19), 6979. https://doi.org/10.3390/ijerph17196979
https://doi.org/10.3390/ijerph17196979...
). As a result, after careful analysis of validity and reliability, some pieces of data were eliminated from the research corpus, and those considered comparable were used instead. As a result, the complete research experience (from recruitment to data analysis) was carried out with families of adolescents aged between 10 and 14, from four cities in the state of Rio Grande do Norte and users of the CRAS in their territory.

The application of the Dyadic Interaction Observation Method

In this section, the method will be described together with its application to the case described.

Data collection. The observation sessions took place at two different times, one week before (pre-test) and between one week and one month after (post-test) the family have taken part in the PFF 10-14. The location was chosen according to the family’s suggestion: at the CRAS where they were registered or at their own home, when contextual variables prevented or made it difficult for them to go to the CRAS.

On the appointed day and time, the room made available by CRAS or the space provided by the family was prepared in advance by a pair of observers, previously trained in applying the method. In order to avoid inconveniences and maximize the quality of the filming, environmental issues such as lighting, external noise, the arrangement of chairs, which were always positioned side by side, and the positioning of the camera were managed. The participants were then invited to sit facing the camera, and given general information about the filming, such as the impossibility of talking to the observers and voice pitch, and about the research method. They were told that the method consisted of four cards which would be read out by the observer and then the dyad would have to talk about the content of the card.

The situations on the cards were based on the family functioning model proposed by Beavers and Hampson (2000Beavers, R., & Hampson, R. B. (2000). The Beavers Systems Model of family functioning. Journal of Family Therapy, 22(2), 128-143. https://doi.org/10.1111/1467-6427.00143
https://doi.org/10.1111/1467-6427.00143...
). The authors evaluate family interaction based on two dimensions: competence and style. The first relates to structure, i.e. the information available in the system and the flexibility of the family, including its ability to adapt to new situations and to negotiate. The second, family style, is linked to the quality of family interaction, i.e. achieving satisfaction and support. These cards were part of a set of eight cards describing everyday family situations and issues pertinent to the adolescents’ age group, such as school issues and living together in the family environment. The cards were based on studies by Kreppner and Ulrich (1996Kreppner, K., & Ulrich, M. (1996). Familien-Codier-System (FCS): Beschreibung eines codiersystems zur beurteilung von kommunikationsverhalten in familiendyaden [Family Coding System (FCS): Description of a coding system for assessing communication behavior in family dyads]. Max-Planck Institut für Bildungsforschung.) and Villas Boas (2013Villas Boas, A. C. V. B. (2013). Violência física contra a criança: Fatores de risco e proteção e padrões de interação na família. [Physical violence against children: Risk and protective factors and patterns of interaction in the family] [Tese de doutorado não publicada]. Universidade de Brasília. https://repositorio.unb.br/handle/10482/15387
https://repositorio.unb.br/handle/10482/...
) and their content was adapted to the local culture, based on hypothetical situations that require negotiation of interests and emotional support (See Figure 1). A validation study of the use of the cards was carried out previously, still within the scope of the research project.

Figure 1
Situations proposed in the Cards for Observing Dyadic Family Interaction

When the dyad verbalized that they were ready to start, the observers began filming. The initial instruction was given by one of the observers:

I have some cards that describe everyday situations in families. I’m going to read the cards out loud and then I’d like you to talk about it for about two to three minutes. After that, I’ll read the next card and you should talk about it, continuing until the last card. It’s important that you both have the chance to give your opinion on the subject of the cards.

The observer then read out the first card and finished with the command: “Discuss the topic”. The same command was repeated for the other cards.

The observation sessions lasted an average of 2 minutes and 58 seconds for each family and 40.76 seconds for each card. Despite the instruction to make the discussion last longer, all the dyads ended the subject before the time allotted for each card. As a result, more than four cards were read at some points during the collection process, in order to broaden the discussion and produce data for analysis.

Data analysis. Data analysis took place in six phases: (1) post-production, (2) validation of the category system, (3) training for judges, (4) data extraction, (5) concordance analysis and (6) data analysis comparing the collection times, i.e. pre-test versus post-test.

Post-production

The first stage is called post-production. It involves creating new video material by assembling scenes to help identify similarities, differences and/or details of the dyad’s interaction (Kreppner, 2012Kreppner, K. (2012). Aplicando a metodologia de observação em psicologia do desenvolvimento e da família [Applying observational methodology in developmental and family psychology]. Juruá.). Specifically, using Windows Movie Maker software, each session was cut and reassembled. Thus, the sections on reading the card - which were removed from the new material - were separated from the sections on the discussion on the subject, which were kept for the next stage; and a black screen was inserted for five seconds, signaling the end of the discussion and the switch to the next card. This task was carried out by one of the members of the research group who did not work on data collection, extraction or analysis because she was aware of what the pre-test interactions were and what the post-test interactions were.

Validation of the category system

In the second stage, the system of categories was validated. To do this, the category system proposed by Kreppner and Ulrich (1996Kreppner, K., & Ulrich, M. (1996). Familien-Codier-System (FCS): Beschreibung eines codiersystems zur beurteilung von kommunikationsverhalten in familiendyaden [Family Coding System (FCS): Description of a coding system for assessing communication behavior in family dyads]. Max-Planck Institut für Bildungsforschung.) and adapted by Villas Boas (2013Villas Boas, A. C. V. B. (2013). Violência física contra a criança: Fatores de risco e proteção e padrões de interação na família. [Physical violence against children: Risk and protective factors and patterns of interaction in the family] [Tese de doutorado não publicada]. Universidade de Brasília. https://repositorio.unb.br/handle/10482/15387
https://repositorio.unb.br/handle/10482/...
) was used, and its relevance and suitability for the data in this study were analyzed. This analysis was carried out by the first author of this article, confirmed by the second author, and guided by a senior researcher experienced in the method.

For this task, all the footage was watched for the first time. Two categories were eliminated: (1) Who picks up the card was eliminated because no family manually accessed the cards once they had been read by the observer. This was done to standardize the procedure for subjects (mothers) who verbalized reading difficulties. (2) Body orientation was another eliminated category, as the seats provided made it difficult for participants to move around. Thus, the system was made up of 11 categories of analysis, 2 of which refer to formal aspects, 6 to verbal aspects, 2 to non-verbal aspects, and 1 to the overall aspect of dyadic interaction. Four categories are classified into levels, namely relative speaking time, insertion of self, tension, and proximity; the others are classified into subcategories. Figure 2 shows the coding system in categories, subcategories, and levels.

Figure 2
Categories and subcategories of the Evaluation System

Training for judges

The training for judges was online and synchronous and covered topics on (a) the family system, (b) the observation method in general, and (c) the specific observation method adopted. Themes (a) and (b) were the subject of textual discussion using book chapters and scientific articles. Theme (c) was not only discussed but also trained in the form of workshops. Initially, the textual description of the categories of analysis was shared for the initial reading of four judges, who were undergraduate students in Psychology and linked to the Scientific Initiation Program of their Higher Education Institution. After reading the material, there were two meetings: one to discuss the categories and the other to practice data extraction using a recording clip. As there were still doubts about the categories and there was a low level of agreement among the judges, video material was prepared to illustrate each of the categories in all their subcategories and levels, so that the judges could resolve any inconsistencies or doubts in the categorization of the interactions. Two more meetings were held to align data extraction. The training was concluded when acceptable agreement rates were repeatedly obtained in different categories.

Data extraction

In this fourth stage, both versions of the category system described above (textual and video) were used to carry out data extraction, i.e. identifying and coding categories and subcategories/levels. First, the situation (the video of the interaction fostered by each card) was viewed three times, to familiarize with the participants, the environment, and the language used and to check for any complications in the recording. The situation was then watched three more times and one of the 11 categories was recorded, and so on for each category. In the case of categories coded for each member of the dyad, the previous procedure was repeated for coding for each of them. Thus, each situation was viewed 36 times, independently, by judges who were unaware of the time of collection, whether pre-test or post-test, who then coded the situation.

Agreement analysis

Agreement among observers serves several purposes, such as “reducing the possibilities of error and inconsistency during collection, increasing the reliability and quality of the data, minimizing possible biases that a given observer may have (or develop) in relation to the target behavior” (Sella et al., 2020Sella, A. C., Santos, J. J. S., Cavalcante, R. D. P., Gomes, S. N. B., Santana, S. L. S., & Ribeiro, D. M. (2020). Concordância entre observadores e fidelidade de implementação no Brasil: Uma revisão [Observer agreement and implementation fidelity in Brazil: A review]. Acta Comportamentalia, 28(1), 53-71. https://www.revistas.unam.mx/index.php/acom/article/view/75181
https://www.revistas.unam.mx/index.php/a...
, p. 55). Calculating agreement between observers can be done in two ways: inter-observer or intra-observer (Dessen, 2013Dessen, M. A. (2013). Tecnologia de vídeo: Registro de interações sociais e cálculos de fidedignidade em estudos observacionais [Video technology: Recording social interactions and reliability calculations in observational studies]. Psicologia: Teoria e Pesquisa, 11(3), 223-227. https://periodicos.unb.br/index.php/revistaptp/article/view/17254
https://periodicos.unb.br/index.php/revi...
). In the first option, a researcher and judges are needed, the records are compared and the level of agreement is calculated. This can be calculated using the percentages of agreement as a reference, with those with indices greater than 75% being considered adequate; and can be calculated using the Kappa index or Interclass Correlation Coefficient (ICC) depending on the type of variable (Courtright, 2020Courtright, J. A. (2020). Observational methods. In J. van den Bulck, D. R. Roskos-Ewoldsen (Eds.), The international encyclopedia of media psychology (pp. 1-6). Willey Blackwell.). In the study reported here, an inter-observer analysis was carried out between a researcher and two judges. A moderate Fleiss Kappa (k = .42; p < .001; 95% CI [.38, .47]) was obtained for the nominal categories, and a substantial ICC (ICC= .75; p < .001; 95% CI [.74, .82]) for the ordinal categories.

If it is impossible to carry out the analysis among observers, due to a lack of financial and/or human resources, the second option can be used, i.e. intra-observer analysis. In this case, the observer extracts all the categories and then does it again after a few days. In the end, her/his first analysis is compared with her/his second analysis (Dessen, 2013Dessen, M. A. (2013). Tecnologia de vídeo: Registro de interações sociais e cálculos de fidedignidade em estudos observacionais [Video technology: Recording social interactions and reliability calculations in observational studies]. Psicologia: Teoria e Pesquisa, 11(3), 223-227. https://periodicos.unb.br/index.php/revistaptp/article/view/17254
https://periodicos.unb.br/index.php/revi...
). The calculation is the same as for the inter-observer analysis.

Data analysis comparing collection times

At this stage, once acceptable levels of agreement have been obtained, the researcher knows the dyad’s pattern of interaction and sets out to understand the data based on theory. The aim is to explain the functioning and analyze the dynamics and structures of the family system, in an attempt to find the deep structure that is responsible for what is apparent, what is seen publicly (Kreppner, 2012Kreppner, K. (2012). Aplicando a metodologia de observação em psicologia do desenvolvimento e da família [Applying observational methodology in developmental and family psychology]. Juruá.). In the study adopted as an illustration, the aim was to compare family interaction before and after, with a view to verifying the effect of PFF 10-14 on the outcome of family interaction. Thus, qualitative analyses were carried out comparing how the dyad behaved before (pre-test) participating in the program and how they behaved after (post-test) their participation. Discussions of the findings were based on the systemic side of Family Studies, particularly those models that more openly admit topographical-behavioral analyses.

Having said that, the concept of family interaction is adopted as a relational, dynamic, complex, and historical process, which involves the co-construction of meanings, in verbal and non-verbal terms, and which takes place at the level of content (literal) and relationship (Turner & West, 1998Turner, L. H., & West, R. (1998). Perspectives on family communication. Mayfield.). The quality of this interaction, in turn, can be understood through the dimensions of (1) intensity: the degree or magnitude to which the interaction takes place, for example, when something, independent if communicated by whisper or shout, has the same effect; (2) content and verbal material, which can be understood as positive, negative or neutral; (3) non-verbal communication: facial expressions and expressive gestures of emotion; (4) emotions that have an impact on the other’s behavior; (5) motivation and willingness to interact, as well as affection, beliefs, closeness, intimacy, autonomy and independence (Hinde, 1997Hinde, R. A. (1997). Relationships: A dialectical perspective. Psychology Press.). The theoretical proposals of Beavers and Hampson (2000Beavers, R., & Hampson, R. B. (2000). The Beavers Systems Model of family functioning. Journal of Family Therapy, 22(2), 128-143. https://doi.org/10.1111/1467-6427.00143
https://doi.org/10.1111/1467-6427.00143...
), Hinde (1997Hinde, R. A. (1997). Relationships: A dialectical perspective. Psychology Press.), and Turner and West (1998Turner, L. H., & West, R. (1998). Perspectives on family communication. Mayfield.) can be aligned with others from Systemic Family Theory to shed light on both the family’s systemic functioning and the influence it can have on adolescents.

Possibilities and limits of applying the dyadic interaction observation method

In the light of the stages of implementation of the method of direct and systematic observation of behavior, possibilities and limits are identified in its application. In the data collection phase, four issues emerged: the location, the card reading, the resources, and discussion duration; while in the analysis phase: the presence of a work team.

The location of the data collection may have hindered family interaction and audio quality. When data collection took place in the CRAS, there was, sometimes, no room that could be isolated and set aside for this purpose, which is why there were frequent interruptions when staff came in. As the family taking part in the research was a user of this CRAS, the staff were known to them, so there was a pause until it was ascertained who was entering the room and the family continued to interact about the card. When the collection took place in the home, other stimuli in the environment took the dyads’ attention away from the recording: a third child demanding attention, a television on, and motorcycle noise in the street. Then the focus returned to the recording and the method was applied again. In line with previous studies, it can be seen that being in a familiar environment reduces the novelty of the scenario itself; however, distracting elements inherent to it can compromise the dyad’s attention and concentration on the task (Pontes & Brino, 2022Pontes, L. B., & Brino, R. F. (2022). Programa Parental ACT: Um piloto com medidas observacionais da interação cuidador-criança [ACT Parenting Program: A pilot study with observational measures of parent-child interaction]. Psicologia: Teoria e Pesquisa , 38, e38313. https://doi.org/10.1590/0102.3772e38313.pt%20
https://doi.org/10.1590/0102.3772e38313....
). It should be noted that it is up to each research team to define the best observation scenario since the context can have an impact on the mother-child interaction (Dittrich et al., 2017Dittrich, K., Fuchs, A., Führer, D., Bermpohl, F., Kluczniok, D., Attar, C. H., Jaite, C., Zietlow, A.-L., Licata, M., Reck, C., Herpertz, S. C., Brunner, R., Mohler, E., Resch, F., Winter, S. M., Lehmkuhl, U., & Bödeker, K. (2017). Observational context of mother-child interaction: Impact of a stress context on emotional availability. Journal of Child and Family Studies , 26(6), 1583-1591. http://dx.doi.org/10.1007/s10826-017-0678-8
http://dx.doi.org/10.1007/s10826-017-067...
).

As far as reading the card is concerned, the method originally proposed by Kreppner (2012Kreppner, K. (2012). Aplicando a metodologia de observação em psicologia do desenvolvimento e da família [Applying observational methodology in developmental and family psychology]. Juruá.) involves placing the card on a table in front of the dyad and the person who picks up the card is a category to be analyzed. However, the difficulty some participants had in reading and/or understanding the text led to the introduction of the card being changed. Thus, all the cards were read to ensure homogeneity in the application of the method. As a result, it was impossible to analyze this variable. If, on the one hand, there was a limitation in an analysis category that was “lost”; on the other hand, it was possible to expand the viability and applicability of the method by including, as potential participants, a non-literate population.

The project’s design called for the use of cameras on tripods so that the recordings were high-quality. With the scarcity of resources, a barrier faced by the majority of Brazilian researchers, the pairs of observers who collected this data started using the cameras on their own laptops or the cameras on their cell phones. This was also due to the fact that the data was collected in four different municipalities and it was not possible, given the time available for data collection, to wait until the end of the collection in one place to transport the equipment to another. Recording problems did occur, but they did not prevent data extraction. Adopting a longer collection time or more technological equipment are some possibilities for addressing the issue.

Moreover, the dyad discussion lasted less than expected on all cards and for all dyads, resulting in the need to use more than four cards in each session, as initially proposed. Consistently with this, this dilemma was also found by Pontes & Brino (2022Pontes, L. B., & Brino, R. F. (2022). Programa Parental ACT: Um piloto com medidas observacionais da interação cuidador-criança [ACT Parenting Program: A pilot study with observational measures of parent-child interaction]. Psicologia: Teoria e Pesquisa , 38, e38313. https://doi.org/10.1590/0102.3772e38313.pt%20
https://doi.org/10.1590/0102.3772e38313....
). Furthermore, some interactions had to be excluded from the analysis because they lacked information that could consistently characterize the interaction. At first glance, two points can be detected. The first is that the lack of interaction was due to the very fact that the dyad was being observed, a factor that can be minimized if it is possible for a third person, previously trained and known within the family, to carry out the recordings, as Oliveira et al. (2020Oliveira, J. J. M. D., Schmidt, C., & Pendeza, D. P. (2020). Intervenção implementada pelos pais e empoderamento parental no Transtorno do Espectro Autista [Intervention implemented by parents and parental empowerment in Autism Spectrum Disorder] Psicologia Escolar e Educacional, 24, 1-10. http://dx.doi.org/10.1590/2175-35392020218432
http://dx.doi.org/10.1590/2175-353920202...
) opted for. Secondly, because these were adolescents and mothers with low levels of education - since they themselves claimed to have reading difficulties - the content or complexity of the discussion was limited. In addition, their children’s developmental stage, marked by the gradual development of independence (Lordello, 2015Lordello, S. R. (2015). Envolvendo o adolescente na discussão sobre drogas: Qual sua contribuição ao debate? [Involving adolescents in the discussion about drugs: what is their contribution to the debate?] In M. F. O. Sudbrack, M. I. G. Conceição, I. M. Silva, & S. R. Lordello (Orgs.), ABRAMD: Compartilhando saberes [ABRAMD: Sharing knowledge] (Vol. 1, pp. 121-138). Technopolitik. ), may have reduced their engagement in activities that they consider unattractive, in the company of those responsible or that would highlight family conflicts. If this is not the case, the situations proposed in the cards may not have taken into account the specificities of the family and the micro-culture, which is why it is suggested that the cards should, whenever possible, be co-created with representatives of potential participants. Nonetheless, the non-verbal manifestations, also captured in the method, provided valuable information and the use of more than four cards did not cause any problems neither for the dyad nor for the research team.

Lastly, as the data was analyzed, the importance of a work team was highlighted. The presence of senior researchers, experienced in the research method, who acted as advisors or supervisors, was essential to validate the inclusion or not of categories in the study; to guide the training of judges, since the use of the textual description of the categories was not sufficient; and to expand the options for calculating the concordance analysis. Similar difficulties were encountered in previous studies (Aguiar et al., 2022Aguiar, L., Carlini, N. C., Jr., & Portes, J. R. M. (2022). Características das interações das díades mãe-criança com deficiência auditiva durante contação de história de livro infantil [Characteristics of hearing-impaired child-mother interactions during story time with children’s book]. Revista Educação Especial, 35. http://dx.doi.org/10.5902/1984686X42598
http://dx.doi.org/10.5902/1984686X42598...
). In addition, members of the team, although not experienced in the method, contributed to post-production, ensuring that the judges were unaware of whether the recording they were viewing was before or after that dyad’s participation in the Program; and were essential in extracting data and categorizing the interactions. Students from the final years of undergraduate and postgraduate studies can make up this research network, contributing as assistants and research partners, making the method more robust and the results obtained more reliable. The geographic dispersion of the team, with members from various regions of Brazil, helped to enable the research to rely on key researchers in their field at the same time.

Potentialities and challenges for the field of observational research in psychology

This research experience introduced the use of the dyadic interaction observation method in the field of evaluating preventive programs in ATOD. Through this method, it was possible to examine changes in mother-child interaction, one of the outcomes expected by the PFF 10-14, alongside other data collection strategies such as scales, interviews, and focus groups. In addition, it allowed authors to learn a number of lessons that relate to the potential and challenges of its use:

  • (1) The set of procedures that precede the data analysis - i.e. post-production, validation of the category system, training for judges, data extraction, and concordance analysis - indicates that the method is expensive in terms of time, and the schedule should take this into account.

  • (2) Having a research team with different levels of experience is essential so that all the stages of the data collection and analysis process are to be carried out with greater robustness and reliability.

  • (3) When developing the project, the conditions for participants to go to the data collection setting and financial resources for the purchase and distribution of recording equipment should be assessed and provided for, in the case of a survey involving several municipalities.

  • (4) When developing the cards, more than eight cards should be drawn up with situations that will serve as a basis for interaction. Thus, if more than four cards are used in the pre-test, they will not be repeated in the post-test. In addition, the cards should be co-created with potential participants to maximize the relevance of the situations to the family and its micro-culture.

  • (5) Before data collection begins, an agreement must be reached with the data collection setting, taking into account the conditions for its use: a well-lit, quiet environment, and available furniture that can be made available to the team, without access by third parties.

The main contributions of this study lie in problematizing the application of the observation method as an option for investigating dyadic interactions in psychological science. In this regard, researchers have made progress in two aspects that are dear to Psychology: the study of dyads as a unit of analysis and the validity and methodological rigor of the study. Firstly, communication and family interaction are factors that can be configured as risk or protective factors for various developmental and health outcomes in children and adolescents. The analysis of these processes, given the relational presupposition imposed, must be carried out using the source and the audience as the unit of analysis, i.e. at least two people. The presentation of a systematized method for this purpose increases the range of options available to researchers interested in the subject and the strategy.

Secondly, given the varied objects of study and their levels of complexity, as well as the diversity of theoretical-methodological perspectives and the breadth of the field of Psychology, the validity and methodological rigor of the proposal in question favor scientific legitimacy (Seidl et al., 2021Seidl, E. M. F., Queiroz, E., Iglesias, F., & Neubern, M. (2021). Apresentação [Presentation]. In E. M. F. Seidl, E. Queiroz, F. Iglesias, & M. Neubern. Estratégias metodológicas de pesquisa em Psicologia Clínica: Possibilidades e avanços [Research methodological strategies in Clinical Psychology: Possibilities and advances] (pp. 11-15). CRV.). Future studies aimed at understanding family interaction could benefit from the use of dyad observation, either by replicating, adapting, or generating methodological innovations based on the method described here, especially for triangulation with verbal data.

Once the necessary infrastructure is in place, it is suggested that this method be used with dyads of different compositions: couples, teachers and students, dads and children, among others, as a way of broadening the possibilities of use. It would also be advisable to use it with different populations, such as families from other regions of Brazil. Finally, the triangulation of methods, with the use of interviews preceding or succeeding direct and systematic observation of dyadic interaction, would certainly provide robust data for understanding the meanings, dynamics, and relational patterns, with broad applicability in multiple contexts.

Acknowledgments:

To Ingrid Gomes Abdala, Larissa Nobre-Sandoval, Jorge Gato, Ana Carolina Barbosa de Mello and Maria Júlia Barbosa de Mello, for their contribution and participation in different stages of the research. To Ana Carolina Villas-Boas and Professor Maria Auxiliadora Dessen, for their guidance and supervision.

References

  • Abdala, I. G., Murta, S. G., Menezes, J. C. L., Nobre-Sandoval, L. A., Gomes, M. S. M., Duailibe, K. D., & Farias, D. A. (2020). Barriers and facilitators in the Strengthening Families Program (SFP 10-14) implementation process in Northeast Brazil: A retrospective qualitative study. International Journal of Environmental Research and Public Health, 17(19), 6979. https://doi.org/10.3390/ijerph17196979
    » https://doi.org/10.3390/ijerph17196979
  • Aguiar, L., Carlini, N. C., Jr., & Portes, J. R. M. (2022). Características das interações das díades mãe-criança com deficiência auditiva durante contação de história de livro infantil [Characteristics of hearing-impaired child-mother interactions during story time with children’s book]. Revista Educação Especial, 35. http://dx.doi.org/10.5902/1984686X42598
    » http://dx.doi.org/10.5902/1984686X42598
  • Aspland, H., & Gardner, F. (2003). Observational measures of parent-child interaction: An introductory review. Child and Adolescent Mental Health, 8(3), 136-143. https://doi.org/10.1111/1475-3588.00061
    » https://doi.org/10.1111/1475-3588.00061
  • Beavers, R., & Hampson, R. B. (2000). The Beavers Systems Model of family functioning. Journal of Family Therapy, 22(2), 128-143. https://doi.org/10.1111/1467-6427.00143
    » https://doi.org/10.1111/1467-6427.00143
  • Bennetts, S. K., Mensah, F. K., Green, J., Hackworth, N. J., Westrupp, E. M., & Reilly, S. (2017). Mothers’ experiences of parent-reported and video-recorded observational assessments. Journal of Child and Family Studies, 26(12), 3312-3326. https://dx.doi.org/10.1007/s10826-017-0826-1
    » https://dx.doi.org/10.1007/s10826-017-0826-1
  • Concato, J., Shah, N., & Horwitz, R. I. (2000). Randomized, controlled trials, observational studies, and the hierarchy of research designs. New England Journal of Medicine, 342(25), 1887-1892. https://doi.org/10.1056/nejm200006223422507
    » https://doi.org/10.1056/nejm200006223422507
  • Courtright, J. A. (2020). Observational methods. In J. van den Bulck, D. R. Roskos-Ewoldsen (Eds.), The international encyclopedia of media psychology (pp. 1-6). Willey Blackwell.
  • Cummings, E. M., Bergman, K. N., & Kuznicki, K. A. (2014). Emerging methods for studying families as systems. In S. M. McHale, P. Amato, & A. Booth (Eds.), Emerging methods in family research (pp. 95-108). Springer.
  • Dessen, M. A. (2013). Tecnologia de vídeo: Registro de interações sociais e cálculos de fidedignidade em estudos observacionais [Video technology: Recording social interactions and reliability calculations in observational studies]. Psicologia: Teoria e Pesquisa, 11(3), 223-227. https://periodicos.unb.br/index.php/revistaptp/article/view/17254
    » https://periodicos.unb.br/index.php/revistaptp/article/view/17254
  • Dittrich, K., Fuchs, A., Führer, D., Bermpohl, F., Kluczniok, D., Attar, C. H., Jaite, C., Zietlow, A.-L., Licata, M., Reck, C., Herpertz, S. C., Brunner, R., Mohler, E., Resch, F., Winter, S. M., Lehmkuhl, U., & Bödeker, K. (2017). Observational context of mother-child interaction: Impact of a stress context on emotional availability. Journal of Child and Family Studies , 26(6), 1583-1591. http://dx.doi.org/10.1007/s10826-017-0678-8
    » http://dx.doi.org/10.1007/s10826-017-0678-8
  • Foster, B. J., & Stewart, P. A. (2017). Observational research methods and politics. In S. A. Petersen, & A. Somit (Eds.), Handbook of Biology and Politics (pp. 324-342). Edward Elgar.
  • Gil, A. C. (2019). Métodos e técnicas de pesquisa social [Methods and techniques of social research] (6a ed.). Atlas.
  • Hawes, D. J., & Dadds, M. R. (2006). Assessing parenting practices through parent-report and direct observation during parent-training. Journal of Child and Family Studies , 15(5), 555-568. https://doi.org/10.1007/s10826-006-9029-x
    » https://doi.org/10.1007/s10826-006-9029-x
  • Hinde, R. A. (1997). Relationships: A dialectical perspective. Psychology Press.
  • Kreppner, K., & Ulrich, M. (1996). Familien-Codier-System (FCS): Beschreibung eines codiersystems zur beurteilung von kommunikationsverhalten in familiendyaden [Family Coding System (FCS): Description of a coding system for assessing communication behavior in family dyads]. Max-Planck Institut für Bildungsforschung.
  • Kreppner, K. (2012). Aplicando a metodologia de observação em psicologia do desenvolvimento e da família [Applying observational methodology in developmental and family psychology]. Juruá.
  • Kumpfer, K. L., Molgaard, V., & Spoth, R. (1996). The Strengthening Families Program for prevention of delinquency and drug use. In Peters, R. D., & McMahon, R. J. (Eds.), Preventing childhood disorders, substance abuse, and delinquency (pp. 241-268). Sage.
  • Lindahl, K. M., Malik, N. M., & Wigderson, S. (2019). Observational methods: Overview. In B. H. Fiese, M. Celano, K. Deater-Deckard, E. N. Jouriles, & M. A. Whisman (Eds.), APA handbook of contemporary family psychology: Foundations, methods, and contemporary issues across the lifespan (pp. 281-297). American Psychological Association.
  • Lordello, S. R. (2015). Envolvendo o adolescente na discussão sobre drogas: Qual sua contribuição ao debate? [Involving adolescents in the discussion about drugs: what is their contribution to the debate?] In M. F. O. Sudbrack, M. I. G. Conceição, I. M. Silva, & S. R. Lordello (Orgs.), ABRAMD: Compartilhando saberes [ABRAMD: Sharing knowledge] (Vol. 1, pp. 121-138). Technopolitik.
  • Oliveira, J. J. M. D., Schmidt, C., & Pendeza, D. P. (2020). Intervenção implementada pelos pais e empoderamento parental no Transtorno do Espectro Autista [Intervention implemented by parents and parental empowerment in Autism Spectrum Disorder] Psicologia Escolar e Educacional, 24, 1-10. http://dx.doi.org/10.1590/2175-35392020218432
    » http://dx.doi.org/10.1590/2175-35392020218432
  • Palazzi, A., Schmidt, B., & Piccinini, C. A. (2021). Observação on-line das interações familiares: Considerações para a pesquisa [Online Observation of Family Interactions: Considerations for Research]. Revista de Psicologia da IMED, 13(2), 159-174. https://doi.org/10.18256/2175-5027.2021.v13i2.4350
    » https://doi.org/10.18256/2175-5027.2021.v13i2.4350
  • Pinheiro-Carozzo, N. P., Murta, S. G., Vinha, L. G. A., Silva, I. M., & Fontaine, A. M. G. V. (2021). Beyond effectiveness of the Strengthening Families Program (10-14): A scoping REAIM-based review. Psicologia: Reflexão e Crítica, 34, 1-16. https://doi.org/10.1186/s41155-021-00182-z
    » https://doi.org/10.1186/s41155-021-00182-z
  • Pontes, L. B., & Brino, R. F. (2022). Programa Parental ACT: Um piloto com medidas observacionais da interação cuidador-criança [ACT Parenting Program: A pilot study with observational measures of parent-child interaction]. Psicologia: Teoria e Pesquisa , 38, e38313. https://doi.org/10.1590/0102.3772e38313.pt%20
    » https://doi.org/10.1590/0102.3772e38313.pt%20
  • Pope, C., & Allen, D. (2020). Observational methods. In C. Pope, & N. Mays (Eds.), Qualitative research in health care (4th ed., pp. 67-81). Wiley Blackwell.
  • Seidl, E. M. F., Queiroz, E., Iglesias, F., & Neubern, M. (2021). Apresentação [Presentation]. In E. M. F. Seidl, E. Queiroz, F. Iglesias, & M. Neubern. Estratégias metodológicas de pesquisa em Psicologia Clínica: Possibilidades e avanços [Research methodological strategies in Clinical Psychology: Possibilities and advances] (pp. 11-15). CRV.
  • Sella, A. C., Santos, J. J. S., Cavalcante, R. D. P., Gomes, S. N. B., Santana, S. L. S., & Ribeiro, D. M. (2020). Concordância entre observadores e fidelidade de implementação no Brasil: Uma revisão [Observer agreement and implementation fidelity in Brazil: A review]. Acta Comportamentalia, 28(1), 53-71. https://www.revistas.unam.mx/index.php/acom/article/view/75181
    » https://www.revistas.unam.mx/index.php/acom/article/view/75181
  • Turner, L. H., & West, R. (1998). Perspectives on family communication. Mayfield.
  • Villas Boas, A. C. V. B. (2013). Violência física contra a criança: Fatores de risco e proteção e padrões de interação na família. [Physical violence against children: Risk and protective factors and patterns of interaction in the family] [Tese de doutorado não publicada]. Universidade de Brasília. https://repositorio.unb.br/handle/10482/15387
    » https://repositorio.unb.br/handle/10482/15387
  • Funding: Secretaria Nacional de Políticas sobre Drogas do Ministério da Justiça do Brasil (SENAD/MJ) (Protocol No. 08129.014870/2015-16). Fundação de Amparo à Pesquisa e ao Desenvolvimento Científico e Tecnológico do Maranhão - FAPEMA (Notice BD 46/2017).

Edited by

Associate editor: Denise Ruschel Bandeira

Publication Dates

  • Publication in this collection
    04 Dec 2023
  • Date of issue
    2023

History

  • Received
    08 Mar 2023
  • Reviewed
    09 July 2023
  • Accepted
    18 July 2023
Universidade de São Paulo, Faculdade de Filosofia Ciências e Letras de Ribeirão Preto, Programa de Pós-Graduação em Psicologia Av.Bandeirantes 3900 - Monte Alegre, 14040-901 Ribeirão Preto - São Paulo - Brasil, Tel.: (55 16) 3315-3829 - Ribeirão Preto - SP - Brazil
E-mail: paideia@usp.br