Validity and reliability of an instrument to measure factors associated with screen time in Brazilian students

Validade e confiabilidade de um instrumento para avaliar fatores associados ao tempo de tela em escolares brasileiros

Abstract

This study analyzed the validity and reliability of an instrument with scales measuring factors associated with screen time in adolescents. Two cross-sectional studies included students for analyses of validity and internal consistency (n=1138, 51.5% boys) and test-retest reproducibility (n=194, 56.2% boys). Individual (attitude, self-efficacy and expectations), social (behaviors, rules, beliefs and family support and support of school members) and physical environmental (home and bedroom equipment) factors of screen time were evaluated. All the scales of the instrument met the statistical criteria of the validity of the exploratory factorial analysis (p<0.001). The exploratory factorial analysis indicated five one-dimensional and three two-factor scales (self-efficacy, expectations and familiar behavior), with factorial loads of 0.56 to 0.88. There were significant correlations between the scores of almost all the scales and the reported screen time (p<0.05). Eight of nine scales/subscales showed Cronbach’s α value being higher than 0.70, and ICC ranged from 0.63 to 0.83. In conclusion, the instrument presented scales with acceptable validity and reliability and can be used to measure factors associated with screen time in Brazilian students.

Key words
Sedentary lifestyles; Associated factors; Questionnaire; Validity tests; Reproducibility of results

Resumo

Este estudo analisou a validade e a confiabilidade de um instrumento com escalas de fatores associados ao tempo de tela em adolescentes. Dois estudos transversais incluíram escolares para análises de validade (n=1138, 51,5% meninos) e reprodutibilidade teste-reteste (n=194, 56,2% meninos) do instrumento. Fatores individuais (atitude, autoeficácia e expectativas), do ambiente social (comportamentos, regras, crenças e suporte familiar e suporte dos membros da escola) e físico (casa e no quarto) associados ao tempo de tela foram avaliados. Todas as escalas do instrumento atenderam os critérios estatísticos de validade da análise fatorial exploratória (p<0.001). A análise fatorial exploratória indicou cinco escalas unidimensionais e três bi-dimensionais (autoeficácia, expectativas e comportamento familiares), com cargas fatoriais de 0,56 a 0,88. Houve correlações significativas entre quase todas as escalas e o tempo de tela reportado (p<0,05). Oito das nove escalas apresentaram um valor de α de Cronbach superior a 0,70, e o CCI variou de 0,63 a 0,83. O instrumento apresentou escalas que se apresentaram com validade e confiabilidade aceitáveis, podendo ser utilizado na mensuração de fatores associados ao tempo de tela em escolares brasileiros.

Palavras-chave
Estilos de vida sedentários; Fatores associados; Questionário; Testes de validade; Reprodutibilidade de resultados

Introduction

Reducing screen time seems to be an emerging topic in public health promotion in Brazilian children and adolescents, as supported by the Brazilian Society of Pediatrics¹. One of the relevant pathways for effective interventions in reducing screen time among young people is understanding the factors on which to focus during the intervention strategies/actions (i.e., modifiable factors that can influence screen time)².

A theoretical framework that is used to understand the factors associated with screen time is the socio-ecological framework. This framework emphasizes that factors from different levels may explain why and how young people had specific behaviors: these can be individual (e.g., self-efficacy and expectations), social (e.g., social support of friends and family) and physical environmental (e.g., house equipment) factors^3,4. The SOS-framework (Systems of Sedentary Behaviors) is another approach that suggest the potential determinants of sedentary behavior (including screen time) across the life course; this approach also suggests that factors from different levels (specially, considering the interaction between factors from psychological to environments levels) should be considered in studies on determinants of screen time and other sedentary behaviors².

Some studies have analyzed factors associated with screen time^4,5, but a few studies have examined a comprehensive set of factors at different levels of influences. Thus, which factors and levels influence on screen time among young people is an unanswerable question³. A possible explanation for a few studies on this topic is the scarcity of valid and reliable instruments to measure the behavior (screen time) and its correlates^3,6,7. Few studies presented validated and reliable instruments on individual and environmental factors associated with screen time in children and adolescents^3,7, and in Brazil, to our knowledge, no instrument was found with validity and reliability indicators. Thus, this is a gap in the national and international literature⁷. Filling this gap is relevant, because validated and reliable scales are relevant to adequately measure screen time correlates and to understand the effectiveness of screen time interventions on modifiable determinants of this behavior.

Therefore, the present study aimed to examine the validity and reliability of an instrument for measuring individual, social environmental and physical environmental factors associated with screen time among Brazilian students.

Method

The instrument used in this study was developed in order to measure factors associated with screen time in an intervention study that aimed for active and healthy lifestyle promotion among students in Fortaleza, northeastern Brazil – the “Fortaleça sua Saúde” program. A detailed description of the population of interest, sampling and intervention program is in the trial record (ClinicalTrials.Gov) and in a previous publication⁸.

This instrument was constructed considering a socio-ecological approach of sedentary behavior^7,9. The developed instrument focused on evaluating modifiable factors of screen time in order to be attempted in intervention studies. Thus, most of the scales considered items that included sentences focused on reducing screen time (i.e., attitude, self-efficacy, family’s support and school members’ support for reducing screen time). Other scales focused on measuring the construct related with screen time, such as the screen equipment in the bedroom⁸.

During the selection of scales and items for this instrument, different scales and items were extracted and adapted from other instruments previously validated for adolescents^10-15. This decision was made because a single instrument was not considered a broad evaluation of modifiable factors of screen time. Also, adaptations on the scales were performed in order to consider a specific type of sedentary behavior (screen time), considering that most of previous scales were based on total sedentary behavior). The used items and scales were in English and, thus, they had a process of translation and language adaptation in order to consider relevant expressions and contexts related to adolescent’s behavior. All process was performed by two experts that are familiar with development and validation of instrument, as well as terms of the literature on screen time and sedentary behavior. A full version of the instrument was evaluated by five stakeholders with several publications on sedentary behavior and theoretical and practical experience in the validation process of instruments. After stakeholder suggestions, the corrected version was filled out by the students. The final version of the instrument is in Chart 1.

Chart 1
Instrument.

Attitude toward reducing screen time use	Give your opinion on the following statements:
	Decreasing the time in front of the TV, computer or video games is important for you.	Not important; A little important; Important; Very important
	Decreasing the time in front of the TV, computer or video games is either boring or fun for you.	Very boring; Boring; Fun; Great fun
	Decreasing the time in front of the TV, computer or video games is good or bad for your health.	Very bad; Bad; Good; Very good
Self-efficacy toward reducing screen time use	I think I am able...
	Individual limits and abilities
	...to limit (decrease) my time watching TV for two hours a day.	Strongly Disagree; Disagree; Agree; Strongly Agree
	...to turn off the TV, even though I am watching a program that I like.
	...not to watch TV on most days with school classes.
	...to limit (decrease) my time using the computer/video games for two hours a day.
	...to turn off the computer/video game, even though I'm doing something I enjoy (favorite games, chatting).
	...not to not use computer/video games on most days with school classes.
	...to determine limits for how long I am in front of the TV, computer or video games.
	...to stop watching TV or using computer/video games in my leisure time to do physical activity.
	Confidence
	...to plan what I am going to watch on TV during the week.	Strongly Disagree; Disagree; Agree; Strongly Agree
	...to watch TV doing other activities that require body effort (clean, play) rather than sitting.
	...to plan what I am going to access on the computer or play video games during the week.
Expectations of screen time use	State how much you disagree or agree with the following statements:
	Positive expectations
	I think it is very relaxing when I sit in front of the TV.	Strongly Disagree; Disagree; Agree; Strongly Agree
	I feel good (happy) when I'm at the computer (talking or playing) or playing the video game.
	I get excited (agitated) when I'm using the computer or video game.
	Using the computer or video games is my way of connecting to the world (making friends).
	My friends would be sad if I shortened my time talking to them on the computer.
	I like to watch TV or use the computer/video games for many hours at a time.
	Watching TV or using the computer/video games is one of the things I enjoy doing in my leisure time.
	I watch TV or use the computer/video games to escape the world (obligations, discussions, problems).
	Negative expectations
	Watching TV or using the computer/video games traps me from doing important things (studying, eating).	Strongly Disagree; Disagree; Agree; Strongly Agree
	I get lazy after spending many hours in front of the TV, computer or video games.
	I feel pain in the body (back, legs) after spending many hours in front of the TV, computer or video games. Watching TV or using the computer/video games burns my eyes and leaves me with a headache.
Family model of screen time use	During a typical week, how often...
	TV use of household members
	...Do your parents (mother and/or father) watch TV?	Never; Sometimes; Often; Always
	...Do other people in your household watch TV?
	...Do you and your parents watch TV together?
	...Do you and others in your household watch TV together?
	Computer/games use of household members
	...Do your parents use the computer/video games?	Never; Sometimes; Often; Always
	...Do other people in your household use the computer/video games?
	...Do you and your parents use the computer/video games together?
	...Do you and others in your home use the computer/video games together?
Family rules of screen time use	During a typical week, how often do your parents...
	...limit the time that you can watch TV (e.g., only after dinner or when you finish your homework)?	Never; Sometimes; Often; Always
	...watch what you're watching on TV?
	...turn the TV off if they think you're watching TV for a long time?
	...limit the amount of time you can spend on the computer/video games (e.g., only after dinner or when you finish your homework)?
	...observe what you are accessing/playing on the computer or video game?
	...turn off computer/video games if they think you're playing for a long time?
Family beliefs of screen time use	In general, the people of my house…
	...think watching TV or using the computer/video games is a cool thing to do.	Strongly Disagree; Disagree; Agree; Strongly Agree
	...prefer to sit in front of the TV, computer or video games rather than doing physical activity.
	...think it's safer for me to stay at home in front of TV, computer or video games rather than doing physical activity outside the home.
Family support for reducing screen time use	In general, the people of my house…
	...encourage me to decrease the time watching TV or using the computer/video games.	Strongly Disagree; Disagree; Agree; Strongly Agree
	...comment to me that a lot of time in front of the TV or computer/video games can do harm to my health.
	...help me think about how I can decrease the time in front of the TV or computer/video games.
	...praise me when I spend less time in front of TV or computer/video games and more time doing physical activity.
	...prevent me from watching TV or using the computer/video games when I do something wrong.
School members' support for reducing screen time use	In general, the people of my school (colleagues, teachers or coordinators)
	...encourage me to decrease the time watching TV or using the computer/video games.	Strongly Disagree; Disagree; Agree; Strongly Agree
	...comment to me that a lot of time in front of the TV or computer/video games can do harm to my health.
	...help me think about how I can decrease the time in front of the TV or computer/video games.
	...praise me when I spend less time in front of the TV or computer/video games and more time doing physical activity.
Items of physical environmental variables	Number of TVs at home	Does not have; one; two; three or more
	Number of Video games at home
	Number of Computers with Internet access at home
	Number of Computers without Internet access at home
	Number of TVs in the bedroom
	Number of Video games in the bedroom
	Number of Computers with Internet access in the bedroom
	Number of Computers without Internet access in the bedroom

Study samples

Study 1 - Validity internal consistency

One sample was evaluated in order to examine the validity and internal consistency (reliability) of the instrument. For this aim, 1,272 students from 40 classes of 7-9 grades in all six public secondary full-time schools in Fortaleza (Brazil) were eligible; they were invited to participate in the “Fortaleça sua Saúde” program⁸. Of these, 87 were not present for the data collection; four students refused to participate, and 47 questionnaires were excluded, due to completion errors. A total of 1,138 students had valid data. This sample size was adequate for estimating validity and reliability parameters, considering the rate of approximately 20 individuals for each item of the instrument¹⁶.

Study 2 - Test-retest reproducibility (reliability)

Another sample was used to verify the test-retest reproducibility of the instrument. For this, two public schools in Fortaleza, independent of the schools in Study 1, were intentionally selected. Students from the 12 classes of 7-9 grades were eligible. In total, 194 students contributed test-retest data. This sample allows for the identification of ICC greater than 0.20 as statistically significant (α=0.05) and 80% power (β=0.20) with two instrument applications.

Instruments

The instrument examined in this study was initially composed of 60 items that were structured in nine scales. A detailed description of each scale is in Table 1. Three scales were related to individual factors. The attitude for reducing screen time scale (3 items) represents the affective and instrumental aspects for reducing screen time, considering contrasting adjectives, such as fun–boring, important–negligible, health-harmful¹³. The self-efficacy for reducing screen time scale represents the students’ perceptions of the confidence and abilities to reduce their screen time^12,13. The expectations of screen time scale measured the students’ perception on the positive (8 items) and negative (4 items) expectations about their screen time^13,14,17. Five were social/environmental factor scales. The scale of family’s screen time use represents the students’ perception on the frequency of screen time among parents and other family’s members, that could be performed alone or together with the students (8 items)^14,18. One scale evaluated the students’ perception of family rules of screen time (six items)¹¹, and another scale was focused on measuring the family’s beliefs on screen time (3 items)¹⁷. Also, the family’s support for reducing screen time (5 items) and the schools’ members support for reducing screen time (4 items) were evaluated in two separated scales^11,15,17. Finally, the scales on the physical environmental variables included eight items of the screen equipment (e.g., computer with Internet access, etc.) in the students’ homes and bedrooms^11,14.

Table 1
Description of the scales used in this instrument for measuring factors associated to screen time among Brazilian students. Fortaleza, Brazil, 2014.

Factors associated with screen time use	Description and instruments used as reference	n of itens	Scale
Scales of individual variables
Attitude for reducing screen time use*	Students' perception of affective and instrumental aspects that are generally related to the behavior in focus13; in this instrument, the reduction of the screen time use.	3	A four-point semantic differential scale with binary pairs of adjectives (trivial/important, boring/fun, harmful/health)
Self-efficacy for reducing screen time use*	Students' perception of abilities (8 items) and confidence (3 items) for reducing their screen time use10,12,13.	11	Likert with 4 points, 1"Strongly disagree" to 4 "Strongly agree"
Negative and positive expectations of screen time use**	Students' perception on the positive (8 items) and negative (4 items) expectations about their screen time use13,14,17.	12	Likert with 4 points, 1"Strongly disagree" to 4 "Strongly agree"
Scales of social environmental variables
Family screen time use	Students' perception of how often the parents and other family members use TV and computer/video games, considering individual use (4 items) and together with them (4 items)14,15.	8	Never, rarely, frequently, always
Family rules of screen time use*	Students' perception of how often a household member gave different types of rules for screen time use (e.g., limit screen use to only 2 hours per day)11,15.	6	Never, rarely, frequently, always
Family beliefs on screen time use	Students' perception of family beliefs on screen time use11,15,17.	3	Likert with 4 points, 1"Strongly disagree" to 4 "Strongly agree"
Family support for reducing screen time use*	Students' perception of support for reducing screen time use from the household members11,15,17.	5	Likert with 4 points, 1"Strongly disagree" to 4 "Strongly agree"
School members support for reducing screen time use*	Students' perception of support for reducing screen time use from school members (friends, teachers, coordinators)11,15,17.	4	Likert with 4 points, 1"Strongly disagree" to 4 "Strongly agree"
Items of physical environmental variables
Home and bedroom screen equipment	Items (TV, computer with and without internet access, and video games) in their home and in their bedroom that may encourage screen time use11,14.	8	Number of items (0, 1, 2, and 3 or more)

• ^*
  These item scales were inverted in order to standardize all scale scores (higher scores were favorable to screen time use).
• ^**
  Only the scores of negative expectations of screen time use subscale were inverted.

Students self-reported gender, age and grade. Students reported screen time during free time, using four standardized questions¹⁹. These questions were related to habitual daily time watching TV or using computer/video games on weekdays and weekend days separately. These questions were combined in order to estimate the total screen time as well as the time watching TV or using computer/video games on habitual days.

Procedures

Study 1 - Validity and internal consistency (reliability)

Evaluators previously trained in theoretical explanations and practical simulations administered the instrument with students in the classroom, without the presence of teachers. Prior to administration, the evaluators provided instructions and read each question aloud. Then, the students answered the questions. Data collection was conducted in November and December 2014.

Study 2 - Test-retest reproducibility (reliability)

An identical application was also performed during the test-retest data collections. Data collection was applied (test) and repeated (retest) at an interval of 15 days. In the first application, students received the instrument in an envelope to complete. In the second application, the same procedure was carried out. Data collections were conducted in June 2014.

Data analysis

We used mean, standard deviation and frequency for a description of the sample characteristics. All analyses were performed using the software SPSS 21.0 (SPSS, Inc., Chicago, IL) with a significance level of 5%.

Study 1 - Validity and internal consistency (reliability)

Construct (factorial) validation was analyzed using exploratory factorial analysis, considering the principal component method with Promax rotation, because correlations between the constructs are expected by the theory. This procedure was performed in order to define factors and items that were extracted in each scale (Table 1). For this, factors were retained if their self-values were equal to or greater than one, and items with a factor loading equal to or greater than 0.4 were considered relevant¹⁶. The adequacy of the variable items for factor analysis was assessed using the Kaiser Meyer-Olkin (KMO) index, Bartlett’s Sphericity Test (BST) and the measure of sampling adequacy (MSA). Satisfactory values were considered if KMO and MSA ≥0.60 and BST with statistical significance at p<0.05¹⁶. Items on screen equipment were evaluated separately (not analyzed as a scale), because each item may have an isolated association with screen time among students, i.e., they do not build a construct (factor)⁹.

Construct validity was assessed using Spearman’s correlation coefficient (rho) between the scale scores and reported total screen time. We standardized the scores of the items in a positive direction relationship with the screen time. Items with higher values representing conditions that were less favorable to screen time had their values reversed before the sum of the scale scores.

Internal consistency was tested using Cronbach’s α, because it is the main parameter used in studies on reliability of scales, including in studies on sedentary behaviors⁶. Values α≥0.70 and ranging from 0.60 to 0.69 were considered adequate and substantial, respectively²⁰.

Study 2 - Test-retest reproducibility (reliability)

Test-retest reliability was assessed using the ICC with a confidence interval of 95% (95%CI) between test-retest measurements. ICC values ≥0.70 were considered adequate, and values from 0.60 to 0.69 were considered substantial²⁰.

Ethical Considerations

Parents/guardians of the students authorized participation in this study by signing the informed consent. The National Research Ethics System approved this research project.

Results

Characteristics of samples

The final samples were 1,138 and 194 in studies 1 and 2, respectively. No statistically significant difference between samples was found for gender, age groups and school grade and time watching TV (p>0.05). A sample of test-retest reproducibility (Study 2) showed a higher proportion of students who reported using computer/video games two or more hours per day (63.9% versus 48.7%; p<0.05, Table 2).

Table 2
Characteristics of the study samples. Fortaleza, Brazil, 2014.

Descriptive Variables	Sample of validation and internal consistency analyzes (n=1,138)		Sample of test–retest reproducibility analyzes (n=194)		X2	p-value
	n	%	n	%
Gender					1.46	0.226
Boys	558	51.5	109	56.2
Girls	526	48.5	85	43.8
Age groups (years)					0.094	0.760
11-12	169	14.9	34	17.6
13-14	642	56.6	101	52.3
15	204	18.0	40	20.7
16 or more	120	10.6	18	9.3
Grades					6.561	0.065
7th	474	41.7	71	36.6
8th	400	35.1	73	37.6
9th	264	23.2	50	25.8
Reported TV time use					1.96	0.187
<2 hours/day	374	29.4	57	29.4
≥2 hours/day	710	70.6	137	70.6
Reported computer/games time use
<2 hours/day	556	51.3	70	36.1	15.43	<0.001
≥2 hours/day	528	48.7	124	63.9

Validity and internal consistency

Exploratory factor analysis

All results for the KMO and MSA tests were higher than 0.60 and statistically significant for Bartlett’s sphericity test (p<0.001), indicating that the exploratory factor analysis was adequate in the tested scales (Table 3).

Table 3
Analyses of validity and reliability parameters of scales of factors associated to screen time among Brazilian students. Fortaleza, Brazil, 2014.

Scales (n items and % of variance explained by these items)	Description	Factor loading		Reliability
	Mean (SD)	1	2	α**	ICC	95%CI
Scales and subscales of individual aspects of screen time use
Attitude for reducing screen time use (3 items, 57.9%)	7.40 (2.08)	KMO=0.63 Bartlett's test p<0.01		0.64	0.71	0.63	0.79
Unimportant/important	2.50 (1.00)*	0.81	-		0.72	0.64	0.81
Boring/Funny	2.89 (0.83)*	0.73	-		0.70	0.62	0.79
Bad/good for health	2.01 (0.90)*	0.74	-		0.68	0.60	0.77
Self-efficacy for reducing screen time use (11items, 49.6%)	27.29 (5.69)	KMO=0.88 Bartlett's test p<0.01		0.84	0.63	0.55	0.72
Individual limits and abilities (8 items, 38.8%)	20.22 (4.76)			0.85	0.71	0.63	0.80
To limit time watching TV for 2 hours a day	2.34 (0.82)*	0.67	-		0.65	0.57	0.74
To turn off the TV	2.80 (0.84)*	0.69	-		0.85	0.77	0.94
Do not watch TV on most school days	2.46 (0.82)*	0.59	-		0.64	0.56	0.73
To limit time using computer/games for 2 hours a day	2.49 (0.86)*	0.76	-		0.65	0.57	0.74
To turn off computer/games	2.71 (0.87)*	0.76	-		0.73	0.65	0.82
Do not use computer/games on school days	2.51 (0.86)*	0.70	-		0.69	0.61	0.78
To determine limits in screen time use	2.44 (0.83)*	0.73	-		0.66	0.58	0.75
To change screen time for physical activity in free-time	2.48 (0.94)*	0.66	-		0.73	0.65	0.82
Confidence (3 items, 10.7%)	7.07 (1.68)			0.61	0.64	0.56	0.73
To schedule the time watching TV on weekdays	2.38 (0.75)*	-	0.78		0.66	0.58	0.75
To watch TV doing other activities	2.38 (0.80)*	-	0.56		0.61	0.53	0.70
To schedule the time using computer/games on weekdays	2.30 (0.82)*	-	0.78		0.63	0.55	0.72
Expectations with screen time use (12 items, 50.7%)	31.63 (4.98)	KMO=0.86 Bartlett's test p<0.01		0.71	0.74	0.66	0.83
Positive expectations (8 items, 33.5%)	22.29 (4.65)			0.85	0.78	0.70	0.87
It is relaxing to watch TV	2.86 (0.76)	0.56	-		0.69	0.61	0.78
I feel good (happy) when I am using computer/games	3.03 (0.75)	0.76	-		0.64	0.56	0.73
I get excited (agitated) when I am using computer/games	2.79 (0.83)	0.72	-		0.69	0.61	0.78
Connecting with friends using computer/games	2.84 (0.85)	0.73	-		0.70	0.62	0.79
Cut the time using computer is not good to my friends	2.54 (0.88)	0.57	-		0.66	0.58	0.75
I like to use screen for many hours at the same time	2.74 (0.88)	0.79	-		0.76	0.68	0.85
I prefer to use screen during free-time	2.91 (0.84)	0.77	-		0.67	0.59	0.76
I use screen to escape the world (e.g., problems)	2.59 (0.90)	0.62	-		0.61	0.53	0.70
Negative expectations (4 items, 17.2%)	9.34 (2.58)			0.70	0.65	0.57	0.74
It trap me from doing important things (e.g., studying)	2.26 (0.85)*	-	0.68		0.62	0.54	0.71
I get lazy	2.22 (0.86)*	-	0.72		0.58	0.50	0.67
I feel pain in the body (back, legs)	2.45 (0.89)*	-	0.79		0.69	0.61	0.78
It burns my eyes and leaves me with a headache	2.41 (0.91)*	-	0.74		0.66	0.58	0.75
Scales and subscales of social environmental variables
Family model of screen time use (8 items, 55.5%)	18.39 (4.59)	KMO=0.64 Bartlett's test p<0.01		0.73	0.74	0.66	0.83
TV use of household members (4 items, 34.8%)	11.06 (2.94)			0.74	0.70	0.62	0.79
Parents' TV watching	3.00 (0.96)	0.73	-		0.69	0.61	0.78
Household's TV watching	3.03 (0.98)	0.74	-		0.73	0.65	0.82
Parents and son/daughter watching TV together	2.50 (1.00)	0.77	-		0.70	0.62	0.79
Households and son/daughter watching TV together	2.53 (1.00)	0.76	-		0.67	0.59	0.76
Computer/games use of household members (4 items, 20.7%)	7.32 (2.83)			0.70	0.64	0.56	0.73
Parents' computer/games using	1.75 (0.96)	-	0.69		0.72	0.64	0.81
Household's computer/games using	2.36 (1.15)	-	0.67		0.60	0.52	0.69
Parents and son/daughter using computer/games together	1.45 (0.80)	-	0.79		0.65	0.57	0.74
Households and son/daughter using computer/games together	1.77 (0.96)	-	0.76		0.68	0.60	0.77
Family rules of screen time use (6 items, 57.6%)	18.80 (4.69)	KMO=0.84 Bartlett's test p<0.01		0.85	0.69	0.61	0.78
Limiting TV time	3.17 (0.99)*	0.72	-		0.66	0.58	0.75
Observe the TV content	2.84 (1.09)*	0.69	-		0.64	0.56	0.73
Turn off TV	3.27 (1.00)*	0.79	-		0.62	0.54	0.71
Limiting computer/games use	3.19 (1.02)*	0.81	-		0.72	0.64	0.81
Observe the computer/games content	3.04 (1.06)*	0.75	-		0.58	0.50	0.67
Turn off computer/games	3.30 (1.01)*	0.79	-		0.66	0.58	0.75
Family beliefs of screen time use (3 items, 52.9%)	7.57 (1.87)	KMO=0.62 Bartlett's test p<0.01		0.55	0.66	0.58	0.75
Using screen is cool/fun	2.66 (0.81)	-	0.68		0.53	0.45	0.62
Using screen is better than perform physical activity.	2.38 (0.85)	-	0.77		0.58	0.50	0.67
Using screen is safer than perform physical activity outside the home.	2.53 (0.91)	-	0.74		0.59	0.51	0.68
Family support for reducing screen time use (5 items, 54.2%)	12.46 (3.19)	KMO=0.83 Bartlett's test p<0.01		0.78	0.67	0.59	0.76
To encourage to decrease screen time	2.55 (0.85) *	0.74	-		0.55	0.47	0.64
To speak that screen time is unhealthy	2.23 (0.84) *	0.76	-		0.62	0.54	0.71
To help to decrease screen time	2.54 (0.83) *	0.80	-		0.54	0.46	0.63
To laud reducing screen time	2.54 (0.89) *	0.75	-		0.70	0.62	0.79
To limit screen time when I do something wrong.	2.60 (0.93) *	0.64	-		0.63	0.55	0.72
School members support for reducing screen time use (4 items, 68.7%)	11.20 (2.93)	KMO=0.81 Bartlett's test p<0.01		0.85	0.69	0.61	0.78
To encourage to decrease screen time	2.90 (0.88)*	0.83	-		0.65	0.57	0.74
To speak that screen time is unhealthy	2.62 (0.90)*	0.83	-		0.62	0.54	0.71
To help to decrease screen time	2.80 (0.86)*	0.88	-		0.59	0.51	0.68
To laud reducing screen time	2.89 (0.90)*	0.78	-		0.70	0.62	0.79
Items of physical environmental variables
Number of TV at home	2.00 (0.76)				0.78	0.70	0.87
Number of Video games at home	0.53 (0.64)				0.81	0.73	0.90
Number of Computer with internet access at home	0.98 (0.96)				0.72	0.64	0.81
Number of Computer without internet access at home	0.22 (0.53)				0.66	0.58	0.75
Number of TV in bedroom	0.65 (0.52)				0.76	0.68	0.85
Number of Video games in bedroom	0.30 (0.50)				0.83	0.75	0.92
Number of Computer with internet access in bedroom	0.55 (0.71)				0.71	0.63	0.80
Number of Computer without internet access in bedroom	0.13 (0.39)				0.70	0.62	0.79

• ^*
  These item scales were inverted in order to standardize all scale scores (higher scores were favorable to screen time use).
• ^**
  Cronbach's α was used to represent the internal consistency of the total scale and excluding each item.

A single factor was identified for attitude for reducing screen time (the proportion of explained variance by the items was 57.9%). Two factors were identified in the scale of self-efficacy for reducing screen time, which were labeled as individual limits and abilities and confidence. The total variance explained by the items was 39.3% and 10.8%, respectively. Two factors were extracted with all items of the scale expectations of screen time: positive expectations (33.3% of total variance) and negative expectations (17.5% of total variance, Table 3).

Two factors were identified in the scale of family’s screen time use: family’s TV use (35.0% of total variance) and family’s computer/games use (20.7% of total variance). One factor was extracted for each scale: family’s rules of screen time, family’s beliefs on screen time, family’s support for reducing screen time and school members’ support for reducing screen time. The total variance explained by the items was 57.9%, 53.5%, 54.2% and 68.7%, respectively (Table 3).

Construct validity

The scores of almost all scales and subscales presented statistically significant correlations with total screen time. The significant correlations coefficients ranged from -0.068 (p<0.001, subscale of self-efficacy related to confidence for reducing screen time) to 0.462 (p<0.001, subscale of positive expectations). In general, similar results were found when time watching TV and time using computer/games were analyzed separately (Table 4).

Table 4
Spearman correlation (rho) between scale scores and reported daily screen time use among students. Fortaleza, Brazil, 2014.

Scales	Reported daily screen time use in leisure-time
	Total screen time use	TV time use	Computer/games time use
Scales and subscales of individual aspects of screen time use
Attitude for reducing screen time use	0.255**	0.180**	0.198**
Self-efficacy for reducing screen time use	0.200**	0.157**	0.136**
Individual limits and abilities	0.335**	0.209**	0.280**
Confidence	-0.068*	-0.016	-0.090**
Expectations with screen time use	0.393**	0.196**	0.392**
Positive expectations	0.462**	0.234**	0.447**
Negative expectations	0.104**	0.025	0.131**
Scales and subscales of social environmental variables
Family model of screen time use	0.283**	0.225**	0.235**
TV use of household members	0.166**	0.287**	0.001
Computer/games use of household members	0.236**	0.013	0.359**
Family rules of screen time use	0.001	0.003	-0.014
Family beliefs of screen time use	0.154**	0.154**	0.098**
Family support for reducing screen time use	0.010	0.052	-0.031
School members support for reducing screen time use	0.035	0.007	0.042
Items of physical environmental variables (screen equipment)
Number of TV at home	0.127**	0.129**	0.080**
Number of Video games at home	0.223**	0.081**	0.297**
Number of Computer with internet access at home	0.244**	-0.002	0.364**
Number of Computer without internet access at home	-0.015	0.024	-0.028
Number of TV in bedroom	0.109**	0.134**	0.055
Number of Video games in bedroom	0.215**	0.103**	0.248**
Number of Computer with internet access in bedroom	0.235**	0.003	0.348**
Number of Computer without internet access in bedroom	-0.006	-0.011	0.008

• ^*
  p<0.05;
• ^**
  p<0.01.

Considering the items of physical environment, the quantity of screen equipment at home and in the bedroom showed a statistically significant correlation with screen time. All significant correlations ranged from 0.109 (p<0.001, number of TVs in the bedroom) to 0.244 (p<0.001, number of computers with Internet access at home). Also, similar results were found when time watching TV and time using computer/games were analyzed separately (Table 4).

Internal consistency (reliability)

Eight of nine scales showed adequate Cronbach’s α values (≥0.70), ranging from 0.70 (subscale of negative expectations of screen time) to 0.85 (subscale of self-efficacy: individual limits and abilities for reducing screen time, positive expectations of screen time, family rules of screen time and school members’ support for reducing screen time). The scale of attitude for reducing screen time (α=0.64) and the subscale of self-efficacy: confidence for reducing screen time (α=0.61) showed substantial internal consistency. The scale of family’s beliefs on screen time had a Cronbach’s α of 0.55 (Table 3).

Study 2 - Test-retest reproducibility (reliability)

The ICC of the scales ranged from 0.63 (self-efficacy for reducing screen time) to 0.78 (positive expectations of screen time). Considering the physical environmental variables (screen equipment), the ICC ranged from 0.66 (number of computers without Internet access at home) to 0.83 (number of video games in the bedroom).

Discussion

The present study aimed to examine the validity and reliability of an instrument with nine scales that represents potential modifiable factors associated with screen time among Brazilian students. In general, scales had acceptable validity, internal consistency, and test–retest reproducibility. Valid and reliable measures of individual, social and physical environmental factors associated with screen time are fundamental for exploring and understanding the factors that could be target of surveillance and interventions on screen time among young people.

The attitude for reducing screen time scale was confirmed with all three items, with a significant correlation with screen time among students. The attitude is a fundamental construct for determining an individual’s intention to perform behavior, for example, when the child or adolescent has a positive attitude towards reducing screen time, it is probable that this reduction occurs⁴. This factor is frequently evaluated in physical activity intervention studies⁴, but studies on this factor for sedentary behavior and screen time are limited, mainly because scales on attitude for screen time are scare⁴. Considering that attitude is considered by an international transdisciplinary framework for the studies on sedentary behavior across the life course², we suggest the use of the attitude for reducing screen time scale in intervention studies, because it may be important to understand whether interventions are effective on this construct and whether it helps in reducing the screen time among young people.

The scale of self-efficacy for reducing screen time was confirmed with eleven items and a two-factor structure (individual limits and abilities and confidence) and scores with statistically significant correlation with screen time (rho=0.200). Differently, Norman et al.¹³ found a scale of self-efficacy for reducing sedentary behaviors (including other behaviors besides screen time) in 11- to 15-year-old American adolescents composed by one factor (seven items) – this was one of the first studies focused on developing and validating a scale on factors associated with screen time. Difference between scales may represent the fact that various cultures might affect the self-efficacy systems. Students’ perception of abilities and confidence is relevant to define individual strategies for a specific behavior, but studies measuring self-efficacy as a potential determinant of screen time³ or in interventions studies for reducing screen time⁴ are still scare. Thus, we believe the measurement of self-efficacy for reducing screen time is a relevant aspect of the proposed instrument and in future intervention studies, because it can help to understand different facets of the self-efficacy and how they can explain screen time changes among young people.

Outcome expectation is a factor mentioned in the Social Cognitive Theory and Health Belief Model that has been studied as a correlate of physical activity²¹, but studies on screen time are insufficient. In our study, negative expectations and positive expectations constructs were identified and they had significant correlations with screen time. Similarly, Norman et al.¹³ developed a “decisional balance” scale with negative and positive expectations of screen time. The existence of two factors indicates the importance of both the benefits and the costs related to observed behavior, reflecting the level of motivation to change behavior. The scale on outcome expectations of screen time had satisfactory validity parameters and, thus, studies measuring these constructs help to understand whether the perception of benefits and costs is associated with screen time, as well as how interventions focused on these constructs can be effective on reducing screen time among adolescents.

The family’s screen time use scale presented the highest correlation coefficient (rho=0.283) with screen time between social environmental scales, and the family’s beliefs on screen time scale presented a significant correlation with screen time (rho=0.138). Members of the family may be considered the primary agents for the development of healthy behavior in children and adolescents¹¹, and family-related factors were considered in the pioneer studies on scales of factors associated with screen time among young people^13,17. Besides some studies that presented paternal and maternal rules as a significant determinant to screen time among children^22,23, our results showed that family’s rules of screen time and family’s support for reducing screen time scales were not significantly correlated with screen time (rho<0.010). Based on our results, the scales on family’s screen time use and family’s beliefs on screen time can be considered in studies on family correlates of screen time or the effectiveness of family-based interventions on screen time-related outcomes. However, we stimulate futures studies on the validity of scales on family-related factors of screen time, because most of the scales proposed in our study had trivial correlation coefficients.

Results from the school members’ support for reducing screen time scale demonstrated the existence of one factor with four items. Understanding the impact of social support on health behaviors can help to develop future health promotion interventions⁷. Specifically among adolescents, teachers and friends have a greater contribution to the adoption of behaviors because they tend to be more independent from their families. However, in our study, the scores of this scale did not have a significant correlation with screen time; possibly, because the screen use (TV, computer or video games) tends to be performed in the home context. Thus, using the scale of school members’ support for reducing screen time may be important in studies on the effectiveness of school-based interventions on screen time-related outcomes, in order to identify changes on an intervention process element i.e., whether the school context change the way they discuss on screen time and health. However, studies on school members’ support for screen time and other sedentary behaviors are still a relevant literature gap³.

Most of screen items at home and in the bedroom showed a significant positive correlation with screen time. Studies have shown that physical environment is a strong factor associated with screen time²⁴. For instance, a longitudinal study with school-aged children found that the presence of a computer in the bedroom was positively associated with screen time²⁵. Focus on these screen equipment in both contexts seems to be relevant to understand environmental factors related to screen time and to identify whether interventions with strategies that use electronic monitoring devices and limits for screen use are effective in reducing screen time among young people²⁶.

The majority of scales showed adequate internal consistency with Cronbach’s alpha values above 0.70. Only the family’s beliefs on screen time scale obtained a value below substantial, possibly explained by the low number of items comprising the scale²⁷. In addition, all scales showed substantial or appropriate values in the test-retest reproducibility (ICC ranging from 0.63 to 0.83). Few studies have investigated the reproducibility of the measures about individual, social and physical environmental factors associated with screen time in adolescents. Singh et al.²⁸ evaluated the test-retest reproducibility using an instrument with 24 items of potential screen-viewing determinants in young people and identified ICC, ranging from 0.48 to 0.92. Approximately 76% of the items had ICC as good; however, the classification used by the authors consider values above 0.60 as good, different from our study²⁸. In another study, which evaluated the ICC test-retest values of parents’ rules and guidance in relation to screen time among Chinese children, the authors showed substantial reliability (ICC ranging from 0.72 to 0.86)²⁹. These evidences show the complexity of measuring psychosocial factors associated with screen time in children and adolescents.

According to a review³⁰, no study that focused on assessing the measurement properties of self-report questionnaires about sedentary behavior (screen time) were considered both valid and reliable. Moreover, to the authors’ knowledge, this is the first study to examine psychometric propriety of the questionnaire about individual, social and physical environmental factors associated with screen time for Brazilian children and adolescents. Finally, other strengths were the sample with sufficient statistical power and the different statistical procedures used to make inferences, both for the analysis of validity and for the reliability of instrument¹⁶.

This study has some limitations. First, some scales were developed with the objective of measuring the factors associated with screen time, while others were directed to interventions aimed at reducing screen time. Therefore, the items those higher values represent conditions that were less favorable to screen time had their values reversed, and studies using these scales should consider the represented constructs. Also, some scale had a low quantity of items, and it may explain some results of psychometric indicators²⁷.

A comprehensive approach of screen time has been stimulated, and we need to improve the screen time measurement. The proposed scales are not a final step, but a step forward to achieve a best evidence practice in screen time measurement and intervention in youth. Thus, future studies testing these scales and/or improving them with new scales are also needed. Specifically, considering that our study was limited to evaluate a few psychometric parameters of validity and reliability, other statistical parameters (e.g., confirmatory factor analysis of these constructs) are stimulated in order to understand the factors that explain screen time among young people.

In conclusion, the instrument included scales that showed acceptable validity and reliability. Future epidemiological and intervention studies can use the proposed scales in order to identify factors associated with screen time, as well as to identify mediators of the effect of interventions on reducing screen time in schoolchildren.

• Financial support
  There was no financial funding to perform this review. Individual grants for VC Barbosa Filho from the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES), AS Lopes and AS Bandeira from the Conselho Nacional de Ciência e Tecnologia (CNPQ). J Mota was supported by grants: FCT: SFRH/BSAB/142983/2018 and UID/DTP/00617/2019 as well as Programa de Bolsas Santander Universidades 2018. The funding agencies had no participation in the interpretation, analysis, writing and approval of this manuscript.

Acknowledgements

We thank the Municipal Education Department for technical support and authorization to the study execution. We thank all members of the school community (managers, teachers, parents and students) who were involved in this study. We thank the “Fortaleça sua Saúde” Working Group, which is detailed in a previous publication (http://www.biomedcentral.com/1471-2458/15/1212).

References

• ¹
  Sociedade Brasileira de Pediatria (SBP). Promoção da Atividade Física na Infância e Adolescência [Internet]. 2017 [acessado 2017 Set 21]. Disponível em: ://www.sbp.com.br/fileadmin/user_upload/ 19890d-MO-Promo_AtivFisica_na_Inf_e_Adoles. pdf
  » http://www.sbp.com.br/fileadmin/user_upload/ 19890d-MO-Promo_AtivFisica_na_Inf_e_Adoles. pdf
• ²
  Chastin SFM, De Craemer M, Lien N, Bernaards C, Buck C, Oppert J-M, Nazare J-A, Lakerveld J, O'Donoghue G, Holdsworth M, Owen N, Brug J, Cardon G, DEDIPAC consortium, expert working group and consensus panel. The SOS-framework (Systems of Sedentary behaviours): an international transdisciplinary consensus framework for the study of determinants, research priorities and policy on sedentary behaviour across the life course: a DEDIPAC-study. Int J Behav Nutr Phys Act 2016; 13:83.
• ³
  Stierlin AS, De Lepeleere S, Cardon G, Dargent-Molina P, Hoffmann B, Murphy MH, Kennedy A, O'Donoghue G, Chastin SFM, De Craemer M, DEDIPAC consortium. A systematic review of determinants of sedentary behaviour in youth: a DEDIPAC-study. Int J Behav Nutr Phys Act 2015; 12:133.
• ⁴
  Van Stralen MM, Yildirim M, Velde SJ, Brug J, van Mechelen W, Chinapaw MJ. What works in school-based energy balance behaviour interventions and what does not? A systematic review of mediating mechanisms. Int J Obes 2011; 35(10):1251-1265.
• ⁵
  Garcia JM, Agaronov A, Sirard JR, Whaley D, Rice DJ, Weltman A. psychosocial and friend influences on objective sedentary behavior and screen time: A mixed methods analysis. J Phys Act Health 2017; 14(3):213-221.
• ⁶
  Lubans DR, Hesketh K, Cliff DP, Barnett LM, Salmon J, Dollman J, Morgan PJ, Hills AP, Hardy LL. A systematic review of the validity and reliability of sedentary behaviour measures used with children and adolescents. Obes Rev 2011; 12(10):781-99.
• ⁷
  Salmon J, Tremblay MS, Marshall SJ, Hume C. Health Risks, Correlates, and Interventions to Reduce Sedentary Behavior in Young People. Am J Prev Med 2011; 41(2):197-206.
• ⁸
  Barbosa Filho VC, Lopes AS, Lima AB, Souza EA,Gubert FA, Silva KS, Vieira NFC, Trompieri Filho N,Araújo TS, Bruin PFC, Mota J, “Fortaleça sua Saúde” Working Group. Rationale and methods of a cluster-randomized controlled trial to promote active and healthy lifestyles among Brazilian students: the “Fortaleça sua Saúde” program. BMC Public Health 2015; 15:1212.
• ⁹
  Owen N, Salmon J, Koohsari MJ, Turrell G, Giles-Corti B. Sedentary behaviour and health: mapping environmental and social contexts to underpin chronic disease prevention. Br J Sports Med 2014; 48(3):174-177.
• ¹⁰
  Gebremariam MK, Totland TH, Andersen LF, Bergh IH, Bjelland M, Grydeland M, Ommundsen Y, Lien N. Stability and change in screen-based sedentary behaviours and associated factors among Norwegian children in the transition between childhood and adolescence. BMC Public Health 2012; 12:104.
• ¹¹
  Hardy LL, Baur LA, Garnett SP, Crawford D, Campbell KJ, Shrewsbury VA, Cowell CT, Salmon J. Family and home correlates of television viewing in 12-13 year old adolescents: the Nepean Study. Int J Behav Nutr Phys Act 2006; 3:24.
• ¹²
  Jago R, Baranowski T, Watson K, Bachman C, Baranowsi JC, Thompson D, Hernández AE, Venditti E, Blackshear T, Moe E. Development of new physical activity and sedentary behavior change self-efficacy questionnaires using item response modeling. Int J Behav Nutr Phys Act 2009; 6:20.
• ¹³
  Norman GJ, Vaughn AA, Roesch SC, Sallis JF, Calfas KJ, Patrick K. Development of decisional balance and self-efficacy measures for adolescent sedentary behaviors. Psychol Health 2004; 19(5):561-575.
• ¹⁴
  Salmon J, Hume C, Ball K, Booth M, Crawford D. Individual, social and home environment determinants of change in children's television viewing: The Switch-Play intervention. J Sci Med Sport 2006; 9(5):378-387.
• ¹⁵
  Tandon PS, Zhou C, Sallis JF, Cain KL, Frank LD, Saelens BE. Home environment relationships with children's physical activity, sedentary time, and screen time by socioeconomic status. Int J Behav Nutr Phys Act 2012; 9(1):88.
• ¹⁶
  Hair JF, Black WC, Babin BJ, Anderson RE. Multivariate Data Analysis [Internet]. Prentice Hall; 2009 [acessado 2017 Ago 25]. Disponível em: ://gen.lib.rus.ec/book/index.php?md5=FCDB08391BCEC7C365BB1D252D4BA523
  » http://gen.lib.rus.ec/book/index.php?md5=FCDB08391BCEC7C365BB1D252D4BA523
• ¹⁷
  Norman GJ. Psychosocial and Environmental Correlates of Adolescent Sedentary Behaviors. Pediatrics 2005; 116(4):908-916.
• ¹⁸
  Salmon J, Timperio A, Telford A, Carver A, Crawford D. Association of family environment with children's television viewing and with low level of physical activity. Obesity 2005; 13(11):1939-1951.
• ¹⁹
  Silva KS, Lopes AS, Hoefelmann LP, Cabral LGA, De Bem MFL, Barros MVG, Nahas MV. Health risk behaviors Project (COMPAC) in youth of the Santa Catarina State, Brazil: ethics and methodological aspects. Rev Bras Cineantropometria Amp Desempenho Hum 2013; 15(1):1-15.
• ²⁰
  Brown H, Hume C, ChinAPaw M. Validity and reliability of instruments to assess potential mediators of children's physical activity: A systematic review. J Sci Med Sport 2009; 12(5):539-548.
• ²¹
  Williams DM, Anderson ES, Winett RA. A review of the outcome expectancy construct in physical activity research. Ann Behav Med 2005; 29(1):70-79.
• ²²
  Cillero IH, Jago R, Sebire S. Individual and social predictors of screen-viewing among Spanish school children. Eur J Pediatr 2011; 170(1):93-102.
• ²³
  Gingold JA, Simon AE, Schoendorf KC. Excess screen time in US children: association with family rules and alternative activities. Clin Pediatr (Phila) 2014; 53(1):41-50.
• ²⁴
  LeBlanc AG, Katzmarzyk PT, Barreira TV, Broyles ST, Chaput J-P, Church TS, Fogelholm M, Harrington DM, Hu G, Kuriyan R, Kurpad A, Lambert EV, Maher C, Maia J, Matsudo V, Olds T, Onywera V, Sarmiento OL, Standage M, Tudor-Locke C, Zhao P, Tremblay MS, ISCOLE Research Group. Correlates of Total Sedentary Time and Screen Time in 9-11 Year-Old Children around the World: The International Study of Childhood Obesity, Lifestyle and the Environment. PloS One 2015; 10(6):e0129622.
• ²⁵
  Atkin AJ, Corder K, van Sluijs EM. Bedroom media, sedentary time and screen-time in children: a longitudinal analysis. Int J Behav Nutr Phys Act 2013; 10(1):137.
• ²⁶
  Buchanan L, Rooks-Peck CR, Finnie RKC, Wethington HR, Jacob V, Fulton JE, Johnson DB, Kahwati LC, Pratt CA, Ramirez G, Mercer SL, Glanz K, Community Preventive Services Task Force. Reducing Recreational Sedentary Screen Time. Am J Prev Med 2016; 50(3):402-415.
• ²⁷
  Scatena LM, Wysocki AD, Beraldo AA, Magnabosco GT, Brunello MEF, Ruffino AN, Nogueira JA, Silva Sobrinho RA, Brito EWG, Alexandre PBD, Monroe AA, Villa TCS. Validity and reliability of a health care service evaluation instrument for tuberculosis. Rev Saúde Pública 2015; 49:7
• ²⁸
  Singh AS, Vik FN, Chinapaw MJ, Uijtdewilligen L, Verloigne M, Fernández-Alvira JM, Stomfai S, Manios Y, Martens M, Brug J. Test-retest reliability and construct validity of the ENERGY-child questionnaire on energy balance-related behaviours and their potential determinants: the ENERGY-project. Int J Behav Nutr Phys Act 2011; 8(1):136.
• ²⁹
  Huang Y-J, Wong SH, Salmon J, Hui SS. Reliability and validity of psychosocial and environmental correlates measures of physical activity and screen-based behaviors among Chinese children in Hong Kong. Int J Behav Nutr Phys Act 2011; 8:16.
• ³⁰
  Hidding LM, Altenburg TM, Mokkink LB, Terwee CB, Chinapaw MJM. Systematic review of childhood sedentary behavior questionnaires: What do we know and what is next? Sports Med 2017; 47(4):677-699.

Edited by

Chief Editors: Romeu Gomes, Antônio Augusto Moura da Silva

Publication Dates

• Publication in this collection
  15 Mar 2021
• Date of issue
  Mar 2021

History

• Received
  22 Jan 2018
• Accepted
  04 May 2019
• Published
  06 May 2019