Thematic trends in fact-checking in Brazil’s COVID-19 infodemic: the first year in focusAbstract
Fact-checking agencies were pivotal in countering misinformation during the COVID-19 pandemic. We investigated the predominant thematic orientations within Brazilian fact-checking agencies during the pandemic’s initial year. Brazil’s case is relevant because it has a high traffic of false information and was the second country most affected by the pandemic. From Latam Chequea Coronavirus’s database, 838 assertions checked of fact-checking were analysed by automated content analysis, allowing the identification of the thematic incidence through descending hierarchical classification and factorial correspondence analysis. The emerged themes encompassed vaccine, virus origin, social isolation, economic relief, disease mortality, and treatment. Fact-checking agencies assigned varying levels of importance to specific themes, which may reinforce the idea of possible non-neutrality fact-checking journalism. Most categories displayed an irregular publication pattern, indicating they were not part of continuous communicative processes. This research addresses methodological, temporal, and sampling gaps by employing automated analysis, covering the pandemic’s initial year, and utilising a substantial dataset from the global South. The study contributes to the ongoing discourse on fact-checking agencies’ roles and limitations.
Key words
automated content analysis; COVID-19; fact-checking; fake news; infodemic; disinformation
INTRODUCTION
Fact-checking, a journalistic practice encompassing the scrutiny of information disseminated across both conventional and digital media, has undergone a transformative evolution (Cotter et al. 2022). It initially emerged in response to the post-truth media landscape, subsequently advancing towards the professionalisation of journalism, wherein the emphasis shifted towards anchoring reporting within factual veracity. This progression extended into a comprehensive framework characterised by diverse participants, practices, principles, routines, and interwoven tools. This study aligns with Krause et al. (2020) assertion regarding the imperative comprehension of fact-checking’s nuances.
Despite its producers designating fact-checking as a journalistic genre underpinned by factual accuracy and political impartiality, it is imperative to acknowledge that the arena of social communication has long contested this professed communicative even-handedness (Manovich 2001). Additionally, fact-checking agencies often operate under the assumption that individuals engage with information objectively and rationally, neglecting the potent impact of cognitive biases and the role of emotions, values, and worldviews within this cognitive process (Chou et al. 2021, Carrasco-Farré 2022).
Contrary to the purported impartiality associated with fact-checking, scholarly literature has illuminated proclivities, inclinations, and discernible selection biases inherent in the production of fact-checking across diverse contexts (Uscinski Butler 2013, Rodríguez-Pérez 2020). Rodríguez-Pérez (2020), in particular, underscores that the most pronounced jeopardy to the integrity of fact-checking resides in the realm of selection.
Given news agencies’ commercial motivations, the audience’s discernment is pivotal as a criterion for claim selection, albeit without uniform robustness. This selection effect can engender the categorisation of certain assertions as more or less veracious, a judgement potentially influenced by journalists’ ideological preferences due to the dearth of rigorous criteria. Uscinski Butler (2013) elucidate a range of practices that divulge discernible orientations within fact-checking, manifesting in decisions regarding the curation of facts subject to scrutiny, the conflation of factual fragments, causal attributions, prognostications, and implicit evaluative frameworks. Notably, these orientations find manifestation in the examination of political information.
Graves (2017) discerns that contextual contingencies imbue factual investigations with rationale, assumptions, values, and often latent methodologies, reflective of appropriateness within specific social cohorts, an observation broadly applicable to all fact-checking agencies. While the genesis of this practice lies within the political domain, its contemporary purview encompasses a spectrum of subjects (Allcott Gentzkow 2017, Krause et al. 2020), including the pandemic. In this scenario, the proliferation of false information, concomitant with distinct political affiliations among its producers and disseminators, has been underscored by research (Biancovilli et al. 2021, Gehrke Benetti 2021, Cotter et al. 2022, Calvillo et al. 2020). Falsehoods about the pandemic wield particularly pernicious consequences for the populace (Carrion-Alvarez Tijerina-Salina 2020), thereby casting fact-checking agencies as indispensable bulwarks against the proliferation of misinformation.
In the milieu of proliferating false information within the sphere of social communication, fact-checking agencies have undertaken the critical task of scrutinising pandemic-related information. As Cotter et al. (2022) elucidate, amid the pandemic, fact-checking evolved into a “go-to response for managing information pollution,” assuming the mantle of dispelling misleading narratives (Siwakoti et al. 2021). Scholars have demonstrated that the efficacy of fact-checking wanes when confronted with individuals’ entrenched beliefs, revealing its limitations in combating misinformation (Chou et al. 2021). Conversely, Schuetz et al. (2021) posit that heightened fact-checking during the pandemic correlated with favourable behaviours, such as increased adherence to protective measures like hand hygiene.
The confluence of escalating disinformation, the evolution of media platforms, and pandemic-centred exploratory inquiries accentuate the imperative of comprehending fact-checking agencies’ roles within society and media. This necessitates an evaluation of these agencies’ veracity and impartiality. It is within this context that the present study is situated. Accordingly, our study endeavours to assess whether Brazilian fact-checking agencies during the pandemic’s first year exhibited thematic orientations and if divergent themes were discernible across agencies. Importantly, selecting themes for evaluation inherently signals agency orientations, reflecting the ideological nuances underlying content, as Volosinov (1973) postulates.
This research addresses the dichotomy between fact-checking agencies’ professed neutrality and the literature’s acknowledged non-neutrality. Thus, this study contributes to the ongoing scholarly discourse on fact-checking agencies’ objectivity while also unveiling novel avenues for comprehending disinformation patterns and the interplay among fact-checking agencies, the media, and the public.
Ideology and neutrality of communication and fact-checking
As Volosinov (1973) underscored, communication is a socially generated process involving organised individuals who engage in interactions to share meanings. This perspective necessitates recognising that messages are shaped by specific individuals, each of whom cannot be regarded in an abstract, generic manner. These individuals engaged in communication are inherently moulded by their social class. Their existence and manner of engagement with the world are intrinsically linked to this class positioning. Consequently, any communication analysis cannot afford to overlook the fundamental influence of class determinations; they are not merely peripheral considerations.
Furthermore, it is crucial to acknowledge that communication is an integral outcome of social production and reproduction activities. Hence, communication is inherently rooted in its historical context. In this light, while communication remains subject to historical, productive, and technical developments, it also establishes multifaceted relationships with various layers of societal production and reproduction, as elucidated by Williams (1980). Fuchs (2020) additionally highlights that communication is an inherent part of economy, politics and culture that organises, produces and reproduces social relations.
The relationship between communication and social practice imposes essential characteristics on this process. In light of the division of society into social classes, these characteristics impose different ideological orientations on the objects of communication (Volosinov 1973). Volosinov (1973) explains that communication is executed through ideological signs, which reflect and refract reality according to the interests of social classes. For example, the diverse meanings of the word “work” (derived from the perspectives of a worker, an executive, and a shareholder or company owner) result from the varying ideological interpretations shaped by the social classes individuals belong to.
Consequently, no communicative process can be considered neutral or impartial; it will inevitably embody the ideological orientations of the social groups responsible for the communication. However, despite the structural and coherent nature of these considerations, academic and journalistic literature interpretations assert the possibility of neutrality, as seen in the case of fact-checking. Neglecting the non-neutrality of communication means not recognising that “ideology hides the true essence and state of the world behind false appearances and communicates these false appearances as truths and nature” (Fuchs 2020, p. 232).
For instance, Elizabeth (2014) from the American Press Institute explicitly states: “Fact-checkers investigate verifiable facts, and their work is free from partisanship, advocacy, and rhetoric.” Amazeen points out that “the viability of fact-checking depends on it being generally accepted as unbiased. The challenge, however, lies in its very essence: making judgments about the factual accuracy of claims without inviting allegations of bias” (Amazeen 2015, p. 4). The author further contends that the selection effects and other issues perpetuated by verification journalism result from inconsistent fact-checking practices that mirror the ideological orientations of the media organisations producing the messages (Amazeen 2015). Therefore, if all communication is fundamentally ideological, as explained by Volosinov (1973), it is improbable that any communicative practice will erase the inherent orientations present in communication. At most, a particular practice can conceal or obscure the ideologies embedded in the discourse produced.
In addressing the epistemology of fact-checking, Uscinski Butler (2013) point out that verification news generally extends beyond facts to encompass verifications of allegations that are not strictly factual. They argue that “fact-checking organisations often go beyond merely ‘checking facts’ in an attempt to ascertain the truth behind many statements” (Uscinski Butler 2013, p. 163). In their critique, Uscinski Butler (2013) elucidate five aspects that elucidate the characteristics of fact-checking, challenging the notion of impartiality: the selection effect, conflating several facts or dissecting whole, causal claims, predicting the future, and implicit selection criteria.
Regarding the selection effect, Uscinski Butler (2013) posit that producers of verification news make choices about which statements to verify, as it would be practically impossible to verify all claims made by politicians. The questions raised by the authors are: “Which actors will have their statements checked? Which particular statements will be checked?” These questions reveal an inherent bias in the very selection of the allegations to be verified. Given the interests in the news market, public interest becomes a determining criterion for allegation selection, although this criterion is not always rigorously applied. The selection effect can influence the interpretation of certain individuals’ claims, potentially categorising them as more or less untrue. Furthermore, these interpretations may be biased, as the ideological orientation of the journalist can sway the selection process:
“(...) the fact checker must rely on her gut feeling about what is interesting, and the cognitive function of ideology is precisely to highlight some facts as more interesting than others—because they represent aspects of reality that are made pertinent by the ideology”. (Uscinski Butler 2013, p. 166).
Uscinski Butler (2013) further emphasise that it is a common practice in fact-checking to amalgamate multiple statements into a single fact: “In essence, fact-checking is inherently political, as the journalist, when selecting evidence and breaking it down into discrete ‘facts,’ essentially mirrors the actions of the politician being scrutinised” (Uscinski Butler 2013, p. 167).
It is crucial to recognise that understanding a statement hinges on the context in which it is made; any assertion by an individual takes place within a specific communication context. Consequently, news reports occasionally underscore the significance of context. However, context analysis necessitates subjective judgments that extend beyond mere fact verification. In the examination of fact-checking, it is imperative to bear in mind that the
“Fact-checkers are participants in the political argument; the problem is that they present “the facts” as if they are so clear that they stand alone, and that their content, their context, and their classification as one or several facts are self-evident”. (Uscinski Butler 2013, p. 168).
Another notable characteristic of verification messages is the inclusion of causal claims, which endeavour to establish relationships between various facts. However, it is essential to note that these causal claims are not rooted in scientific methodologies. Scientific statements typically find it challenging to establish explanatory patterns based on dichotomies, a practice commonly employed in verification messages (Uscinski Butler 2013).
Additionally, a practice often scrutinised in journalism is verifying claims about the future. These future-oriented claims are frequently encountered in the political arena, particularly as political proposals are inherently oriented toward anticipated future outcomes. Given that the future is inherently unobservable in the present, verification practices compare asserted predictions with other predictions, often originating from experts in the respective field. However, this introduces two significant challenges: (i) it allows fact-checkers to select the theoretical-methodological framework of the expert, as even scientific predictions can vary based on the theoretical-methodological approach adopted; (ii) it effectively compares a claim with a possibility, as the expert’s prediction may also contain inaccuracies.
Uscinski Butler’s (2013) delineation of five critical points provides a foundational understanding for critically examining the practice of fact-checking. This work, in particular, concentrates on scrutinising the selection effect, which can manifest through the thematic concentration of facts verified by specific fact-checking agencies. It is worth noting that while the selection effect can encompass other facets of communication, such as the subject or the context/situation, this study’s scope remains focused on the selection effect alone.
Fact-checking agencies practices in the Global South and Brazil: empirical underpins
Siwakoti et al. (2021) conducted an investigative study concerning multiple fact-checking agencies, observing an escalation in posting frequency and engagement. Notably, the Brazilian agency Lupa experienced a pronounced surge in activity, leading to the identification of a correlation between this increase and the socio-political context of the respective countries. This prompts the necessity to delve into the mechanisms of fact-checking agencies to bolster their credibility, as suggested by Krause et al. (2020), and to implement additional measures against misinformation and disinformation, as highlighted by Chou et al. (2021) and Krause et al. (2020). In this context, comprehending the non-neutral nature of fact-checking agencies takes on significant importance. This understanding contributes to readers’ insight into these entities’ operational strategies and decisions, thus enhancing their work’s credibility.
Ceron et al. (2021) identified a gap in research concerning the global South, where only a limited number of studies have been conducted. Investigations have explored false information in the Latin American context, encompassing fact-checking agencies, policies, and government responses to counter false information (Ceron et al. 2021, Nieves-Cuervo et al. 2021, Pita 2021, Gehrke Benetti 2021, Biancovilli et al. 2021). Brazil, in particular, stands out due to its high mortality rate, limited governmental countermeasures against misinformation, and paradoxically, a substantial role in generating such content (Nieves-Cuervo et al. 2021, Pita 2021, Gehrke Benetti 2021).
Biancovilli et al. (2021) thoroughly examined 232 instances of false information propagated by the Brazilian fact-checking agency Lupa. Their qualitative classification encompassed content analysis, dissemination on social media platforms, and recurring themes. Subsequently, a quantitative analysis categorised the content based on themes, utilising categories developed collaboratively by two researchers after analysing a sample of 20 news stories. Emotional sentiments within texts were also analysed. An intercoder reliability of 81% indicated substantial agreement among researchers. The study revealed a prevalence of false information on Facebook (76%), followed by WhatsApp (10%). Misinformation predominantly comprised fabricated content (53%), false context (34%), and misleading content (13%). Fabricated content often pertained to health tips (92.9%), and virtual scams (88.9%) were predominantly fabricated.
Similarly, Gehrke Benetti (2021) examined 407 instances of false content identified by fact-checking agencies within the collaborative platform Latam Chequea Coronavirus from March to July 2020. Thematic categorisation encompassed China, contagion, cures, data, economics, politics, and other themes. Quantitative analysis revealed politics (25.55%) as the predominant theme, followed by cures (20.64%), data (19.66%), and contagion (18.43%). Qualitative analysis indicated alignment with President Jair Bolsonaro’s perspectives on the pandemic, often utilising the strategy of false context.
Ceron et al. (2021) examined Twitter accounts from two Brazilian fact-checking outlets, Aos Fatos and Lupa. The study scrutinised topics debunked by these agencies fortnightly throughout the pandemic, yielding a dataset of 5115 tweets from January to July 2020. The study adopted a mixed methods approach, integrating time series, network analysis, and close reading. The study highlighted the dynamics of information dissemination beyond national boundaries, illustrating the intersection of politics and health crises during this period. Souza Junior (2021) also analysed 11 posts from a Twitter thread commenting on local pandemic trajectories, deaths, and governmental responses. The aim was to qualitatively explore prevalent perspectives and associated frameworks, shedding light on power dynamics and resistance within pandemic narratives.
Certain methodological, temporal, and sample-related gaps persist despite these research endeavours. Notably, there is a need for more diverse methodological approaches, a broader temporal focus, and a more expansive sample size to comprehensively understand the choices made by fact-checking agencies. Furthermore, thematic trends within fact-checking agencies remain unexplored. This article is set apart from previous studies due to its in-depth investigation into multiple fact-checking agencies as well as a comprehensive database analysed using a mixed methods approach.
To address these gaps, our research aims to identify potential orientations or tendencies exhibited by fact-checking agencies in analysing information during the initial year of the COVID-19 pandemic. Using data from the first year of the pandemic is crucial as it captures the initial and often dynamic response to an unprecedented crisis. This period provides unique insights into the rapid spread of misinformation, evolving fact-checking practices, and the foundational moments that shape the trajectory of information verification in a crisis. By examining validated facts from four Brazilian fact-checking agencies affiliated with Latam Chequea Coronavirus—namely Lupa, Aos Fatos, Estadão Verifica, and AFP Checamos—we aim to contribute to the debate on fact-checking practices. The analysis of fact-checked themes provides a multifaceted perspective on information dynamics. Additionally, this approach assists in tracking evolving trends in misinformation, contributing to a more comprehensive awareness of the dynamic landscape of false information and its societal impact.
An analytical perspective on this issue aids in enhancing comprehension of fact-checking methodologies. It empirically enhances the discourse surrounding the inherent non-neutrality within this journalistic domain. Observing patterns in pandemic coverage potentially reveals the sway of editorial influences on fact-checking journalism, thereby underscoring the intrinsic interplay between journalistic practices and vested interests, a facet acknowledged by relevant agencies. Furthermore, an enhanced understanding of fact-checking can serve as a valuable tool in the fight against disinformation. It is essential to clarify that the revelation of selection biases and the absence of neutrality is not intended to discredit fact-checking efforts but to uncover this journalistic genre’s core attributes. This endeavour enriches ongoing discussions by its foundation upon a meticulously assembled and diverse database, which encapsulates data from the early phases of the pandemic.
MATERIALS AND METHODS
In this section, we describe the background of our data—the Latam Chequea Coronavirus database and Brazil’s fact-checking agencies—as well as the stages of capturing, organising, triaging, and analysing the data, which are applied to support automated content analysis. Figure 1 presents an overview of this process, detailed in the following sub-sections.
Decoding the data context: an examination of fact-checking agencies
Given the seriousness of the pandemic in Brazil and the vigorous action of national fack-checking agencies, this study aims to determine the predominant thematic orientations in fact-checks conducted in Brazil during the first year of the pandemic and assess whether significant variations in themes were observed across different fact-checks during the first year of the pandemic. The analysis relied on the data made available by the Latam Chequea Coronavirus, a collaborative network between 17 Iberic-American countries formed by 34 fact-checking organisations (Chequeado 2014) and which relies on an international database elaborated by the International Fact-Checking Network (IFCN).
The network provides access to data on the facts checked by four Brazilian agencies: Lupa, Aos Fatos, Estadão Verifica and AFP Checamos, which subsidised this work. It is noteworthy that other academic investigations also used the same database (e.g., López-Pujalte Nuño-Moral 2020, Gehrke Benetti 2021) for other purposes, which shows its relevance and the possibility of consolidating a robust understanding of the dissemination of fake news about the COVID-19 pandemic and fact-checking in Iberic-American countries.
To understand the characteristics of fact-checking about the pandemic in Brazil, we deemed it essential to understand the practices of the agencies that check the facts that constitute our corpus since their principles and editorial orientations determine the false information to be checked (Table I). Lupa and Aos Fatos are Brazil’s most well-known fact-checking organisations (Ceron et al. 2021). While Estadão Verifica is a section of one of the leading newspapers in Brazil, and AFP Checamos is a section of one of the main outlets of the world. It is worth highlighting that every agency indicates in one way or another their appreciation for the impartiality and/or neutrality of their work, a condition that has been questioned for years by the specialised fact-checking literature (Uscinski Butler 2013).
The main characteristics of the four fact-checking agencies whose assertions checked compose the corpus of analysis.
Bearing in mind the aims of this work, in addition to these general characteristics, it is also essential to point out that the agencies agree with the definition that fact-checking compares information present in statements to the facts they refer to check their veracity (Uscinski Butler 2013, Amazeen 2015, Rodríguez-Pérez 2020). On this matter, Amazeen (2015, p. 3) argues that “as a form of accountability journalism, dedicated fact-checking is committed to publicising errors or falsehoods regardless of the source”. Moreover, the opportunity to produce fact-checking requires accepting their impartiality, albeit contradictory, since the judgement of a claim as true or false indicates an attitude toward the fact-checked (Amazeen 2015).
Corpus organisation and triage
We selected from the Latam Chequea Coronavirus database exclusively the information published by Brazilian fact-checking agencies in the first year of the pandemic, comprehending the interval between the origin of the disease and the beginning of December 2020 (available on the international database in March 2021). The information released by Latam Chequea Coronavirus includes fact-checking publications by the agencies after identifying false information disseminated in verbal, image, or audio-visual forms. Our corpus was constituted by a total of 838 facts checked by checking news published as texts in Brazilian Portuguese on the websites of four agencies: Lupa, Aos Fatos, AFP and Estadão Verifica.
The corpus (Table II, see Methods last section) was parametrised according to the specifications of IRaMuTeQ (e.g., removing characters and standardising hyphenated words) so that it could be legible and processed by the software for the automated content analysis (Cao et al. 2018) (more details in the next section). The metadata (agency’s name and the publication month) was used as a complementary analysis after the first stage of analysis (DHC and FCA) due to the supposition that the themes addressed by the fact-checking news could be temporally concentrated due to new facts and their presence in the media.
This research only covers fake messages in textual format, and the automated content analysis used the software’s most recent Brazilian Portuguese version.
Automated content analysis as a method
Automated content analysis techniques were employed to systematically describe textual information (traditionally qualitative data) published by the four fact-checking agencies. Hence, we relied empirically on technological tools to help in data capture, organisation, triage, analysis, and interpretation (Wiedemann 2016, Grimmer Stewart 2013, Lahlou 2012). Automated content analysis techniques are particularly useful for large textual documents or corpus and for making inferences systematically with reduced subjectivity, which is why we adopted this approach. There are several techniques (e.g., machine learning, text mining, sentiment classification) supported by software and different statistic packages; they have been used as crucial tools to understand multiple social-mediatic texts (e.g., de Morais Santos et al. 2022, Ceron et al. 2021, Smallman 2016, Stockwell et al. 2009).
The present analyses were assisted by IRaMuTeQ, an open-source software that conducts statistical analysis on textual corpora using software R (Lahlou 2012). We chose this method because (i) IRaMuTeQ makes it possible to obtain a series of indicators and to conduct textual statistical analysis that enables the identification of tendencies and particularities on the database, and (ii) one of the program outputs is the organisation of the database according to similarities, which helps us to extract meanings from texts in the same category, facilitating the interpretative stage of content analysis. This way, IRaMuTeQ’s organisation, visualisation and analysis tools were used to process the data.
In particular, we conducted two analyses that contributed to the understanding of the structure and the organisation of discourses present in the corpus: the algorithm of Reinert (1990) to create clusters via Descending Hierarchical Classification (DHC) followed by the factorial correspondence analysis (FCA), as detailed later on. Hence, our unit of analysis was the word in context, and we extracted from them indicators such as co-occurrences, hierarchies, and frequencies, which were considered by the algorithm for the classification and, afterwards, for the FCA. Through the software tools, even having the word as the central unit of analysis, we can quickly evaluate the word in context in the corpus. This facilitates the analysis steps for extracting meanings and synthesis.
Analysis of the content checked by fact-checks to identify thematic orientations
The first stage of the analysis employed the Reinert (1990) method, which generated a dendrogram for the Descending Hierarchical Classification (DHC). We used the Reinert algorithm because it reduces subjectivity in interpreting the corpus and increases the chances of replicating analyses (Cervi 2018). The analysis relied on lexical (not syntactical) analysis and considered the presence and the form of words in a text. This means we are emphasising at this stage the analysis and understanding of the words themselves, regardless of their grammatical function or structure within a sentence. In the present analysis, we sought a clear definition for clusters with unique vocabulary to identify different discourses regarding the pandemic (Kronberger Wagner 2002). We describe the details of this analysis below.
The software organised texts into fragments that served as the basis for the matrices of presence and absence, utilising lemmatised vocabulary—words that have undergone the process of lemmatisation. This process reduces inflected or derived forms of words to their base forms, allowing for the accurate grouping of semantically similar terms under a single lemma, thereby enhancing text analysis (Manning et al. 2008). This approach facilitates more effective information retrieval and linguistic analysis (Terämä et al. 2016). Concerning this particular corpus, the text fragments were news headlines (with the assertions checked), which were used in totality for this analysis stage, given their length and content.
The matrixes were crossed in repeated tests of the X 2 type to obtain a definitive classification by cluster similarities. The dendrogram generated at the end of this process enabled the identification (now conducted by the researchers) of more and less similar classes and their labels, which were selected according to the most frequent words in each cluster and Pearson’s X 2 statistics. By comparing X 2 – which indicated whether the presence of the vocabulary was statistically different among clusters – we identified the theme of each class. Therefore, it was possible to create thematic categories (clusters) without prior criteria and conduct a posteriori analysis based on the corpus vocabulary (a bottom-up approach) where decision criteria are based on statistical metrics.
Regarding parameters, our corpus met the analysis criteria since the texts focused on one subject (the pandemic) and 82.58% of the classified segments – the base value is 75% (Camargo et al. 2018) (Table II). Based on DHC results, we labelled the classes using the cluster’s lexical content as a source, especially the X 2 values that indicate the statistical difference in the presence of specific words in each cluster.
The second analysis stage was the FCA, conducted with the results of Reinert’s Algorithm. These processes enabled the identification of associations of dependence and independence for the classes represented in the cartesian plane. FCA contributes to understanding the relationships in the classes’ lexical content due to the proximity between words and classes. For all analyses, we indicated verbs and nouns as main categories and the remaining word classes (e.g., adverbs, adjectives) as supplementary to facilitate the interpretation of meanings conveyed by the graphs. The graph output of DHC and FCA generated by the software were lightly edited to provide better visual exposure to the results.
Finally, the corpus data was explored in the third analysis stage through complementary descriptive steps. In the initial analysis, the focus was on theme frequencies derived from the DHC (it’s important to note that this analysis followed the statistical test). Organising the data analysis this way allowed for the creation of frequency tables and the exploration of thematic trends and temporal relationships between agencies only after verifying the thematic differences that emerged.
RESULTS
This hierarchical procedure started by considering all data as belonging to a single cluster and then dividing them into subgroups successively until more homogeneous clusters were formed. The result was a tree of clusters representing different levels of clustering. Figure 2 gives the first stage of the corpus vocabulary analysis, which is the result of Reinert’s method. It also provides a vision of the internal organisation of each cluster based on the list of more frequent forms (lemmatised words) in each of them and the values of X 2 (the form’s force of association with the class). By identifying the clusters and their internal content, it was possible to derive the thematic categories of analysis.
Dendrogram of the corpus resulting from DHC. The classes were named based on the most frequent forms (lemmatised words) and indicated in each cluster in descending order of X 2.
We identified six stable classes in the corpus, each with segments of texts with a similar internal vocabulary. It was observed that class 3: Social isolation (which has 24.4% of the text segments classified within the corpus), is divided from the others on a first level. On the second level, class 4: Economic relief (13.4%) and class 5: Mortality of the disease (21.1%), form an external cluster similar to one another. Finally, in the internal cluster are class 1: Vaccine (13.4%) and class 2: Origin of the virus (12.9%), which are set apart from class 6: Treatment (14.7%).
The form “vaccine” (X 2 = 370.3) is especially significant in the lexical determination of class 1: Vaccine. In this class, the subjects revolve around the name, the origin and the developers of the vaccine, as well as around the lies about the side effects, as identified in the passages (translated freely from Brazilian Portuguese):
The coronavac, a vaccine being developed in China and to be tested in Brazil was tested only in monkeys; A vaccine made by Roche Medical Company cures COVID-19 in 3 hours; The coronavac vaccine uses cells of aborted foetuses; The side effects of coronavac are more lethal than COVID 19; The vaccines against COVID-19 cause irreversible DNA mutations.
As displayed in Figure 2, classes 1 and 2 are grouped, indicating similarity in the lexical content. In class 2: Origin of the virus, we observe that the form “coronavirus” (X 2 = 169.7) is the most relevant, followed by various forms with X 2 close to one another and regarding false facts related to the origin of the virus, as seen in the messages: China planned coronavirus; China has benefited financially from the new coronavirus; A text shared on WhatsApp says that the Chinese created the coronavirus and the main cities in China have not suffered from the pandemic.
In class 6: Treatment, chloroquine is the form that stands out (X 2 = 104.9) in association with other forms that refer to false information about the cure and the treatment of the disease, as seen in the passages:
The FDA, Food and Drug Administration has approved the use of chloroquine in all COVID-19 patients in the United States; Global research shows that hydroxychloroquine is the most effective drug against COVID-19; A video statement of a doctor who claims that a random double-blind study in the Medical Centre Henry Ford proves that hydroxychloroquine is effective against COVID-19, it also says that changes in the protocol of hydroxychloroquine were made based on research removed from The Lancet; Self hemotherapy cures COVID-19.
The forms “death” (X 2= 286.3) and “number” (X 2 = 112.4) are strongly associated with class 5: Mortality of the disease, but the first is the most relevant in the lexical determination of the class. Both forms are followed by others that refer to monitoring deaths during the pandemic: register, data, registration, decrease, decline and death. The analysis of the elementary context units of this class indicates that the facts checked refer to the possible manipulation of numbers on the pandemic, particularly the number of deaths, as shown in the examples: A table comparing the deaths caused by COVID-19 with other causes shows that the new coronavirus is harmless; Deaths by a hit and run are counted as COVID-19 in SP, Brazil; A decree by SP governor increases the number of deaths by COVID-19 artificially; New health minister in Brazil Nelson Teich is investigating the statistics of COVID-19 due to alleged fraud, therefore the number of cases and deaths has decreased in the country during the week of April 24.
In class 4: Economic relief, we observe a strong association between the forms “government” (X 2 = 92.3) and “assistance” (X 2 = 65.3). Analysis of these and other forms (family, emergency, federal, urgent, distribute) in the context units of the class indicates it contemplates facts checked regarding the availability of economic resources to the population during the pandemic:
The governor of RJ will pay inmates a special monthly subsidy of $836 reais because of COVID-19 while the emergency relief for poor people that are not convicted criminals is just $600 reais; Cuba, Argentina and Venezuela, unlike Brazil, did not pay economic relief to the poorest population during the pandemic; 170 millions of people have requested the economic relief provided by the Brazilian government during the COVID-19 pandemic.
Finally, in class 3: Social isolation, the most external of the classes, we identified that the form “isolation” (X 2 = 147.5) has a prominent association. The class is different from the others as it conveys a series of forms referring to verbal-visual information of messages, which is expressed in the frequency and relevance of forms “show”, “video”, “photo”, and “caption”, as seen in the context units of the class: A video shows animals in a market in Wuhan; A photo of several people fallen in the middle of the street allegedly in China; A video shows the alleged owner of a hotel in New York jumping from a window after firing his employees. The hotel closed due to social isolation policies; A video shows police officers beating a woman. According to the subtitles, they are military policemen in SP, and they beat her because she disrespected social isolation. Therefore, the characteristic forms of this class refer to social isolation and the structural elements of misinformation and disinformation, which are essential to understanding the meanings intended by the producers.
Class distinctions and overlaps in the cluster
Despite the possibility of forming a first vision of the differences and similarities between classes in the DHC cluster, correspondence analysis provides a clearer view of oppositions and overlapping for the lexical content of classes. This way, in Figure 3, the proximity between keywords and clusters represents stronger associations. It indicated the isolation of class 3 (social isolation) and the similarities between classes 1, 2, and 6 (the more central region of the graph), opposed to classes 4 and 5 (lower to the right).
Correspondence analysis of DHC clusters considering the news’ original language, Brazilian Portuguese. The size of words reflects the importance of that word in the class (X 2 indicates the association of the word in the class). The largest terms are the first ones in the list for each DHC.
By exploring the data composing each class using the software, it was also possible to display the importance of the sources and months in each. Our analysis in this step indicated that sources have different degrees of importance to specific facts (Table III). To explore this result, we also assessed the absolute frequency with which a source checked a given class of facts by category (Table IV). Beyond the classes, Table IV indicates the composition relevant to understanding the facts that the agencies checked. However, it is important to note that this interpretation is already being made based on the statistical tests that separated the clusters.
Absolute frequency (n) and relative frequency (RF) of facts checked by source in the composition of each class.
While Table III shows that the class: 1 Vaccine and class: 5 Mortality and disease were the only ones that did not have a relevant determination from the sources, Table IV shows that the sources Lupa, Estadão and Aos Fatos verified a similar number of facts for each of these categories (providing evidence as to why they were not determinant in these classes). It is important to stress that those agencies broadly equivalently checked facts in absolute numbers but were unequal in comparing the ratio and the total of facts checked. Proportionally, the facts checked about the disease’s mortality comprised 17.43% of the fact-checking done by Lupa, 23.9% by Estadão, and 23.5% by Aos Fatos.
Analogously, upon a comparison of Tables III and IV, a discernible pattern emerges that sheds light on the substantial influence carried by the assertions put forth by agency Lupa in the determination of class 3: Social isolation. Noteworthy is that Lupa’s scrutinisation encompassed 32.0% of the facts under examination. This value contrasts the percentages allocated by Estadão and Aos Fatos, which reported 22.4% and 16.1%, respectively, of the total facts checked about Social isolation. This data qualitatively establishes a hierarchical distinction regarding the prevalence of fact-checking emphasis accorded to Social isolation across these agencies.
Similarly, an analogous trend is observed within class 4: Economic relief. Lupa exhibited a noticeably higher frequency in its fact-checking endeavours concerning this particular theme than the other agencies. However, it is worth noting that the magnitude of fact-checking activity targeting Economic relief remained below that of the activity focused on Social isolation. Furthermore, an aspect revealed by the analysis is the relatively diminished attention afforded by Aos Fatos towards fact-checking regarding Economic relief, indicating a nuanced discrepancy in their fact-checking priorities.
The facts checked by the agency Aos Fatos, in turn, were relevant to determine the categories of Origin of the virus and Treatment. Each one represented 17.97% of the facts checked by the agency. However, it must be observed that, despite the agency’s importance in determining these categories, Aos Fatos checked more facts about the disease’s Mortality.
Temporal influences on cluster dynamics and the flow of information
Alongside the significance of the sources in class determination, the findings highlight the influence of temporal factors on cluster formation, as depicted in Table III. Across most categories, a consistent pattern emerged, characterised by alternating months. This temporal arrangement suggests that the themes associated with specific clusters were not sustained through continuous communicative processes but manifested intermittently.
For instance, the class related to Vaccines provides a notable illustration of this trend. It predominantly encompasses facts validated in July and October, coinciding with the signing of cooperation agreements and vaccine procurement by the Brazilian government, respectively. Noteworthy is the Senate’s approval of economic relief for low-income informal workers in Brazil in March 2020. Furthermore, on September 16th, the federal government enacted a provisional measure to regulate economic relief measures.
Similarly, the months of May and August played pivotal roles in shaping the Mortality class. May corresponded to a significant surge in fatalities during the initial pandemic wave in Brazil, while August marked the commencement of this wave’s decline. The alignment between critical temporal junctures and the composition of distinct clusters suggests a more intricate association rather than mere coincidence. This points to a possible connection wherein the propagation of misinformation, as indicated by fact-checking activities, closely mirrored the populace’s information demands. This interplay between information demand and the prevalence of accurate or inaccurate content holds significance within information dissemination dynamics.
To visualise the analysis of the results on the temporality of the messages, we produced a graphic indicating the distribution of each agency’s fact-checking volume (Figure 4). These results indicate similar general patterns in the volume of fact-checking production in agencies (except AFP) despite temporal variation between sources.
Hence, examining temporal aspects associated with the scrutinised themes appears to correlate with the volume of false information circulating within the media landscape. This observation suggests a plausible connection between the prevalence of specific themes and the temporal distribution of fact-checking efforts.
Moreover, the intermittent nature of the validated facts may indicate a deliberate strategy employed in disinformation campaigns. While this conjecture warrants future empirical validation by other studies, the recurrence of intermittent fact-checking could potentially reinforce previously disseminated false content. In this context, the similar trends observed in the production of checked facts across different agencies could result from these underlying patterns.
DISCUSSION
In this study, our objective was to analyse whether fact-checking agencies in Brazil during the initial year of the pandemic exhibited thematic orientations and if significant variations existed in themes among different agencies. The extensive dataset encompassing the first year of the pandemic, analysed using automated content analysis techniques, revealed distinct thematic fluctuations within the fact-checking content. Subsequently, we examine these thematic categories within the pandemic context while drawing parallels to previous research findings.
The scholarly literature has previously underscored the significance of verbal-visual composition in conferring legitimacy to false information (Cao et al. 2020, Lima et al. 2021), as is evident in category 3: Social Isolation. Curiously, despite its apparent commonality in false information, the isolated categorisation of this theme struck an atypical chord (Figure 3). This might be attributed to Lupa’s categorisation methodology, which prominently considers the means through which false information was disseminated, thereby revealing a distinct agency stance by accentuating medium and information dissemination. This potentially underscores the role of verbal-visual composition primarily seen in social media platforms as primary sources of false information (Biancovilli et al. 2021, Gehrke Benetti 2021, Cotter et al. 2022, Turk 2018, Nieves-Cuervo et al. 2021).
Furthermore, when examining class 2: Origin of the Virus, a xenophobic undertone emerges, corroborated by a holistic examination of the verified claims. As asserted by Chou et al. (2021), xenophobia can render individuals more susceptible to specific forms of misinformation and future correction endeavours. Notably, these false claims often revolve around perceptions of China gaining advantages for its population, effectively interlinking class 2 with class 1: Vaccine. This proximity stems from the widespread circulation of misinformation concerning the ‘coronavac,’ the Chinese-developed vaccine in collaboration with Brazil’s Butantan Institute. China serves as the nexus bridging both categories: positively in vaccine discussions and derogatorily in discussions about the virus’s origin. This intricate relationship underscores the relevance of this multifaceted issue within the pandemic discourse, an assertion corroborated by authoritative bodies like the WHO.
Addressing class 6, which is composed of the same sets as classes 1 and 2, it is noteworthy that the majority of fact-checks concerning chloroquine pertain to institutions, subjects, or practices that lend an air of legitimacy to false information due to their association with the scientific domain. Often, these fact-checks entail recommendations from pharmaceutical companies, medical professionals, or research findings, echoing similar observations from previous research. This strategic approach aligns with most studies concerning false information and fact-checking agencies. In classes 2 and 6, the proximity to scientific knowledge plays a pivotal role in legitimising social practices, contrasting to the fate of class 1: Vaccine, where scientific and technological facets assume a negative stance.
It is also important to correlate these results with the work of Massarani et al. (2023), which shows that 39% of the messages received by Lupa in Brazil quote supposedly scientific sources and with the work of Andrade et al. (2023), which shows that out of 473 fact-checking messages analysed, 32% make inappropriate use of scientific knowledge in an attempt to legitimise false claims.
The convergence of classes 4 and 5 stems from the paramount role of the government in both categories. While class 5 explicitly addresses the government’s role in economic relief, class 4 implicitly reflects governmental influence through ministerial statements. Moreover, terms like “register,” “number,” and “deaths” connote the government’s responsibility to monitor the spread of disease. Scholars have already explored the political dimensions of the pandemic and misinformation (Allcott Gentzkow 2017, Krause et al. 2020, Calvillo et al. 2020), thus highlighting the acknowledgement of this dimension by the fact-checking agencies.
A critical observation pertains to aligning our findings with the conclusions of Barcelos et al. (2021), which emphasised prevalent themes such as politics, prevention, epidemiology, and statistics in Brazilian false messages. Although divergent in methodology and sources, the thematic commonalities underscore a tendency for false information to be disseminated and subsequently verified by fact-checking agencies during the pandemic. This concordance indicates thematic inclinations in misinformation and the corresponding fact-checking coverage. This alignment with other studies by Pita (2021), Nieves-Cuervo et al. (2021), and Siwakoti et al. (2021) accentuates the role of political orientations in counteracting false narratives during the pandemic. Our findings are also particularly relevant since evidence expansion is pertinent, especially in topics and databases that have not yet received much attention.
We ascertain that Lupa and Aos Fatos displayed pronounced thematic selections by examining thematic orientations and potential agency variations. These two agencies constitute the only legitimate fact-checking entities in the dataset. This alignment is consistent with Ceron et al. (2021) assertion that Lupa and Aos Fatos are the most established fact-checking entities. While Lupa exerts a decisive influence over the categories of Social Isolation and Economic Relief, Aos Fatos occupies a pivotal role in shaping the themes of the Origin of the Virus and Treatment. This discernment in agency-driven thematic emphasis sheds light on the possible deliberate orientation adopted by these agencies in assessing misinformation.
Determining the central aspect that influences the guidelines of verification agencies requires further research, and it was not our purpose to address this issue. However, Volosinov (1973) makes it clear that all communication is ideological, while its scope is determined by the social groups communicating. Although all the verification agencies analysed are part of the establishment, they are governed by different ideological nuances, which we consider to be the key reason for our results’ different verification approaches. In any case, we emphasise that investigating how different ideologies can guide verification news seems to be a relevant follow-up to this research, as the results will enhance our understanding of this journalistic genre.
Furthermore, our empirical analysis indicates that 41% of the false messages within the dataset were cross-verified by multiple fact-checking agencies, indicating a context that transcends simple collaboration. While potential collaborative dynamics between agencies could underpin such divergence, current evidence does not confirm this proposition, warranting future investigations. This observation underscores each agency’s distinct orientations, echoing Uscinski Butler (2013) contention. These orientations extend beyond mere neglect of other topics and instead reveal the selection criteria guiding the agencies, with the result suggesting the prevalence of preferred thematic subjects amidst the pandemic. This calls attention to the themes not addressed due to this partial thematic overlap, prompting us to explore disregarded subjects as those highlighted in our study attain prominence.
The discussion surrounding the findings of this study could be significantly expanded by delving into the implications of thematic variations among fact-checking agencies and their potential effects on public perception and trust in verified information. For instance, the differing thematic emphases —such as vaccine hesitancy— highlight how specific narratives can shape audience attitudes towards health communications. Recent research indicates that framing health information can significantly influence public trust and behaviour (Roozenbeek et al. 2020). Furthermore, as Chou et al. (2021) noted, variations in the thematic focus of fact-checking agencies can mitigate or exacerbate misinformation, depending on how narratives are presented and contextualised. This suggests that a more nuanced understanding of how different agencies communicate their findings could provide critical insights into fostering public confidence in verified information, particularly during health crises where misinformation can have dire consequences. Thus, exploring the intersection of thematic variation, public perception, and the trustworthiness of fact-checked information represents an important avenue for future research (Uscinski Butler 2013).
Exploring the concept of neutrality within fact-checking practices reveals a complex and often debated terrain within communication theory. Theories such as agenda-setting and framing demonstrate that media actors, including fact-checking agencies, significantly shape narratives through their choices in information presentation (McCombs Shaw 1972). This dynamic is particularly relevant during the COVID-19 pandemic, where fact-checkers operate in a landscape filled with misinformation and varying public perceptions. Research has shown that framing information by fact-checking organisations can profoundly influence public trust and behaviour, revealing the inherent tension between striving for objectivity and the subjective nature of information dissemination (Bennett Iyengar 2008, Lewandowsky et al. 2017). By integrating these theoretical perspectives, a deeper understanding of how neutrality is operationalised in fact-checking practices during crises can be achieved.
The intention here is not to judge the orientations unearthed in this study, as these orientations can be rationalised through diverse aspects linked to fact-selection criteria. These orientations likely stem from audience dynamics, ideological considerations, and the agencies’ policy stances, but their comprehensive comprehension mandates further inquiry (by using, for instance, other databases or methodological approaches).
Consequently, our endeavour involves merging the present study’s findings with existing research on COVID-19 coverage, bridging methodological gaps through automated content analysis, temporality by focusing on the pandemic’s first year, and inclusivity by considering an expansive database from the global South. This comprehensive approach suggests the agencies’ partiality, enriching the discourse on fact-checking agencies’ roles and limitations during crises.
These findings fill a distinct gap within the domain by offering a mixed methods analysis yielding a systematic classification of false claims verified by Brazilian fact-checking agencies (Gehrke Benetti 2021, Biancovilli et al. 2021). Additionally, despite methodological differences, our study reveals thematic overlaps with prior research, thus enriching the comprehension of fact-checking agencies’ roles and choices, as delineated by Krause et al. (2020) and Uscinski Butler (2013). Our research also aligns with previous studies on these agencies, such as López-Pujalte Nuño-Moral (2020) and Gehrke Benetti (2021). This underscores the necessity for deeper fact-checking investigations to comprehend their nuances within communication crises, such as the COVID-19 pandemic.
Considering its informative and exploratory nature, this study harbours limitations that future research can address. Qualitative analysis delving into the discursive characteristics of the whole checked content within the socio-economic and political contexts of the pandemic stands as a prospective avenue. Such qualitative inquiry would afford greater insights into disinformation patterns and concurrently unveil patterns of reliable information, cultivating a more transparent fact-checking culture.
Additionally, in the context of future exploration to enrich the debate about (non)neutrality in the (fake) news selection process, it would be pertinent to explore the active versus passive roles of fact-checking agencies in news selection and the implications of these approaches on their practices. Investigating in depth how different agencies fact-check the same piece of news could provide valuable insights into the nuances of fact-checking processes. Such an inquiry could contribute to a more comprehensive understanding of misinformation’s evolving landscape and the strategies agencies employ to combat it.
From a theoretical perspective, our empirical investigation into fact-checking agencies stands to contribute significantly. Firstly, the concrete data and evidence generated herein (when combined with others) can lay the basis for developing theoretical frameworks elucidating these agencies’ roles and impacts in the information ecosystem. These models can enrich our comprehension of how fact-checking agencies contribute to fostering accuracy and reliability within the news domain. Secondly, empirical studies such as ours provide a discerning analysis of the methodologies and approaches adopted by these agencies, triggering reflective consideration of their practices. This comprehensive understanding affords deeper insights into the challenges and limitations encountered by these entities, thereby catalysing theoretical and pragmatic enhancements within the field. Lastly, the empirical examination of fact-checking agencies catalyses probing pertinent theoretical inquiries about the ethical and societal ramifications of these practices, encompassing their implications for free expression, the media’s role, and public trust in journalism. These theoretical reflections hold relevance for the broader discourse on information quality and democracy’s vitality, underpinning future research endeavours and interventions involving fact-checking agencies.
ACKNOWLEDGMENTS
This work did not receive funding from funding agencies.
REFERENCES
- ALLCOTT H GENTZKOW M. 2017. Social Media and Fake News in the 2016 Election. J Econ Perspect 31: 211-236.
- AMAZEEN MA. 2015. Revisiting the epistemology of fact-checking. Crit Rev 27: 1-22.
- ANDRADE HP, FERREIRA CS LIMA GS. 2023. Tipologia de mensagens falsas sobre a COVID-19 no Brasil. RECIIS 17: 550-572.
- BARCELOS TDN, MUNIZ LN, DANTAS DM, COTRIM JUNIOR DF, CAVALCANTE JR FAERSTEIN E. 2021. Análise de fake news veiculadas durante a pandemia de COVID-19 no Brasil. Rev Panam Salud Publica 45: e65.
- BENNETT WL IYENGAR S. 2008. A new era of minimal effects? The changing foundations of political communication. J Commun 58: 707-731.
- BIANCOVILLI P, MAKSZIN L JURBERG C. 2021. Misinformation on social networks during the novel coronavirus pandemic: a quali-quantitative case study of Brazil. BMC Public Health 21: 1-10.
- CALVILLO DP, ROSS BJ, GARCIA RJ, SMELTER TJ RUTCHICK AM. 2020. Political ideology predicts perceptions of the threat of COVID-19 (and susceptibility to fake news about it). Soc Psychol Pers Sci 11: 1119-1128.
-
CAMARGO BV, JUSTO AM FORTE T. 2018. IRaMuTeQ Tutorial (R INTERFACE for multidimensional analysis of texts and questionnaires). Available at: http://iramuteq.org/documentation/fichiers/IRaMuTeQ%20Tutorial%20translated%20to%20English_17.03.2016.pdf Accessed 20 Sept 2022.
» http://iramuteq.org/documentation/fichiers/IRaMuTeQ%20Tutorial%20translated%20to%20English_17.03.2016.pdf - CAO J, QI P, SHENG Q, YANG T, GUO J LI J. 2020. Exploring the Role of Visual Content in Fake News Detection. In: Shu K, Wang S, Lee D Liu H (Eds), Disinformation, Misinformation, and Fake News in Social Media. Lecture Notes in Social Networks. Springer, Cham.
- CARRASCO-FARRÉ C. 2022. The fingerprints of misinformation: how deceptive content differs from reliable sources in terms of cognitive effort and appeal to emotions. Humanit Soc Sci Commun 9: 162.
- CARRION-ALVAREZ D TIJERINA-SALINA PX. 2020. Fake news in COVID-19: A perspective. Health Promot Perspect 10: 290.
- CERON W, GRUSZYNSKI SANSEVERINO G, DE-LIMA-SANTOS MF QUILES MG. 2021. COVID-19 fake news diffusion across Latin America. Soc Netw Anal Min 11: 1-20.
- CERVI EU. 2018. Análise de conteúdo automatizada para conversações em redes sociais online: uma proposta metodológica. Anpocs 48: 26.
-
CHEQUEADO. 2014. Latam Chequea Coronavirus. Available at: https://chequeado.com/latamcoronavirus/ Accessed 30 Sept 2022.
» https://chequeado.com/latamcoronavirus/ - CHOU WYS, GAYSYNSKY A VANDERPOOL RC. 2021. The COVID-19 Misinfodemic: moving beyond fact-checking. Health Educ Behav 48: 9-13.
- COTTER K, DECOOK JR KANTHAWALA S. 2022. Fact-Checking the Crisis: COVID-19, Infodemics, and the Platformization of Truth. Soc Media Soc 8: 1.
-
ELIZABETH J. 2014. Who are you calling a fact-checker?. American Press Institute. In: https://www.americanpressinstitute.org/fact-checking-project/fact-checker-definition/
» https://www.americanpressinstitute.org/fact-checking-project/fact-checker-definition/ - FUCHS C. 2020. Communication and Capitalism: A critical Theory. University of Westminster Press: London.
- GEHRKE M BENETTI M. 2021. Disinformation in Brazil during the COVID-19 pandemic: topics, platforms, and actors. Fronteiras-Estudos Midiáticos 23(2): 14-28.
- GRAVES L. 2017. Anatomy of a Fact Check: Objective Practice and the Contested Epistemology of Fact-Checking. Commun Cult Crit 10: 518-537.
- GRIMMER J STEWART BM. 2013. Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political Anal 21: 267-297.
- KRAUSE NM, FREILING I, BEETS B BROSSARD D. 2020. Fact-checking as risk communication: the multi-layered risk of misinformation in times of COVID-19. J Risk Res 23: 1052-1059.
- KRONBERGER N WAGNER W. 2002. Keywords in context: Statistical analysis of text features. Qualitative researching with text, image and sound: A practical handbook, p. 299-317.
- LAHLOU S. 2012. Text mining methods: An answer to Chartier and Meunier. Pap Soc Represent 20: 1-7.
- LEWANDOWSKY S, ECKER UK COOK J. 2017. Beyond misinformation: Understanding and coping with the “post-truth” era. J Appl Res Mem Cogn 6: 353-369.
- LIMA GS, CALAZANS MM MASSI L. 2021. Mensagens falsas sobre o novo coronavírus: legitimidade e manipulação na luta de classes. Chasqui Rev Latinoam Comun 1: 259-280.
- LÓPEZ-PUJALTE C NUÑO-MORAL MV. 2020. La “infodemia” en la crisis del coronavirus: Análisis de desinformaciones en España y Latinoamérica. Rev Esp Doc Cient 43: e274.
- MANNING CD, RAGHAVAN P SCHÜTZE H. 2008. Introduction to Information Retrieval. Cambridge University Press.
- MANOVICH L. 2001. The Language of New Media. Cambridge: The MIT Press.
- MASSARANI L, MEDEIROS A, WALTZ I LEAL T. 2023. Disinformation about COVID-19 in Ibero-America: an analysis based on fact-checking. TSN, Rev Estud Int 14: 67-79.
- MCCOMBS ME SHAW DL. 1972. The agenda-setting function of mass media. Pub Opin Q 36: 176-187.
- MORAIS-SANTOS N, COTTONE PF, ANTLOGA C, HOCHDORN A, CARVALHO AM BARBOSA MA. 2022. Female entrepreneurship in Brazil: how scientific literature shapes the socio-cultural construction of gender inequalities. Humanit Soc Sci Commun 9: 1-10.
- NIEVES-CUERVO GM, MANRIQUE-HÉRNANDEZ EF, ROBLEDO-COLONIA AF ARDILA EKG. 2021. Infodemia: noticias falsas y tendencias de mortalidad por COVID-19 en seis paises de América Latina. Rev Panam Salud Publica 45: 1-8.
- PITA M. 2021. Disinformation during the pandemic and Latin America’s Regulatory Response. Montevideo: United Nations Educational, Scientific and Cultural Organization, UNESCO.
- REINERT M. 1990. Alceste une méthodologie d’analyse des données textuelles et une application: Aurelia De Gerard De Nerval. BMS 26: 24-54.
- RODRÍGUEZ-PÉREZ C. 2020. Una reflexión sobre la epistemología del fact-checking journalism: retos y dilemas. Rev Comun 19: 243-258.
- ROOZENBEEK J, SCHNEIDER CR, DRYHURST S, KERR J, FREEMAN AL, RECCHIA G VAN DER LINDEN S. 2020. Susceptibility to misinformation about COVID-19 around the world. R Soc Open Sci 7: 201199.
- SCHUETZ SW, SYKES TA VENKATESH V. 2021. Combating COVID-19 fake news on social media through fact checking: antecedents and consequences. Eur J Inf Syst 30: 376-388.
- SIWAKOTI S, YADAV K, BARILETTO N, ZANOTTI L, ERDOGDU U SHAPIRO JN. 2021. How COVID drove the evolution of fact-checking. Harvard Kennedy School Misinformation Review.
- SMALLMAN M. 2016. Public Understanding of Science in turbulent times III: Deficit to dialogue, champions to critics. Public Underst Sci 25: 186-197.
- SOUZA-JÚNIOR J. 2021. Transmediatisation of the COVID-19 crisis in Brazil: The emergence of (bio-/geo-)political repertoires of (re-)interpretation. Humanit Soc Sci Commun 8: 210.
- STOCKWELL P, COLOMB RM, SMITH AE WILES J. 2009. Use of an automatic content analysis tool: A technique for seeing both local and global scope. Int J Hum Comput Stud 67: 424-436.
- TERÄMÄ E, SMALLMAN M, LOCK SJ, JOHNSON C AUSTWICK MZ. 2016. Beyond academia–Interrogating research impact in the research excellence framework. PLoS ONE 11: e0168533.
- TURK Z. 2018. Technology as Enabler of Fake News and a Potential Tool to Combat It. Document prepared for Policy Department for Economic, Scientific and Quality of Life Policies, European Parliament.
- USCINSKI JE BUTLER RW. 2013. The epistemology of fact-checking. Crit Rev 25: 162-180.
- VOLOSINOV VN. 1973. Marxism and the philosophy of language. London: Seminar Press.
- WIEDEMANN G. 2016. Text mining for qualitative data analysis in the social sciences, Vol 1, Wiesbaden, Germany: Springer Fachmedien Wiesbaden.
- WILLIAMS R. 1980. Problems in materialism and culture: Selected essays. Verso.
