Systematic Review Support software system: web-based solution for managing duplicates and screening eligible studies

ABSTRACT

Objective:

To describe the main functions of the “Systematic Review Support” web-based system for removing duplicate articles and aiding eligibility analysis during the process of conducting systematic review studies.

Methods:

The system was developed based on the incremental build model using the Agile methodology. The software is proprietary source code and was published on a proprietary platform. The architecture of the production environment allows the infrastructure used to increase or decrease according to demand. The system functions are presented with insertion of screenshots of the interfaces of the version for personal computers during the simulation of a systematic review.

Results:

After importing the files containing the abstracts retrieved from the Pubmed, Embase, and Web of Science databases, the system identifies and removes duplicates for later reading and analysis of title and abstract, a stage which can be performed by one or more reviewers independently. After unblinding of reviewers, the decisions on the eligibility of the studies are compared automatically to help the researchers reach a consensus on any disagreements. Results can be filtered and a PDF produced containing the eligible studies.

Conclusion:

Version 1.0 of the system is available on the web (sysrev.azurewebsites.net) to assist researchers in the initial stages of systematic reviews.

Keywords:
Databases, bibliographic; Cloud computing; Systematic review; Software; Information technology

RESUMO

Objetivo:

Descrever as principais funcionalidades do sistema “Apoio à Revisão Sistemática” na identificação e exclusão de artigos duplicados e no auxílio na análise de elegibilidade durante a condução de estudo de revisão sistemática.

Métodos:

O sistema foi desenvolvido com base em um modelo de processo incremental, utilizando-se metodologia Ágil. É de código fechado e foi publicado em plataforma proprietária. O ambiente de produção onde o sistema foi implantado possui arquitetura que permite que a infraestrutura utilizada aumente ou diminua conforme a demanda. As funcionalidades foram apresentadas com inserção de imagens das interfaces da versão para computadores, simulando uma revisão sistemática.

Resultados:

Após a importação dos resumos recuperados nas bases de dados PubMed, Embase e Web of Science, o sistema permite a identificação e eliminação de duplicatas para posterior leitura e análise de título e resumo, etapa que pode ser realizada por mais de um revisor de maneira independente. Após a quebra do cegamento entre os revisores, as respostas sobre a elegibilidade dos estudos podem ser comparadas automaticamente para facilitar a resolução de divergências pelos pesquisadores. É possível filtrar os resultados e gerar um arquivo PDF com os estudos elegíveis.

Conclusão:

A versão 1.0 do sistema “Apoio à Revisão Sistemática” encontra-se disponível na web (sysrev.azurewebsites.net) para auxiliar pesquisadores nas etapas iniciais de um estudo de revisão sistemática.

Palavras-chave:
Bases de dados bibliográficos; Computação em nuvem; Revisão sistemática; Software; Tecnologia da informação

INTRODUCTION

The number of health-related scientific articles published in databases is constantly increasing. Reading and critically evaluating this production represents a challenge for researchers, so that systematic review studies are essential tools for scientific updating¹.

Unlike traditional literature reviews, systematic and scoped reviews require protocols with the criteria to be followed to guarantee the reliability of the results and the reproducibility of the process².

The search for studies in different databases generates a considerable number of identical articles, which are called duplicates. Their elimination is usually labor intensive and requires time and supporting computational tools³. A multidisciplinary team with enough members is also necessary, especially in the eligibility phase, due to the demand for time. At least two reviewers are required for initial study selection, quality analysis, and data extraction from scientific publications to ensure comprehensiveness of decisions and avoid bias¹,⁴.

The use of automation or semi-automated processes for the selection of articles in systematic reviews is a reason for study and debate at the International Collaboration for Automation of Systematic Reviews (ICASR), favoring the construction and evaluation of computational tools that contribute to the quality of studies of review of evidence, as they reduce selection bias, time, and workload⁵.

On The Systematic Review Toolbox platform, free and paid software can be found that help in one or more stages of the process of different types of reviews, such as the identification of duplicate articles and the selection of studies. Among them are the Brazilian computational tools Parsifal and StArt⁶.

Despite the existence of different software, in practice, it is observed that students use non-specific programs for this purpose, such as text editors, spreadsheets and reference managers, which require manual work to complete the necessary tasks. Thus, the need for methodological and technological support that contributes to the rapid dissemination of knowledge for application in the health area is identified⁷.

Thus, the objective of this study was to describe the main functionalities of the “Support for Systematic Review” system in the identification and exclusion of duplicate articles and in assisting in the analysis of eligibility during the conduction of systematic literature reviews.

METHODS

This methodological research describes the main functionalities of the computer system called “Support for Systematic Review” (Apoio à Revisão Sistemática) (version 1.0). This web system was created to assist researchers in the initial phases of a systematic review project:

1. Organization of abstracts, with identification and exclusion of duplicate studies; and

2. Eligibility analysis, which includes the definition of eligibility criteria, selection of studies by reading titles and abstracts, according to established criteria, and comparing decisions on the eligibility of studies to resolve disagreements.

An abstract is the initial part of a scientific article, usually presented in a paragraph that highlights the main elements of the research (background, objectives, methods, results, and conclusion). It is preceded by its reference (authorship, title, journal, date of publication) and indexed in the databases to facilitate search⁸. In the developed system, the term “summary” was considered to include its reference.

For the construction of the system, the following items were considered:

• Target audience: professionals and researchers, undergraduate and graduate students from programs in the health area, whose objective is to carry out a literature review.

• Objective: to provide an efficient alternative to manual work, containing specific functions to optimize researchers’ time in the initial stages of systematic review studies and ensure the production of reliable results.

• Infrastructure: access to the system via computer, tablet or smartphone, considering the different types of hardware, screen sizes, and operating systems available in the market. We opted for development on a web platform to access the system on a device with an updated browser connected to the internet.

• Language: Portuguese.

• Ensuring the security of the data stored in the system.

• Priority to ease of use and the construction of a friendly and responsive interface (adaptable to different screen sizes).

An incremental process model⁹ was used, which makes it possible to add new functionalities to each version of the system. Thus, the functionalities referring to the other phases of a systematic review may be implemented in later versions. The Agile¹⁰ methodology was adopted to deliver a version with new features, after each development and testing period. This methodology has the following principles: to satisfy the user through the continuous and early delivery of value-added software; deliver working software in a few weeks, with the shortest time preferred; work together (users and developers) on the entire project; talk face-to-face with the development team to convey information more efficiently and effectively; keep continuous attention on technical excellence and good design to increase agility; seek simplicity as an essential element.

Kanban¹¹ was adopted to monitor the activities under development; thus, the functionalities and requirements defined for the delivery were transformed into cards classified according to To Do, Doing, and Done status. The cards were arranged on a panel according to the version to be delivered. The system was developed with closed source code and published on a proprietary platform (Azure DevOps, by Microsoft)¹² to control activities, display cards, store source code versions and publish versions on the production server.

Functional requirements were established based on theoretical references¹,¹³,¹⁴ and on the experience acquired in the elaboration of review studies. Technical requirements were defined in partnership with a systems analyst. System development activities were organized by an information technology (IT) professional, voluntarily and independently, according to the functionalities described in Chart 1.

Chart 1
Functions defined for the development of the system “Support for Systematic Review”, version 1.0.

Functionality	Description
User identification	It is about creating, changing data, authentication and permissions of a user. Anonymous access to system functionality should not be allowed.
Identification of the systematic review	It is about creating, changing and deleting a systematic review.
Import of abstracts	This involves importing the abstracts into the system's database through text files exported from the source databases, such as Medline via PubMed, Embase and Web of Science.
Organization of abstracts	The organization of studies refers to ordering, identification, separation into groups and removal of duplicate abstracts.
Eligibility process	It involves the reading and analysis of titles and abstracts by the reviewers to assist in the selection of eligible studies. It includes the automatic comparison of decisions and the identification of disagreements between reviewers on the eligibility of abstracts. The main reviewer is allowed to include eligibility criteria, to send an invitation to other reviewers and to decide whether or not to keep blinding among researchers. Reviewers are allowed to obtain a PDF file containing all abstracts.

• Source: prepared by the authors.

For the development of the system, a set of three software components was defined for its constitution:

1. User interface (UI), responsible for displaying information and entering data by users;

2. Application programming interface (API), responsible for communicating data with the user interface and for executing the processing; and

3. Database (DB), responsible for storing and retrieving data from each revision.

The concepts and technologies used in each component are detailed below.

User interface

For the construction of the UI, the concept of Single Page Application¹⁵ was used, in which there is a single HTML page that alternates its visible elements according to user interaction and data update. This approach allows for a more dynamic user experience as only the elements where data has been updated are redrawn. Another advantage of this concept is that part of the processing, referring to the rules for presenting information and entering data by the user, is performed on the device used, allowing more users to access the system without this processing being added to the API server.

The programming language used for the development of the UI was TypeScript¹⁶, using the Angular framework, version 9¹⁷. For the page layout, the admin model was used, which has: a side menu, fixed header and the rest of the page reserved for displaying information. The tool used to build the page layout was the Bootstrap framework¹⁸.

To adapt the UI to different screen sizes, we used the responsive development of pages through Media Queries¹⁹ defined in the Cascading Style Sheet (CSS)²⁰.

Application programming interface

The API was developed using the C#²¹ programming language, with the .Net Core framework, version 3.0²². The use of this framework allows hosting the API on Windows or Linux servers, allowing greater flexibility in defining the infrastructure.

For communication with the UI, the Asp. Net Core Web API framework, version 3.0²³, was used, which allows communication of the Representational State Transfer (REST) type²⁴, in which most of the information is not stored in the server's memory. In this way, a greater number of users can use the system without increasing memory consumption to keep their information.

The UI access authorization to use the API is done through Json Web Token (JWT)²⁵, in which the user's access information is transmitted through an encrypted token, scheduled to expire after a certain time interval.

Communication with the database was developed using the concept of Object Relational Mapping (ORM)²⁶, whereby queries and data updates are generated by the ORM framework, reducing development time and allowing the database management system of data to be changed without having to develop new queries and specific updates for a new management system.

Database

The database chosen for storing revision and user data was Microsoft SQL Server²⁷, as its performance supports large volumes of data. The data model chosen was a hybrid of the relational model, in which data is separated into related tables, with a non-relational model, in which data is stored in JavaScript Object Notation (JSON) objects²⁸. Tables were created to store data from users, reviews, their abstracts and groups, authors, eligibility criteria, comments, and reviewers.

Production environment

The production environment chosen for publishing the software components necessary for the system to work was Azure²⁹, which uses cloud services architecture. This architecture allows the infrastructure used to increase or decrease according to the system's demand, paying only for what was used in memory, processing, and storage.

We created a web application service for the user interface, a web application service for the API, and a Microsoft SQL Server relational database service for storing the data.

Publishing of UI and API components is done by auto publishing, configured in Azure DevOps¹², allowing the storage of published versions and reverting to a previous version, in case the publication has problems.

The database is updated using Data Definition Language (DDL) Strucutred Query Language (SQL) scripts, generated with a database structure comparison tool provided by the Visual Studio Community³⁰. The comparison is made between the local database used for development and the database service in production. Scripts are also created to return to the previous version, if necessary.

For security in the transfer of information between client and server, the HTTPS protocol was used, through Secure Socket Layer (SSL), which encrypts all information, preventing unauthorized persons from viewing it over the network.

Tests performed

The developer performed specific tests for each requirement created. The bugs identified were fixed prior to deployment to the production environment.

Description of the computer system

After the system was implemented, its functionalities were described according to the menu, simulating its use in conducting a systematic review study on cystic fibrosis³¹,³². The search phrases (Chart 2) were inserted into the databases to obtain the files to be imported into the system. Screen images were inserted to visualize the functionalities of version 1.0, for access through computers (desktop and notebook).

Chart 2
Description of the data used in the application of the system.

Criterion	Data description
Database	PubMed, Embase, and Web of Science.
Search phrase	“cystic fibrosis” AND (“child” OR “adolescent”) AND (“nutrition assessment” OR “nutritional status” OR “body composition” OR “anthropometry” OR “Absorptiometry, Photon” OR “electric impedance” OR “electric conductivity” OR “body mass index” OR “waist circumference” OR “skinfold thickness” OR “body weight” OR “body height”).
Publication period	2008–2018.
Languages	English and Portuguese.
Participants	Children and adolescents diagnosed with cystic fibrosis.
Type of study	Observational or clinical studies.
Outcome (primary or secondary)	Anthropometric and body composition measurements and indices obtained by dual-energy X-ray absorptiometry and electrical bioimpedance.
Eligibility Criteria	Children and adolescents with cystic fibrosis; age group between six and 18 years; assessment of nutritional status with at least one measure or anthropometric and body composition index.

• Source: prepared by the authors.

Ethical aspects

The work followed the terms of Resolution of the National Health Council (Conselho Nacional de Saúde – CNS) 466/12 and was approved by the Research Ethics Committee of the School of Public Health of Universidade de São Paulo (CAAE 36397420.8.0000.5421).

RESULTS

The “Support for Systematic Review” system is available at the electronic address (sysrev.azurewebsites.net). Explanatory videos can be accessed on a platform (www.apoioarevisaositematica.com.br) for self-learning and better use of its main features: the identification and exclusion of duplicates, assistance in the selection of eligible studies by reading titles and abstracts and assistance in resolving disputes.

Identification and exclusion of duplicates

First, one needs to register. In the “Reviews” (Revisões) menu, researchers are allowed to create, change or delete a systematic review. The system allows the inclusion of one or more bibliographic review projects as a main reviewer and as a guest reviewer. In the “Groups” (Grupos) menu, the researcher can create, change or delete one or more groups to import the abstracts as needed. In the “Summary” (Resumos) menu, the files that contain the records found in the electronic databases can be imported into the groups created by the researcher. After import, the status of each summary is set by the system as “Imported” (Importado).

After clicking on the “Duplicates” (Duplicados) button, researchers must select a priority database, which will serve as a reference for the system to assign the “Duplicate” status to existing abstracts in the other groups. By clicking on the “Exclude duplicates” (Excluir duplicados) button, the studies to be included in the eligibility phase are obtained.

By using the system, a review was created with the name “Anthropometric assessment in cystic fibrosis”. Next, three groups were created with the names of the origin databases, namely PubMed, Embase, and Web of Science. Each file was imported into the system and saved in the group according to its origin. In all, 834 records were imported, resulting in 644 studies after excluding duplicates (Figure 1).

Figure 1
Screen of the “Abstracts” (Resumos) menu after deleting duplicate articles.

Assistance in the selection of eligible studies

In the eligibility phase, in the “Criteria” (Critérios) menu, researchers are allowed to include, change, and exclude the eligibility criteria predefined in the research protocol.

In the “Reviewers” (Revisores) menu, researchers are allowed to include or remove reviewers for reading and analyzing titles and abstracts. Each guest reviewer is notified of their inclusion in a review.

In the “Abstracts” (Resumos) menu, after clicking on “View” (Visualizar), review participants can access the summaries and independently decide on the eligibility criteria by selecting from the “Complies” (Atende) list an answer for each criterion (yes, no or not clear). Next, they must select a result for each abstract (eligible, non-eligible or unclear), and it is possible to create comments.

In the application of the system, it is possible to notice that the eligibility criteria are presented on the same page of the summary to facilitate and guide analysis. The selection of responses in the “Complies” (Atende) and “Result” (Resultado) lists is not mandatory, but a status should be included in the “Result” (Resultado) list to favor the comparison of responses in the verification of the existence of divergences (Figure 2).

Figure 2
Screen containing abstract and eligibility criteria.

Assistance in resolving disputes

At an appropriate time, the primary reviewer may break the blinding to release results on the eligibility of abstracts to participants. In the “Abstracts” (Resumos) menu, by clicking on “Show comparison” (Exibir comparação), the system automatically compares the answers, and the differences pointed out can be resolved by the participants. At this stage, discussions can take place in person or remotely, provided by the sharing of data by the system. Reviewers can make the necessary changes directly into the system and download a PDF file with eligible abstracts. In the “Dashboard” menu, the main reviewer can check, through follow-up charts, the progress and completion of the abstracts reading by the invited reviewers and identify the existence of divergences (Figure 3).

Figure 3
Screen of the “Abstracts” (Resumos) and “Dashboard” menu showing the comparison of answers on the eligibility of the studies.

DISCUSSION

This study presented the “Support to Systematic Review” system, which excludes duplicate articles, assists in the selection of eligible studies and in the resolution of divergences, organizing teamwork.

These functionalities aim at the rational use of time in the initial phase of a systematic review. Time savings are found in research on the influence of the use of automation tools in review studies. In it, among 189 participants, 79% reported using them in various tasks of the eligibility stage. As participants were divided into systematic reviews, guides and health technology evaluators, it was observed that 80, 88, and 73%, respectively, reported time savings when using automated tools³³.

The use of computational tools is strongly encouraged in review studies¹⁴. Considering that a typical systematic review can take at least a year to be produced³⁴, the use of the system is important in these phases, which are usually very time consuming when performed manually.

A software frequently used in systematic reviews is EndNote, a reference manager that allows the identification of duplicates and the automatic insertion of citations in text editors³⁵, but which does not allow the inclusion of eligibility criteria and the comparison of decisions about eligibility of studies. Another software, Rayyan, has functionalities similar to those existing in “Support for Systematic Review”, with the difference that it is presented in English and requires the researcher to make a decision on the existence of duplicates based on similarity analysis, in addition to allowing the inclusion of keywords and terms that are highlighted in the abstract to facilitate their analysis³⁶.

A common problem in the identification of duplicate studies is the errors that occur due to variations in the records of studies in the databases, mainly in the name of authors and journals. With this, computational tools can point out a study as a duplicate when in fact it is not (false-positive); or they may point out as not duplicated a study that actually is (false-negative)³⁷. A false-positive study, when excluded, can introduce a selection bias if it is not retrieved later by the researcher. The evaluation of the performance of the “Support for Systematic Review” in the identification of duplicates showed a sensitivity of 98.4%, specificity of 99.8%, and accuracy of 99.6%. The sensitivity result was higher than that of EndNote (sensitivity: 59.4%, specificity and accuracy: 100%) and that of Rayyan (sensitivity: 96.4%, specificity: 100%, accuracy: 99.3%) (data under publication).

In the eligibility phase, one of the advantages of “Support for Systematic Review” in the selection of studies is the ease of use, the reproducibility of the process and the participation of several reviewers in an organized and independent way, with the possibility of breaking the blinding in time defined by the main reviewer. The system is practical for storing eligibility criteria and sharing data between reviewers. On the same screen, an abstract and a form with the eligibility criteria for marking responses are presented, facilitating the abstract selection process.

The system is geared toward the production of typical systematic reviews, but is extended to researchers who opt for bibliographic surveys without the formalities of a systematic review. A simulation of a systematic review was carried out using a set of studies on cystic fibrosis, however different topics may be used in other studies, provided that the surveys are carried out in the bibliographic databases enabled in this system. Images of the interfaces of the computer version were inserted, but users can also access the system through smartphones and tablets, as it is responsive to different screen sizes.

We can identify as weaknesses the availability of version 1.0 only in Portuguese, the proprietary code, and the reduced number of databases. However, the Portuguese language version favors its use in other countries that have this language as their mother tongue. The number of databases included can be increased, prioritizing those used in the health area⁴,³⁸. The system has the potential for improvements, and may include quality assessment instruments, data extraction by more than one evaluator and comparison of these results.

As potentialities, there is the use of the Agile methodology, which provides the possibility of constantly adding new resources and improvements, such as the implementation of the Latin American and Caribbean Literature on Health Sciences (Lilacs), the inclusion of checkboxes for deleting false negatives, a recycle bin for retrieving false positives and exporting text files for use in reference managers. As it is a web tool, all updates can be accessed immediately by users.

As a vision of the future, it is intended to expand its functionality according to the steps of a systematic review and improve the usability and accuracy of the duplicate identification process, as well as move toward the (semi) automation of the selection of studies.

The production of scientific knowledge can be reinforced in the presence of computational tools that reduce manual work and allow better synthesis and use of information. Thus, it is expected that the system will be used in Portuguese-speaking countries for the continuous production of knowledge syntheses for application in health actions.

The “Support for Systematic Review” system is available on the web for the exclusion of duplicates and assistance in the selection of eligible studies and resolution of divergences in systematic literature reviews, ensuring greater speed, reliability, and reproducibility for the completion of these steps.

• FUNDING: none.

REFERENCES

• ¹
  Egger M, Smith GD, Altman DG. Systematic reviews in health care: meta-analysis in context. 2^nd ed. London: BMJ Publishing Group; 2001.
• ²
  Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol 2018; 18(1): 143. https://doi.org/10.1186/s12874-018-0611-x
  » https://doi.org/10.1186/s12874-018-0611-x
• ³
  Lefebvre C, Glanville J, Briscoe S, Featherstone R, Littlewood A, Marshall C, et al. Technical supplement to chapter 4: searching for and selecting studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston MS, Li T, Page MJ, et al., eds. Cochrane Handbook for Systematic Reviews of Interventions Version 6.2. [Internet]. 2021 [cited on May 3, 2022]. Available at: https://training.cochrane.org/handbook/current/chapter-04-technical-supplement-searching-and-selecting-studies#section-4-3/
  » https://training.cochrane.org/handbook/current/chapter-04-technical-supplement-searching-and-selecting-studies#section-4-3/
• ⁴
  Muka T, Glisic M, Milic J, Verhoog S, Bohlius J, Bramer W, et al. A 24-step guide on how to design, conduct, and successfully publish a systematic review and meta-analysis in medical research. Eur J Epidemiol 2020; 35(1): 49-60. https://doi.org/10.1007/s10654-019-00576-5
  » https://doi.org/10.1007/s10654-019-00576-5
• ⁵
  O’Connor AM, Glasziou P, Taylor M, Thomas J, Spijker R, Wolfe MS. A focus on cross-purpose tools, automated recognition of study design in multiple disciplines, and evaluation of automation tools: a summary of significant discussions at the fourth meeting of the International Collaboration for Automation of Systematic Reviews (ICASR). Syst Rev 2020; 9(1): 100. https://doi.org/10.1186/s13643-020-01351-4
  » https://doi.org/10.1186/s13643-020-01351-4
• ⁶
  Marshall C, Sutton A, O’Keefe H, Johnson E, eds. The systematic review toolbox [Internet]. 2022 [cited on Jul. 28, 2022]. Available at: http://www.systematicreviewtools.com/
  » http://www.systematicreviewtools.com/
• ⁷
  Miranda ES, Figueiró AC, Potvin L. Are public health researchers in Brazil ready and supported to do knowledge translation? Cad Saude Publica 2020; 36(4): e00003120. https://doi.org/10.1590/0102-311X00003120
  » https://doi.org/10.1590/0102-311X00003120
• ⁸
  Universidade de São Paulo. Faculdade de Saúde Pública. Guia de apresentação de teses. 2ᵃ ed. atual. São Paulo: Faculdade de Saúde Pública da USP; 2017 [Internet]. [cited on May 03, 2022]. Available at: http://www.biblioteca.fsp.usp.br/guia
  » http://www.biblioteca.fsp.usp.br/guia
• ⁹
  Pressman RS, Maxim BR. Engenharia de software: uma abordagem profissional. 8ᵃ ed. Porto Alegre: AMGH; 2016.
• ¹⁰
  Beck K, Beedle M, van Bennekum A, Cockburn A, Cunningham W, Fowler M, et al. Manifesto para desenvolvimento ágil de software [Internet]. [cited on May 03, 2022]. Available at: https://agilemanifesto.org/iso/ptbr/manifesto.html
  » https://agilemanifesto.org/iso/ptbr/manifesto.html
• ¹¹
  Kniberg H, Skarin M. Kanban and Scrum making the most of both [Internet]. 2010 [cited on May 3, 2022]. Available at: https://www.infoq.com/minibooks/kanban-scrum-minibook/
  » https://www.infoq.com/minibooks/kanban-scrum-minibook/
• ¹²
  Microsoft. Azure devops documentation [Internet]. 2015 [cited on May 3, 2022]. Available at: https://docs.microsoft.com/en-us/azure/devops/
  » https://docs.microsoft.com/en-us/azure/devops/
• ¹³
  Centre for Reviews and Dissemination. Systematic reviews. CRD's guidance for undertaking reviews in health care. York: University of York; 2009 [Internet]. [cited on May 3, 2022]. Available at: https://www.york.ac.uk/crd/guidance/
  » https://www.york.ac.uk/crd/guidance/
• ¹⁴
  Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. 2021 [cited on May 03, 2022]. Available at: http://training.cochrane.org/handbook/
  » http://training.cochrane.org/handbook/
• ¹⁵
  Santamaria JMA. The single page interface manifesto [Internet]. 2015 [cited on May 3, 2022]. Available at: http://itsnat.sourceforge.net/php/spim/spi_manifesto_en.php/
  » http://itsnat.sourceforge.net/php/spim/spi_manifesto_en.php/
• ¹⁶
  TypeScript, TypeScript documentation [Internet]. [cited on May 3, 2022]. Available at: https://www.typescriptlang.org/docs/
  » https://www.typescriptlang.org/docs/
• ¹⁷
  Introduction to the angular docs [Internet]. [cited on May 3, 2022]. Available at: https://angular.io/docs/
  » https://angular.io/docs/
• ¹⁸
  Introduction [Internet]. [cited on May 3, 2022]. Available at: https://getbootstrap.com/docs/4.1/getting-started/introduction/
  » https://getbootstrap.com/docs/4.1/getting-started/introduction/
• ¹⁹
  Marcotte E. Responsive web design [Internet]. 2010 [cited on May 3, 2022]. Available at: https://alistapart.com/article/responsive-web-design/
  » https://alistapart.com/article/responsive-web-design/
• ²⁰
  W3C. Cascading style sheets. What is CSS [Internet]. [cited on Aug. 30, 2021]. Available at: https://www.w3.org/Style/CSS/
  » https://www.w3.org/Style/CSS/
• ²¹
  Microsoft. C# documentation [Internet]. [cited on May 3, 2022]. Available at: https://docs.microsoft.com/en-us/dotnet/csharp/
  » https://docs.microsoft.com/en-us/dotnet/csharp/
• ²²
  Microsoft. What is .NET? Introduction and overview [Internet]. [cited on May 3, 2022]. Available at: https://docs.microsoft.com/en-us/dotnet/core/introduction/
  » https://docs.microsoft.com/en-us/dotnet/core/introduction/
• ²³
  Microsoft. Visão geral do ASP.NET core [Internet]. [cited on May 3, 2022]. Available at: https://docs.microsoft.com/pt-br/aspnet/core/introduction-to-aspnet-core?view=aspnetcore-3.0/
  » https://docs.microsoft.com/pt-br/aspnet/core/introduction-to-aspnet-core?view=aspnetcore-3.0/
• ²⁴
  Fielding RT. Representational State Transfer (REST). In: Fielding RT. Architectural styles and the design of network-based software architectures [dissertation] Irvine: University of California; 2000. [cited on May 3, 2022]. Available at: https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm
  » https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm
• ²⁵
  JWT. Introduction to JSON web tokens [Internet]. [cited on May 3, 2022]. Available at: https://jwt.io/introduction/
  » https://jwt.io/introduction/
• ²⁶
  Hibernate. What is object/relational mapping? [Internet]. [cited on May 3, 2022]. Available at: https://hibernate.org/orm/what-is-an-orm/
  » https://hibernate.org/orm/what-is-an-orm/
• ²⁷
  Microsoft. SQL server technical documentation [Internet]. [cited on May 03, 2022]. Available at: https://docs.microsoft.com/en-us/sql/sql-server/
  » https://docs.microsoft.com/en-us/sql/sql-server/
• ²⁸
  Introducing JSON [Internet]. [cited on Apr. 4, 2021]. Available at: https://www.json.org/json-en.html
  » https://www.json.org/json-en.html
• ²⁹
  Microsoft. Azure documentation [Internet]. [cited on May 3, 2022]. Available at: https://docs.microsoft.com/en-us/azure/
  » https://docs.microsoft.com/en-us/azure/
• ³⁰
  Microsoft. Visual studio documentation [Internet]. [cited on May 3, 2022]. Available at: https://docs.microsoft.com/en-us/visualstudio/windows/?view=vs-2019/
  » https://docs.microsoft.com/en-us/visualstudio/windows/?view=vs-2019/
• ³¹
  Egan ME, Greene DM, Voynow JA. Fibrose cística. In: Kliegman RM, Stanton BF, St Geme III JW, Schor NF, Behrman RE. Nelson: Tratado de pediatria. 20ᵃ ed. Rio de Janeiro: Elsevier; 2018. p. 2098-112.
• ³²
  Hauschild DB, Rosa AF, Ventura JC, Barbosa E, Moreira EAM, Ludwig Neto N, et al. Association of nutritional status with lung function and morbidity in children and adolescents with cystic fibrosis: a 36-month cohort study. Rev Paul Pediatr 2018; 36(1): 31-8. http://dx.doi.org/10.1590/1984-0462/;2018;36;1;00006
  » http://dx.doi.org/10.1590/1984-0462/;2018;36;1;00006
• ³³
  Scott AM, Forbes C, Clark J, Carter M, Glasziou P, Munn Z. Systematic review automation tools improve efficiency but lack of knowledge impedes their adoption: a survey. J Clin Epidemiol 2021; 138: 80-94. https://doi.org/10.1016/j.jclinepi.2021.06.030
  » https://doi.org/10.1016/j.jclinepi.2021.06.030
• ³⁴
  Borah R, Brown AW, Capers PL, Kaiser KA. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open 2017; 7(2): e012545. https://doi.org/10.1136/bmjopen-2016-012545
  » https://doi.org/10.1136/bmjopen-2016-012545
• ³⁵
  Thomson Reuters. EndNote X9. Quick reference guide [Internet]. [cited on Jul. 29, 2022]. Available at: https://support.clarivate.com/Endnote/servlet/fileField?entityId=ka14N000000EcsXQAS&field=CA_Attachment_1__Body__s
  » https://support.clarivate.com/Endnote/servlet/fileField?entityId=ka14N000000EcsXQAS&field=CA_Attachment_1__Body__s
• ³⁶
  Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan-a web and mobile app for systematic reviews. Syst Rev 2016; 5(1): 210. https://doi.org/10.1186/s13643-016-0384-4
  » https://doi.org/10.1186/s13643-016-0384-4
• ³⁷
  Rathbone J, Carter M, Hoffmann T, Glasziou P. Better duplicate detection for systematic reviewers: evaluation of systematic review assistant-deduplication module. Syst Rev 2015; 4(1): 6. https://doi.org/10.1186/2046-4053-4-6
  » https://doi.org/10.1186/2046-4053-4-6
• ³⁸
  Gupta S, Rajiah P, Middlebrooks EH, Baruah D, Carter BW, Burton KR, et al. Systematic review of the literature: best practices. Acad Radiol 2018; 25(11): 1481-90. https://doi.org/10.1016/j.acra.2018.04.025
  » https://doi.org/10.1016/j.acra.2018.04.025

Publication Dates

• Publication in this collection
  17 Oct 2022
• Date of issue
  2022

History

• Received
  16 May 2022
• Reviewed
  01 Aug 2022
• Accepted
  16 Aug 2022