Precision over quantity: identifying trustworthy genomic risk stratification models

Costa Filho, Valbert Oliveira; Passos, Pedro Robson Costa

doi:10.31744/einstein_journal/2025CE1726

Dear Editor,

The Brazilian Society of Medical Genetics and Genomics recently issued a statement against using polygenic risk scores for breast cancer risk stratification in Brazilian women.¹ In fact, an ongoing surge could be observed in genomic signatures and prognostic models, with over 200 LASSO-based TCGA signatures appearing in Scientific Reports last year, based on a focused keyword-based search, several of questionable quality. Discerning reliable models from unreliable ones is crucial for clinicians in order to avoid poorly validated tools.

The first step is to assess how the model was built and whether it was properly validated. Currently, we distinguish two common validation types. The first involves splitting the patient dataset into proportions such as 80/20 or 70/30 for model training and testing. Unfortunately, this approach is less effective, as machine learning after a single random training-test split might yield unreliable results.²The second and more robust method involves validation using an entirely external dataset from independent studies. This approach is more reliable, as internal data splitting often fails to truly validate a model and cannot replace independent replication across different researchers, populations, and methodologies.³

The next step is the assessment of the area under the receiving operating curve (AUC), representing the integral of all points along the curve and capturing both sensitivity and specificity to provide a comprehensive measure of the overall performance of the model. In general, a model with an AUC <0.6 could be considered unreliable and acceptable models should display an AUC of at least 0.7.⁴Subsequently, evaluating the biological plausibility of the genes included in the model is essential. For instance, models in which a tumor suppressor expression predicts worse survival or an oncogene expression predicts better prognosis, could lack reproducibility. Models and signatures should (or should aim to) retain biological relevance and plausibility. Therefore, assessing whether the applied genes have been validated in the disease through methods beyond transcriptomics (e.g., real-time PCR or immunohistochemistry) is also necessary.

Finally, most models and signatures are developed using publicly available datasets, such as those from TCGA or the Gene Expression Omnibus, and are typically produced and evaluated retrospectively. To ensure reproducibility and enable the integration of these models into clinical practice, model validation in prospective studies would be essential. This step is crucial, as prospective validation provides an important first assessment of the real-world machine learning model performances.⁵

These steps could potentially prove particularly valuable for clinicians, helping them avoid poorly designed studies, as this emerging field is underrepresented in the undergraduate curricula and its literature could be hard to navigate. Figure 1 presents a flowchart, summarizing the algorithm for identifying robust models.

Figure 1
Algorithm for identifying robust models

REFERENCES

¹ Sociedade Brasileira de Genética Médica (SBGM). Posicionamento da SBGM sobre o uso de escores de risco poligênico para estratificação de risco para câncer de mama em mulheres brasileiras. Porto Alegre: SBGM; 2025 [citado 2025 Mar 8]. Disponível em: https://www.sbgm.org.br/detalhe.aspx?id=6379&area=4
» https://www.sbgm.org.br/detalhe.aspx?id=6379&area=4
² An C, Park YW, Ahn SS, Han K, Kim H, Lee SK. Radiomics machine learning study with a small sample size: Single random training-test set split may lead to unreliable results. PLoS One. 2021;16(8):e0256152.
³ Källberg H, Alfredsson L, Feychting M, Ahlbom A. Don't split your data. Eur J Epidemiol. 2010;25(5):283-4.
⁴ Polo TC, Miot HA. Aplicações da curva ROC em estudos clínicos e experimentais. J Vasc Bras. 2020;19:e20200186.
⁵ Brajer N, Cozzi B, Gao M, Nichols M, Revoir M, Balu S, et al. Prospective and external evaluation of a machine learning model to predict in-hospital mortality of adults at time of admission. JAMA Netw Open. 2020;3(2):e1920733.

Associate Editor:
Kenneth Gollob Hospital Israelita Albert Einstein, São Paulo, SP, Brazil ORCID: https://orcid.org/0000-0003-4184-3867

Publication Dates

Publication in this collection
01 Aug 2025
Date of issue
2025

History

Received
8 Mar 2025
Accepted
27 Mar 2025

This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

[1] ¹ Sociedade Brasileira de Genética Médica (SBGM). Posicionamento da SBGM sobre o uso de escores de risco poligênico para estratificação de risco para câncer de mama em mulheres brasileiras. Porto Alegre: SBGM; 2025 [citado 2025 Mar 8]. Disponível em: https://www.sbgm.org.br/detalhe.aspx?id=6379&area=4
» https://www.sbgm.org.br/detalhe.aspx?id=6379&area=4

[2] ² An C, Park YW, Ahn SS, Han K, Kim H, Lee SK. Radiomics machine learning study with a small sample size: Single random training-test set split may lead to unreliable results. PLoS One. 2021;16(8):e0256152.

[3] ³ Källberg H, Alfredsson L, Feychting M, Ahlbom A. Don't split your data. Eur J Epidemiol. 2010;25(5):283-4.

[4] ⁴ Polo TC, Miot HA. Aplicações da curva ROC em estudos clínicos e experimentais. J Vasc Bras. 2020;19:e20200186.

[5] ⁵ Brajer N, Cozzi B, Gao M, Nichols M, Revoir M, Balu S, et al. Prospective and external evaluation of a machine learning model to predict in-hospital mortality of adults at time of admission. JAMA Netw Open. 2020;3(2):e1920733.