Data-driven methods distort optimal cutoffs and accuracy estimates of depression screening tools: a simulation study using individual participant data

the Depression Screening Data (DEPRESSD) EPDS Group

doi:10.1016/j.jclinepi.2021.03.031

Data-driven methods distort optimal cutoffs and accuracy estimates of depression screening tools: a simulation study using individual participant data

the Depression Screening Data (DEPRESSD) EPDS Group

Jewish General Hospital
McGill University
Keele University, School of Medicine
University of Calgary
McGill University
McGill University
University of Amsterdam
University of York
Stanford University
Concordia University
Johns Hopkins University School of Medicine
Universite de Montreal
University of Alberta
University of Toronto
Sapienza University of Rome
Fundación Arriaran–Facultad de Medicina Universidad de Chile
Avenida Universidad
Federal Neuropsychiatric Hospital
Birkbeck, University of London
Rajarajeswari Medical College and Hospital
School of Nursing
University Medical Center Hamburg-Eppendorf
University of Sydney
Lithuanian University of Health Sciences
Federal University of Uberlândia
University of Rochester School of Medicine and Dentistry
Federal University of Minas Gerais
University of São Paulo
University of New South Wales
University of Geneva
Mount Carmel Hospital Malta
University of Southampton, Faculty of Medicine
University of Minho
Monash University
Universidad Peruana Cayetano Heredia, Facultad de Medicina Alberto Hurtado
Hospital Clinic
University of Florence
King’s College London
Ahfad University for Women
Danderyd Hospital
Private Practice
Athens University Medical School
Chulalongkorn University
Catholic University of Croatia
University of Tokyo
University of Kinshasa
Albert Schweitzer Ziekenhuis
Halifax Health
University of the Witwatersrand, Johannesburg
Bristol Medical School
Castle Peak Hospital Hong Kong
Uppsala University
University of Oxford
University of Malawi
China Medical University College of Medicine
University of Zagreb
Northwestern University Feinberg School of Medicine
Institute of Mental Health
MTA-SZTE “Lendület” Mycobiome Research Group
San Paolo Hospital
Deakin University
Yale University School of Medicine

Research output: Contribution to journal › Article › peer-review

19 Scopus citations

Abstract

Objective: To evaluate, across multiple sample sizes, the degree that data-driven methods result in (1) optimal cutoffs different from population optimal cutoff and (2) bias in accuracy estimates. Study design and setting: A total of 1,000 samples of sample size 100, 200, 500 and 1,000 each were randomly drawn to simulate studies of different sample sizes from a database (n = 13,255) synthesized to assess Edinburgh Postnatal Depression Scale (EPDS) screening accuracy. Optimal cutoffs were selected by maximizing Youden's J (sensitivity+specificity–1). Optimal cutoffs and accuracy estimates in simulated samples were compared to population values. Results: Optimal cutoffs in simulated samples ranged from ≥ 5 to ≥ 17 for n = 100, ≥ 6 to ≥ 16 for n = 200, ≥ 6 to ≥ 14 for n = 500, and ≥ 8 to ≥ 13 for n = 1,000. Percentage of simulated samples identifying the population optimal cutoff (≥ 11) was 30% for n = 100, 35% for n = 200, 53% for n = 500, and 71% for n = 1,000. Mean overestimation of sensitivity and underestimation of specificity were 6.5 percentage point (pp) and -1.3 pp for n = 100, 4.2 pp and -1.1 pp for n = 200, 1.8 pp and -1.0 pp for n = 500, and 1.4 pp and -1.0 pp for n = 1,000. Conclusions: Small accuracy studies may identify inaccurate optimal cutoff and overstate accuracy estimates with data-driven methods.

Original language	English
Pages (from-to)	137-147
Number of pages	11
Journal	Journal of Clinical Epidemiology
Volume	137
DOIs	https://doi.org/10.1016/j.jclinepi.2021.03.031
State	Published - Sep 2021
Externally published	Yes

Keywords

Accuracy estimates
Bias
Cherry-picking
Data-driven methods
Depression
Optimal cutoff

Access to Document

10.1016/j.jclinepi.2021.03.031

Cite this

@article{ddd71ecc6b6f4414b8e291082ccf0cd6,

title = "Data-driven methods distort optimal cutoffs and accuracy estimates of depression screening tools: a simulation study using individual participant data",

abstract = "Objective: To evaluate, across multiple sample sizes, the degree that data-driven methods result in (1) optimal cutoffs different from population optimal cutoff and (2) bias in accuracy estimates. Study design and setting: A total of 1,000 samples of sample size 100, 200, 500 and 1,000 each were randomly drawn to simulate studies of different sample sizes from a database (n = 13,255) synthesized to assess Edinburgh Postnatal Depression Scale (EPDS) screening accuracy. Optimal cutoffs were selected by maximizing Youden's J (sensitivity+specificity–1). Optimal cutoffs and accuracy estimates in simulated samples were compared to population values. Results: Optimal cutoffs in simulated samples ranged from ≥ 5 to ≥ 17 for n = 100, ≥ 6 to ≥ 16 for n = 200, ≥ 6 to ≥ 14 for n = 500, and ≥ 8 to ≥ 13 for n = 1,000. Percentage of simulated samples identifying the population optimal cutoff (≥ 11) was 30\% for n = 100, 35\% for n = 200, 53\% for n = 500, and 71\% for n = 1,000. Mean overestimation of sensitivity and underestimation of specificity were 6.5 percentage point (pp) and -1.3 pp for n = 100, 4.2 pp and -1.1 pp for n = 200, 1.8 pp and -1.0 pp for n = 500, and 1.4 pp and -1.0 pp for n = 1,000. Conclusions: Small accuracy studies may identify inaccurate optimal cutoff and overstate accuracy estimates with data-driven methods.",

keywords = "Accuracy estimates, Bias, Cherry-picking, Data-driven methods, Depression, Optimal cutoff",

author = "\{the Depression Screening Data (DEPRESSD) EPDS Group\} and Bhandari, \{Parash Mani\} and Brooke Levis and Dipika Neupane and Patten, \{Scott B.\} and Ian Shrier and Thombs, \{Brett D.\} and Andrea Benedetti and Ying Sun and Chen He and Rice, \{Danielle B.\} and Ankur Krishnan and Yin Wu and Marleine Azar and Sanchez, \{Tatiana A.\} and Chiovitti, \{Matthew J.\} and Nazanin Saadat and Riehm, \{Kira E.\} and Mahrukh Imran and Zelalem Negeri and Boruff, \{Jill T.\} and Pim Cuijpers and Simon Gilbody and Ioannidis, \{John P.A.\} and Kloda, \{Lorie A.\} and Ziegelstein, \{Roy C.\} and Liane Comeau and Mitchell, \{Nicholas D.\} and Marcello Tonelli and Vigod, \{Simone N.\} and Franca Aceti and Rub{\'e}n Alvarado and Cosme Alvarado-Esquivel and Bakare, \{Muideen O.\} and Jacqueline Barnes and Bavle, \{Amar D.\} and Beck, \{Cheryl Tatano\} and Carola Bindt and Boyce, \{Philip M.\} and Adomas Bunevicius and \{Castro e Couto\}, Tiago and Chaudron, \{Linda H.\} and Humberto Correa and \{de Figueiredo\}, \{Felipe Pinheiro\} and Valsamma Eapen and Nicolas Favez and Ethel Felice and Michelle Fernandes and Barbara Figueiredo and Fisher, \{Jane R.W.\} and Vega-Dienstmaier, \{Johann M.\}",

note = "Publisher Copyright: {\textcopyright} 2021 Elsevier Inc.",

year = "2021",

month = sep,

doi = "10.1016/j.jclinepi.2021.03.031",

language = "Ingl{\'e}s",

volume = "137",

pages = "137--147",

journal = "Journal of Clinical Epidemiology",

issn = "0895-4356",

}

TY - JOUR

T1 - Data-driven methods distort optimal cutoffs and accuracy estimates of depression screening tools

T2 - a simulation study using individual participant data

AU - the Depression Screening Data (DEPRESSD) EPDS Group

AU - Bhandari, Parash Mani

AU - Levis, Brooke

AU - Neupane, Dipika

AU - Patten, Scott B.

AU - Shrier, Ian

AU - Thombs, Brett D.

AU - Benedetti, Andrea

AU - Sun, Ying

AU - He, Chen

AU - Rice, Danielle B.

AU - Krishnan, Ankur

AU - Wu, Yin

AU - Azar, Marleine

AU - Sanchez, Tatiana A.

AU - Chiovitti, Matthew J.

AU - Saadat, Nazanin

AU - Riehm, Kira E.

AU - Imran, Mahrukh

AU - Negeri, Zelalem

AU - Boruff, Jill T.

AU - Cuijpers, Pim

AU - Gilbody, Simon

AU - Ioannidis, John P.A.

AU - Kloda, Lorie A.

AU - Ziegelstein, Roy C.

AU - Comeau, Liane

AU - Mitchell, Nicholas D.

AU - Tonelli, Marcello

AU - Vigod, Simone N.

AU - Aceti, Franca

AU - Alvarado, Rubén

AU - Alvarado-Esquivel, Cosme

AU - Bakare, Muideen O.

AU - Barnes, Jacqueline

AU - Bavle, Amar D.

AU - Beck, Cheryl Tatano

AU - Bindt, Carola

AU - Boyce, Philip M.

AU - Bunevicius, Adomas

AU - Castro e Couto, Tiago

AU - Chaudron, Linda H.

AU - Correa, Humberto

AU - de Figueiredo, Felipe Pinheiro

AU - Eapen, Valsamma

AU - Favez, Nicolas

AU - Felice, Ethel

AU - Fernandes, Michelle

AU - Figueiredo, Barbara

AU - Fisher, Jane R.W.

AU - Vega-Dienstmaier, Johann M.

PY - 2021/9

Y1 - 2021/9

N2 - Objective: To evaluate, across multiple sample sizes, the degree that data-driven methods result in (1) optimal cutoffs different from population optimal cutoff and (2) bias in accuracy estimates. Study design and setting: A total of 1,000 samples of sample size 100, 200, 500 and 1,000 each were randomly drawn to simulate studies of different sample sizes from a database (n = 13,255) synthesized to assess Edinburgh Postnatal Depression Scale (EPDS) screening accuracy. Optimal cutoffs were selected by maximizing Youden's J (sensitivity+specificity–1). Optimal cutoffs and accuracy estimates in simulated samples were compared to population values. Results: Optimal cutoffs in simulated samples ranged from ≥ 5 to ≥ 17 for n = 100, ≥ 6 to ≥ 16 for n = 200, ≥ 6 to ≥ 14 for n = 500, and ≥ 8 to ≥ 13 for n = 1,000. Percentage of simulated samples identifying the population optimal cutoff (≥ 11) was 30% for n = 100, 35% for n = 200, 53% for n = 500, and 71% for n = 1,000. Mean overestimation of sensitivity and underestimation of specificity were 6.5 percentage point (pp) and -1.3 pp for n = 100, 4.2 pp and -1.1 pp for n = 200, 1.8 pp and -1.0 pp for n = 500, and 1.4 pp and -1.0 pp for n = 1,000. Conclusions: Small accuracy studies may identify inaccurate optimal cutoff and overstate accuracy estimates with data-driven methods.

AB - Objective: To evaluate, across multiple sample sizes, the degree that data-driven methods result in (1) optimal cutoffs different from population optimal cutoff and (2) bias in accuracy estimates. Study design and setting: A total of 1,000 samples of sample size 100, 200, 500 and 1,000 each were randomly drawn to simulate studies of different sample sizes from a database (n = 13,255) synthesized to assess Edinburgh Postnatal Depression Scale (EPDS) screening accuracy. Optimal cutoffs were selected by maximizing Youden's J (sensitivity+specificity–1). Optimal cutoffs and accuracy estimates in simulated samples were compared to population values. Results: Optimal cutoffs in simulated samples ranged from ≥ 5 to ≥ 17 for n = 100, ≥ 6 to ≥ 16 for n = 200, ≥ 6 to ≥ 14 for n = 500, and ≥ 8 to ≥ 13 for n = 1,000. Percentage of simulated samples identifying the population optimal cutoff (≥ 11) was 30% for n = 100, 35% for n = 200, 53% for n = 500, and 71% for n = 1,000. Mean overestimation of sensitivity and underestimation of specificity were 6.5 percentage point (pp) and -1.3 pp for n = 100, 4.2 pp and -1.1 pp for n = 200, 1.8 pp and -1.0 pp for n = 500, and 1.4 pp and -1.0 pp for n = 1,000. Conclusions: Small accuracy studies may identify inaccurate optimal cutoff and overstate accuracy estimates with data-driven methods.

KW - Accuracy estimates

KW - Bias

KW - Cherry-picking

KW - Data-driven methods

KW - Depression

KW - Optimal cutoff

UR - https://www.scopus.com/pages/publications/85105570449

U2 - 10.1016/j.jclinepi.2021.03.031

DO - 10.1016/j.jclinepi.2021.03.031

M3 - Artículo

C2 - 33838273

AN - SCOPUS:85105570449

SN - 0895-4356

VL - 137

SP - 137

EP - 147

JO - Journal of Clinical Epidemiology

JF - Journal of Clinical Epidemiology

ER -

Data-driven methods distort optimal cutoffs and accuracy estimates of depression screening tools: a simulation study using individual participant data

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this