Measurement of the Inter-Rater Reliability Rate Is Mandatory for Improving the Quality of a Medical Database: Experience with the Paulista Lung Cancer Registry

Autor(es)

Lauricella, Leticia L. ^{[1, 2]} ; Costa, Priscila B. ^[2] ; Salati, Michele ^[3] ; Pego-Fernandes, Paulo M. ^{[2, 4]} ; Terra, Ricardo M. ^{[1, 2]}

Afiliação do(s) autor(es)

^[1] Inst Canc Estado Sao Paulo, Sao Paulo - Brazil

^[2] Univ Sao Paulo, Fac Med, Thorac Surg Div, 44 Dr Eneas de Carvalho Aguiar Ave, B 2, Room 9, BR-05403000 Sao Paulo, SP - Brazil

^[3] Univ Ancona, Sch Med, Ancona - Italy

^[4] Univ Sao Paulo, Fac Med, Hosp Clin, Thorac Surg Div, Inst Coracao, Sao Paulo - Brazil

• Tipo de documento: Artigo Científico
• Fonte: JOURNAL OF THE AMERICAN COLLEGE OF SURGEONS; v. 226, n. 6, p. 1128-1136, JUN 2018.
• Citações Web of Science: 0

Resumo

BACKGROUND: Database quality measurement should be considered a mandatory step to ensure an adequate level of confidence in data used for research and quality improvement. Several metrics have been described in the literature, but no standardized approach has been established. We aimed to describe a methodological approach applied to measure the quality and inter-rater reliability of a regional multicentric thoracic surgical database (Paulista Lung Cancer Registry). STUDY DESIGN: Data from the first 3 years of the Paulista Lung Cancer Registry underwent an audit process with 3 metrics: completeness, consistency, and inter-rater reliability. The first 2 methods were applied to the whole data set, and the last method was calculated using 100 cases randomized for direct auditing. Inter-rater reliability was evaluated using percentage of agreement between the data collector and auditor and through calculation of Cohen's kappa and intraclass correlation. RESULTS: The overall completeness per section ranged from 0.88 to 1.00, and the overall consistency was 0.96. Inter-rater reliability showed many variables with high disagreement (>10%). For numerical variables, intraclass correlation was a better metric than inter-rater reliability. Cohen's k showed that most variables had moderate to substantial agreement. CONCLUSIONS: The methodological approach applied to the Paulista Lung Cancer Registry showed that completeness and consistency metrics did not sufficiently reflect the real quality status of a database. The inter-rater reliability associated with k and intraclass correlation was a better quality metric than completeness and consistency metrics because it could determine the reliability of specific variables used in research or benchmark reports. This report can be a paradigm for future studies of data quality measurement. (C) 2018 by the American College of Surgeons. Published by Elsevier Inc. All rights reserved. (AU)

• Processo FAPESP: 12/51764-8 - Implementação do registro paulista de tratamento cirúrgico do câncer de pulmão
• Beneficiário: Ricardo Mingarini Terra
• Modalidade de apoio: Auxílio à Pesquisa - Pesquisa em Políticas Públicas para o SUS