INTRODUCTION
Acute abdominal pain is one of the most common patient complaints, and it accounts for 11% to 12% of all patients who visit the emergency department (ED) each year [1]. Performing a proper differential diagnosis is critical for the management of these patients, and abdominal computed tomography (CT) is often essential to this process. CT has been shown to be superior to other diagnostic methods for patients presenting to the ED with trauma, acute abdominal pain, or for geriatric patient evaluation [2-4]. Several studies have since reported the increased utilization of CT in the adult ED [5-7].
Today, it has become more important than ever to interpret a CT scan accurately given the clinical repercussions of its use; CT is a high-value diagnostic tool that is used with an increased frequency and has a potential impact on patient care and disposition. However, many hospitals do not always have a radiologist available. Therefore, the ability of emergency physicians to interpret CT findings appropriately is critical in the patient care process, and efforts to reduce interpretation errors are necessary. Identifying factors that affect incorrect interpretations of abdominal CT and addressing them in educational programs may help improve the interpretation capability of physicians in training.
Some studies have reported on discrepancies in interpretation between radiology specialists and residents of other departments, but there has been no research on a checklist-based CT learning program for residents. This study aimed to identify the factors that are mainly associated with the incorrect interpretation of abdominal CT by novice postgraduate year-1 (PGY-1) residents in a checklist-based learning program [8,9]. We also analyzed the characteristics of incorrectly interpreted CT.
METHODS
Study design and setting
A convenience sample of patients who visited the ED and underwent abdominal CT from March 2016 to February 2018 was included in this retrospective analysis of a prospective observational cohort. During this study period, a checklist-based CT learning program for PGY-1 residents had been in place in the EDs of three tertiary academic hospitals (Appendix 1). The checklists were revised over time by resident educational staff during the study period [8,9] based on the frequencies and diagnoses of the CT scans performed in the ED. The checklist was completed by the PGY-1 residents in the checklist-based CT learning program immediately after the CT scan. The three hospitals operate EDs with approximately 50,000 to 60,000 patient visits per year and run a 4-year resident training program, during which time the residents circulate between each hospital. This study conformed with the ethical norms and standards as set out in the Declaration of Helsinki. The institutional review board of Korea University Ansan Hospital approved the study (AS15030). The requirement for informed consent was waived by the board, as this study was performed as part of an education program.
Study protocol
We included patients who visited the ED and underwent an abdominal CT examination, as ordered by a PGY-1 resident, during their ED stay. CT images were excluded if interpretations from both a board-certified radiologist and an ED PGY-1 resident were not available. Patients who visited the ED for trauma, those who underwent imaging as part of a routine disease follow-up for the evaluation of postoperative complications, and patients under 15 years of age were also excluded from the analysis. Fig. 1 shows a schematic flow chart of the study. The CT protocols ranged from non-contrast to intravenous contrast-enhanced three-phase scans depending on the clinically suspected disease entity. CT scanners with 64 or 256 detector rows were used.
Outcome measures
A trained researcher (registered nurse) entered the results from the checklists recorded by the PGY-1 and the radiologists’ reports into a database. Variables in the database included patients’ age, sex, time of interpretation (on-duty vs. off-duty; on-duty time was defined as 8 a.m. to 5 p.m., weekday vs. weekend), reason for the visit, chief complaints, suspected disease, and interpretations by the PGY-1 resident and radiologist. Two board-certified emergency physicians independently assessed the discrepancies between the interpretations of PGY-1 residents and radiologists. The assessors also evaluated whether a resident’s interpretation was clinically acceptable, despite not being identical to that of a radiologist. When the assessors were not in agreement, the decision was discussed among the assessors until a consensus was reached.
The assessors determined the interpretations as either correct or incorrect. Interpretations were judged to be correct if both were identical and clinically acceptable, while any other case was defined as incorrect. Incorrect interpretations were further divided into three subgroups: false negative, false positive, and misinterpretation. A false negative was defined as interpreting an abnormal CT image as normal, and a false positive was defined as interpreting a normal CT image as abnormal. Misinterpretation was defined as the PGY-1 resident correctly identifying that the CT was abnormal but incorrectly identifying the pathological lesion.
Data analysis
The distributions of categorical data are presented as percentages. Continuous variables that were not normally distributed were presented as medians with interquartile ranges. Categorical variables were compared using the chi-squared test, and continuous variables were evaluated using the Wilcoxon rank sum test. To assess inter-rater agreement, Cohen’s kappa coefficient was calculated; its value ranged from 0 (perfect discordance) to 1 (perfect accordance). Multivariable logistic regression analysis was performed to evaluate the factors associated with incorrect interpretation; the model was adjusted for potential confounders such as patient age, sex, time of interpretation, and organ systems. Data analyses were performed using RStudio ver. 3.5.1 (RStudio Inc., Boston, MA, USA), and P-values <0.05 were considered statistically significant.
RESULTS
Baseline characteristics of enrolled patients
During the study period, 2,202 abdominal CT scans were performed in the ED. Of these, several were excluded because checklists were not completed (n=36), scans were not formally read by radiology specialists (n=256), or they involved cases of patients with trauma, patients who underwent routine disease follow-up, or patients who were under 15 years of age (total, n=282). Ultimately, 1,628 checklists completed by PGY-1 emergency residents were included in the analysis (Fig. 1).
Among the 1,628 CT scans, 1,358 (83.4%) were assessed as correct and 270 (16.6%) were incorrect. The incorrect interpretation rates for female and male patients were 19.6% (158/808) and 13.7% (112/820), respectively. Interpretation accuracy varied widely across organ categories (Fig. 2). The biliary tract is the organ system that was most frequently incorrectly interpreted (28.4%, 48/169). The second highest incorrect interpretation rate occurred in cases of no abnormal organs (28.2%, 96/340). The urinary system was the most correctly interpreted organ system (95.8%, 365/381).
There were no significant differences between the two groups regarding characteristics (age, duty time [on-duty vs. off-duty], and day of the week [weekday vs. weekend or holiday]). PGY-1 residents also displayed non-significant heterogeneity in their CT interpretation ability (Table 1).
Characteristics of incorrect interpretation
Table 2 shows the characteristics of incorrect interpretation according to organ categories. In the misinterpretation subgroup, the organ system incorrectly interpreted most frequently was the biliary tract (28.9%, 43/149). Hollow viscus disease had the highest proportion of incorrect interpretations in both the false negative (24.0%, 6/25) and false positive (32.3%, 31/96) groups.
Factors affecting the accuracy of novice emergency residents’ CT interpretations
Multivariable logistic regression analysis, which was used to analyze the factors that affected incorrect interpretations, showed that organ categories were the major determinants of incorrect interpretation (Fig. 3). Patient age, sex, and the time of interpretation did not affect the interpretation status.
In descending order, the statistically significant adjusted odds ratios for incorrect interpretation by organ systems are as follows: biliary tract, no abnormal organs, liver, genital system, hollow viscus, and pancreas, with the urinary system as the reference.
DISCUSSION
Biliary tract disease was found to be a major factor associated with the incorrect preliminary interpretation of abdominal CTs, among various confounders such as patient age, sex, and time of interpretation. We did not find any association between incorrect interpretations and duty time or weekend duty. An accurate and confident interpretation of normal CT images must be ensured for high-quality emergency care. However, ED residents showed high false-positive interpretation rates of normal CT images.
Although the use of CT as a diagnostic tool has increased because of its accessibility and convenience within the ED, there are few educational programs that help novice emergency physicians improve their interpretation capability. We developed a checklist-based learning program, revised over time, to enable novice emergency residents to interpret CT images easily [8,9].
Diagnostic errors are among the main causes of medical malpractice, and they can lead to adverse outcomes such as treatment delay, maltreatment, unnecessary use of medical resources, and patient disability or death [10,11]. As diagnostic technologies evolve, the CT scan has been shown to have great advantages in the ED, such as an increased level of diagnostic certainty and use in early decision-making and formulation of treatment plans [12]. The ability to interpret abdominal CT images correctly is important for accurate and rapid decision-making in the ED, where highly heterogeneous patients, including mild to critically ill, trauma, pediatric, and geriatric patients, visit simultaneously.
Most previous studies examined the discrepancies in interpretation between radiology specialists and radiology residents. The overall discrepancy rate varied from 1% to 10% depending on the type of modality (CT or magnetic resonance imaging) and body parts scanned [13-17]. In our study, the discrepancy rate between preliminary interpretations by novice emergency residents and the final radiologists’ reports was 16.6%, which was higher than that of previous studies. This is most probably due to differences in the experience and level of education of the residents included in the study with respect to CT interpretation.
False-positive interpretation rates were also higher in our study than in previous studies [17]. This could be explained by several possible reasons, one of which might be the residents’ experience levels. Resident learner feedback revealed that PGY-1 residents are often not fully confident about their interpretations and show a tendency to fill in the checklist with any findings related to the chief complaint. Another possible reason is that emergency physicians are clinicians who encounter patients firsthand. Therefore, they are prone to interpreting diagnostic images with the underlying assumption that there must be a pathologic lesion to be found. Radiologists, on the other hand, often take an objective approach to image interpretation. This result shows that the ability to identify normal CT images, an aspect that is often overlooked in emergency care, requires improvement and is a potential training goal for novice emergency residents. False-positive interpretations can lead to unnecessary laboratory tests, incorrect disposition, and increased ED stay.
Ruutiainen et al. found that major discrepancies increased significantly during the final two hours of consecutive overnight call shifts and that this finding could be related to either fatigue or circadian desynchronization [18]. However, in our study, we did not find any statistically significant relationship between incorrect interpretations and shift time or holiday and weekend duty.
This study has several limitations. First, the degree of subjectivity in the assessment of checklists cannot be ruled out. One of the most important prerequisites of this study is consistency among assessors. Although a high agreement rate of judgment supports the objectivity of our outcome measure, we cannot completely discount the subjective nature of human assessment. To mitigate this, we analyzed 100 pilot test cases and conducted extensive discussions. There was a 91.5% agreement in the assessment with substantial inter-rater reliability (kappa=0.75; 95% confidence interval, 0.70–0.80) [19]. Second, the checklist alone may not always accurately reflect the residents’ interpretation abilities. The checklist was developed to have a simple and focused format for user convenience, compliance, and rapid completion. Therefore, it may not be appropriate to judge PGY-1 residents’ image interpretation skills based on checklist accuracy alone. However, it is a quick, objective assessment tool, and its data can be coded and compared against radiologists’ reports. Third, for abdominal CT interpretations, there may be differences in interpretation results depending on the residents’ training periods [8]. However, the monthly analysis could not be performed for the residents in this study. Fourth, whether the CT scan was enhanced with contrast could affect residents’ preliminary interpretation. However, the number of cases of non-enhanced CT in our study was very small, so the presence of contrast was not included as a study variable. Finally, ED patient characteristics may vary by region, culture, insurance, institute, and country, thereby reducing the generalizability of our findings.
In conclusion, the presence of biliary tract disease is a major factor associated with the incorrect preliminary interpretation of abdominal CT scans by PGY-1 ED residents. Furthermore, no association was found between incorrect interpretations and the time of interpretation. Additionally, PGY-1 residents showed high false-positive interpretation rates for normal CT images. These are two areas on which emergency residents’ training should focus to improve abdominal CT interpretation accuracy and the quality of emergency care.
PDF Links
PubReader
ePub Link
XML Download
Full text via DOI
Download Citation 
Print 
