Agreement between self-reported fractures in a clinical trial with New Zealand Accident Compensation Corporation claims data

Clinical studies that use routinely gathered data from registries or insurance claims are becoming increasingly popular as more and more clinical data are stored electronically and some databases may be easier to access.¹ These data have well described limitations when they are repurposed for use in clinical research.¹ The use of registry data has been touted as a disruptive technology that will transform clinical trials,² but, although the potential is acknowledged, there are numerous challenges with the conduct and interpretation of such trials.³

We recently completed a 10-year randomised, placebo-controlled trial of zoledronate given once or every 5 years in 1,054 women aged 50–60 years at baseline in Auckland.⁴ During the study, participants informed us when they had a fracture, either directly or as part of routine 6-monthly questionnaires, and we obtained the relevant radiology report or imaging and confirmed the presence or absence of a fracture. Thus, data were obtained for verified self-reported fractures for these 1,054 women for up to 10 years.

We planned a cost-effectiveness analysis to complement the clinical trial results and therefore obtained Accident Compensation Corporation (ACC) claims data for fractures for the 1,054 women over the 10-year period to estimate the total cost of each fracture event. ACC is the national New Zealand no-fault accident claims organisation which is funded through levies. There were marked differences between the number of women with a verified self-reported fracture and an ACC fracture claim. The primary aim of this study was to assess agreement between self-reported fractures in a clinical trial with the administrative claims dataset covering the same study sample from ACC. The secondary aim was to determine whether there were differences in treatment estimates for the efficacy of zoledronate on fracture incidence for self-reported verified fractures from the clinical trial and fractures reported in the ACC claims database.

Methods

The study and its protocol have been published in full.⁴ The relevant parts are that 1,054 women aged 50–60 years were enrolled into the 10-year study and 1,003 (95%) completed 10 years of follow up. They were asked to contact us if they had a fracture or symptoms consistent with a fracture. Every 6 months they were sent a questionnaire which asked whether they had had a recent fracture. At each participant’s in-centre visits (at 5 and 10 years) fracture details were checked for the entire duration of their study participation. Once a fracture was reported in any of these ways, we obtained the radiology report (most often an X-ray report), and/or the relevant imaging when necessary, and an investigator (MB) confirmed whether there was a fracture or not. Thus, a self-reported fracture was verified when either examination of the imaging or a radiologist report confirmed it. All trial fractures (but no morphometric vertebral fractures) were included in these analyses.

After the last participant had completed their 10-year visit, we applied to ACC for fracture claims data for the 1,054 women for the time-period they were in the study. Ethical approval to do this was obtained from the Northern A Health and Disabilities Ethics Committee and participants gave written informed consent. As our primary interests were cost data, we sought ACC data on the date and details of the event, the diagnosis, Read codes, International Classification of Diseases (ICD) codes and all relevant costs. Data were linked through the National Health Index (NHI) provided to ACC by us, together with the first and last visit dates for each participant in the study. It took 172 days from initial request to ACC until approval for data access was granted and an agreement signed, and less than 1 month from provision of the list of NHI identifiers to receipt of claims data.

Initially, we only sought the ACC data for all events with an ACC classification of “fracture”. However, it quickly became apparent that there was a marked difference between the number of events obtained from ACC data and the self-reported verified events during the study. After a discussion with and permission from ACC, we requested and obtained data for all claims made by participants during the study period, to determine whether we could resolve the discrepancies. Once the full set of claims was obtained, for each person we reviewed all dates and descriptions for ACC claims for fracture, and all ACC claims up to 7 days before or after a self-reported fracture, in an attempt to match an ACC claim to a self-reported fracture. The choice of 7 days was a pragmatic one, in an attempt to capture all verified fractures without reviewing every ACC claim.

One clinician (MB) searched the ACC dataset, identified ACC claims within the 7-day period of the fracture, reviewed all information in the ACC dataset about each claim, including the accident cause, ACC diagnosis, accident description, Read code, and ICD-9 and ICD-10 codes. Based on the information provided, each claim was categorised as consistent with or not consistent with the fracture. Where there were multiple claims, each claim was assessed independently, but only one claim was ascribed to each fracture. Sixteen women had fracture events in which they had more than one fracture (25 total fractures). For these events, we treated the fractures separately. All 16 events had a matching claim (11 classified as fracture, 5 other classification). The matching ACC claim was applied to all the fractures in the event.

Statistical analysis

Proportions are presented as counts and percentages. Relative risks and 95% confidence intervals (CIs) are presented for the risk of at least one fracture with zoledronate compared to controls. All analyses were done using R 4.42 (R Core Team, 2024, R Foundation for Statistical Computing).

Results

In the trial, 356 self-reported fractures in 248 women were verified (“trial fractures”). In the initial ACC request for fracture claims, there were 328 claims from 238 women for the period of the study that were classified as fracture: “ACC fracture claims” (note these are classified as “fracture/dislocation” by ACC). Table 1 shows that there were large numbers of both trial fractures without a matching ACC fracture claim and ACC fracture claims without a matching trial fracture. Two hundred and eleven out of 356 (59%) trial fractures had a matching ACC fracture claim, and 211/328 (64%) ACC fracture claims had a matching trial fracture.

View Table 1–4.

When we obtained all ACC claims for all participants during the study time frame, 976 women made 5,897 ACC claims. After manually matching ACC claims with each trial fracture, we identified a matching ACC claim for 340 of the 356 trial fractures (96%). Table 2 shows the details of the matching ACC claims for all the trial fractures and by the randomised treatment group. The proportion of matching ACC claims were similar across treatment groups. Table 3 shows the number of women with at least one trial fracture by treatment group classified by any match with an ACC claim. When we compared the pooled zoledronate groups with the control group (that is zoledronate-zoledronate [ZZ] and zoledronate-placebo [ZP] vs control [placebo-placebo, PP], Table 3), the relative risk (95% CI) for a first fracture was 0.80 (0.64–1.0, 248 women with first trial fractures, 153/703 vs 95/351), 0.86 (0.64–1.16) for matching ACC fracture claim with first trial fracture (158 women, 100/703 vs 58/351), and 0.87 (0.69–1.09) for first ACC fracture claims (238 women, 151/703 vs 87/371).

Fifty-nine percent of fractures had a matching ACC fracture claim, 36% a non-fracture claim and 4% no matching claim. The most common non-fracture matching claim was soft-tissue injury (31%), followed by laceration/puncture/sting (4%). There were some differences in the proportions of ACC claims by treatment group, but they were usually only small. Under the assumptions that the verified self-reported trial fractures are the gold standard, that no women with an ACC fracture claim without a reported trial fracture actually had a fracture, and women with a trial fracture with an ACC claim for something other than fracture are treated as not having an ACC fracture claim, the sensitivity, specificity, positive predictive value and negative predictive value for an individual having at least one ACC claim for fracture, are 0.64 (95% CI 0.57–0.70), 0.92 (95% CI 0.90–0.94), 0.71 (95% CI 0.65–0.77) and 0.89 (95% CI 0.87–0.91) respectively.

Table 4 shows the individual trial fracture type with the related ACC claim type. Hip, femur, arm, shoulder and wrist fractures all had concordance of >85%. Knee, leg, tarsal, rib, sternum and spine all had <40% concordance. It is possible that axial/central fractures are more likely to have an ACC fracture claim, but there were obvious outliers (e.g., metacarpal 78%, spine 37%).

Discussion

In this clinical trial where data on fractures were systematically gathered and verified, only 59% of events had a matching ACC claim classified as a fracture, although 96% of events did have a matching ACC claim. Soft-tissue injury claims were made for 31% of verified fractures. Conversely, 64% of ACC fracture claims were associated with a fracture event during the study, but 36% had no associated verified fracture. The proportions of verified fractures with matching fracture or non-fracture claims were similar across treatment groups. There was no clear pattern in fracture types to explain the differences between the ACC claim classifications. The main end points of the trial were women with at least one fracture or fracture type. For all fractures, relying on matching ACC claims and verified fractures or ACC claims alone reduced the estimate of treatment efficacy and widened the uncertainty in the estimate.

These data indicate that repurposing ACC fracture claims data for a clinical trial has significant limitations. It is likely that a number of fractures may be missed by relying solely on ACC data (false negatives) and also that a number of events may be misclassified as a fracture (false positives). Collectively, this may not introduce a directional bias in the results of a clinical trial, but it will introduce error and noise and potentially introduce a bias towards the null (as seen in these current analyses). Error in the assessment of fracture may lead to a true effect being obscured because a larger sample size is required to identify any differences due to the treatment. In our trial of 1,054 women followed for 10 years, there were 356 fracture events in 248 women. The relative risk was lower and the CIs smaller (indicating more precise estimates) for treatment effects of zoledronate in the analysis of verified fractures compared to analyses of the ACC claims data, whether utilising all fracture claims or just those with a matching verified fracture. If there are relatively few events, adding noise could lead to substantial departure from the true treatment estimate in either direction, simply by chance. Thus, for a trial with relatively few events, the use of ACC fracture data alone as the source of major fracture outcomes does not seem appropriate.

One way around this would be to obtain both fracture and soft-tissue injury ACC claims data. In this case, 92% of verified fractures would have been identified, but there would be a very large number of soft-tissue injury claims without fracture, biasing any analyses toward the null. Retrospective adjudication of the extensive number of claims would be impractical and prohibitively time-consuming.

We do not know why many fractures were classified in the ACC claim as soft-tissue injury. We searched for similar information about misclassification of injuries in ACC claims but were unable to locate any relevant research. It is possible that the person who completed the ACC form chose what appeared to be the best diagnosis at the time: some fractures may not have been apparent or may have been a lesser injury compared to the presenting problem.

It took close to 6 months from submission of the request to ACC until a contract was signed to obtain the data. ACC staff said the initial delay of 4 months was caused by a gap in their triaging process. Once regular contact was established the process was smooth, and it took about 2.5 months until the final data were obtained. Others have described the length of time to access data from various New Zealand registries, which was 441 days from enquiry to data receipt for ACC.⁵ Researchers planning to utilise ACC data should factor such durations into their study design.

If the clinical trial is large and many participants and events are anticipated, it is possible that the noise from misclassification is only small compared to the treatment effect and it may have little impact on the treatment effect estimate or its precision. We are aware of at least one clinical trial involving >5,000 participants with extended follow-up of up to 10 years that used ACC data, at least in part, for the outcome of fractures.^6,7 The authors reported that nearly half the fracture data came from ACC in their 10-year analysis (491/1,016 participants), but they did not report an analysis by initial randomisation, and we are unaware as to whether they compared ACC data with that from other sources such as self-reports or hospital discharge data.

For non-randomised clinical research the same caveats apply. For research where there are likely to be a relatively small number of events and it is critical that each event is accurately classified, the use of ACC fracture data to determine the fracture outcome seems unwise. However, in very large research studies, for example assessing fracture outcomes for the entire population or assessing secular trends in fractures, it may be that misclassification will not introduce differential bias for most analyses and so ACC fracture claims data could be a very valuable data source for analyses.

This research has limitations. This was a single clinical trial carried out in a single centre in a highly motivated population of early post-menopausal women. Results may be different in different populations. It is possible that some women did not self-report fractures during the trial, despite the multiple prompts to do so. We did not contact participants to find out more information about the ACC fracture claims that lacked a matching fracture event and so it is possible that some of these claims were indeed fractures that were missed. However, we do not think it is plausible that 105 women (10% of the study population) failed to report 117 fractures.

In summary, while using routinely gathered data for clinical trials has recently been promoted enthusiastically, the use of ACC fracture claims data alone for clinical trials is probably not wise, unless there is substantial tolerance for misclassification of events in the study design. On the other hand, when precise individual data are not essential, such as in very large non-randomised clinical studies, ACC fracture claims data could be very useful since about 60% of ACC fracture claims had a verified fracture event. For “major” fractures, such as wrist, shoulder, pelvis, hip and femur, the proportions were higher still, but for some of these categories there were few events, and confirmation of the high concordance in different populations would be useful.