Refining predictive risk models for stroke in atrial fibrillation: a scoping review and meta-analysis for Aotearoa New Zealand, Māori and Pacific peoples

Reducing health inequities in Aotearoa New Zealand is a healthcare priority, and a number of strategies have been employed to facilitate this—one example being Health New Zealand – Te Whatu Ora’s Assessment and Management of Cardiovascular Risk, which recommends screening of cardiovascular risk 10 years earlier in Māori than in non-Māori.¹ Such recommendations signal the differing risk profile as determined by age, ethnic and socio-demographic factors, comorbidities, etc. In Aotearoa New Zealand, ethnicity is a significant marker of health needs as highlighted in a recent editorial.² As a result, there is growing interest in developing new risk scores and adapting existing risk stratification tools for the Aotearoa New Zealand context. For example, there is currently a research programme underway called the Vascular Risk Equity in Aotearoa New Zealand (VAREANZ), which uses large-scale cohorts in primary and secondary care settings as a basis for developing vascular risk prediction equations and quality improvement initiatives.³ In this manuscript we will describe the reasons and a strategy for developing/refining risk models using our area of interest as an example—stroke prevention in atrial fibrillation (AF).

The CHA₂DS₂ VASc score (Congestive heart failure/left ventricular dysfunction [1 point], Hypertension [1 point], Age ≥75 [2 points], Diabetes [1 point], previous Stroke/transient ischaemic attack [TIA]/thromboembolism [2 points]—Vascular disease such as ischaemic heart disease [1 point], Age 65–74 [1 point], and female sex [1 point]) is the most common risk score used to predict the risk of thromboembolism in patients with AF.⁴ The annual risk of ischaemic stroke in untreated patients is estimated to be 0.2%, 0.6% and 2.2% for those with a CHA₂DS₂ VASc of 0, 1 and 2 respectively.⁵ Other risk stratification scores exist, such as the Global Anticoagulant Registry in the FIELD—Atrial Fibrillation (GARFIELD-AF) or those that include biomarkers such as Anticoagulation and Risk Factors in Atrial Fibrillation (ATRIA).^6,7 These scores have marginally improved performance in risk prediction; however, this is offset by increased complexity or need for biomarkers. The American Heart Association and European Society of Cardiology recommend the use of CHA₂DS₂ VASc. The consensus view is that ischaemic stroke risk assessment in AF should balance simplicity and practicality against calibration to improve usability in clinical practice.^8,9

Based on 2023 Aotearoa New Zealand Census ethnic data, almost 18% (887,493) report their ethnicity as Māori, the Indigenous people of New Zealand.¹⁰ A further 8.9% (442,632) identify as Pacific peoples. The higher incidence of traditional vascular risk factors (e.g., hypertension, diabetes and obesity) and non-traditional factors (e.g. rheumatic heart disease) may contribute to the higher age-adjusted rates of AF in Māori and Pacific peoples.¹¹ Further, Māori and Pacific peoples are less likely to access healthcare services and may experience disparities in the quality of care they receive.¹² Therefore, comorbidities such as heart failure, diabetes and hypertension may be undetected for a longer time or be suboptimally managed. These factors likely contribute significantly to Māori and Pacific peoples experiencing stroke on average 14 years younger than NZ Europeans (61.5 years vs 75.4 years, p<0.001) based on data from the fifth Auckland Regional Community Stroke Study, 2020–2021 (unpublished data). Further, the prevalence of AF was similar across ethnic groups in this cohort (NZ Europeans: 23.4%, Māori: 24.1%, Pacific peoples: 19.3%), suggesting that the attributable risk of stroke associated with AF varies between ethnic groups.

Many commonly used risk assessment scores available, including CHA₂DS₂ VASc, have been developed and validated in predominantly European cohorts, accounting for the variation in discriminative performance in other settings.^4,13 Given the differences in risk factor prevalence and potential ethnic differences in their impact, it is important to develop ethnic-specific algorithms. However, little is known about the accuracy of stroke risk prediction in Māori and Pacific populations.

In this scoping review, we aim to determine:

1. If stroke risk from AF varies by ethnicity within populations.
2. If the rates of stroke secondary to AF vary by time and geographical locations according to CHA₂DS₂ VASc score.
3. The performance of CHA₂DS₂ VASc across geographical locations.

We will also provide an overview of methodological errors in predictive risk models and optimisation strategies.

Methods

PubMed, Scopus and EMBASE were searched using a mix of the following terms as Medical Subject Heading or its equivalent: atrial fibrillation, ethnicity, race, indigenous, outcomes, stroke, risk prediction, CHA₂DS₂ VASc, model performance, discrimination, c-statistics, anticoagulation. The search strategy focussed on published peer-reviewed English language publications and “grey” literature (including conference abstracts) from December 1995 to November 2024. Titles were screened and, if appropriate, the abstract then main text were reviewed for relevance sequentially. For objective 1, studies were included if the sampled population had AF, were representative of the baseline population (i.e., we excluded studies of sub-group populations, e.g. diabetes, chronic kidney disease, stroke cohorts, post-surgery, etc.) and reported stroke rates stratified by ethnicity. For eligible studies, we conducted a meta-analysis of aggregate data using a random-effects model, anticipating significant heterogeneity between studies. Parameters were estimated using restricted maximum likelihood.¹⁴ For objective 2, cohort studies were included across a range of periods and geographical locations if CHA₂DS₂ VASc was used for stroke risk assessment, the sampling technique was unselected, the size exceeded 5,000 patients and stroke rates were stratified by anticoagulation status. For objective 3, we included cohort studies across geographical regions that reported c-statistics as a performance measure. Studies reporting comparison of cohorts from different countries were not permitted due to the risk of misclassification of variables. When no other option was available, data extraction involved approximations based on the visual data from graphs. Summary tables were created. Statistical analysis was performed using STATA BE17.

Finally, a summary of papers describing methodological errors, statistical analysis and model evaluation is provided. Data were recorded independently. Records retrieved were catalogued in Zotero. Duplicates were removed by automation supplemented by manual checking. This manuscript was written in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) checklist. ¹⁵

Results

View Figure 1–2, Table 1–3.

In our scoping review, most records (1,083/1,162, 93%) were excluded following a brief review of the titles/abstracts due to ineligibility, duplicate manuscripts or only age-adjusted stroke rates reported, which prohibited calculation of absolute rates. Of the remaining 79 manuscripts, a further 52 were removed following review for reasons documented in Figure 1. We found few studies assessing stroke risk in AF in multi-ethnic populations. Included cohort studies were rated medium- to high-quality on the Newcastle Ottawa Scale for assessing quality of non-randomised studies in meta-analysis.¹⁶ Methodological biases included a high proportion of imputed ethnicity (e.g., up to 70% in one study),¹⁷ prevalence rates of stroke/TIA used to determine stroke risk as opposed to follow-up stroke incidence rates (a better marker of causality) and results presented graphically only. Further, some studies reported time-to-event outcomes, whereas we reported binary outcomes, which may have led to imprecision in the data reported.

Objective 1

Our scoping review included relevant manuscripts from the United States, New Zealand and Australia. In the United States, observational data demonstrated an increased risk of stroke in individuals of African American race with AF. ¹⁸ In an attempt to address this, a retrospective cohort study of approximately half a million patients with AF was used to derive a modified AF risk assessment, CHA₂DS₂-VASc-R, which allocated an additional point for African American race. ¹⁹ This improved the predictive ability of the score and the fit of the model. Similarly, a large observational study of young low-risk AF patients in Taiwan found an elevated stroke rate compared with an age-matched cohort without AF. ²⁰ Subsequently, a retrospective cohort study demonstrated improved ischaemic stroke risk prediction in this population with AF using a modified CHA₂DS₂ VASc, which assigned one point for age (50–74 years). ²¹ In this study, a theorised cut-off score of one in males or two in females would yield a net clinical benefit with anticoagulation, with a potential 30% relative risk reduction of ischaemic stroke in patients. Similar findings of elevated stroke risk have been reported in observational studies of Māori, Pacific peoples, Australian Aboriginals and Hispanics with AF compared with other local ethnicities (Table 1). ^{11,17,18,22–33} We performed a meta-analysis using a random-effects model due to high heterogeneity between studies; however, this strategy had limited effect and significant heterogeneity remained (Table 1).

Our meta-analysis found no significant difference in stroke risk due to AF in Māori and Pacific peoples. This may be the result of methodological errors/limitations, and higher-quality prospective studies are required to resolve this. A funnel plot was constructed to assess potential publication bias among the included studies. The plot revealed asymmetry, with smaller studies disproportionately reporting more significant results (Figure 2). However, Egger’s regression test revealed a p>0.05, suggesting no significant asymmetry; hence, the findings of the funnel plot may be due to study heterogeneity. There were more and larger cohort studies assessing stroke risk in African American and Hispanic individuals, in which we found a significantly higher stroke rate (odds ratio [OR] 1.44 [95% confidence interval (CI) 1.25–1.66], p<0.001 and OR 1.11 [95% CI 1.05–1.18], p<0.001 respectively).

Objective 2

We included eight studies across 21 years that reported stroke rates in non-anticoagulated cohorts of over 5,000 patients across geographical locations (Table 2). ^{13,21,34–39} Overall stroke rates ranged from 0.6/100 to 6.8/100 person-years across studies and 0.8/100 person-years to 3.7/100 person-years in those with a CHA₂DS₂ VASc score of 2, confirming significant heterogeneity in stroke risk between populations. Other potential reasons for these differences are described in the discussion.

Objective 3

Model performance can be assessed by determining calibration (how well the model fits the data), discrimination (how well the model distinguishes between those with and without the outcome of interest) and net reclassification index (the net benefit, e.g., when an existing model is updated with a new predictor).⁴⁰ We used c-statistics since this was the most reported measure.

Factors such as population characteristics, study biases and confounders led to differences in model discrimination across settings. In the studies identified, CHA₂DS₂ VASc c-statistic varied from 0.55 (poor discrimination) to 0.80 (excellent discrimination) across locations and periods studied (Table 3).^{13,35,36,38,41–45} This finding has been observed in other risk scores, including GARFIELD-AF.⁴⁶

Summary of findings

In this scoping review, we found moderate- to high-quality evidence from predominantly retrospective observational cohort studies to suggest that stroke risk in AF is modified by ethnicity/race. We found significant findings in African Americans and Hispanics. Though other studies found elevated age-adjusted rates in Māori, Pacific peoples and Australian Aboriginals, we were unable to demonstrate this, likely due to the quality of published studies and eligibility criteria for this scoping review/meta-analysis. These findings may be pertinent to Aotearoa New Zealand, where almost half of Māori and Pacific AF patients aged <65 years are at high risk of stroke compared with 22% of those from other ethnicities, suggesting risk model refinement is desirable.²⁶

We also found variation in stroke risk in different study populations due to inherent differences in study population and study methodology, which also caused similar variation in CHA₂DS₂ VASc model performance. For objective 2, there was insufficient evidence to indicate that improvements in primary prevention have led to reduced stroke rates over time.

We acknowledge that ethnicity is a social construct and, to our knowledge, no ethnicity-based biological mechanisms explain the increased risk of stroke in AF. However, it is also established, particularly in New Zealand, that ethnic differences in outcomes and disparities in treatment stem from health inequities in healthcare delivery and are a critical social determinant of health for reasons discussed above. There may be other important risk factors not yet established. In the REasons for Geographic and Racial Differences in Stroke (REGARDS) cohort, traditional risk factors and socio-economic factors accounted for only one-half of the excess stroke incidence in Blacks, while the remainder was attributed to other unknown factors and pathways.¹⁸ It may also be the case that the relative weight of each predictor within the model varies between ethnicities; based on individual-level data from the South London Stroke Register, hypertension carried a higher population-attributable risk for stroke for individuals of Black Caribbean or African ethnicity.⁴⁷ Another consideration for the elevated stroke risk in African Americans and Hispanics is the lower rates of anticoagulation and rhythm control strategies reported in the literature.⁴⁸

The development, refinement and use of risk scores

Risk models are frequently developed using retrospective observational data from registries or cohort studies since these are cheaper and more convenient. However, the identified predictors and outcomes are associated with selection and measurement bias (limited by the intended use of the database) and missing information. Further, risk models are often derived from insurance claims records, or hospital-based cohorts, which are likely to have higher baseline risk profiles compared with the intended population, a form of confounding by indication. Though data from randomised controlled trials may seem a tempting source to develop risk models, strict inclusion/exclusion criteria limit generalisability. Prospective cohort studies are the preferred method for developing a risk model for these reasons.⁴⁹ Studies have demonstrated significantly different observed outcome rates; for instance, stroke rates from AF increase from an annualised rate of 1.22% in prospective studies to 3.8% in retrospective studies. Unfortunately, prospective studies are potentially expensive and time consuming.⁵⁰

Other methodological errors include heterogeneity in the predictor and outcome definition and the timing of model application. In one systematic review of 33 studies including 151 different predictors, one-third of predictors were categorised at high risk of error.⁵¹ Finally, few risk prediction studies include a sample size calculation and therefore may be under-powered to detect a significant finding. As a result of this heterogeneity, significant variation in observed outcomes can be seen between studies, even in those with supposedly similar risks.

It is accepted that a much better alternative to developing new models in new patient samples is to update existing prediction models and adjust or recalibrate them to the local circumstances or validation sample available.⁴⁰ For stroke prevention in AF, we have highlighted the evidence to suggest that modification of CHA₂DS₂ VASc according to ethnicity or local factors improves risk prediction caused by heterogeneity in predictor effects.^19,21 Given the limitations of risk prediction tools, some organisations including the European Society of Cardiology recommend that clinicians should use locally validated tools and take an individualised approach to thromboembolic risk, considering each patient’s unique factors. The decision to start anticoagulation must balance population-level evidence with patient-specific risks, comorbidities and their risk tolerance.^9,52,53 As such there is an interest in developing predictive risk measures for such purposes. The challenge lies in their development, which we will summarise.

Once the optimal cohort has been identified and predictor and outcome data collected, a multivariable regression analysis determines the strength of associations (coefficients) between predictors and outcomes. These coefficients can refine existing risk models for the intended population or assess the importance of new predictors. The model may then be categorised into risk profiles based on sensitivity, specificity and cost effectiveness. Detailed guidance on risk model development is available, and consulting a biostatistician is advised.^40,54,55

Risk models should be considered geographically and temporally relevant. Over time, vascular event rates may decline due to better risk management. As an example of such “model deterioration”, the Framingham Stroke Risk Score, developed in the 1990s, later over-estimated risk and was withdrawn for ischaemic heart disease prediction.^56,57 Similarly, emerging predictors (e.g., biomarkers) may improve model performance, highlighting the need for regular refinement of risk scores.⁷

There are two important guidelines for the development of a risk model, covering such aspects as defining and selecting predictors, and methods to reduce bias. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) is a guideline for reporting studies developing or validating a prediction model.⁵⁸ The Prediction model Risk Of Bias ASsessment Tool (PROBAST) helps researchers assess the methodology, risk of bias and applicability of a study that develops, validates or updates a diagnostic or prognostic prediction model.⁵⁹

Strengths of this scoping review include the use of a range of databases to obtain relevant studies; the application of strict inclusion/exclusion criteria; and the inclusion of a range of ethnicities. We acknowledge the limitations of scoping reviews, such as potential selection bias from limited databases. The methodology of this review is also subject to limitations. First, some relevant studies may have been excluded due to a lack of sufficient information in the title or abstract to indicate the manuscript met the inclusion/exclusion criteria. Secondly, some important papers could not be included as they reported only age-adjusted rates. Thirdly, inclusion was limited by language as only English texts were considered. Fourthly, some studies reported time-to-event outcomes, whereas we reported binary outcomes, which may have led to imprecision in the data reported. Finally, though the European Society of Cardiology has recently endorsed CHA₂DS₂ VA (i.e., without criterion for birth sex or gender) as the risk score of choice, there have been no studies in an ethnically/racially diverse population to date, and hence this was not included in this study. ⁹

Conclusion

The main results show that risk scores like CHA₂DS₂ VASc, while widely used, may require refinement for populations with unique risk profiles, improving risk prediction caused by heterogeneity in predictor effects. We encourage the refinement of widely accepted risk scores for Aotearoa New Zealand based on our unique population characteristics, as well as the need to update existing risk models. We encourage researchers to consider the potential pitfalls, good practices and available guidelines of model development to improve predictive accuracy and utility.