The readability of online laryngectomy patient information: how do Australia and New Zealand compare?

The functions of the larynx are producing speech, airway patency and protecting the lungs from aspiration. Carcinoma of the larynx is one of the most common head and neck cancers, and was estimated to affect 592 Australians in 2013.¹ Because of its important functions, laryngectomy is associated with a significant impact on quality of life and psychosocial wellbeing.² The surgery will negatively impact speech, swallow function and body image; thus, patient education is an extremely important part of pre-operative counselling and informed consent.³ An important part of the education process, not only for patients but also for friends and relatives who might not be able to be a part of in-person education, is website materials.

Patient education prior to surgery is a requirement of pre-operative decision making, which can come from many sources, including face-to-face education, written handout material and internet-based resources. Over 50% of patients search for resources online prior to making the decision to proceed with an elective surgery.⁴

Numerous websites from Australia, New Zealand and worldwide give varying degrees of information surrounding laryngectomy, including on surgery, recovery and long-term outlook.

Aim

The aim of our research is to analyse the readability of Australian– and New Zealand–produced patient education materials that are freely available online. Additionally, we aimed to compare these to internationally produced materials.

Methods

Article selection

Searches were undertaken in Melbourne, Australia in November 2023. The search engine “Google Chrome” was used, and location services were turned off. The internet browser cookies, cache and history were cleared, and an incognito window was used.

Searches were performed using the terms “laryngectomy”, “laryngectomy patient information”, “voice box removal” and “voice box removal patient information”.

The first 50 search results from Australian/New Zealand and international websites (excluding advertisements or sponsored results) for each term were screened for applicability and duplicates excluded.

Inclusion criteria:

• English language
• Written material
• Material aimed at patients

Exclusion criteria:

• Non-English websites
• Video or diagram materials
• Materials aimed at healthcare providers

Twenty-six websites were included for analysis from the above search structure.

A further search was conducted to include only websites published in Australia/New Zealand. The above search terms (“laryngectomy”, “laryngectomy patient information”, “voice box removal” and “voice box removal patient information”) were again used, and the first 50 search results were reviewed. The inclusion and exclusion criteria were not changed. Of the 50 websites, eight were deemed appropriate for inclusion. Of these eight websites, five were included in the previous 26 websites. Thus, a total of 29 websites (eight from Australia/New Zealand and 21 from overseas) were included in the analysis. See the Appendix for further information.

Readability analysis

The primary outcomes measures were the Flesch–Kincaid Grade Level, the Flesch Reading Ease Score, the Gunning Fog Index, the Coleman–Liau Index, the SMOG (Simple Measure of Gobbledygook) Index, the Automated Readability Index and the Linsear Write Formula.

The body of each text (not including author information, references, titles and advertisements) was copied into the automatic score generators and a score derived.

Flesch–Kincaid Grade Level, Gunning Fog Index, Coleman–Liau Index, SMOG Index, Automated Readability Index and Linsear Write Formula

These measures are tools used to estimate how accessible a body of text is to a person with a certain level of school education.⁵ They use characteristics of the text, such as the number of words in a sentence, the length of words used, the number of syllables in words, average letters per 100 words and average number of sentences per 100 words to estimate the difficulty of reading the text and comprehending the message of the text.^5,6

These scores are based on years of education completed, where a score of 6 represents an article that someone with 6 years of education (i.e., the 6th Grade, between 11 and 12 years old) can understand and a score of 7 represents an article that someone educated for 7 years (i.e., the 7th Grade, between 12 and 13 years old) can understand. ^7,8 A score of above 12 indicates that the passage of text requires more than a high school education to be able to read and comprehend the text.

Flesch Reading Ease Score

The Flesch Reading Ease Score is a reliable and valid tool that determines the readability of a body of text by giving it a score out of 100.^9,10 The higher the number, the easier the text is to read.⁹ Scores above 60 indicate that the text is easy to follow, and deemed by some to be the appropriate level for written patient information.⁷ In research by Wang et al. the Flesch Reading Ease Score and Flesch–Kincaid formulas were the most commonly used formulas in medical research.¹¹

Statistical analysis

Independent Samples t-Test was used to calculate p-values. Inter-rater variability did not need to be analysed as the aforementioned measures are computer generated.

Ethics

Given the nature of the review, no hospital ethics approval was required.

Results

Overall, 29 articles were analysed using the above scores. The mean Gunning Fog Index for all sites was 11.93 (95% confidence interval [CI] 11.32–12.53) and the mean Flesch–Kincaid Grade Level was 9.45 (95% CI 8.85–10.06), correlating to reading levels of Grade 11 and 9 respectively. The mean Gunning Fog Index for Australian/New Zealand materials was 12.79 (95% CI 12.14–13.44), compared with 11.6 (95% CI 10.93–12.27) for international materials (Table 1).

View Table 1–3.

The mean Flesch Reading Ease Score for all articles was 57.57 (95% CI 54.29–60.86), with a mean of 55.8 (95% CI 52.45–59.15) for Australian/New Zealand materials and 58.25 (95% CI 54.51–61.98) for international materials (Table 2).

The average readability of the Australian/New Zealand materials analysed was at or above a 9th Grade level for all grade markers of readability used. Seven of the eight Australian/New Zealand articles analysed had a Gunning Fox Index above 10, and none had a score that indicated they would be accessible to someone with a level of schooling equivalent to the 6th Grade (see Table 3). The average grade level readability score of internationally produced materials was lower than the Australian/New Zealand average for all markers except the Coleman–Liau Index, indicating that non-Australian/New Zealand materials tended to be more accessible to readers.

Of the Australian/New Zealand materials analysed using the Gunning Fog Index and the Flesch–Kincaid Grade Level, 87.5% and 75%, respectively, required a 10th Grade education or higher to read and understand the text.

Discussion

Readability is defined by Dale and Chale as “The sum total (including all the interactions) of all those elements within a given piece of printed material that affect the success a group of readers has with it. The success is the extent to which they understand it, read it at an optimal speed and find it interesting.”¹² A readability formula is a mathematical formula that gives a number rating to indicate the reading difficulty of a text.¹³

This study indicates that written patient information on the internet is at a high risk of not being fully understood by patients considering laryngectomy. Overall, based on the seven markers of readability used, Australian and New Zealand written information on laryngectomy found on the internet was more complex and difficult to read than international materials; however, the result was only statistically significant for one marker of readability (Linsear Write Formula). Two other markers, the Flesch–Kincaid Grade Level and the Automated Readability Index, approached significant difference (p=0.07 and p=0.06 respectively). Most materials found were not tailored to the level of education and reading level recommended by national health organisations and are at risk of not being fully understood by patients considering laryngectomy.

During our study, we employed a thorough search strategy and believe that the included resources make up the vast majority that patients in Australia or New Zealand would retrieve during their search for written patient information. We used a large number of readability calculators that analyse different aspects of the written text and addressed a major gap in Australian/New Zealand literature surrounding the readability of laryngectomy resources.

Our study does have some limitations. Readability formulas assess the linguistic characteristics of a written text, but do not take in to account other aspects that influence understanding, such as diagrams, charts and videos. Furthermore, they do not take into account a patient’s health literacy level, and level of understanding of laryngectomy prior to accessing written materials. Additionally, although government bodies recommend a certain level of readability for written health information materials, the average reading levels of laryngectomy patients is not known, and thus we cannot state with total certainty that patients cannot understand a certain body of text—we assume they have similar reading levels to the rest of the population. Another weakness is that the total number of Australian– or New Zealand–made patient information materials is limited, and only eight articles that were deemed appropriate for inclusion were retrieved. Thus, although all markers of readability showed that Australian– and New Zealand–produced materials tended to be more difficult to read, only one marker was able to show this to be a statistically significant difference.

These findings are in keeping with other studies within the literature investigating online patient information tools for procedures within otolaryngology. Wu et al. investigated online resources for functional endoscopic sinus surgery and found average ease of reading scores were at a minimum university education reading level.¹⁴ Similarly, a North American study by Wong et al., also investigating patient education resources in laryngectomy, found that the reading capacity of an average American adult was less than the average reading difficulty of the resources investigated.⁶ Our study adds to the findings of Wong et al. about internet-based laryngectomy resources, and also identifies that Australian and New Zealand resources on laryngectomy are potentially more difficult to read and understand than other resources available.

So that patients can fully comprehend the written information given to them, it is important to direct written patient information at a level that is accessible to most people. The American Medical Association recommends that patient information sheets should be written at or below a 6th Grade level to ensure written information can be understood by most patients.^6,15 In Australia, it is estimated that 13% of people read at or below a primary school level, 44% read at or below a Year 10 level, and only 15% read above a high school level.¹⁶ South Australia Health recommends that health information materials be written at an 8th Grade level or lower.¹⁷

These issues are further compounded by the fact that the average laryngectomy patient may have poor baseline education levels, with one study of patients undergoing laryngectomy demonstrating 37% had failed to graduate from high school.¹⁸

Based on the estimated reading levels of the Australian population that state 57% of the population reads at or below a Year 10 level, using the Flesch–Kincaid Grade Level, the Gunning Fog Index, the Coleman–Liau Index, the Automated Readability Index and the Linsear Write Formula, only 43% of the Australian population can potentially read and comprehend the Australian/New Zealand patient information materials on laryngectomy.

Additionally, the average Flesch Reading Ease Score for Australian/New Zealand articles was 55.8, which correlates to “fairly hard” to read, rather than the “easy” or “very easy” categories that are usually recommended for texts aimed at the general population. Similarly, the average Flesch Reading Ease Score for internationally produced materials was 58.23.

Based on our research, Australian/New Zealand and internationally produced online resources on laryngectomy are at risk of not being understood by a large number of patients considering laryngectomy. This gap in the readability of articles and population literacy levels could potentially hinder patient comprehension and engagement, and result in problems regarding informed consent and patient satisfaction, which can subsequently worsen patient outcomes. Because written online patient information is an important means for patients to gain information on laryngectomy, these patients might benefit from the creation of written information that is easier to read and comprehend.

This paper only analyses written patient information on the internet, and as such further research into how accessible other internet resources, such as videos and pictures, are to patients with laryngeal cancer considering laryngectomy would be useful.

Conclusion

The decision to undergo laryngectomy is a life-altering decision for a patient to make, and it is important that information on the subject is available at a level that is accessible to the majority of the public. Overall, laryngectomy patient information found on the internet is at a high risk of not being fully understood by members of the public. Notably, overall, patient information on Australian/New Zealand websites tends to be less readable than information on international websites. Thus, some thought should go into creating patient information that is less complex and more likely to be understood by more of the population.

View Appendix.