The quality of online consumer health information at the intersection of complementary and alternative medicine and arthritis

Complementary and alternative medicine (CAM) use is prevalent among patients living with arthritis. Such patients often seek information online, for the purpose of gaining a second opinion to their healthcare provider or even self-medication. Little is known about the quality of web-based consumer health information at the intersection of CAM and arthritis; thus, investigating the quality of websites containing this information was the purpose of this study. Four unique search terms were searched on Google across four English-speaking countries. We assessed the first 20 results of each search, including them if they contained CAM consumer health information for the treatment and/or management of arthritis. Eligible websites were assessed in duplicate using the DISCERN instrument, which consists of 16-items designed to assess quality. Of total of 320 webpages, 239 were duplicates, and a total of 38 unique websites were deemed eligible and assessed using the DISCERN instrument. The mean summed DISCERN scores across all websites was 55.53 (SD = 9.37). The mean score of the overall quality of each website was 3.71 (SD = 0.63), thus the majority of websites are ranked as slightly above ‘fair’ quality. Eligible websites generally received scores better than ‘moderate’ in terms of overall quality. Several shortcomings included a lack of transparency surrounding references used and underreporting of risks associated with treatment options. These results suggest that health providers should be vigilant of the variable quality of information their patients may be accessing online and educate them on how to identify high quality resources.

synovial fluid tests, and by using imaging techniques such as x-rays [3]. Arthritis is most common among women. Approximately 20 million people worldwide were living with rheumatoid arthritis in 2017 with over a million new cases being diagnosed each year [4]. Although arthritis affects all ages, the risk increases as individuals grow older. If left untreated, it can negatively impact physical functioning and quality of life [2].
Many arthritis patients seek out various treatment options to help them manage and cope with their condition, one of which includes complementary and alternative medicine (CAM), which has gained popularity. CAM refers to a wide range of therapies and health care practices that are not considered mainstream or standard medical practice. "Complementary" medicine can be defined as an unconventional practice that is used in combination with conventional medicine [5], while "alternative" medicine is primarily used in place of conventional health care [5]. Furthermore, the term "integrative practice" takes place when conventional and complementary approaches are used collaboratively [5,6]. CAM treatments are popular among patients with arthritis for a variety of reasons. A Malaysian study found that people with arthritis typically use CAM because they believe there are fewer side effects associated with such therapies in comparison to conventional medicine [7]. Studies from South Korea, India, Turkey, and Nigeria have found that the majority of patients turn to CAM therapies because they believe it helps to alleviate pain, helps control morning stiffness, and is less expensive [8][9][10][11]. One report from the United Kingdom found that approximately 60% of arthritis patients have used CAM therapies at least once [12]. One Canadian study found that 33.9% of patients with juvenile idiopathic arthritis used CAM therapies [13]. Several studies from the United States have estimated that 30-41% of adults with arthritis use CAM [14][15][16][17], while CAM use among youth with juvenile idiopathic arthritis has been found to be 72% [18]. Another study found that 40% of Australians who suffer from osteoarthritis use CAM [19]. Some of the most commonly-used CAM therapies for arthritis include dietary supplements (including glucosamine sulphate and chondroitin), chiropractic and manual therapies, massage therapy, herbal medicine, acupuncture, and meditative therapies (including yoga and tai chi). These are purportedly used to improve damaged cartilage, manage pain/stress, and increase mobility [20,21].
One study that reviewed the quality of online content found that information relating to osteoarthritis is generally 'fair' in quality [22]. Another research study investigating the quality of complementary and integrative health information on the internet typically found that most websites did well in terms of information quality relating to ownership, currency, interactivity and navigability, while results varied for authorship, balanced information and the use of sources [23]. Another study on the quality of herbal product information on the internet found serious shortcomings in terms of providing specific safety information [24]. To our knowledge, no study has specifically focused on the quality of online information at the intersection of CAM and arthritis. The objective of this study is to analyse the quality of consumer health information on CAM therapies that a "typical" patient with arthritis may encounter online. This study is of value to both healthcare providers and researchers, as it provides them with a greater understanding and insight into what types of information patients with arthritis may be accessing on their own.

Search strategy
A cross-sectional survey of the quality of websites containing CAM consumer health information for the treatment and/or management of arthritis was undertaken. JYN developed the search strategies which were executed by AV on Google on May 5th, 2020 as follows: 'alternative medicine for arthritis', 'complementary and alternative medicine for arthritis', 'complementary medicine for arthritis', and 'integrative medicine for arthritis'. All searches were conducted on Google Chrome in incognito mode to ensure that the search results retrieved were unaffected by prior browsing history and run across four English-speaking countries: Australia (Google.com.au), Canada (Google.ca), the United Kingdom (Google.co.uk), and the United States (Google. com). This was done to obtain a more internationally representative selection of commonly-visited websites. The four search strategies were based on the most common terms used to refer to CAM [6] and were designed to be simple, so as to replicate how a "typical" patient with limited knowledge of medical or research topics may seek health-related information online. We elected to search only Google as it comprises over 90% of the search engine market share used worldwide as of August 2020 [25], thereby retrieving websites that are much more likely to be accessed and read by patients. It is worth noting that the four next most popular search engines (Bing, Yahoo!, Baidu, and Yandex) only hold between 0.44 and 2.78% of the search engine market share each [25]. Additionally, we only assessed the first 20 webpage results of each Google search, given that it is known that internet users typically only view the first page of results that are generated and may occasionally look through the second search engine results page [26]. With four search terms being generated across four countries (for a total of 16 searches and 20 websites for each), we sought to assess a total of 320 websites.

Eligibility criteria
AV and TJ reviewed the 320 websites yielded across all searches and removed all duplicates. Once all of the URLs were collected, the authors assessed the inclusion criteria for each website. Websites were screened for eligibility and included if they contained at least one webpage that contained CAM consumer health information for the treatment and/or management of arthritis. For the purpose of this study, we identified and included CAM therapies based on the operational definition provided by Cochrane, as follows: https://cam.cochrane.org/ operational-definition-complementary-medicine.
In addition, websites had to have been published in the English-language and be publicly available. Ineligible websites that were excluded were as follows: Wikipedia pages, videos without text content (i.e. YouTube links), Google images, broken URLs, websites published in non-English languages, eBooks, peer-reviewed journals and articles, forums, and news articles that did not contain any consumer health information. A finalized list was developed to undergo further evaluation of content.
Data extraction and website quality assessment AV and TJ data extracted the following items: website URL, website type, types of arthritis discussed, types of CAM therapies, types of non-CAM therapies (if present), whether the website appeared in more than one search (different search terms and/or regions), whether the website was certified by Health on the Net Foundation Code of Conduct (HONcode), as well as the scores for the 16 DISCERN questions. Different webpages from the same website captured by searches were considered a single item for the DISCERN instrument quality assessment; we therefore conducted a quality assessment of websites and not individual webpages. The DISCERN instrument is divided into a series of 16 key questions including an overall rating on the quality of the information being assessed. It is designed to examine what information is being provided rather than simply looking at how it is presented. It is organized into three sections: Section 1 (Questions 1-8), which address the reliability of the source, Section 2 (Questions 9-15), which analyses the specific details regarding treatment options, and Section 3 (Question 16), which is the overall quality rating of the publication [27]. To receive a 'good' quality rating, the majority of the questions must be rated highly on a five-point rating scale ranging from 'No' to 'Yes' [27]. A rating of five indicates that the answer to the question is a definite 'Yes', while a rating of one indicates a definite 'No'. The ratings in between signify that the criterion for the question was only partially fulfilled. Questions that encompass DISCERN criteria include analysing the aims of the publication, determining if the information is relevant, identifying if sources and their dates were made explicit, judging if the information is balanced and unbiased, enabling readers to pursue further sources of information and clarifying that more than one treatment option is available, identifying areas of uncertainty, providing background information on how a treatment works, describing the benefits and risks of the treatment/lack of treatment, highlighting effects on the overall quality of life, and determining if the publication (website) supports collaborative decision-making [27]. The assessor is then asked to rate the publication's overall quality (Question 16) by taking all of these areas into account and summarizing the results based on how the majority of the questions were answered. This final question is an intuitive summary of the previous 15 items of the DISCERN instrument [27].
We developed a standardized procedure to data extract and assess each website for quality using the DISCERN assessment by first conducting a pilot test on three separate websites independently, followed by a meeting to discuss and resolve any discrepancies (JYN, AV, TJ). Following this, AV and TJ completed the data extraction and assessed the quality of consumer health information using the DISCERN instrument, for each eligible website. The individual scores were reviewed by JYN and any remaining discrepancies were resolved between AV and TJ without unduly altering original assessor ratings. Averages were calculated across the two authors' scores for every DISCERN question and for all websites assessed. In addition to the five-point scale intuitive summary score for Question 16, an overall DISCERN score sum of questions 1-15 was also calculated (ranging from a score between 15 and 75). The mean score and standard deviation were determined for each DISCERN item as well as the total mean score for all 16 items.

Search results
A total of 320 webpages were reviewed, of which 239 were duplicates, and 81 were unique. Twenty-seven websites were not eligible as they were peer-reviewed academic articles (n = 20), Google images (n = 3), invalid addresses (n = 2), and news articles (n = 2). One additional website did not mention CAM treatment for arthritis from a patient/consumer health information point of view, resulting in a total of 53 eligible webpages. After collapsing webpages from the same website, a total of 38 websites were deemed eligible and assessed using the DISCERN instrument. This process is depicted in Fig. 1.

General characteristics of eligible websites
We organized eligible websites based on the following categories: commercial, government, health portal, news, non-profit, professional, and other. For a website to be categorized as commercial, it would have to either purchase or sell a product or service with the intent of making a profit. An example of a commercial website in this study is the Praana website (https://praanaim.com/). Government websitessuch as HealthLink British Columbia (https://www.healthlinkbc.ca/)are websites that are controlled by a governmental body. A health portal is any website that primarily discusses health topics, such as WebMD (https://www.webmd.com/). The news categorization refers to websites like Medical News Today (https://www.medicalnewstoday.com/) that take information from newspapers, magazines and television and are a source of consumer health information. Non-profit websites are often linked to charities or organizations that are not aiming to make a profit, as seen with the Arthritis Foundation website (https://www. arthritis.org/). The majority of websites assessed (n = 13) were categorized as professional, meaning that they were either affiliated with health experts, a university/hospital, or an authorized organization, as seen in the example with the Mayo Clinic website (https://www.mayoclinic. org/). Websites that did not fit into any of the previously mentioned categories, were classified as 'other', which for example was the case for the Creaky Joints website (https://creakyjoints.org/). Common types of CAM therapies discussed were dietary and herbal supplements, which according to the National Center for Complementary and Integrative Health, includes vitamins, minerals, herbs, botanicals, amino acids, enzymes, tissues from organs or glands, and extracts [28]. Dietary and herbal supplements were mentioned in nearly all the websites assessed aside from one (n = 37), proving to be the most frequent CAM therapy mentioned. Acupuncture was the next most common therapy (n = 30), followed by exercise (n = 26), when including tai chi and yoga. The overwhelming majority of websites (89.47%) also provided information about non-CAM therapiesthe most common types being pharmaceutical medications and surgery. Most websites (n = 29) appeared in more than one search, including those that were generated from the same country but through a different search term. Table 1 depicts some of the general characteristics of the eligible websites.

DISCERN instrument ratings
The total mean score across all websites for the sum of questions 1 through 15 in the DISCERN instrument was 55.53 (SD = 9.37) out of 75. The total mean score for question 16, which assessed the overall rating of each website was 3.71 (SD = 0.63) on a five-point scale. This shows that on average, most websites achieved a moderately high overall rating. Thus, the quality of most sources was better than 'fair', but still had room for improvement in terms of obtaining the optimal standard. The Mayo Clinic and Upto-Date websites scored the highest with both achieving a DISCERN score sum of 69.50. The overall quality rating was a score of 4.64 (SD = 0.05) for both of these webpages. Conversely, the Dr. Weil and Robert Schad Naturopathic Clinic websites scored the lowest, with scores of 38.00 and 32.00 respectively; the overall quality ratings were 2.53 and 2.14 respectively. A breakdown of the DISCERN instrument ratings can be viewed in Table 2, which includes the total means and standard deviations for each question.

Trends identified across resources assessed Website aims
The DISCERN instrument considers a source to be of good quality if the aims of the publication are clearly indicated. Question 1 assesses the extent to which a    website fulfils this requirement. The vast majority of websites ranked above average for this question with a total mean score of 4.53 (SD = 0.56). The "About" page of a website typically included this information in the form of a mission/vision statement, or it could sometimes be found in the introduction of an article. It is here where most websites identified their target audience and what kind of information users could expect to find within the resource.

Use of references
Question 4 of the DISCERN instrument evaluates whether or not the sources used to compile the information on the website are made clear to the viewer. To achieve a good ranking for this question, the website should include both in-text citations and a reference list at the end of the article. Twenty-two out of 38 websites scored a 3 or below on this question, meaning that the majority of websites did not include both in-text citations and a formal reference list. The total mean score was 2.76 (SD = 1.51). Some websites included a reference list but failed to have in-text citations and vice versa, while others showed no evidence of referencing at all. Only 4 websites achieved a perfect ranking of 5 for this question.

Risks associated with treatment
To achieve a high ranking for question 11 of the DISC ERN instrument, a website must include a description of the risks associated with each treatment that is discussed. The mean score for this question was 2.78 (SD = 1.13). None of the sources assessed received a perfect score of 5, which is of concern given that patients should be made aware of what to expect from a treatment, especially if any adverse side effects are known. Many websites did not provide enough detail about the disadvantages of a treatment, or only mentioned the risks for some of the CAM therapies discussed but failed to provide adequate details.

Variety of treatment options
The overwhelming majority of websites analysed in this study scored highly on question 14 of the DISCERN instrument. The question assesses whether or not a source demonstrates that there are various treatment options for arthritis patients. Thirty-four out of the 38 websites assessed received a perfect score of 5, and no website was graded lower than 4. This provided a mean score of 4.91 (SD = 0.28). Various forms of CAM therapies were often discussed, including dietary and herbal supplements, acupuncture, exercise, and manual therapies. Most of the websites provided additional information regarding conventional treatment options as well, such as pharmaceuticals, surgery, steroid injections and assistive devices.

Shared decision-making
Question 15 of the DISCERN instrument assesses the extent to which a website provides patients with support for shared decision-making. The majority of the websites assessed received a high ranking, with a mean score of 4.54 (SD = 0.98). Only 4 websites scored below a 3 for this question. Most websites recommended that patients do not take any dietary and herbal supplements without first consulting with their physician, stating that such products may interact with medications or potentially cause harm. Many sources also contained disclaimers about how website content should not be a substitute for professional medical advice and making healthcare decisions in collaboration with a healthcare provider.

Recommended websites for patients and consumers
The websites that scored highest in this study (an overall ranking of 4 or above for Question 16) may serve as good quality sources for practitioners to recommend to arthritis patients who seek additional CAM consumer health information about arthritis. These websites were similar in the sense that they typically scored a high rating above 4 across many of the DISCERN questions. Each of the highest scoring websites received a score of 5 for question 8 (which assesses whether the website refers to any areas of uncertainty with the treatment) and question 14.
Furthermore, these websites clearly outlined areas of uncertainty in the research known about the disease, and highlighted the various treatment options that are available to arthritis patients. The majority of these websites provided clear cited sources and, in our judgement, provided information which remained unbiased. Many of these websites also referred consumers to further material on the topic discussed, and provided sufficient details surrounding the impacts a treatment option could have on a patient's overall quality of life. Additional details about these websites are provided in Table 3.

Discussion
The purpose of this study was to assess the quality of online CAM consumer health information for the treatment and/or management of arthritis. Given the high prevalence of CAM use among patients with arthritis and their increasing reliance on the internet for health information, it is important to critically evaluate the online resources that this unique patient population accesses. This is especially crucial given the fact that nearly half of all arthritis patients who utilize CAM therapies do not disclose this to their physicians [29]. Healthcare providers should, therefore, be aware of the online information that their patients may consult, either as a second opinion or in their absence, to help them navigate appropriate resources and discern between websites of high quality and those providing misinformation or unsafe recommendations. The present study found that the majority of eligible websites assessed were of 'fair' quality. Twenty-one out of the 38 sources analysed received a moderate overall score (ranging from 2.90-3.97 out of 5). The total mean score of 3.71 (SD = 0.63) and average DISCERN score of 55.53 (SD = 9.37) suggests that most websites were slightly above adequate quality but still had some shortcomings. It should be noted that 14 out of 38 websites achieved a high overall ranking of 4 or above and can be deemed of 'good' quality. The websites that scored highest in this study may serve as useful resources to healthcare providers who seek to recommend high-quality resources for their patients. While no websites assessed received an overall score below 2 (i.e. 'poor' quality), three websites ranked below moderate (ranging from 2.14-2.67), suggesting that these websites had some serious shortcomings (including a lack of transparency in terms of the sources used, dates of the articles, areas of uncertainty, and the risks associated with each treatment while also not providing further reading).
This study also found that the majority of sources outlined the aims of the website, provided a wide variety of treatment options, and encouraged the use of shared decision-making between the patient and those who are involved in their care. In particular, adequate discussion surrounding different treatment options are beneficial for patients, so that they may be aware of the options that are available to them. While it is reassuring that numerous websites encourage patients to consult their healthcare providers before making any treatment decisions on their own, practitioners and researchers should be cautioned that the majority of websites did not report all of the risks that are associated with each CAM treatment and did not always provide explicit details about the references used to support their claims made about these therapies. These findings illuminate an aspect of patient behaviour outside of a clinical setting, given that many are unaware that various CAM therapies may lack efficacy and/or safety [30]; it is well-documented in the literature that, for example, many patients believe that natural remedies do not have any harmful side effects or that "natural equals safe", which is an issue that healthcare providers need to be cognizant of and should strongly advocate that their patients discuss their use of or concerns associated with CAM with them as opposed to self-medicating [30].
To our knowledge, this is the first study to assess the quality of websites containing CAM consumer health information specifically in the context of arthritis. Despite this, our findings can be compared to other studies that have assessed the quality of consumer health information on related topics. A previous study analysed the readability and quality of online information on osteoarthritis, including conventional forms of treatment. This study found that website information was considered 'fair' and that the readability was either equivalent to or more challenging than the recommended standard, as outlined by both the Journal of the American Medical Association and the DISCERN criteria [22,31]. Prior to this, another study found that 44% of web search results related to arthritis promoted alternative medicine and that 80% of these websites had clear financial aims [32]. Another study also highlighted the importance of recommending websites that draw on evidence-based medical literature; an example of such information could include a systematic review that identified the quantity and assessed the quality of CAM recommendations for osteoarthritis and rheumatoid arthritis evidence-based clinical practice guidelines [33]. Guidelines and other evidence-based resources could be used as a starting point for the improvement of consumer health information on websites such as the ones we assessed in the present study.
Additionally, several studies have evaluated the quality of online CAM information as it relates to other conditions. A recent study assessed the quality of online consumer health information at the intersection of CAM and low back pain using the DISCERN instrument, finding that quality was quite variable across different websites. The authors found that the mean overall rating across all websites was 3.47 (SD = 0.70), while the summed DISCERN scores ranged from 25.5-68.0, with a mean of 53.25 (SD = 10.41) [34]. Another study also used the DISCERN instrument to evaluate information provided on chronic pain websites, and found that the overall quality of most were of moderate quality with several shortcomings. The overall average DISCERN score for this study was 55.90 (however, they scored this total to include question 16, out of 80) [35].
United Kingdom and the United States) in order to yield a more internationally-representative sample of websites. Another strength of this study included the use of the DISCERN instrument, which has been found to be both reliable and valid in assessing the quality of consumer health information [36]. Additional strengths include the fact that we also piloted conducting data extractions and applying the DISCERN instrument; during the pilot and full-assessments, both steps were conducted independently and in duplicate, before all three authors met to discuss and resolve discrepancies.
One limitation includes the fact that our search only captured a cross-sectional snapshot of website information; since the internet is a dynamic source that is subject to change, the results from this study may differ if replicated across future searches. Additionally, since the search strategy for this study only included the first 20 websites that were generated for each search term, less popular websites remain unassessed for quality. Furthermore, if different search terms were used, variable results could have been generated. Yet another limitation would include the fact that the internet searches were only conducted in English, as were the eligibility criteria, thereby excluding all websites that were written in other languages. Studies on the quality of non-English online consumer health information are also limited, thus it is unclear as to whether our findings are generalizable to non-English websites and their respective patient populations that access and read them. Multimedia platforms like videos and images were excluded, even though these may also contribute to an understanding of patient health-seeking behaviour in the context of CAM information for arthritis. Furthermore, the DISCERN instrument is primarily designed to assess the quality of health information, as opposed to determining the accuracy of the content or the safety of the information written in the publication [27].

Conclusion
The purpose of the present study was to assess the quality of online CAM consumer health information for the treatment and/or management of arthritis, that a 'typical' patient may access. While the internet gains increasing prominence as a source of information for CAM treatment choices, healthcare providers must be aware of the quality of consumer health information that their patients with arthritis may read and rely upon. Following a quality assessment using the DISCERN instrument, it was found that the majority of sources evaluated were slightly above average (3.71 out of 5), which suggests that patients should not solely base their CAM treatment decisions on the information found on these websites alone. Further discussion with a healthcare practitioner who can refer patients to reputable sources of consumer health information is warranted.