BRONZE

The Quality of Dissemination of Information on the Internet of COVID-19 and Multiple Sclerosis

This project aims to quantify the amount of accurate information spread throughout the internet with the use of the DISCERN instrument, HONcode Seal, and the Flesch Kincaid Reading Level.
Aanisa Abeer
Grade 11

Problem

Multiple sclerosis has been found to show poorer results in patients who acquire COVID-19.21 Patients who take disease modifying therapies (DMT’s), used to help alleviate damage caused by multiple sclerosis, have been found to have an increased risk of acquiring COVID-19.212223 People in the MS society require more care than the general public and because there is very little known about the COVID-19 disease, multiple sclerosis patients are often worried about the risk the coronavirus can have on them.

The central research problem of this study is how reliable will the information on the internet be, as COVID-19 key terms and multiple sclerosis is searched on different search engines. Not many health studies have been conducted on the novel coronavirus in relation to multiple sclerosis. This study aims to find how accurate and reliable the information on the internet is, especially since COVID-19 is a recent and upcoming disease with little known information in relation to multiple sclerosis.

Method

The reason why this experiment chose to use the DISCERN instrument, HONcode seal and the Flesch Kincaid Reading level, is because they have found to be the most effective and least time-consuming methods. The DISCERN tool was first developed in 1998, having it been around for a while and many infodemiologies have used this tool.15 The JAMA Benchmark, used in various other studies, proves to be vaguer and will not serve a greater purpose in the overall results of the experiment. The HONcode seal is an excellent method of analyzing and reassuring the quality of information as proved by the conclusion in the paper on Perthes disease. The best method to use would be a mixture of the DISCERN Instrument and the HONcode seal. These methods prove to be the most useful because of the reliability and efficiency of these methods.

Another method used in this experiment includes the Flesch Kincaid Reading level which is equivalent to the U.S grade level of education. A list of number between 1- 18 show the required education to be able to understand a certain text. This method was found to be highly valuable as it has been used in similar studies and accurately depicts what a grade level of a certain text may entail.

To conduct the experiment on the quality of information involving COVID-19 and multiple sclerosis, multiple search engines will be used to conduct an experimental study. Imitating a simple search on the internet, the experiment will begin by browsing through the different hits shown from order of relevance, from top to bottom. What goes into the search bar is "COVID- 19 and Multiple Sclerosis."

This experiment aims to have two methods of evaluation. The first method of evaluation will be through analyzing the different devices used to determine the quality of the data. The five search engines will remain the same and when the top hits are received, each devices result will be compared.

According to the DISCERN handbook, the quality of the website is determined as follows: excellent=63–75, good=51–62, fair=39–50, poor=27–38 and very poor=15–26. 24 Scores will be given out of 80 for each link evaluated. The experiment will quantify what DISCERN score an article receives and if the category falls into ‘excellent,’ ‘good,’ or ‘fair,’ it will be seen whether there is a presence of the HONcode seal on the websites as well.

To access the Flesch Kincaid reading level which is readily available on Microsoft Word, each article/link will be copied and pasted onto a blank Word document. Once formatted, the accesibility tab under Review, will help rate the articles grade level on a score of 1-18.

Another method of evaluation in this experiment includes the evaluation of the different search engines used. These search engines are: Bing, Google, Swisscows, DuckDuckGo, and Yahoo. Once these websites are opened, the DISCERN Instrument, HONcode seal and the Flesch Kincaid reading level will be used. It will be compared wether the rating of one score affected the other and how the instruments intertwine when each of the different search engines are evaluated.

To refrain from previous search history influencing the experiment, previous search history will be deleted and personal accounts will be logged out of. Once the top 20 links of each search are found, these links will be used to answer the DISCERN questionnaire, copied and pasted onto Microsoft Word to determine the reading scale through the Flesch Kincaid Grade Level test and it will be determined whether each of the websites holds the HONcode seal11,15,25.

The experiment conducted will take approximately two to three hours at one sitting. Ethical considerations regarding participation and anonymity will not be required as this is a personal experiment conducted through an electronic device(s). The results will help quantify the number of websites with HONcode seal, the Flesch Kincaid Grade Level and the average score of the DISCERN questionnaire11,17,18.

Research

Coronavirus:

In 2003, a new member of the coronavirus, known as SARS-CoV, or Acute Respiratory Syndrome, emerged into the human population, a similar pandemic to today’s COVID-19, quickly fading in numbers.1 Today, SARS- CoV-2, a new member of the coronavirus family and a global health emergency, named Coronavirus Disease of 2019 (COVID-19) causing an aggressive lung disease, is quickly spreading through humans across the globe, displaying its impact on the world.2 The virus has reached nearly every country worldwide in less than 6 months. Several of these countries are already enduring a second wave outbreaks.3 Since the first epidemic of SARS, the coronaviruses have harmed millions of people worldwide. Considering the unfavorable outcomes of the current COVID-19 epidemic, developing successful therapeutic strategies is necessary to cope with the absence of effective drugs, high mortality rate, and the potential of the virus to cause further epidemics.4 In a recent study conducted on the mental health among medical and nursing staff, it was determined that 50.4% of people who took part of the study had accessed psychological resources available through media on mental health self-help coping methods.5

 

Quality of Medical Information on the Internet:

 

One of the fundamental issues in society is the disseminating of false or unsourced information spread across the media. This information cannot be stopped or simply removed therefore it is important to educate oneself and those around to acquire relevant information from responsibly sourced media. Although the internet provides immense opportunities, it also lowers the cost of generating and disseminating information, allowing misinformation and sensationalized stories to take a hold. This causes a rapid global spread of information through one small initiation that webs to different claims.6 Dr. Brittany Seymour, an assistant professor of oral health policy and epidemiology at Harvard University, shares, “With 80% of people seeking health information online, viral misinformation poses a growing threat to public health.”7 A recent quality assessment of information of COVID-19 on the internet showed that in the United States, 81% of adults access the internet and 53.1% look for health information online.8 In another study conducted on what young adults find most important in literature out of the five dimensions of information- quality completeness of information, understandability of information, relevance of information, depth of information, and accuracy of information- the completeness of information and understandability of information were rated as the two most important quality dimensions by the study participants.9 This study shows that young adult readers do not place the importance to accuracy as its first impression while reading medical information on the internet. The accuracy of information is not deemed as valuable and therefore its potential harm is unnoticed to the reader's eye.

In the 2000’s, misinformation was widespread during the early years of the HIV epidemic. It too, suffered from conspiracy theories, rumours, and misinformation for many years, with the effects still visible to this day. Their argument continues that HIV does not exist, or cause AIDS, and that its cure can cause damage to human health.10

With respect to COVID-19, this disease too has seen its fair share of sourced and unsourced information11. As COVID-19 turned into a full-fledged public health crisis in February, multiple theories regarding the virus’ origin have taken hold on the internet, all with a similar theme: the virus was artificially created in a lab by a rogue government with an agenda. This misinformation originated from social media accounts and websites with no credible evidence to support their claims10. These claims continue to gain readers despite scientists from multiple nations analysing the genome of COVID-19 and coming to the decisive conclusion that the virus originated in nature from an animal source.10,12,13

 

Previous Studies:

 

The experiment that this research aims to conduct has been used in various instances. In 2018, an analysis of cancer related tweets was conducted. Using the Python Web Crawler, the conclusions arrived at by the experiment stated that 30% of them were found to have misinformation in them.14 A more recent study, analyzed the misinformation of COVID-19 on the internet. Tools such as the HONcode seal; a seal which proves the credibility of the information, JAMA network; an online network of sourced journals, and DISCERN; a questionnaire to assess the quality of written information, were used to quantify the reliable sites that appeared from those hits. The study concluded that out of the first 110 websites that were critically analyzed, only 1.8% of the websites had the HONcode seal, The JAMA benchmark showed that 39.1% of the websites did not have any of the categories required by this tool, and only 10.0% of the websites had the four quality criteria required by JAMA. The DISCERN score showed that 70.0% of the websites were evaluated as having a low score and none were rated as having a high score.11

Tools such as JAMA, DISCERN Instrument, and the Flesch Reading Scale have been used in many studies before.11,15–19 The Flesch Kincaid Reading Level (FRE), a tool provided by Microsoft Word helps users identify the grade level of a text by simply inserting the document into Word17. This grade level is identified by measuring semantic and syntactic difficulty. In a study conducted using the FRE tool, websites relating to anorexia nervosa were analyzed.20 These websites showed that the DISCERN score ranged from 37 to 71 representing “fair” information. The FRE score ranged from 28-64, which is a complicated range of text.20 In 2015, a research on online information about Perthes disease was conducted. Two of the three methods of assessment included the DISCERN Instrument and the presence of the HONcode seal. Results showed that only six of the forty-five websites found had the HONcode seal and the mean DISCERN score was 53.1(/80). In conclusion, the websites with the HONcode seal proved to have a greater DSICERN score.16

Multiple sclerosis has been found to show poorer results in patients who acquire COVID-19.21 Patients who take disease modifying therapies (DMT’s), used to help alleviate damage caused by multiple sclerosis, have been found to have an increased risk of acquiring COVID-19.212223 People in the MS society require more care than the general public and because there is very little known about the COVID-19 disease, multiple sclerosis patients are often worried about the risk the coronavirus can have on them.

 

The central research problem of this study is how reliable will the information on the internet be, as COVID-19 key terms and multiple sclerosis is searched on different search engines. Not many health studies have been conducted on the novel coronavirus in relation to multiple sclerosis. This study aims to find how accurate and reliable the information on the internet is, especially since COVID-19 is a recent and upcoming disease with little known information in relation to multiple sclerosis.

Data

•In assessing the quality of the unique websites, of a maximum score of 80, the overall mean DISCERN Score was 50.52
•The average Flesch Kincaid Reading Level on a scale of 1-18 was 8.26.
•HONcode seals were averagely found to be present 38% of the time. 
 
  Discern Scores Data
Link Bing Yahoo DuckDuckGo Google Swisscows
1 41 44 55 56 49
2 43 53 52 53 47
3 47 47 58 55 57
4 52 49 57 58 55
5 57 52 59 49 61
6 49 42 62 44 55
7 52 58 34 47 52
8 58 44 55 39 38
9 44 56 52 46 60
10 45 52 41 44 51
STDEV 5.846176 5.396501 8.631338251 6.190495 6.80277721
Average 49.88 48.63 54.00 50.13 51.75

 

  Flesch Kincaid reading level Data
Link Bing Yahoo DuckDuckGo Google Swisscows
1 7.8 5.5 12.2 12.2 8.6
2 4.7 7.6 14.1 9.7 6.8
3 5.5 2.9 11.2 8.3 13.3
4 4.2 4.8 13.6 5.9 12.3
5 7.4 6.4 10.2 8.7 7.3
6 2.9 8.5 17.2 12.8 8.4
7 3.2 7.5 8.7 13.6 6.9
8 5.9 3.2 5.9 13.2 7.3
9 8.2 6.1 6.9 9.6 5.5
10 4.3 6 9.9 9.3 10.8
STDEV 1.888238 1.830149 3.435905186 2.514867 2.57414495
           
           
Average 5.41 5.85 10.99 10.33 8.72

 

 

Prescence of HONcode seal:

Bing: 2/10

Yahoo: 3/10

DuckDuckGo: 6/10

Google: 4/10

Swisscows: 4/10

Conclusion

A total of 50 websites were reviewed in January 2021, 10 for each of the five search engines (Google, Bing Yahoo, DuckDuckGo and Swisscows). A total of 37 out of 50 websites were eligible for examination after the removal of 13 duplicate websites. When examining the number of websites classified under each of the characteristics evaluated it was found that 38% of the websites offered presented HONcode seal of approval. The mean (SD) DISCERN value for Google was 49.1, Yahoo 49.7, Swisscows 52.5, DuckDuckGo 53.5 and Bing 48.8.

The average Flesch Kincaid Reading Level was found to be 8.26 for all search engines.

In addition, links with higher DISCERN scores provided the HONcode seal. This proved the HONcode seal was used in legitimate websites where information was found to have a higher quality of information.

Flesch Kincaid scores depended on the search engines themselves, as some search engines were used to present websites directly related to the HONcode seal rather than articles without.

When comparing all five of the search engines used in this study, the rank of highest to lowest reliability reads:

1. DuckDuckGo

2. Swisscows

3. Bing

4. Google

5. Yahoo

When compared to previous studies conducted, which have used the three instrument used in this study, results form this study show similar findings to the original studies. A study conducted in early 2020 on the term "Wuhan Coronavirus", (PMID: 32250960) concluded that results displayed low DISCERN scores, none in the excellent range. These results match with the present study which similarly showed little excellent scores and had a high number of links with relatively low scores. Presence of HONcode seal in the current study was much higher than the presence of the HONcode seals in the April study, which could mean that over time, websites with HONcode seals appeared far often than websites without. As for the Flesch Kincaid Grade level, most data remained in the grade 7- 8 category, described as an "average" score. A study conducted last November analyzing the readability of online COVID-19 health information (PMID: 33183297)  showed DISCERN scores with only 17.2% universal readability which produced similar results within this study.

Overall, the conclusions arrived at by this quick study prove that while the internet can be a resourceful tool in accessing trivial amount of information in a short amount of time, not all sources or articles are properly sourced and therefore not 100% factual unless confirmed. While there were links that provided HONcode seals and a high DISCERN scores, there were others too, with DISCERN scores as low as 19. The Flesch Kincaid Reading Level also provided insight on the extreme ease of readability of each of the sources. The average Flesch Kincaid score, 8.26, displays a fairly easy read in articles that have the power to decide one's wellness, a readability score that is deemed far too low for that matter.

Citations

References:

 

1.        Zhong NS, Zheng BJ, Li YM, et al. Epidemiology and cause of severe acute respiratory syndrome (SARS) in Guangdong, People’s Republic of China, in February, 2003. Lancet (London, England). 2003;362(9393):1353-1358. doi:10.1016/s0140-6736(03)14630-2

2.        Schoeman D, Fielding BC. Coronavirus envelope protein: current knowledge. Virol J. 2019;16(1):69. doi:10.1186/s12985-019-1182-0

3.        Dong E, Du H, Gardner L. An interactive web-based dashboard to track COVID-19 in real time. Lancet Infect Dis. 2020;20(5):533-534. doi:10.1016/S1473-3099(20)30120-1

4.        Khan S, Siddique R, Shereen MA, et al. Emergence of a Novel Coronavirus, Severe Acute Respiratory Syndrome Coronavirus 2: Biology and Therapeutic Options. J Clin Microbiol. 2020;58(5):e00187-20. doi:10.1128/JCM.00187-20

5.        Kang L, Ma S, Chen M, et al. Impact on mental health and perceptions of psychological care among medical and nursing staff in Wuhan during the 2019 novel coronavirus disease outbreak: A cross-sectional study. Brain Behav Immun. 2020;87:11-17. doi:10.1016/j.bbi.2020.03.028

6.        Wang Y, McKee M, Torbica A, Stuckler D. Systematic Literature Review on the Spread of Health-related Misinformation on Social Media. Soc Sci Med. 2019;240:112552. doi:10.1016/j.socscimed.2019.112552

7.        Vogel L. Viral misinformation threatens public health. CMAJ. 2017;189(50):E1567-E1567. doi:10.1503/cmaj.109-5536

8.        Din HN, McDaniels-Davidson C, Nodora J, Madanat H. Profiles of a Health Information-Seeking Population and the Current Digital Divide: Cross-Sectional Analysis of the 2015-2016 California Health Interview Survey. J Med Internet Res. 2019;21(5):e11931-e11931. doi:10.2196/11931

9.        Tao D, LeRouge C, Smith KJ, De Leo G. Defining Information Quality Into Health Websites: A Conceptual Framework of Health Website Information Quality for Educated Young Adults. JMIR Hum factors. 2017;4(4):e25-e25. doi:10.2196/humanfactors.6455

10.      Mian A, Khan S. Coronavirus: the spread of misinformation. BMC Med. 2020;18(1):89. doi:10.1186/s12916-020-01556-3

11.      Cuan-Baltazar JY, Muñoz-Perez MJ, Robledo-Vega C, Pérez-Zepeda MF, Soto-Vega E. Misinformation of COVID-19 on the Internet: Infodemiology Study. JMIR public Heal Surveill. 2020;6(2):e18444-e18444. doi:10.2196/18444

12.      Zhou P, Yang X-L, Wang X-G, et al. A pneumonia outbreak associated with a new coronavirus of probable bat origin. Nature. 2020;579(7798):270-273. doi:10.1038/s41586-020-2012-7

13.      Lu R, Zhao X, Li J, et al. Genomic characterisation and epidemiology of 2019 novel coronavirus: implications for virus origins and receptor binding. Lancet (London, England). 2020;395(10224):565-574. doi:10.1016/S0140-6736(20)30251-8

14.      Chen L, Wang X, Peng T-Q. Nature and Diffusion of Gynecologic Cancer-Related Misinformation on Social Media: Analysis of Tweets. J Med Internet Res. 2018;20(10):e11515-e11515. doi:10.2196/11515

15.      Fan KS, Ghani SA, Machairas N, et al. COVID-19 prevention and treatment information on the internet: a systematic analysis and quality assessment. BMJ Open. 2020;10(9):e040487. doi:10.1136/bmjopen-2020-040487

16.      Nassiri M, Bruce-Brand RA, O’Neill F, Chenouri S, Curtin P. Perthes Disease: The Quality and Reliability of Information on the Internet. J Pediatr Orthop. 2015;35(5). https://journals.lww.com/pedorthopaedics/Fulltext/2015/07000/Perthes_Disease__The_Quality_and_Reliability_of.20.aspx

17.      Jayasinghe R, Ranasinghe S, Jayarajah U, Seneviratne S. Quality of online information for the general public on COVID-19. Patient Educ Couns. Published online August 7, 2020:S0738-3991(20)30402-X. doi:10.1016/j.pec.2020.08.001

18.      Fahy E, Hardikar R, Fox A, Mackay S. Quality of patient health information on the Internet: reviewing a complex and evolving landscape. Australas Med J. 2014;7(1):24-28. doi:10.4066/AMJ.2014.1900

19.      Fang IE. By computer flesch’s: Reading ease score and a syllable counter. Behav Sci. 1968;13(3):249-251. doi:https://doi.org/10.1002/bs.3830130312

20.      Arts H, Lemetyinen H, Edge D. Readability and quality of online eating disorder information—Are they sufficient? A systematic review evaluating websites on anorexia nervosa using DISCERN and Flesch Readability. Int J Eat Disord. 2020;53(1):128-132. doi:https://doi.org/10.1002/eat.23173

21.      Giovannoni G, Hawkes C, Lechner-Scott J, Levy M, Waubant E, Gold J. The COVID-19 pandemic and the use of MS disease-modifying therapies. Mult Scler Relat Disord. 2020;39:102073. doi:10.1016/j.msard.2020.102073

22.      Vishnevetsky A, Levy M. Rethinking high-risk groups in COVID-19. Mult Scler Relat Disord. 2020;42:102139. doi:10.1016/j.msard.2020.102139

23.      Willis MD, Robertson NP. Multiple sclerosis and the risk of infection: considerations in the threat of the novel coronavirus, COVID-19/SARS-CoV-2. J Neurol. 2020;267(5):1567-1569. doi:10.1007/s00415-020-09822-3

24.      Som R, Gunawardana NP. Internet chemotherapy information is of good quality: assessment with the DISCERN tool. Br J Cancer. 2012;107(2):403-404. doi:10.1038/bjc.2012.223

25.      Hernández-García I, Giménez-Júlvez T. Assessment of Health Information About COVID-19 Prevention on the Internet: Infodemiological Study. JMIR public Heal Surveill. 2020;6(2):e18717-e18717. doi:10.2196/18717

Acknowledgement

Dr. Beatriz Garcia Diaz

Webber Academy

 

Dr. Carlos Camara- Lemarroy

Cumming School of Medicine

University of Calgary

 

Ms. Bogusia Gierus

Webber Academy