AMA Journal of Ethics^®

Abstract

Internet technology makes information from both peer-reviewed sources and crowdsourced content, such as Wikipedia, instantly accessible. Health sciences education must adapt by providing learners with the skills needed to effectively and appropriately access and use information. In this article, we introduce a conceptual framework for teaching and learning using crowdsourced content. Using this framework, we show how educators can help learners develop the skills they need for critically assessing information quality, acquiring knowledge, and making clinical decisions.

Using Digital Sources as a Resource in Medical Education

In our digital age, there is a plethora of medical information of varying reliability instantly available. This phenomenon encouraged the coining of the term “e-health,” which was first defined in 2001 as “the intersection of medical informatics, public health and business, referring to health services and information delivered or enhanced through the Internet.”¹ The World Health Organization redefined this term in 2005 as “the cost-effective and secure use of information and communications technologies [ICT] in support of health and health-related fields, including health care services, health surveillance, health literature, and health education, knowledge and research.”² Since the emergence of e-health, it is estimated that 1 in 3 American adults have used the internet to try to figure out a medical condition.³ The internet is also widely used by children and adolescents, with a significant number of them actively seeking health information online.^4,5

It is no surprise that current health professions students, many of whom were born in this digital era, are using the internet as a resource throughout their education. Marc Prensky coined the terms “digital natives” and “digital immigrants” to describe those who were born into the digital world and those who were not.⁴ He discusses the dichotomy in learning and teaching methodologies between the two groups and argues that educators must “learn to communicate in the language and style of their students” who are digital natives.⁴

Many educators are concerned about students’ reliance on crowdsourced information as a starting point for learning. Crowdsourcing is the practice of soliciting group contributions, often from an online community or forum, to guide selection of services or products or to gather ideas and suggestions. Often crowdsourced material, such as that seen on Wikipedia, is anonymous and might not be written by experts. Rather than dissuading students from using such material, we propose helping students learn how to effectively navigate resources such as Wikipedia.

The Changing Landscape of Information and Resources

Students consult online sources for a number of reasons, whether to supplement lecture material, clarify unfamiliar terms or disease processes, or answer an attending physician’s questions during rounds. Guidelines are available to students describing how to effectively evaluate websites’ merits and reliability. One university’s library system has a specific “rubric” to evaluate websites. The rubric suggests appraising the website’s currency (“When was the site last updated?”), authority (“Who is the author or creator?”), validity (“Is the information accurate or valid?”), point of view or bias (“What is the website’s point of view?”), and audience (“Who was the website created for?”).⁶ Together, these proposed criteria are intended to help students evaluate important information regarding the accuracy, perspective, and relevance of information on websites, which should inform their decision about whether to use them.

Many medical schools provide students with similar guidance on how to evaluate online information and how to search for online peer-reviewed sources using the UpToDate^® and PubMed databases. However, even with this guidance, crowdsourced resources such as Wikipedia continue to supplement peer-reviewed sources in students’ education. Consider the following experiences students shared with us about using crowdsourced resources.

When I had difficulty understanding a problem-based learning [PBL] case, I would use Wikipedia to get an overall sense of the topic. For example, I struggled initially with learning about Cushing’s syndrome. When I first googled Cushing’s syndrome, the first link was to Wikipedia. I used that to get an overview of the topic before I went to my other sources (textbooks, Pathoma). I often found Wikipedia was easier to understand than some of my medical textbooks.

I had several apps on my phone to help with drugs and mechanisms of action. However, if the apps weren’t working or I was short on time, I would quickly google drugs. I typically did this if I needed information that I thought was common knowledge—drug classes, side effects, etc. I never used Wikipedia for things such as current guidelines or recommendations; I instead referenced UpToDate or the USPSTF [US Preventative Services Task Force] guidelines because I trusted those more.

These examples of students’ use of Wikipedia provide both evidence that students are using crowdsourced information throughout medical school and insight into how they are using this information.

Consistent with these examples, one study suggests that Google and Wikipedia are frequently used as starting points for locating information, even though students rate these platforms as having significantly lower quality and reliability than peer-reviewed sources.⁷ A 2012 survey conducted at one medical institution found that 94% of medical students reported using Wikipedia, stating its articles were both easy to access and easy to understand.⁸ Although ease of access and understandability are typically not used as criteria to determine a website’s merit, it appears these criteria are still important for students when they are searching for information online. Although there is little information available as to what medical information students are specifically searching for on Wikipedia, one study found that there was “a significant correlation between the year of medical school and the use of Wikipedia as the initial resource, with older years less likely to use Wikipedia as the first resource,”⁸ suggesting that as students progress through their graduate education they rely less on crowdsourced material found online. This evidence of students’ use of Google and Wikipedia highlights the need for faculty understanding of when and how students use a variety of information sources to augment their learning.

There has also been some debate as to how accurate crowdsourced resources such as Wikipedia are. Physicians and faculty often dissuade students from using resources like Wikipedia since they appear to be inaccurate and lack a traditional fact-checking system.⁹ Anecdotal evidence highlights this theme. One student told us:

I never wrote a paper in college or medical school where I cited Wikipedia because I knew Wikipedia was typically frowned upon. If a student included information in our PBL discussion from Wikipedia, it was usually said in a joking manner. “Oh, I know this is from Wikipedia, but…” It was always qualified first with, “I’m not sure how accurate this is.”

Several articles have examined Wikipedia’s accuracy. In 2005, the peer-reviewed journal Nature compared the accuracy of scientific articles in Wikipedia to those in Encyclopaedia Britannica.¹⁰ While both are considered encyclopedias, Wikipedia is a free online encyclopedia that anyone can edit as compared to an established source such as Encyclopaedia Britannica. Analysis of 42 scientific articles determined that both references contained 4 serious errors.¹⁰ Another review conducted in 2011 on the accuracy of Wikipedia’s medical entries found mixed results.¹⁰ Despite the warnings that crowdsourced resources are unreliable because they contain inaccuracies, such sources continue to be used by students, physicians, and the general public.¹⁰ Medical content on Wikipedia, which contained over 155 000 articles by the end of 2013, was viewed more than 4.88 billion times that year alone.¹¹ One study even suggests that “the creation of a Wikipedia article leads scientists to use similar words in later scientific work,” which is ultimately influencing the state of scientific literature.¹² This study also found evidence to suggest that the scientific articles referenced in Wikipedia receive more citations, suggesting Wikipedia is a complement to the traditional journal system.

It is clear that, despite this controversy, students use crowdsourced information such as that found on Wikipedia. For this reason, it is helpful for educators to consider how they might best use these resources in their teaching to help learners understand the benefits and pitfalls of using these resources throughout their medical education.

Frameworks for Incorporating Online Content Into Education

Rather than dwelling on the source, we propose that educators focus on the skills students need for acquiring knowledge in general and for critically appraising the reliability of information presented in the source. When considering reliability, students should be looking at accuracy, consistency, and completeness of information (or “sufficiently sound quality”).¹³ Several competency models or frameworks are useful for thinking through what critically appraising knowledge might look like in practice for educators and future clinicians.

The Technological Pedagogical Content Knowledge (TPACK) is a framework that expands on the work of Lee Shulman¹⁴ by including the knowledge beneficial for educators to teach effectively with technology.¹⁵ The TPACK framework brings together 3 primary forms of knowledge necessary for educators to demonstrate: content knowledge (CK), pedagogical knowledge (PK), and technological knowledge (TK) (see table). In effectively combining these 3 forms of knowledge in their teaching, educators can provide optimal learning experiences for students and help learners begin to be cognizant of the best ways to use various resources.

Table. Descriptions of Types of Knowledge Presented in the Technological Pedagogical Content Knowledge Framework

Knowledge Area	Types of Knowledge Represented
Content knowledge	“Knowledge of concepts, theories, ideas, organizational frameworks,” and knowledge of “established practices and approaches toward developing such knowledge.”16
Pedagogical knowledge	“Knowledge about the processes and practices or methods of teaching and learning…. This generic form of knowledge applies to understanding how students learn, general classroom management skills, lesson planning, and student assessment.”16 It also includes understanding of the materials, programs, and resources that comprise the curriculum and how to develop and manage the curriculum based on students’ needs.
Technological knowledge	“Knowledge about certain ways of thinking about, and working with technology, tools and resources. This includes understanding information technology broadly enough to apply it productively at work and in everyday life, being able to recognize when information technology can assist or impede the achievement of a goal, and being able continually [to] adapt to changes in information technology.”15
Adapted from Kohler,15 Kohler, Mishra.16

Shulman discusses the importance of contextual understanding in teaching and learning;¹⁴ this understanding emphasizes the need for educators to have knowledge of the contexts in which learning occurs (eg, classroom, clinic, hospital) and the character of the communities and cultures within which learning occurs. This approach is especially important in dispelling myths that only certain sources can or should be used for learning, especially in health professions education where just-in-time learning is important. Instead of shying away from using nonpeer-reviewed sources throughout health professions education, educators need to understand the context in which students learn and to help students better navigate the digital landscape.

As digital natives move through our health professions programs, it is critical that educators learn from their students how to address students’ preferences and needs. In discussing millennials in academic medicine, Waljee¹⁷ describes opportunities for flattening the hierarchy through various mentoring activities. Reverse mentoring is one such approach that serves as an effective way to empower learners. It provides students with opportunities to impart their perspectives, skills, and guidance to more senior colleagues while promoting a collaborative environment.¹⁷ Engaging in conversations about information sources and resources can open a dialogue between faculty and students in which all parties benefit. Moreover, given that patients are likely using some of the same sources of information, using a reverse mentoring approach in discussing these sources can be a powerful learning experience for all and preparation for future practice.

Strategies for Moving Forward

While it is helpful for educators to use the TPACK framework to determine students’ current level of knowledge and skills, as we begin to cultivate the additional skills we want students to apply when using online resources, it is equally important for students to take ownership of their learning. Charles Friedman is one educator with a deep interest in how individuals and groups interact with information technology. He has described a 3-competency framework to help prepare health professions students, most of whom are digital natives, navigate their futures as physicians in this digital era.¹⁸ First, it is important that students understand what they do and don’t know. Friedman describes a process of “calibration,” by which students and clinicians have sense when they have reached their limit of knowledge and need to seek help.¹⁸ The second competency is the ability to ask a good question. With access to unlimited information, this competency helps to ensure that students are using resources to “improve their incomplete knowledge.”¹⁸ Finally, the third competency highlights the skills necessary to evaluate and weigh evidence and to make clinical decisions based on the strengths or weaknesses of that evidence.

Together, these competencies provide students a solid framework to further their learning and give educators opportunities to be innovative in their planning, instruction, and assessments. The TPACK framework inspires educators to think in practical terms about how students access and use information in their daily practice. This reflective process provides opportunities not only for students but also for educators to respond in ways that foster growth and learning. Ultimately, students’ and educators’ use of nonpeer-reviewed sources is not going away anytime soon. For this reason, we believe it is important for educators to help students feel comfortable and competent navigating nonpeer-reviewed sources throughout their education and future careers. By learning together, we can create effective teaching and learning experiences for all.