2019/Grants/UX of Wikipedia Credibility

Title:

UX of Wikipedia Credibility

Name:

Amy X. Zhang

Wikimedia username:

amyxzhang

E-mail address:

axzcs.uw.edu

Resume:

https://homes.cs.washington.edu/~axz/papers/CV.pdf

Geographical impact:

English-speaking countries

Type of project:

Research + Output

What is your idea?

Prior research has highlighted a wide range of indicators for evaluating content credibility such as visual appearance, tone, and representative citations. However, it is unclear which types of credibility indicators are most useful to online information consumers and how they should be presented.

In this project, we will take a user-centered approach to addressing credibility. We will examine how end-users respond to different types of credibility signals when we vary the presentation of those signals experimentally. 1) This includes signals on Wikipedia, such as the structure of the page, tone of writing, and placement of citations within Wikipedia articles, as well as associated content such as the presence and content of images or the existence of structured metadata. 2) This also includes Wikipedia content that exists elsewhere on the web, such as within information cards on search and social platforms.

While we are interested in these questions for a range of content on Wikipedia, we are particularly interested in examining contentious topics and Wikipedia articles that are used to refute conspiracy theories and health misinformation, particularly relevant given our current pandemic.

Why is it important?

Misinformation is a growing problem in today's online information ecosystem. Much research on combating online misinformation has focused on ways to give users greater context about the news they see in order to spot misinformation for themselves. This strategy is in many ways preferable to systems that spot and remove misinformation for users, as it ultimately gives users the flexibility to apply their own standards of credibility and measures of trust to evaluate content.

Through the experiments we conduct, we will contribute empirically grounded guidelines on which types of credibility signals we should emphasize in a given context, and how we should present these signals to end-users. These studies will also pave a way to design new tools that better signal credibility.

Is your project already in progress?

As chair of the UX of Credible Content working group at the Credibility Coalition, our working group has been brainstorming and gathering background research on this topic for the last year and a half. Below are several blog posts from our group detailing our work: https://misinfocon.com/exploring-the-design-of-credibility-tools-c198530785a4 https://misinfocon.com/what-does-it-take-to-design-for-a-user-experience-ux-of-credibility-f07425940808 https://misinfocon.com/designing-our-way-to-a-health-information-ecosystem-1efc97fe6000

We have also run a number of design workshops, including a workshop at CredCon, at the Berkman Klein Center, and most recently at WikiConference North America. These workshops have allowed us to work with community members on developing their and our ideas for better signaling credibility in online information.

How is it relevant to credibility and Wikipedia? (max 500 words)

Wikipedia has emerged as a key player in the space of better informing users, as many platforms have turned towards displaying links to Wikipedia or Wikipedia content to help users gain greater context about a topic. However, little is still known about how users determine the credibility of the Wikipedia content they see, and what signals and how those signals are presented affect their trust in the information. The most recent work conducted by the Wikimedia Foundation has examined the impact of citations on trust in Wikipedia (https://meta.wikimedia.org/wiki/Research:The_role_of_citations_in_how_readers_evaluate_Wikipedia_articles). However, there are many additional factors that could relate to user trust. We intend to work with these Wikimedia researchers to come up with a set of indicators to study further.

What is the ultimate impact of this project?

Armed with results from our studies, Wikipedians and platforms will have a better understanding of how to present Wikipedia content in a way that improves credibility in the eyes of users. They may also consider how to tailor presentation to suit different users and different use cases. From these studies, we also aim to develop new computational tools to help automatically spot when certain signals could be improved on a page.

Could it scale?

Yes, we will start out with lab experiments in the 10s of people and then moving towards online experiments in the 100s. Eventually with results from these smaller-scale studies, we could move towards large experiments with collaboration with the Wikimedia Foundation.

Why are you the people to do it?

I have been studying credibility signals with the Credibility Coalition since 2017 and led the group's first research paper arguing for an approach to combat misinformation centered around credibility signals that can be crowdsourced and that are aligned with experts.

Since then, I along with the UX of Credible Content Working Group have been pushing for a greater emphasis on user experience of credibility signals, including how to design interfaces that better convey credibility.

In the past, I have also collaborated with the Wikimedia Foundation on research, and have conducted studies with the English Wikipedia community of their internal processes: https://meta.wikimedia.org/wiki/Research:Supporting_deliberation_and_resolution_on_Wikipedia I have presented on this work at Wikimania 2019 and WikiConference North America.

What is the impact of your idea on diversity and inclusiveness of the Wikimedia movement?

People's notions of what is credible will undoubtedly vary according to their background and identity. We expect to see differences emerge between people based on traits such as their age, country, urban/rural location, political leaning, and educational background. We seek to understand how to design interfaces that appeal to people of diverse backgrounds and that does not alienate marginalized groups.

What are the challenges associated with this project and how you will overcome them?

Our biggest challenges involve: 1) narrowing down to a set of experimental conditions we seek to study more deeply 2) designing prototypes of these different experimental interfaces, and then 3) recruiting a large enough and varied enough set of users to test out the interfaces and provide feedback. For 1) and 2), I will work with a graduate student at University of Washington to design an experiment and develop prototypes. We will use our connections within the UX of Credible Content group to gather feedback. For 3), we will work with our contacts at the Wikimedia Foundation and Credibility Coalition to recruit more widely.

How much money are you requesting?

9,534 U.S. dollars

How will you spend the money?

This money will go towards paying a graduate student to design experiments, develop interfaces, conduct studies, and analyze and write up study results.

How long will your project take?

About 3 months for a smaller-scale study in a lab. Another 2 months to do a larger-scale internet study.

Have you worked on projects for previous grants before?

Yes, my PhD research was funded by grants (Google research grant) and fellowships (NSF graduate research fellowship and Google PhD fellowship): https://homes.cs.washington.edu/~axz/