Difference between revisions of "2019/Grants/Classifying Wikipedia Actors"
Carlinmack (talk | contribs) (Created page with "{{WCNA 2019 Grant Submission |name=Carlin MacKenzie |username=carlinmack |email=carlin.mackenzie{{@}}gmail.com |geography=English |type=Technology |idea=Wikipedia currently ha...") |
Carlinmack (talk | contribs) (Redraft) |
||
| Line 5: | Line 5: | ||
|geography=English |
|geography=English |
||
|type=Technology |
|type=Technology |
||
| − | |idea=Wikipedia currently has good vandalism detection tools for individual edits |
+ | |idea=Wikipedia currently has good vandalism detection tools for individual edits but lacks tooling to report which users repeatedly engage in misconduct over time. This project will create a tool which can identify these users. This will reduce the load on volunteers as they currently need to go through a user's edits manually to identify any patterns of bad behaviour. A machine process can then alert human volunteers where to examine a user’s history. This will enhance the current practice which only catches editors with the most egregious editing histories. |
| − | |||
| ⚫ | |||
| − | |||
| − | This project is currently in progress and I imagine the next steps to be: |
||
| ⚫ | I am currently doing research on misconduct on Wikipedia at the University of Virginia. My project will create a database of all edits on Wikipedia since 2001 for subsequent classification of good and bad actors. We have no funding for our current project, and we know its limits. If I am sponsored, we could pursue this project next year and add the following features and outcomes |
||
* Detection of more complex forms of misconduct, such as complaining or discussion in bad faith |
* Detection of more complex forms of misconduct, such as complaining or discussion in bad faith |
||
| − | |||
* Developing a system that will ingest information in an online fashion and flag users that have reputation scores that are decreasing. This could be integrated into the Recent changes feed. |
* Developing a system that will ingest information in an online fashion and flag users that have reputation scores that are decreasing. This could be integrated into the Recent changes feed. |
||
| + | * Creating online community discussion around adding AI features for reliability to Wikipedia. Regardless of the direction of future technology, the social discussion about human / AI interaction should start now using experiments like this for modelling. |
||
| − | |||
| − | * Creating online community discussion around adding this feature to Wikipedia - many avenues to do this with and without endorsement from the Wikimedia Foundation. |
||
| − | |||
* Creating documentation to make research into Wikipedia more approachable. |
* Creating documentation to make research into Wikipedia more approachable. |
||
| − | |||
* Community review to improve the current Research pages on meta. |
* Community review to improve the current Research pages on meta. |
||
| − | |||
* Creating reports and visualisations of users, misconduct and editing patterns on Wikipedia |
* Creating reports and visualisations of users, misconduct and editing patterns on Wikipedia |
||
| + | |||
| − | |relevance=The less misconduct there is on Wikipedia, the more credible it is. With the tools that I will develop we can hopefully get a handle on the scale of the issue and start thinking about strategies to deal with it. |
||
| + | |relevance=This project is relevant to Wikipedia's credibility because misinformation and misconduct routinely appear in the same topic environments. Where there is misinformation, there tends to be higher rates of misconduct. By studying misconduct, we get benefits including leads on topics where misinformation has a higher risk of appearing, and also we keep conversations around misinformation more civil and enjoyable. |
||
| + | |||
| + | Additionally, this project will give us a handle on the scale of the issue and allow us to create strategies to deal with it. |
||
| + | |||
|importance=There are no tools to detect users who perform misconduct over time which decreases the quality of Wikipedia. This means that we don't know how big the problem is or strategies that are effective. |
|importance=There are no tools to detect users who perform misconduct over time which decreases the quality of Wikipedia. This means that we don't know how big the problem is or strategies that are effective. |
||
| − | Additionally, |
+ | Additionally, this project aims to create documentation around Wikipedia research and improve the resources so that it is more appealing. This is a fertile area of research with applicability for network analysis, natural language processing, classification, prediction and digital humanities. However, starting Wikipedia research requires a large learning curve and the information we create will encourage future research into these areas. |
| + | |||
| − | |scalability=Hopefully the resulting processing techniques could be light enough to be integrated into the Recent changes feed. |
+ | |scalability=Hopefully the resulting processing techniques could be light enough to be integrated into the Recent changes feed. |
| + | |||
| ⚫ | |||
| + | |||
| ⚫ | |people=I, as project lead, am familiar with the Wikipedia research space and I am a Wikipedian of several years. I have already started this project and the database is being created with the requisite data. Additionally, I have collaborators and advisers for this project in academia, the Wikimedia community, the WMF and in industry, which are important to coordinate future strategy. |
||
| ⚫ | |||
| ⚫ | |people=I am familiar with the Wikipedia research space and I am a Wikipedian of several years. I have already started this project and the database is being created with the requisite data. Additionally, I have |
||
|cost=$8,250 |
|cost=$8,250 |
||
| − | |expenses=The money will cover many different costs. |
||
| − | My university |
+ | |expenses=My university will grant some storage and computation resources, but for me to develop this project beyond the pilot demonstration, I need to purchase storage required to host all of Wikipedia. Beyond that, I would like to document and publish this project as a model for anyone to perform research on Wikipedia. |
| + | |||
| ⚫ | |||
| ⚫ | |||
| ⚫ | |||
| ⚫ | |||
| + | Storage and compute space - $750 |
||
| ⚫ | |||
| − | Additionally, I would like to have funds available for: |
||
| ⚫ | |||
| ⚫ | |||
| ⚫ | |||
| ⚫ | |||
| ⚫ | |||
My current research is until June of this year. |
My current research is until June of this year. |
||
Revision as of 23:30, 7 March 2020
Title:
Classifying Wikipedia Actors
Name:
Carlin MacKenzie
Wikimedia username:
carlinmack
E-mail address:
carlin.mackenzie
gmail.com
Resume:
Geographical impact:
English
Type of project:
Technology
What is your idea?
Wikipedia currently has good vandalism detection tools for individual edits but lacks tooling to report which users repeatedly engage in misconduct over time. This project will create a tool which can identify these users. This will reduce the load on volunteers as they currently need to go through a user's edits manually to identify any patterns of bad behaviour. A machine process can then alert human volunteers where to examine a user’s history. This will enhance the current practice which only catches editors with the most egregious editing histories.
I am currently doing research on misconduct on Wikipedia at the University of Virginia. My project will create a database of all edits on Wikipedia since 2001 for subsequent classification of good and bad actors. We have no funding for our current project, and we know its limits. If I am sponsored, we could pursue this project next year and add the following features and outcomes
- Detection of more complex forms of misconduct, such as complaining or discussion in bad faith
- Developing a system that will ingest information in an online fashion and flag users that have reputation scores that are decreasing. This could be integrated into the Recent changes feed.
- Creating online community discussion around adding AI features for reliability to Wikipedia. Regardless of the direction of future technology, the social discussion about human / AI interaction should start now using experiments like this for modelling.
- Creating documentation to make research into Wikipedia more approachable.
- Community review to improve the current Research pages on meta.
- Creating reports and visualisations of users, misconduct and editing patterns on Wikipedia
Why is it important?
There are no tools to detect users who perform misconduct over time which decreases the quality of Wikipedia. This means that we don't know how big the problem is or strategies that are effective.
Additionally, this project aims to create documentation around Wikipedia research and improve the resources so that it is more appealing. This is a fertile area of research with applicability for network analysis, natural language processing, classification, prediction and digital humanities. However, starting Wikipedia research requires a large learning curve and the information we create will encourage future research into these areas.
Is your project already in progress?
How is it relevant to credibility and Wikipedia? (max 500 words)
This project is relevant to Wikipedia's credibility because misinformation and misconduct routinely appear in the same topic environments. Where there is misinformation, there tends to be higher rates of misconduct. By studying misconduct, we get benefits including leads on topics where misinformation has a higher risk of appearing, and also we keep conversations around misinformation more civil and enjoyable.
Additionally, this project will give us a handle on the scale of the issue and allow us to create strategies to deal with it.
What is the ultimate impact of this project?
Could it scale?
Hopefully the resulting processing techniques could be light enough to be integrated into the Recent changes feed.
Other avenues could be a Wikipedia gadget or script that could display scores for users like the existing tool ORES currently does.
Why are you the people to do it?
I, as project lead, am familiar with the Wikipedia research space and I am a Wikipedian of several years. I have already started this project and the database is being created with the requisite data. Additionally, I have collaborators and advisers for this project in academia, the Wikimedia community, the WMF and in industry, which are important to coordinate future strategy.
What is the impact of your idea on diversity and inclusiveness of the Wikimedia movement?
What are the challenges associated with this project and how you will overcome them?
How much money are you requesting?
$8,250
How will you spend the money?
My university will grant some storage and computation resources, but for me to develop this project beyond the pilot demonstration, I need to purchase storage required to host all of Wikipedia. Beyond that, I would like to document and publish this project as a model for anyone to perform research on Wikipedia.
Development - $4000 Community survey - $500 Documentation - $1500 Data analysis, reports and visualisations - $1500 Storage and compute space - $750
How long will your project take?
If I am sponsored, I will be able to pursue this research in September for the duration of the academic year, until June 2021.
My current research is until June of this year.
Have you worked on projects for previous grants before?