6 Ideas to Strengthen Wikipedia(s) with Citizen Behavioral Science

by Max Klein, Julia Kamin, and J. Nathan Matias

Can we decrease harassment on Wikipedia using AI? How can Wikipedia broaden content focused on underrepresented groups? Would new Wikipedia editors be more likely to stay if they were greeted by ‘welcome bots’?

The people who contribute and maintain Wikipedia have many creative ideas for supporting and growing the world’s freely available knowledge resource. This August in Stockholm, just after Wikimania, CAT Lab convened 30 Wikipedians from four continents and fifteen countries to turn those ideas into testable hypotheses.

At our Community Research Summit, the CAT Lab team and other researchers discussed and developed studies to help Wikipedia thrive. By working alongside Wikipedians from eighteen communities – from Azerbaijani to Women in Red – we developed dozens of possible research ideas for the future.

Evidence-Based Community Practices on Wikipedia

Wikipedians are the lifeblood of Wikipedia. They write, edit, organize, illustrate, translate and even write code that underpins the world’s largest online encyclopedia. Wikipedians also co-create the strategy and future direction of their projects through chapters and the Wikimedia Foundation.

When CAT Lab collaborates with Wikipedians, we work together on research to discover the effects of ideas for improving Wikipedia. The knowledge we create together can help Wikipedians make evidence-based decisions towards community goals. The things we learn can also make fundamental contributions to science.

Our gathering in Stockholm gave us a chance to develop new research ideas together, face to face. In the morning, CAT Lab director and founder Nathan Matias led a workshop on how to design experiments in online communities. Participants thought deeply about the goals that matter most and imagined ways to move their Wikipedias toward those goals. Along the way, each was invited to imagine and propose a rough experiment plan to test one of their ideas for change.

In the afternoon, we shared ideas and organized into teams around common goals. Each group worked on one proposed experiment. Groups refined ideas into six research proposals. Here are short summaries of those ideas:

Fundraising banners could ask for money — and knowledge!

Wikipedia relies on small donations to as part of its fundraising model, but do monetary requests crowd out motivation to actually edit the encyclopedia? This team imagined an experimental fundraising campaign in which visitors would be invited to contribute either by making donations or by contributing knowledge.

Training moderators (and technology) to address harassment

As with many online platforms, harassment is an all-too common occurrence on Wikipedia. Still, what to do about it is an open discussion. This team proposed developing an algorithmic measure of harassment, alongside interventions such as one-on-one trainings, on-line and off-line trainings for moderators, and real-time support for moderators.

De-escalating arguments between contributors

While in the process of developing and editing Wikipedia articles, contributors regularly argue about things like word choice, sourcing, and framing of topics. These arguments sometimes escalate into harassment. What if developers built a warning pop-up which asks the users to confirm they want to post in an argument? Could this help reduce conflict?

Automated appreciation messages to retain newcomers

Retaining new editors has long been a challenge for Wikipedia. If new editors were to receive automated messages of encouragement and help, might they be more inclined to stay? Such messages could include statistics and feedback on the user’s edits and opportunities to connect with community mentors.

Making wishlists to increase diversity of photo submissions

The Wiki Loves Monuments project was established by Wikipedians who were concerned about the dearth of photographs of world monuments on the site. The project was highly successful. But one side effect now is drawing in redundant photos of the same buildings (such as the Eiffel Tower) year after year. Would the practice of creating lists of monuments still needing photos and forefronting it in the competition increase the of the variety of photos submitted?

Targeted recruitment of editors for Diversity-focused Wikiprojects

The internal Wikiproject Women in Red is intended to increase participation in editing and developing articles about women — other projects focus on increasing content focused on other underrepresented groups on the site. Sometimes, users may be editing content related to such groups, but they may not know about projects like Women in Red. This team proposed an A/B test that would message users who are already editing relevant articles and invite them to join the Wikiprojects. We would then see if this led them to do more editing in the target areas.

You can find all the study ideas generated at the summit (more than two dozen) in this google doc.

How Can Community Questions Advance Scientific Knowledge?

At CAT Lab, we believe that science can advance when we ask practical questions together with communities – and that community members can help bridge those questions to scientific discovery. As part of the workshop, we asked participants to think about “theories of change” that might explain the positive effects of the interventions they propose. Many of the theories they generated mapped onto questions that are of scientific interest to scholars. Below are four themes among many that emerged from the summit.

Scientific Area: Value and Generosity in Knowledge Communities

The existence and growth of large volunteer peer production projects (like Wikipedias) continues to puzzle social scientists. Unlike money, knowledge on Wikipedia isn’t a limited resource. Goods that people produce are expected to be copied and distributed freely to people who the contributors will never meet (Benkler 2006).

Research with Wikipedians can advance social scientific understanding of value and generosity in these online information communities. Efforts to broaden participation and create supportive environments can help researchers understand the structure and causes of pro-social behavior. By working with multiple communities, we can study if discoveries in one language or community also apply elsewhere (Hill and Shaw 2017).

Scientific Area: Social Norms

When people receive appreciation or warnings or think about the “friendliness” of a community, they are thinking about social norms–people’s beliefs about what other people consider normative (Miller and Prentice 2016; Cialdini and Goldstein 2004). These beliefs do not always accurately match the true or intended norms of a community.

Unlike attitudes and beliefs, norms can change quickly and influence behavior based on new information (Tankard and Paluck 2016). In complex social environments such as or online communities, norms in a sub-network or sub-group of people (what psychologists call “social referents”) may be more influential to someone than the average group norm (Paluck et al 2016).

Communities are especially interested in research on social norms, since they are interested to create perceptions (and reality) of a friendly community. Communities also have questions about the most meaningful sources of normative information. They wonder if messages of encouragement and warning should come from institutions, from high or low status individuals, from automated accounts, and whether the influence of social referents in peer production online is dependent on network structure.

By doing research with communities, we could develop and validate new discoveries about the structure of social referents in large scale information communities and the influence of those referents on prosocial behavior.

Scientific Area: Social Connection

In a flourishing community, social connections bind people together, reinforce common identities, grow a person’s sense of social worth, and motivate pro-social behaviors (Prentice 1994; Bauer et al 2007; Grant & Gino 2010; Levine 1994). In large-scale online collaborations, high volume multi-person activity can sometimes be mistaken for social connection (Ducheneaut 2006). May community liaisons report this lack of social connection as strong potential factor in the loss of new and experienced contributors.

Perhaps because developing and maintaining social connections involves effort and risk, large online communities sometimes struggle to convince people to do this relational labor (Menking 2015). In some cases, communities have (unsuccessfully) attempted to fully automate the functions that social connections might provide (Narayan et al 2017).

By doing research with communities, we hope to replicate and refine theories that link social connections with pro-social behavior- especially as communities seek to formalize and scale the aspects of social connection needed for their collective endeavors to thrive.

Scientific Area: Generalized Knowledge, Machine Learning, and Behavior Change

In recent years, machine learning algorithms have been used to create study samples and direct “precision nudges” to participants (Kleinberg et al 2017).

In our most recent research with Wikipedians, we used community-trained machine learning systems (Halfaker et al 2019) to identify eligible study participants who would be more likely for others to thank on average. While models trained in their context of use have much greater internal validity than more general models (Patton et al 2018; Davidson et al 2019), scientists are unsure how to compare results between communities (Kitchin 2017).

By working with Wikipedians on questions across multiple languages, we may be able to better understand how to make sense of discoveries across contexts that involve machine learning.

Next Steps

From our perspective, the day was a great success. In addition to generating specific study ideas, the summit surfaced a rich picture of goals that are common across Wikipedias along with Wikipedians’ theories on how to move the needle toward reaching those goals.

What will happen to these ideas? Not every idea can come to fruition, but CAT Lab is using ideas as a blueprint our fundraising and our software design in 2020. We left Stockholm feeling inspired, motivated and humbled by the collaborative and creative spirit that drives this community. We can’t wait to start our future collaborations!

If your Wikipedia community is interested to test one of these ideas, or if you are a researcher interested to work with us on these questions, please contact our research manager Julia Kamin <julia.kamin at civilservant.io>.

CAT Lab’s work is financially supported with grants from charitable foundations and through donations. If you are interested to support our work with Wikipedians, please contact Nathan Matias <nathan.matias at cornell.edu>.

Acknowledgments

The ideas in this post are first and foremost ideas from Wikipedians. We’re grateful to everyone who participated for your thoughtful, creative, generous collaboration.

References

• Bauer, T. N., Bodner, T., Erdogan, B., Truxillo, D. M., & Tucker, J. S. (2007). Newcomer adjustment during organizational socialization: A meta-analytic review of antecedents, outcomes, and methods. Journal of Applied Psychology, 92(3), 707–721.Benkler, Y. (2006). The wealth of networks: How social production transforms markets and freedom. Yale University Press.
• Cialdini, R. B., & Goldstein, N. J. (2004). Social influence: Compliance and conformity. Annu. Rev. Psychol., 55, 591–621.
• Davidson, T., Bhattacharya, D., & Weber, I. (2019). Racial Bias in Hate Speech and Abusive Language Detection Datasets. ArXiv Preprint ArXiv:1905.12516.
• Grant, A. M., & Gino, F. (2010). A little thanks goes a long way: Explaining why gratitude expressions motivate prosocial behavior. Journal of Personality and Social Psychology, 98(6), 946.
• Halfaker, A., & Geiger, R. S. (2019). ORES: Lowering Barriers with Participatory Machine Learning in Wikipedia. ArXiv Preprint ArXiv:1909.05189.
• Hill, B. M., & Shaw, A. (2017). Studying Populations of Online Communities. The Handbook of Networked Communication. Oxford University Press, New York, NY.
• Kitchin, R. (2017). Thinking critically about and researching algorithms. Information, Communication & Society, 20(1), 14–29.
• Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017). Human decisions and machine predictions. The Quarterly Journal of Economics, 133(1), 237–293.
• Levine, J. M., & Moreland, R. L. (1994). Group socialization: Theory and research. European Review of Social Psychology, 5(1), 305–336.
• Menking, A., & Erickson, I. (2015). The heart work of Wikipedia: Gendered, emotional labor in the world’s largest online encyclopedia. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, 207–210. ACM.
• Miller, D. T., & Prentice, D. A. (2016). Changing Norms to Change Behavior. Annual Review of Psychology, 67(1), 339–361.
• Narayan, S., Orlowitz, J., Morgan, J., Hill, B. M., & Shaw, A. (2017). The Wikipedia Adventure: Field Evaluation of an Interactive Tutorial for New Users. Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, 1785–1799.
• Patton, D. U., MacBeth, J., Schoenebeck, S., Shear, K., & McKeown, K. (2018). Accommodating grief on twitter: An analysis of expressions of grief among gang involved youth on twitter using qualitative analysis and natural language processing. Biomedical Informatics Insights, 10, 1178222618763155.
• Paluck, E. L., Shepherd, H., & Aronow, P. M. (2016). Changing climates of conflict: A social network experiment in 56 schools. Proceedings of the National Academy of Sciences, 113(3), 566-571.
• Prentice, D. A., Miller, D. T., & Lightdale, J. R. (1994). Asymmetries in attachments to groups and to their members: Distinguishing between common-identity and common-bond groups. Personality and Social Psychology Bulletin, 20(5), 484-493.