Can saying “thank you” make online communities stronger & more inclusive? Or does thanking others for their voluntary efforts have little effect?
To ask this question, we organized 344 volunteers over the last year to thank 2,702 contributors to Wikipedia in Arabic, German, Polish, and Persian languages. Today, we are excited to share the results.
To achieve their goal of making available the sum of all knowledge, each language Wikipedia needs to recruit, retain, and support a growing number of contributors. Yet people often experience negative feedback from Wikipedia’s many quality-protection systems. This feedback can discourage people from continuing to contribute. To increase opportunities for positive feedback, the Wikimedia Foundation created a “Thanks” feature in 2013. Using this feature, anyone with an account can thank someone for a specific edit to Wikipedia.
The simple act of expressing thanks fascinates social scientists. Saying thanks maintains relationships (Algoe 2012), strengthens important social bonds between people (Simpson et al 2018), and can motivate pro-social behavior (Grant et al 2010). Prior research with Wikipedia has found that public rewards can motivate volunteer efforts (Gallus 2017). Because messages of thanks on Wikipedia are private and pseudonymous, they represent a puzzle for social science. That puzzle excited the team at the Citizens and Technology Lab (CAT Lab), whose favorite studies answer practical community questions while also advancing science.
To discover the effect of receiving thanks, we identified 15,558 newcomers and experienced contributors to Wikipedia across four languages. We then encouraged volunteers to thank half of them and observed the effects on their behavior over six weeks. We found that organizing volunteers to thank others:
- increased two-week retention of newcomers and experienced contributors by 2 percentage points on average
- caused people to send more thanks to others by 43% on average
We did not detect any average effect on the time contributed to Wikipedia over six weeks.
How we tested the effect of organizing thanks
To discover the effect of this organized thanks campaign, we designed a randomized trial where some accounts were assigned to receive thanks and others were not. We designed this study in close collaboration with volunteers from each language Wikipedia. Communities reviewed and approved key details at multiple stages. The Princeton and Cornell University ethics boards approved the overall study.
To make the study realistic, 344 experienced Wikipedia editors sent thanks to actual Wikipedia contributors. Accounts were eligible to receive thanks if they had made at least four high quality, non-userpage edits in the last 540 days. In most languages, high quality edits were determined by machine learning. In German, we used flagged revisions (see the research plan for details).
To study newcomers separately from experienced volunteers, we defined newcomers as accounts that were created within the previous 90 days. To include enough newcomers, we organized three waves of the study as new people created accounts. We then randomly assigned newcomer and experienced accounts within languages to receive thanks.
From August 2, 2019 to February 11, 2020, volunteers sent thanks to eligible accounts on Arabic, German, Persian, and Polish Wikipedias. We created a web application that organized thanking by showing volunteers a queue of potential people and edits to thank.
| eligible participants | average outcome values for accounts that were not assigned to be thanked (control group) | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| monthly active | available during the study | included in the study | % of treatment thanked | two week retention rate | pre-study labor hours | difference in daily labor hours | thanks sent | ||
| Newcomer | Polish | 2414 | 2756 | 1653 | 14% | 19% | 0.027 | -0.015 | 0.038 |
| German | 8555 | 10476 | 5834 | 48% | 18% | 0.024 | -0.011 | 0.052 | |
| Arabic | 12281 | 5936 | 3142 | 13% | 11% | 0.020 | -0.010 | 0.032 | |
| Newcomers | 23250 | 19168 | 10629 | 31% | 16% | 0.023 | -0.012 | 0.044 | |
| Experienced | Polish | 1985 | 8189 | 3384 | 72% | 44% | 0.061 | -0.004 | 0.349 |
| Persian | 1899 | 4648 | 2537 | 6% | 31% | 0.028 | 0.007 | 0.264 | |
| Experienced | 3884 | 12837 | 5921 | 43% | 38% | 0.046 | 0.001 | 0.311 | |
Knowing that volunteers might not be able to thank everyone, we designed an intent to treat study. In this blog post, we report the effect of suggesting that volunteers thank someone, whether or not they actually received thanks. That’s why we describe the effects of “organizing appreciation” rather than “receiving appreciation.” Overall, volunteers thanked 35% out of the 7,779 accounts we suggested.
To measure the outcomes, we recorded public counts of contributions to Wikipedia over time. For accounts that received thanks and their matched pair, we started counting from the point they were sent thanks. For all other accounts, we started counting on the day they were added to the study. We also surveyed participants after six weeks. Since only 5% completed the survey, we cannot make confident claims from the survey.
Retaining newcomers and experienced volunteers by saying “thank you”
Does receiving appreciation influence how long someone contributes to Wikipedia?
To find out, we measured two-week retention. This number records whether someone is still contributing to Wikipedia in the second week after an event (like receiving thanks). We designed the measure to be more stable over time than counting whether an editor contributes or not in a given week. In our measure, someone has stopped contributing by the second week if they take no actions in the five weeks after the second week begins. We validated this measure with survival analysis and will use it to study long-term effects on the retention rate over time.
In Arabic Wikipedia, organizing appreciation to eligible newcomers increased the retention rate from 11 percent to 13 percent
We found that being assigned to receive thanks increased two-week retention of Wikipedia contributors by 2 percentage points on average for newcomers and experienced participants across communities (p < 0.001). The newcomer-only effect is similar (p=0.005). For example, in Arabic Wikipedia, sending appreciation to eligible newcomers increased the retention rate from 11 percent to 13 percent.
We found no observable effect on time contributed to Wikipedia
While receiving thanks influences volunteers to stay active for longer, does it also influence how much time they contribute to Wikipedia?
To answer this question, we studied trends in activity over time. We counted the labor hours per day in the six weeks before receiving thanks and subtracted it from the daily labor hours in the six weeks after (Geiger, Halfaker 2013). For example, eligible Polish newcomers contributed an average of 1.6 minutes per day, or 11 minutes a week. Over 42 days, their average contributions declined by almost a minute per day, a difference in daily labor hours of -0.015.
We did not detect an effect from organized thanks on changes in the time that newcomers contribute to Wikipedia on average (p=0.12). It’s possible that a further study with a larger sample might detect an effect. We estimate that 95% of attempted experiments would have an effect that is smaller than 3 minutes per week on average.
Receiving thanks causes people to thank others too
Large-scale cooperation on Wikipedia relies on a complex web of generosity, with people contributing value to others who may never directly help them back. With this in mind, we wanted to learn if receiving appreciation causes people to send further thanks.
Across language Wikipedias, experienced accounts send more thanks than newcomers. They also receive more thanks than newcomers (see our analysis from 2019). On Polish Wikipedia for example, newcomers send 0.05 thanks on average over 42 days, while experienced accounts send 0.34 thanks on average.
Does receiving thanks cause people to send thanks themselves? We found that being assigned to receive thanks caused Wikipedians of all kinds to send 43% more thanks on average (p=0.002). For Polish newcomers on Wikipedia, receiving thanks caused an increase in the predicted thanks per newcomer from 0.05 thanks to 0.08.
Making sense of what we learned about thanking others
Should you thank contributors to Wikipedia and organize groups to do the same? That question can only be answered in context.
On Arabic, German, Persian, and Polish Wikipedias, organized effort to thank contributors for their work increases retention and increases the rate at which people thank others. While receiving thanks may not cause meaningful change in the time people contribute, it does create an opportunity for communities to further engage with volunteers who keep contributing to the site in greater numbers.
Organized effort to thank Wikipedia contributors for their work increases retention and increases the rate at which people thank others
Thanking requires time and effort. Could bots be used to send thanks? Remember: all the thanks in our study were sent by experienced contributors. We have no evidence that sending automated thanks would have a similar effect. Using bots could have negative consequences if people learned they were thanked by a bot.
How do these effects compare to other research on retaining Wikipedia contributors? Since measurements of retention and contributions are not yet standardized, we cannot directly compare effects between studies. Also, unlike other research on newcomer retention, our study focused on accounts with at least four high quality edits.
How should Arabic, German, Persian, and Polish Wikipedias interpret these results? Language-specific effect sizes were not measurably different between languages. We encourage language Wikipedias to learn from the overall results rather than the language-specific ones. Future studies with larger samples might be able to differentiate more clearly between languages.
Any differences between languages, if they exist, could have many explanations, including variation in:
- definitions of eligibility
- baseline values
- the percentage of people actually thanked by volunteers
- random chance
- cultural differences
Do these results apply to languages that weren’t in the study? Your answer depends on your view of cultural differences. If you believe that these effects are stable across cultures, you might expect similar outcomes in your language.
Like us, you might wonder why receiving thanks has these outcomes. With so few survey answers, this study (like many real-world experiments) cannot provide guidance on the beliefs and feelings of the people who received thanks. We wrote more about the science of gratitude in the blog post that announced this project.
Acknowledgments
At CAT Lab, communities are co-partners in our research. We are deeply grateful to the many Wikipedians who advised and supported this work. We are especially indebted to our liaisons from partnering Wikipedias – Reem Al-Kashif, Christine Domgörgen, Mohamed ElGohary, Maria Heuschkel, Amir Ladsgroup, Wojciech Pędzich, Mohsen Salek, and Natalia Szafran-Kozakowska. These liaisons worked closely with us to design the study and guide us from start to finish. This work is a product of their endless wisdom, patience and dedication. We also appreciate the 344 volunteers who generously used our app to send thanks as part of our study. Many also helped us test and fix software bugs to improve the quality of this study.
We are also grateful to Dariusz Jemielniak, Maria Sefidari, and Aaron Halfaker, for early advice on this project, and to the Wikimedia Foundation for helping us understand community decision-making processes. This research was funded by a grant from the Templeton World Charity Foundation.
References
- Algoe, S. B. (2012). Find, remind, and bind: The functions of gratitude in everyday relationships. Social and Personality Psychology Compass, 6(6), 455–469.
- Gallus, J. (2017). Fostering public good contributions with symbolic awards: A large-scale natural field experiment at Wikipedia. Management Science, 63(12), 3999-4015.
- Geiger, R. S., & Halfaker, A. (2013, February). Using edit sessions to measure participation in Wikipedia. In Proceedings of the 2013 conference on Computer supported cooperative work (pp. 861-870). ACM.
- Grant, A. M., & Gino, F. (2010). A little thanks goes a long way: Explaining why gratitude expressions motivate prosocial behavior. Journal of personality and social psychology, 98(6), 946.
- Goel, S., Anderson, A., & Zia, L. (2019, May). Thanks for Stopping By: A Study of “Thanks” Usage on Wikimedia. In Companion Proceedings of The 2019 World Wide Web Conference (pp. 1208-1211).
- Morgan, J. T., & Halfaker, A. (2018, August). Evaluating the impact of the Wikipedia Teahouse on newcomer socialization and retention. In Proceedings of the 14th International Symposium on Open Collaboration (pp. 1-7).
- Narayan, S., Orlowitz, J., Morgan, J., Hill, B. M., & Shaw, A. (2017, February). The Wikipedia Adventure: field evaluation of an interactive tutorial for new users. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (pp. 1785-1799).
- Simpson, B., Harrell, A., Melamed, D., Heiserman, N., & Negraia, D. V. (2018). The Roots of Reciprocity: Gratitude and Reputation in Generalized Exchange Systems. American Sociological Review, 83(1), 88–110.